fix(stage): isolate credentialless author bootstrap and recovery rehearsal
Stage author bootstrap (dedicated repair) / bootstrap (push) Failing after 3m55s

This commit is contained in:
Canina Automation committed 2026-10-28 19:52:55 +00:00
1 parent 6ed0e73311
commit 46da1c5339
30 files changed
+1295 -62

No files matched your search

+40
View File
@@ -0,0 +1,40 @@
# Stage-only public author bootstrap
Implemented locally, **not executed against Stage**. No production mode, commit, push or recovery-gate override.
## Official operation path
1. Review/commit the integrated backend auth, public-author restoration service, Prisma client/schema, versioned SQL and these scripts/workflow together. This shared worktree is not yet a releasable SHA.
2. Independently approve that exact SHA in repository secret `STAGE_AUTHOR_BOOTSTRAP_REVIEWED_SHA` through existing authorized repository administration. No new API token is needed by these scripts. Gitea 1.22 uses exact push refs, not workflow_dispatch.
3. Push the **same reviewed commit**, only when authorized, to `ops/author-bootstrap-stage-rehearse` first, then `ops/author-bootstrap-stage-apply`. The new `.gitea/workflows/stage-author-bootstrap.yml` does not call deploy.sh: the existing normal Stage recovery gate cannot pass before this repair.
4. CI fetches the immutable event SHA and invokes `ci.py`. The driver builds only `backend/Dockerfile` with revision label and records its immutable image ID **before any credential-file prerequisites**. No migrate, seed, app recreation or live DDL occurs during build. Build failure blocks the operation.
5. Existing Stage runtime env is read privately from `canino_backend_stage` into mode0600 runner files, with exact DB URL identity validation. Nothing is printed. Clone config overrides database/Redis with isolated loopback instances while retaining the existing Stage JWT verification configuration. No JWT is minted and no auth guard is bypassed. Do not upload any env files or dumps.
6. Parent browser assistance must provide a currently valid **human admin** bearer privately in a mode0600 regular file visible to the CI runner. Set repository secret `STAGE_AUTHOR_BOOTSTRAP_TOKEN_FILE` to its **path**, not token value. No implicit host path, credential, token or runner mount is invented here. Missing prerequisite blocks repair *after* the image build. `/opt/data/cache/scratch/author-bootstrap-parent-assistance.json` describes the outstanding browser handoff; that agent-local path is not automatically runner-visible.
7. Rehearsal must pass before apply: fixed Stage identity and orphan baseline; retained original full dump; expected original FK restore failure; new clone pre-data/data; two transactional schema applies; authenticated API once; all post-data; new repaired dump restored fully into another clean clone. Original invalid archive is never overwritten or relabeled clean.
8. Apply repeats the entire rehearsal, rechecks Stage identity/lock, applies one live schema transaction in dedicated exact-SHA CI, then starts a **separate temporary maintenance API** with `STAGE_AUTHOR_RESTORATION_MODE=maintenance`. It shares the fixed Stage backend network namespace for existing DB DNS and real Redis blacklist validation, but HTTP is forced to loopback `127.0.0.1:39071`, never published, avoiding existing port3000. It runs immutable image `node` directly, without startup migration/seed. The only business writes are the authenticated restoration API. POST is never automatically retried after ambiguity.
9. Rerun the **unchanged** full Stage recovery gate and normal release acceptance afterward. This receipt is not live recovery/deployment acceptance. If live DDL succeeds but API fails, inspect retained evidence before deciding any recovery; do not blindly rerun the orphan-baseline tool. Prisma ledger remains untouched; SQL is a separately applied Stage bootstrap artifact, not migrate-deploy/resolve.
Clone authentication uses both restored Stage user/session rows and a read-only Stage Redis RDB snapshot in a disposable no-network Redis. An empty blacklist would silently accept revoked tokens and is deliberately not substituted. The clone backend has `STAGE_AUTHOR_RESTORATION_MODE=rehearsal`. Snapshot staleness is bounded by the operation; live authentication independently uses real Stage Redis.
## Practical prerequisites/blockers
- Source is currently uncommitted; a reviewed exact SHA and authorized official ref push remain parent responsibilities.
- Approved browser bearer and secure CI-runner-visible file handoff are outstanding. Do not expose token values in chat/logs or create replacement tokens.
- Host read-only inspection confirmed Stage backend port3000, Redis `redis_stage:6379`, and no explicit `JWT_ACCESS_SECRET`/`JWT_SECRET` env. Existing auth code's configured fallback is reused unchanged; no replacement secret is synthesized. This existing fallback should be reviewed separately, never rotated during repair without coordinating sessions.
- Runner must have Docker/Python/git, sufficient disk for three DB clones, one Redis clone and two full archives, and the existing host deployment lock inode. All operations have bounded individual deadlines. No pre-existing lock helper is reclaimed.
- PostgreSQL image ID is pinned in operation.py and must match the exact Stage DB image or execution fails closed.
- Private env/archive evidence is retained in a mode0700 runner directory, printed before execution. Delete credential copies using an explicitly approved retention/cleanup action afterward; never generic prune/delete other workers' containers or images. Owned disposable containers/anonymous volumes are removed by captured ID in reverse order. Immutable backend image is retained for the subsequent official operation, not discarded as a test image.
## Tests
```
python3 -m unittest discover -s scripts/stage-author-bootstrap -v
python3 -m unittest discover -s scripts/stage-schema -v
python3 scripts/stage-author-bootstrap/operation.py --help
```
Tests exercise the actual CI Python entrypoint with command doubles (build/rehearse/apply, missing token after build, wrong SHA/ref before transport, build failure), maintenance isolation and original/post-data restore failure ordering. They are offline tests, **not CI execution or real database rehearsal**.
For controlled local clone-only diagnostic use, `operation.py rehearse --sha ... --sql-sha256 ... --backend-image sha256:... --clone-env-file ... --clone-token-file ... --evidence ...` accepts explicit private files. Live `apply-schema` additionally requires `--live-env-file`, independently reviewed SHA and the exact dedicated CI event/ref; do not spoof CI variables on SSH. Backend image revision label must match source SHA. SQL digest and committed tool/workflow contents are verified in CI.
CHECK scope is row-local; application auth must deny public-author principals centrally. Authors must be the five genuine configured UUIDs, `isPublicAuthor=true`, role `User_PetOwner`, no mobile/email/password/session/expiry and zero balances. No invented identity, phone number or privileged role is permitted.
+80
View File
@@ -0,0 +1,80 @@
#!/usr/bin/env python3
"""Dedicated exact-ref CI driver. No deploy, migration CLI, token mint or SSH apply."""
import hashlib
import importlib.util
import json
import os
from pathlib import Path
import re
import subprocess
import sys
HERE = Path(__file__).resolve().parent
spec = importlib.util.spec_from_file_location('bootstrap', HERE / 'operation.py')
assert spec is not None and spec.loader is not None
op = importlib.util.module_from_spec(spec)
spec.loader.exec_module(op)
def run(env=os.environ, call=subprocess.run):
sha = env.get('GITHUB_SHA', '')
refs = {'refs/heads/ops/author-bootstrap-stage-rehearse': 'rehearse',
'refs/heads/ops/author-bootstrap-stage-apply': 'apply-schema'}
if (env.get('GITHUB_ACTIONS') != 'true' or not re.fullmatch('[a-f0-9]{40}', sha)
or env.get('STAGE_AUTHOR_BOOTSTRAP_REVIEWED_SHA') != sha
or env.get('GITHUB_REF') not in refs):
raise ValueError('Dedicated reviewed exact-SHA CI required')
def command(args, **kwargs):
return call(args, check=True, timeout=kwargs.pop('timeout', 180), **kwargs)
head = command(['git', '-C', str(op.ROOT), 'rev-parse', 'HEAD'], capture_output=True, text=True).stdout.strip()
if head != sha: raise ValueError('Source SHA mismatch')
evidence = Path(env['AUTHOR_BOOTSTRAP_EVIDENCE'])
evidence.mkdir(mode=0o700, parents=True, exist_ok=False)
os.chmod(evidence, 0o700)
# Build BEFORE credential prerequisites: missing browser-assisted token does
# not prevent the official immutable build needed for this repair.
iid = evidence / 'backend.iid'
command(['docker', 'build', '--label', 'org.opencontainers.image.revision=' + sha,
'--build-arg', 'NPM_REGISTRY=https://registry.parsaaghayi.ir/repository/npm/',
'--iidfile', str(iid), str(op.ROOT / 'backend')], timeout=1800)
image = iid.read_text().strip()
if not re.fullmatch('sha256:[a-f0-9]{64}', image): raise ValueError('Invalid build image ID')
# Reuse existing Stage runtime config privately; never invent JWT secrets.
# CLI writes no configuration to stdout/logs. Runner-local env-file is sent
# by Docker CLI even when its daemon is on another filesystem.
raw = command(['docker', 'inspect', op.BACKEND], capture_output=True).stdout
info = json.loads(raw)[0]
if info.get('Name') != '/' + op.BACKEND or not info.get('State', {}).get('Running'):
raise ValueError('Existing Stage runtime unavailable')
values = dict(x.split('=', 1) for x in info['Config']['Env'])
from urllib.parse import urlparse
url = urlparse(values.get('DATABASE_URL', ''))
if url.hostname != op.STAGE or url.path != '/caninodb_stage' or url.query or url.fragment:
raise ValueError('Existing Stage runtime DB identity mismatch')
if any('\n' in v or '\r' in v for v in values.values()): raise ValueError('Multiline env refused')
live_env = evidence / 'stage.env'
op.private_write(live_env, '\n'.join(k + '=' + v for k, v in values.items()).encode())
clone_env = evidence / 'clone.env'
values.update(REDIS_HOST='127.0.0.1', REDIS_PORT='16379', REDIS_PASSWORD='')
op.private_write(clone_env, '\n'.join(k + '=' + v for k, v in values.items()).encode())
env = dict(env, AUTHOR_BOOTSTRAP_CLONE_ENV_FILE=str(clone_env), AUTHOR_BOOTSTRAP_LIVE_ENV_FILE=str(live_env))
args = [sys.executable, str(HERE / 'operation.py'), refs[env['GITHUB_REF']], '--sha', sha,
'--sql-sha256', hashlib.sha256((op.ROOT / op.SQL_PATH).read_bytes()).hexdigest(),
'--backend-image', image, '--evidence', str(evidence / 'operation')]
for flag, key in [('--clone-env-file', 'AUTHOR_BOOTSTRAP_CLONE_ENV_FILE'),
('--clone-token-file', 'AUTHOR_BOOTSTRAP_TOKEN_FILE')]:
path = env.get(key)
if not path: raise ValueError('Image built; parent-assisted private prerequisite missing: ' + key)
op.private_file(Path(path))
args += [flag, path]
if refs[env['GITHUB_REF']] == 'apply-schema':
path = env.get('AUTHOR_BOOTSTRAP_LIVE_ENV_FILE')
if not path: raise ValueError('Existing private Stage env required')
op.private_file(Path(path))
args += ['--live-env-file', path]
command(args, timeout=1800)
return image
if __name__ == '__main__':
run()
+285
View File
@@ -0,0 +1,285 @@
#!/usr/bin/env python3
"""Explicit Stage-only attribution bootstrap. Never invokes or bypasses recovery gate."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import stat
# Dedicated lifecycle-safe app: no full AppModule startup or listener monkeypatch.
LOOPBACK_JS = "require('./dist/maintenance/main.js').startMaintenance().catch(()=>{console.error('Maintenance startup failed');process.exit(1)});"
BACKEND = 'canino_backend_stage'
def private_file(path):
if path.is_symlink() or not stat.S_ISREG(path.stat().st_mode) or path.stat().st_mode & 0o077:
raise ValueError('Private regular credential files required')
def check_image(d, image, sha):
info = json.loads(d.run(['image', 'inspect', image]))[0]
if info.get('Id') != image or info.get('Config', {}).get('Labels', {}).get('org.opencontainers.image.revision') != sha:
raise ValueError('Backend image revision does not match exact reviewed SHA')
def api(d, backend, token):
# No retry: a lost response can follow a committed transaction.
result = json.loads(d.run(['exec', '-i', backend, 'node', '-e', API_JS], token, timeout=40))
if {a.get('id') for a in result.get('authors', [])} != AUTHOR_IDS:
raise ValueError('API returned unexpected author identities')
return result
def api_ready(d, backend):
for _ in range(30):
try:
if d.run(['inspect', '-f', '{{.State.Running}}', backend]).strip() != b'true':
raise ValueError('Maintenance process exited')
d.run(['exec', backend, 'node', '-e', "fetch('http://127.0.0.1:39071/api/admin/users/stage-author-restoration',{method:'POST',signal:AbortSignal.timeout(2000)}).then(r=>{if(r.status!==401)process.exit(1)}).catch(()=>process.exit(1))"], timeout=5)
return
except subprocess.CalledProcessError:
time.sleep(1)
raise ValueError('Maintenance API readiness failed')
def maintenance(d, image, env_file, token, owned):
# Shares only the fixed Stage backend network namespace, with NO published
# port and loopback-only HTTP. Stage DNS/Redis keep the real blacklist validator.
info = json.loads(d.run(['inspect', BACKEND]))[0]
if info.get('Name') != '/' + BACKEND or not info.get('State', {}).get('Running'):
raise ValueError('Stage backend namespace unavailable')
cid = d.run(['run', '-d', '--pull=never', '--network=container:' + info['Id'],
'--env-file', str(env_file), '-e', 'APP_ENV=stage', '-e', 'PORT=39071',
'-e', 'STAGE_AUTHOR_RESTORATION_ENABLED=true',
'-e', 'STAGE_AUTHOR_RESTORATION_MODE=maintenance', image,
'node', '-e', LOOPBACK_JS]).decode().strip()
owned.append(cid)
api_ready(d, cid)
return api(d, cid, token)
import subprocess
import time
import uuid
ROOT = Path(__file__).resolve().parents[2]
SQL_PATH = 'backend/prisma/migrations/20261028160000_public_author_identity/migration.sql'
POSTGRES_IMAGE = 'sha256:de3a4eab8fdfa507ea92aac488b916b08089e515db49b055fe71dfa271ba3a28'
STAGE = 'canino_db_stage'
AUTHOR_IDS = {'14490d36-52fc-40be-8637-b7b04b8302f8', '4e0d830c-0125-4f08-8cb7-c5ae0a38cee7',
'63d38117-f864-4af6-afa0-ae5b05255f57', '7b1e51d1-8c32-4a3c-804f-948d706aef36',
'f57e0f98-efee-4fa1-90ef-67bf84cafad4'}
ORPHANS_SQL = """SELECT json_build_object('count',count(*),'ids',array_agg(DISTINCT b.author_id::text))
FROM blogs b LEFT JOIN users u ON u.id=b.author_id WHERE b.author_id IS NOT NULL AND u.id IS NULL;"""
def check_orphans(raw):
value = json.loads(raw)
if value.get('count') != 52 or set(value.get('ids') or []) != AUTHOR_IDS:
raise ValueError('Reviewed 52-blog/five-genuine-author orphan baseline changed')
def check_identity(info):
if (info.get('Name') != '/' + STAGE or info.get('Image') != POSTGRES_IMAGE
or not re.fullmatch('[0-9a-f]{64}', info.get('Id', ''))
or not info.get('State', {}).get('Running')
or 'POSTGRES_DB=caninodb_stage' not in info.get('Config', {}).get('Env', [])
or not any(m.get('Destination') == '/var/lib/postgresql/data' and m.get('RW')
and m.get('Type') in ('bind', 'volume') for m in info.get('Mounts', []))):
raise ValueError('Exact Stage database container identity required')
return info['Id']
class Docker:
def run(self, args, data=None, timeout=180):
return subprocess.run(['docker'] + args, input=data,
stdin=subprocess.DEVNULL if data is None else None,
stdout=subprocess.PIPE, stderr=subprocess.PIPE,
timeout=timeout, check=True).stdout
def sql(self, cid, query, live=False):
command = ['sh', '-c', 'exec psql -X -qAt -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"'] if live else [
'psql', '-h', '127.0.0.1', '-U', 'postgres', '-d', 'rehearsal', '-X', '-qAt', '-v', 'ON_ERROR_STOP=1']
return self.run(['exec', '-i', cid] + command, query.encode()).decode().strip()
def transaction(sql):
return "BEGIN; SET LOCAL lock_timeout='5s'; SET LOCAL statement_timeout='60s'; SELECT pg_advisory_xact_lock(72707369); SET LOCAL search_path=public;\n" + sql + '\nCOMMIT;'
def restore(d, cid, backup, section=None):
args = ['exec', '-i', cid, 'pg_restore', '-h', '127.0.0.1', '-U', 'postgres', '-d', 'rehearsal', '--no-owner', '--no-acl', '--exit-on-error']
if section: args += ['--section=' + section]
d.run(args, backup, timeout=300)
def private_write(path, data):
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL, 0o600)
with os.fdopen(fd, 'wb') as f:
f.write(data)
f.flush()
os.fsync(f.fileno())
def readiness(d, cid):
for _ in range(60):
try:
d.run(['exec', cid, 'pg_isready', '-h', '127.0.0.1', '-U', 'postgres', '-d', 'rehearsal'], timeout=5)
return
except subprocess.CalledProcessError: time.sleep(1)
raise ValueError('Final TCP server readiness failed')
# Returns only validated contract metadata. Token never enters argv, env or logs.
API_JS = r'''const fs=require('fs'); const token=fs.readFileSync(0,'utf8').trim();
(async()=>{let r=await fetch('http://127.0.0.1:39071/api/admin/users/stage-author-restoration',{
method:'POST',headers:{Authorization:'Bearer '+token},signal:AbortSignal.timeout(30000)});
if(!r.ok)throw Error('API status '+r.status);let x=await r.json();
if(x.success!==true||x.totalConfigured!==5||x.restoredCount+x.existingCount!==5)throw Error('API contract');
process.stdout.write(JSON.stringify(x));})().catch(()=>{process.exit(1)});'''
def rehearse(d, backup, sql, backend_image, env_file, token, owned, evidence):
def clone():
cid = d.run(['run', '-d', '--pull=never', '--network=none', '--name', 'author-bootstrap-' + uuid.uuid4().hex[:16],
'-e', 'POSTGRES_HOST_AUTH_METHOD=trust', '-e', 'POSTGRES_DB=rehearsal', POSTGRES_IMAGE]).decode().strip()
owned.append(cid)
readiness(d, cid)
return cid
# Preserve proof of the known invalid original: no constraints disabled, no claim of clean recovery.
original = clone()
try:
restore(d, original, backup)
except subprocess.CalledProcessError as e:
error = e.stderr or b''
private_write(evidence / 'original-full-restore.stderr', error)
if b'blogs_author_id_fkey' not in error or b'foreign key constraint' not in error:
raise ValueError('Original restore failed for an unreviewed reason') from e
else:
raise ValueError('Expected original orphan-FK restore failure absent; review baseline before bootstrap')
cid = clone()
restore(d, cid, backup, 'pre-data')
restore(d, cid, backup, 'data')
check_orphans(d.sql(cid, ORPHANS_SQL))
d.sql(cid, transaction(sql))
d.sql(cid, transaction(sql))
# Carry the real token blacklist into the isolated namespace, not an empty
# Redis that would silently accept revoked sessions. SYNC reads keys only.
redis_info = json.loads(d.run(['inspect', 'canino_redis_stage']))[0]
if redis_info.get('Name') != '/canino_redis_stage' or not redis_info.get('State', {}).get('Running'):
raise ValueError('Stage session validator unavailable')
rdb = d.run(['exec', 'canino_redis_stage', 'sh', '-c',
'export REDISCLI_AUTH="$REDIS_PASSWORD"; exec redis-cli --rdb /dev/stdout'], timeout=120)
if not rdb.startswith(b'REDIS'): raise ValueError('Invalid Stage blacklist snapshot')
redis = d.run(['create', '--pull=never', '--network=container:' + cid,
redis_info['Image'], 'redis-server', '--bind', '127.0.0.1',
'--port', '16379', '--save', '', '--appendonly', 'no']).decode().strip()
owned.append(redis)
import io
import tarfile
archive = io.BytesIO()
with tarfile.open(fileobj=archive, mode='w') as tar:
entry = tarfile.TarInfo('dump.rdb'); entry.size = len(rdb); entry.mode = 0o600
entry.uid = 0; entry.gid = 0
tar.addfile(entry, io.BytesIO(rdb))
d.run(['cp', '-', redis + ':/data/'], archive.getvalue())
d.run(['start', redis])
for _ in range(30):
try:
if d.run(['exec', redis, 'redis-cli', '-p', '16379', 'PING']).strip() == b'PONG': break
except subprocess.CalledProcessError: pass
time.sleep(1)
else: raise ValueError('Clone session validator readiness failed')
backend = d.run(['run', '-d', '--pull=never', '--network=container:' + cid,
'--env-file', str(env_file), '-e', 'DATABASE_URL=postgresql://postgres@127.0.0.1:5432/rehearsal',
'-e', 'APP_ENV=stage', '-e', 'PORT=39071', '-e', 'STAGE_AUTHOR_RESTORATION_ENABLED=true',
'-e', 'STAGE_AUTHOR_RESTORATION_MODE=rehearsal',
backend_image, 'node', '-e', LOOPBACK_JS]).decode().strip()
owned.append(backend)
api_ready(d, backend)
result = api(d, backend, token)
restore(d, cid, backup, 'post-data') # Mandatory ALL post-data, exit on first failure.
# A repaired clone is not the original archive: prove full recovery of a NEW archive separately.
repaired = d.run(['exec', cid, 'pg_dump', '-h', '127.0.0.1', '-U', 'postgres', '-d', 'rehearsal', '--no-owner', '--no-acl', '-Fc'])
if not repaired: raise ValueError('Empty repaired clone archive')
private_write(evidence / 'repaired-clone.dump', repaired)
final = clone()
restore(d, final, repaired)
return result, hashlib.sha256(repaired).hexdigest()
def execute(d, mode, sql, backend_image, env_file, token, evidence, live_env_file=None):
owned = []
try:
runner = d.run(['inspect', '-f', '{{.Config.Image}}', 'gitea_runner']).decode().strip()
lock = d.run(['create', '--name', 'canina-stage-deploy-lock', '--pull=never', '--network=none',
'--cap-drop=ALL', '--security-opt=no-new-privileges', '--mount',
'type=bind,src=/opt/services/canina,dst=/host-canina,readonly', '--entrypoint', '/bin/sh', runner, '-c',
'exec 9</host-canina/stage-backend-deploy.lock; flock -n 9 || exit 1; printf "LOCKED\\n"; while :; do sleep 60; done']).decode().strip()
owned.append(lock)
d.run(['start', lock])
for _ in range(10):
if d.run(['logs', lock]).strip() == b'LOCKED': break
time.sleep(1)
else: raise ValueError('Stage deployment lock unavailable')
info = json.loads(d.run(['inspect', STAGE]))[0]
live = check_identity(info)
if d.sql(live, 'BEGIN READ ONLY; SELECT current_database(); COMMIT;', True) != 'caninodb_stage':
raise ValueError('Wrong database')
check_orphans(d.sql(live, ORPHANS_SQL, True))
backup = d.run(['exec', live, 'sh', '-c', 'exec pg_dump -U "$POSTGRES_USER" -d "$POSTGRES_DB" --no-owner --no-acl -Fc'], timeout=300)
if not backup: raise ValueError('Empty Stage archive')
private_write(evidence / 'original-stage-full.dump', backup)
result, repaired_hash = rehearse(d, backup, sql, backend_image, env_file, token, owned, evidence)
if mode == 'apply-schema':
if check_identity(json.loads(d.run(['inspect', STAGE]))[0]) != live:
raise ValueError('Stage container replaced during rehearsal')
if d.run(['inspect', '-f', '{{.State.Running}}', lock]).strip() != b'true':
raise ValueError('Stage deployment lock lost')
d.sql(live, transaction(sql), True)
live_result = None
if mode == 'apply-schema':
live_result = maintenance(d, backend_image, live_env_file, token, owned)
receipt = dict(mode=mode, original_archive_clean=False,
original_backup_sha256=hashlib.sha256(backup).hexdigest(),
repaired_clone_backup_sha256=repaired_hash, clone_full_recovery=True,
live_business_writes=mode == 'apply-schema', live_schema_applied=mode == 'apply-schema', live_api_result=live_result,
prisma_ledger='untouched', api_result=result,
next_gate='Unchanged full Stage recovery gate mandatory; not certified by this receipt')
private_write(evidence / 'receipt.json', json.dumps(receipt, ensure_ascii=False, indent=2).encode())
return receipt
finally:
for cid in reversed(owned):
try: d.run(['rm', '-f', '-v', cid], timeout=30)
except Exception: print('Owned cleanup failed: ' + cid, file=__import__('sys').stderr)
def main():
p = argparse.ArgumentParser(description=__doc__)
p.add_argument('mode', choices=['rehearse', 'apply-schema'])
p.add_argument('--sha', required=True)
p.add_argument('--sql-sha256', required=True)
p.add_argument('--backend-image', required=True)
p.add_argument('--clone-env-file', required=True, type=Path)
p.add_argument('--clone-token-file', required=True, type=Path)
p.add_argument('--live-env-file', type=Path)
p.add_argument('--evidence', required=True, type=Path)
a = p.parse_args()
head = subprocess.check_output(['git', '-C', str(ROOT), 'rev-parse', 'HEAD'], text=True).strip()
if not re.fullmatch('[0-9a-f]{40}', a.sha) or a.sha != head:
raise ValueError('Exact source SHA required')
if not re.fullmatch('sha256:[0-9a-f]{64}', a.backend_image): raise ValueError('Immutable reviewed backend image required')
for file in (a.clone_env_file, a.clone_token_file):
private_file(file)
if a.mode == 'apply-schema':
if a.live_env_file is None: raise ValueError('Private existing Stage runtime env required')
private_file(a.live_env_file)
sql = (ROOT / SQL_PATH).read_bytes()
if hashlib.sha256(sql).hexdigest() != a.sql_sha256: raise ValueError('Reviewed SQL digest mismatch')
if os.environ.get('GITHUB_ACTIONS') == 'true' or a.mode == 'apply-schema':
ref_mode = 'apply' if a.mode == 'apply-schema' else 'rehearse'
if (os.environ.get('GITHUB_ACTIONS') != 'true' or os.environ.get('GITHUB_SHA') != a.sha
or os.environ.get('GITHUB_REF') != 'refs/heads/ops/author-bootstrap-stage-' + ref_mode
or os.environ.get('STAGE_AUTHOR_BOOTSTRAP_REVIEWED_SHA') != a.sha):
raise ValueError('Independent reviewed exact-SHA dedicated Stage CI required')
for path in (SQL_PATH, 'scripts/stage-author-bootstrap/operation.py', 'scripts/stage-author-bootstrap/ci.py', '.gitea/workflows/stage-author-bootstrap.yml'):
if subprocess.check_output(['git', '-C', str(ROOT), 'show', a.sha + ':' + path]) != (ROOT / path).read_bytes():
raise ValueError('Dirty reviewed source refused')
check_image(Docker(), a.backend_image, a.sha)
a.evidence.mkdir(parents=True, mode=0o700, exist_ok=False)
os.chmod(a.evidence, 0o700)
receipt = execute(Docker(), a.mode, sql.decode(), a.backend_image, a.clone_env_file.resolve(),
a.clone_token_file.read_bytes(), a.evidence, a.live_env_file)
print(json.dumps(receipt))
if __name__ == '__main__': main()
+96
View File
@@ -0,0 +1,96 @@
import importlib.util
import json
from pathlib import Path
import subprocess
import tempfile
import unittest
from unittest.mock import patch
HERE = Path(__file__).resolve().parent
spec = importlib.util.spec_from_file_location('ci_driver', HERE / 'ci.py')
ci = importlib.util.module_from_spec(spec)
spec.loader.exec_module(ci)
class WorkflowTests(unittest.TestCase):
def exercise(self, ref, missing=False, build_fail=False):
calls = []
with tempfile.TemporaryDirectory(dir='/opt/data/cache/scratch') as tmp:
root = Path(tmp)
token = root / 'token'
token.write_bytes(b'opaque-test-token')
token.chmod(0o600)
env = {'GITHUB_ACTIONS':'true', 'GITHUB_SHA':'a'*40,
'STAGE_AUTHOR_BOOTSTRAP_REVIEWED_SHA':'a'*40,
'GITHUB_REF':ref, 'AUTHOR_BOOTSTRAP_EVIDENCE':str(root/'evidence')}
if not missing: env['AUTHOR_BOOTSTRAP_TOKEN_FILE'] = str(token)
def call(args, **kw):
calls.append(args)
if args[0] == 'git': return subprocess.CompletedProcess(args, 0, stdout='a'*40+'\n')
if args[:2] == ['docker','build']:
if build_fail: raise subprocess.CalledProcessError(1,args)
Path(args[args.index('--iidfile')+1]).write_text('sha256:'+'b'*64)
if args[:2] == ['docker','inspect']:
return subprocess.CompletedProcess(args,0,stdout=json.dumps([{
'Name':'/canino_backend_stage','State':{'Running':True},
'Config':{'Env':['DATABASE_URL=postgresql://stage:private@canino_db_stage:5432/caninodb_stage',
'REDIS_HOST=redis_stage','REDIS_PORT=6379','JWT_SECRET=existing-test-secret']}}]).encode())
return subprocess.CompletedProcess(args,0)
if missing or build_fail:
with self.assertRaises((ValueError, subprocess.CalledProcessError)): ci.run(env, call)
self.assertFalse(any('operation.py' == Path(a[1]).name for a in calls if len(a)>1))
else:
self.assertEqual(ci.run(env,call),'sha256:'+'b'*64)
operation = calls[-1]
self.assertEqual(operation[2], 'apply-schema' if ref.endswith('apply') else 'rehearse')
self.assertLess(next(i for i,a in enumerate(calls) if a[:2]==['docker','build']),len(calls)-1)
self.assertIn('--live-env-file', operation) if ref.endswith('apply') else self.assertNotIn('--live-env-file', operation)
clone = (root/'evidence/clone.env').read_text()
self.assertIn('REDIS_HOST=127.0.0.1',clone)
self.assertIn('JWT_SECRET=existing-test-secret',clone)
self.assertEqual((root/'evidence/clone.env').stat().st_mode & 0o777,0o600)
return calls
def test_real_ci_driver_rehearse(self): self.exercise('refs/heads/ops/author-bootstrap-stage-rehearse')
def test_real_ci_driver_apply(self): self.exercise('refs/heads/ops/author-bootstrap-stage-apply')
def test_missing_token_still_builds(self):
calls = self.exercise('refs/heads/ops/author-bootstrap-stage-rehearse',missing=True)
self.assertTrue(any(a[:2]==['docker','build'] for a in calls))
def test_failed_build_never_operates(self): self.exercise('refs/heads/ops/author-bootstrap-stage-apply',build_fail=True)
def test_wrong_ref_sha_no_transport(self):
for ref, reviewed in [('refs/heads/develop','a'*40),('refs/heads/ops/author-bootstrap-stage-apply','b'*40)]:
with self.assertRaises(ValueError):
ci.run({'GITHUB_ACTIONS':'true','GITHUB_SHA':'a'*40,'GITHUB_REF':ref,
'STAGE_AUTHOR_BOOTSTRAP_REVIEWED_SHA':reviewed}, lambda *a,**kw:self.fail('transport before binding'))
def test_workflow_exact_refs_and_real_driver(self):
text = (HERE.parents[1]/'.gitea/workflows/stage-author-bootstrap.yml').read_text()
self.assertIn('- ops/author-bootstrap-stage-rehearse',text)
self.assertIn('- ops/author-bootstrap-stage-apply',text)
self.assertIn('scripts/stage-author-bootstrap/ci.py',text)
self.assertNotIn('scripts/deploy.sh',text)
self.assertNotIn('workflow_dispatch',text)
class MaintenanceTests(unittest.TestCase):
def test_no_published_port_real_validator_and_single_post(self):
op=ci.op
calls=[]
class D:
def run(self,args,data=None,timeout=180):
calls.append((args,data))
if args[0]=='inspect': return json.dumps([{'Id':'a'*64,'Name':'/canino_backend_stage','State':{'Running':True}}]).encode()
if args[0]=='run': return b'b'*64
return b''
owned=[]
with patch.object(op,'api_ready'), patch.object(op,'api',return_value={'success':True}) as api:
op.maintenance(D(),'sha256:'+'c'*64,Path('/private/stage.env'),b'opaque',owned)
api.assert_called_once()
command=calls[1][0]
self.assertIn('--network=container:'+'a'*64,command)
self.assertIn('STAGE_AUTHOR_RESTORATION_MODE=maintenance',command)
self.assertIn('PORT=39071',command)
self.assertNotIn('-p',command)
self.assertIn(op.LOOPBACK_JS,command)
self.assertEqual(owned,['b'*64])
if __name__ == '__main__': unittest.main()
@@ -0,0 +1,74 @@
import importlib.util
import json
from pathlib import Path
import subprocess
import tempfile
import unittest
from unittest.mock import patch
HERE=Path(__file__).resolve().parent
spec=importlib.util.spec_from_file_location('op',HERE/'operation.py')
assert spec is not None and spec.loader is not None
op=importlib.util.module_from_spec(spec)
spec.loader.exec_module(op)
class ExecuteTests(unittest.TestCase):
def test_recovery_failure_blocks_live_and_cleans_reverse(self):
self.exercise('recovery')
def test_empty_backup_blocks_rehearsal(self):
self.exercise('empty')
def test_contended_lock_never_removed(self):
self.exercise('lock')
def test_rehearse_never_writes_live(self):
self.exercise(None,mode='rehearse')
def test_apply_ddl_then_authenticated_api(self):
self.exercise(None)
def exercise(self,fail,mode='apply-schema'):
calls=[]
identity={'Id':'a'*64,'Name':'/'+op.STAGE,'Image':op.POSTGRES_IMAGE,
'State':{'Running':True},'Config':{'Env':['POSTGRES_DB=caninodb_stage']},
'Mounts':[{'Destination':'/var/lib/postgresql/data','RW':True,'Type':'volume'}]}
class D:
def run(self,args,data=None,timeout=180):
calls.append(args)
if args[0]=='create':
if fail=='lock': raise subprocess.CalledProcessError(1,args)
return b'b'*64
if args[0]=='logs': return b'LOCKED'
if args[:2]==['inspect',op.STAGE]: return json.dumps([identity]).encode()
if args[0]=='inspect': return b'true'
if 'pg_dump' in ' '.join(args): return b'' if fail=='empty' else b'original'
return b''
def sql(self,cid,query,live=False):
if query==op.ORPHANS_SQL: return json.dumps({'count':52,'ids':list(op.AUTHOR_IDS)})
if 'READ ONLY' in query: return 'caninodb_stage'
calls.append(['live-ddl'])
def rehearsal(d,backup,sql,image,env,token,owned,evidence):
owned.append('c'*64)
calls.append(['recovery'])
if fail=='recovery': raise ValueError('restore failed')
return {'success':True},'hash'
def maintenance(*args):
calls.append(['live-api'])
return {'success':True}
with tempfile.TemporaryDirectory(dir='/opt/data/cache/scratch') as tmp, patch.object(op,'rehearse',side_effect=rehearsal),patch.object(op,'maintenance',side_effect=maintenance):
if fail:
with self.assertRaises((ValueError,subprocess.CalledProcessError)):
op.execute(D(),mode,'SQL','image',Path(tmp)/'env',b'token',Path(tmp),Path(tmp)/'live-env')
self.assertFalse((Path(tmp)/'receipt.json').exists())
self.assertNotIn(['live-ddl'],calls)
self.assertNotIn(['live-api'],calls)
else:
result=op.execute(D(),mode,'SQL','image',Path(tmp)/'env',b'token',Path(tmp),Path(tmp)/'live-env')
self.assertEqual(result['live_business_writes'],mode=='apply-schema')
if mode=='apply-schema':
self.assertLess(calls.index(['recovery']),calls.index(['live-ddl']))
self.assertLess(calls.index(['live-ddl']),calls.index(['live-api']))
else:
self.assertNotIn(['live-ddl'],calls)
self.assertNotIn(['live-api'],calls)
removed=[a[-1] for a in calls if a[0]=='rm']
self.assertEqual(removed, [] if fail=='lock' else ['b'*64] if fail=='empty' else ['c'*64,'b'*64])
if __name__=='__main__': unittest.main()
@@ -0,0 +1,72 @@
import importlib.util
from pathlib import Path
import unittest
HERE = Path(__file__).resolve().parent
class BootstrapTests(unittest.TestCase):
def test_stage_identity_fails_closed(self):
self.assertTrue((HERE / 'operation.py').exists(), 'Stage bootstrap operation is missing')
spec = importlib.util.spec_from_file_location('bootstrap', HERE / 'operation.py')
op = importlib.util.module_from_spec(spec)
spec.loader.exec_module(op)
identity = {'Id': 'a'*64, 'Name': '/canino_db_stage', 'Image': op.POSTGRES_IMAGE,
'State': {'Running': True}, 'Config': {'Env': ['POSTGRES_DB=caninodb_stage']},
'Mounts': [{'Destination':'/var/lib/postgresql/data','RW':True,'Type':'volume'}]}
self.assertEqual(op.check_identity(identity), 'a'*64)
for key, value in [('Name','/canino_db_prod'), ('Image','other'), ('Id','short')]:
with self.subTest(key=key):
with self.assertRaises(ValueError): op.check_identity({**identity,key:value})
class RecoveryTests(unittest.TestCase):
def test_clone_order_and_postdata_failure_no_recovery_claim(self):
spec = importlib.util.spec_from_file_location('bootstrap', HERE / 'operation.py')
op = importlib.util.module_from_spec(spec)
spec.loader.exec_module(op)
from unittest.mock import patch
import subprocess
import tempfile
calls = []
class Transport:
def run(self, args, data=None, timeout=180):
if args[:2] == ['inspect', 'canino_redis_stage']:
return b'[{"Name":"/canino_redis_stage","State":{"Running":true},"Image":"redis"}]'
if args[:2] == ['inspect', '-f']: return b'true'
if args[0] == 'create': return ('b'*64).encode()
if '--rdb /dev/stdout' in ' '.join(args): return b'REDISsnapshot'
if 'PING' in args: return b'PONG'
if args[0] == 'run': return ('a'*64).encode()
if 'pg_restore' in args:
section = next((x for x in args if x.startswith('--section=')), 'full')
calls.append(section)
if section in ('full','--section=post-data'):
raise subprocess.CalledProcessError(1, args, stderr=b'ERROR: insert or update on table "blogs" violates foreign key constraint "blogs_author_id_fkey"')
if 'node' in args and '-i' in args:
calls.append('api')
return __import__('json').dumps({'success':True,'totalConfigured':5,'restoredCount':5,'existingCount':0,'authors':[{'id':x} for x in op.AUTHOR_IDS]}).encode()
return b''
def sql(self, cid, query):
if query == op.ORPHANS_SQL:
return __import__('json').dumps({'count':52,'ids':list(op.AUTHOR_IDS)})
calls.append('ddl')
return ''
with tempfile.TemporaryDirectory(dir='/opt/data/cache/scratch') as tmp:
with patch.object(op, 'readiness'):
with self.assertRaises(subprocess.CalledProcessError):
op.rehearse(Transport(), b'archive', 'DDL', 'image', Path(tmp)/'env', b'token', [], Path(tmp))
self.assertEqual(calls, ['full','--section=pre-data','--section=data','ddl','ddl','api','--section=post-data'])
self.assertFalse((Path(tmp)/'repaired-clone.dump').exists())
class MigrationTests(unittest.TestCase):
def test_strong_check_contract(self):
migration = HERE.parents[1] / 'backend/prisma/migrations/20261028160000_public_author_identity/migration.sql'
self.assertTrue(migration.exists(), 'Versioned migration missing')
sql = migration.read_text()
for fragment in ('DROP NOT NULL', 'is_public_author', 'mobile IS NOT NULL', 'mobile IS NULL',
'email IS NULL', 'password IS NULL', "role = 'User_PetOwner'", 'stage_test_session IS NULL',
'stage_test_generation IS NULL', 'auth_expires_at IS NULL', 'wallet_balance = 0'):
self.assertIn(fragment, sql)
self.assertNotIn('NOT VALID', sql)
self.assertNotRegex(sql.upper(), r'INSERT|UPDATE|DELETE|DISABLE TRIGGER')
if __name__ == '__main__': unittest.main()