Files
canina/docs/orchestration-v4-rollout.md

5.4 KiB

Stage v4 rollout safety

Scope and deployment boundary

Only a push to develop followed by the existing Gitea workflow may roll out this change. Do not run deploy.sh manually. Its CI marker is an accident guard, not an authentication boundary: runner permissions remain the security boundary. main, unknown branches and the former stage-backend-only bypass fail before Docker operations. Production is intentionally blocked pending separate review; no silent baseline/resolve is permitted.

Tracked sequence

  1. Acquire the Stage rollout lock (same lock as the retired backend-only path).
  2. Fetch the branch and build images using the existing workflow. Build failure leaves old applications running.
  3. Stop Stage frontend and backend (maintenance/outage; no promise of a custom 503 page).
  4. Run the newly built backend image as an isolated one-off Prisma CLI job (compose run --rm --no-deps --entrypoint ... migrate deploy). No app command, published service ports or automatic baseline. Existing DB must already be running.
  5. Only on migration success start backend. Require all nine v4 tables, the successful tracked migration receipt and an actual HTTP catalog response with the catalog envelope. HTTP acceptance has bounded retries and timeouts; no response body is logged.
  6. On backend acceptance failure stop both applications and fail. Only after acceptance start frontend. CI success marker follows these gates. The shared support bot is not rebuilt/replaced by this Stage rollout.

Migration failure leaves maintenance in place; do not auto-restart old apps against a partly migrated schema. Diagnose from CI logs, restore from an authorized backup or ship a reviewed forward fix through CI/CD. Never db push, seed, silently migrate resolve, or manually apply live DDL.

Verification (2026-10-22)

Strict RED/GREEN cycles captured missing rollout function, migration failure ordering, missing backend acceptance and missing CI/Production guard before implementation. python3 scripts/tests/test_stage_rollout.py: 3 passed. Shell tests simulate Docker failures to prove sequencing only; they do NOT certify a healthy release. node --test scripts/tests/test_rollout_health.cjs: 4 passed using a local HTTP server (503, HTML, incorrect envelope rejected; valid empty catalog shape accepted). These fixtures are not live health evidence.

python3 scripts/tests/rehearse_rollout.py performs only schema-only pg_dump and migration-history-only data dump from Stage. It restores into a fresh PostgreSQL 16 container with no network, no host ports, no persistent volume, and no customer rows. Prisma runs in that disposable container's network namespace using an existing generated v4 client/dependencies read-only from the core test worktree. Dumps have a private temporary directory and are removed on exit.

Observed output:

Applying migration `20261022000000_add_orchestration_v4_core`
All migrations have been successfully applied.
No pending migrations to apply.
PASS: actual Prisma legacy catalog/support and v4 relation queries
Restored 61 legacy tables; all business tables empty
PASS: unchanged legacy column definitions; 61 legacy queries; 9 v4 tables; migrate deploy repeatable
Disposable database/container removed; no live DDL performed

The first rehearsal hit a read-only nested mount creation error, cleaned up, and was corrected to mount only prisma/ plus dependencies; subsequent rehearsals passed.

Remaining acceptance constraints

  • No push, actual deployment, live DDL, customer export, or Production change was performed. No live v4 health success is claimed.
  • Schema-only rehearsal proves schema/query compatibility, not row-volume performance, locks under traffic, or business-data backfill behavior. All 61 legacy business tables were verified empty.
  • Full image build/start and public frontend/admin/API acceptance still require the authorized CI rollout. The gate currently covers backend schema/catalog; frontend start success is not browser acceptance.
  • Workflow now downloads the deployment entrypoint by triggering github.sha and passes DEPLOY_SHA. The entrypoint rejects missing/malformed SHA values, verifies that the commit belongs to the fetched deployment branch, and checks out that exact commit. Six disposable rollout tests cover sequencing, CI/Production guards, pin wiring and SHA validation. Actual runner execution and strong artifact/release attestation still require live CI acceptance; SHA pinning alone does not authenticate artifact evidence.
  • Existing compose credentials are out of scope; no new credentials are added. Continue restricting runner access and runtime env files. Orchestration mutation/chat/runtime remain disabled in the integration base; this rollout does not enable them.

Completion-branch rehearsal

The repository-relative rehearsal applied both 20261022000000_add_orchestration_v4_core and 20261022190000_harden_orchestration_ledger to a disposable database restored from Stage schema/history only. All 61 legacy business tables were empty, their column definitions were unchanged, nine v4 tables and actual Prisma queries passed, and a second deploy found no pending migrations. No live DDL or release execution occurred.

The workflow shell now stops on download failure. SHA-pinning and six rollout tests plus four HTTP fixtures are verified locally in disposable containers; they are not proof of an actual CI rollout or frontend acceptance.