fix(auth): prepare narrow approved CI schema reconciliation

This commit is contained in:
devops committed 2026-10-25 12:43:20 +00:00
1 parent cf9b80bbb3
commit 688a0e4de5
7 files changed
+246

No files matched your search

@@ -0,0 +1,70 @@
name: Approved narrow auth schema reconciliation
on:
workflow_dispatch:
inputs:
environment:
description: 'stage or production (requires separate explicit schema approval)'
required: true
default: stage
sql_sha256:
description: 'Exact reviewed scripts/auth-schema-reconcile.sql SHA256'
required: true
schema_approval:
description: 'Approval record identifier; production requires separately signed authorization'
required: true
jobs:
reconcile:
runs-on: canina
steps:
- name: Authenticated host narrow schema operation (no deployment)
shell: bash
env:
TARGET_ENV: ${{ github.event.inputs.environment }}
SQL_SHA: ${{ github.event.inputs.sql_sha256 }}
APPROVAL_ID: ${{ github.event.inputs.schema_approval }}
DEPLOY_SHA: ${{ github.sha }}
DEPLOY_KEY: ${{ secrets.DEPLOY_KEY }}
DEPLOY_KNOWN_HOSTS: ${{ secrets.DEPLOY_KNOWN_HOSTS }}
PRODUCTION_SCHEMA_APPROVAL: ${{ secrets.PRODUCTION_AUTH_SCHEMA_APPROVAL }}
REVIEWED_SCHEMA_COMMIT: ${{ secrets.REVIEWED_AUTH_SCHEMA_COMMIT }}
STAGE_AUTH_ACCEPTANCE: ${{ secrets.STAGE_AUTH_SCHEMA_ACCEPTANCE }}
run: |
set -euo pipefail
set +x
[[ "$TARGET_ENV" == stage || "$TARGET_ENV" == production ]]
[[ "$SQL_SHA" =~ ^[0-9a-f]{64}$ && "$DEPLOY_SHA" =~ ^[0-9a-f]{40}$ ]]
[[ "$APPROVAL_ID" =~ ^[A-Za-z0-9_-]{4,100}$ ]]
[[ "$REVIEWED_SCHEMA_COMMIT" == "$DEPLOY_SHA" ]]
schema_ok=no; auth_ok=no
if [[ "$TARGET_ENV" == production ]]; then
[[ -n "$PRODUCTION_SCHEMA_APPROVAL" && "$PRODUCTION_SCHEMA_APPROVAL" == "$DEPLOY_SHA:$SQL_SHA:$APPROVAL_ID" ]]
[[ "$STAGE_AUTH_ACCEPTANCE" == "$DEPLOY_SHA:$SQL_SHA" ]]
schema_ok=yes; auth_ok=yes
fi
[[ -n "$DEPLOY_KEY" && -n "$DEPLOY_KNOWN_HOSTS" ]]
umask 077
scratch=$(mktemp -d)
trap 'rm -rf "$scratch"' EXIT
printf '%s\n' "$DEPLOY_KEY" > "$scratch/key"
printf '%s\n' "$DEPLOY_KNOWN_HOSTS" > "$scratch/known_hosts"
unset DEPLOY_KEY DEPLOY_KNOWN_HOSTS PRODUCTION_SCHEMA_APPROVAL
ssh-keygen -F '[87.248.133.138]:2234' -f "$scratch/known_hosts" >/dev/null
ssh -F /dev/null -T -p 2234 -i "$scratch/key" -o BatchMode=yes -o IdentitiesOnly=yes -o IdentityAgent=none -o StrictHostKeyChecking=yes -o UserKnownHostsFile="$scratch/known_hosts" -o GlobalKnownHostsFile=/dev/null devops@87.248.133.138 "sudo -n bash -s -- $DEPLOY_SHA $TARGET_ENV $SQL_SHA $schema_ok $auth_ok" <<'HOST'
set -euo pipefail
set +x
umask 077
sha=$1; target=$2; sql_sha=$3
[[ ! -e /.dockerenv && $(id -u) == 0 ]]
work=$(mktemp -d /opt/services/canina/auth-reconcile-XXXXXX)
trap 'rm -rf "$work"' EXIT
git clone --quiet --no-checkout /opt/services/canina/stage "$work"
remote=$(git -C /opt/services/canina/stage remote get-url origin)
git -C "$work" fetch --quiet "$remote" "$sha"
git -C "$work" checkout --quiet --detach "$sha"
cd "$work"
if [[ $target == production ]]; then
[[ $4 == yes && $5 == yes ]]
export PRODUCTION_SCHEMA_APPROVED="$sha:$sql_sha" STAGE_AUTH_ACCEPTANCE_APPROVED="$sha:$sql_sha"
fi
GITHUB_ACTIONS=true bash scripts/ci-auth-schema-reconcile.sh "$target" "$sha" "$sql_sha"
HOST
+25
View File
@@ -0,0 +1,25 @@
# Narrow auth schema operation — approval required
## Exact scope
`auth-schema-reconcile.sql` adds only users.auth_expires_at TIMESTAMPTZ NULL, stage_test_session VARCHAR(64) NULL, stage_test_generation UUID NULL (no defaults/backfill), and prepare_workflow_outbox with its original CHECKs, three RESTRICT FKs, task uniqueness, lease and due indexes, including nullable not_before. No existing rows changed, no auth behavior disabled, no Prisma metadata altered.
Read-only production PostgreSQL 16.14 catalogs verified all three users columns and outbox absent; users.id and orchestration_tasks/runs/budget_ledger.id are UUID PKs. Existing users wallet CHECK and orchestration constraints stay intact. Production has six ledger rows (one historical rolled-back admin_modules attempt), no Oct22–24 entries despite orchestration objects existing. Never run generic migrate deploy or mark whole pending history applied.
Historical principal migration at e036c5d has two columns, SHA256 8b4fe1fdedd2ae459b720f78479cfcb13da465eec821b422ece5e9bb41dc47ae; 80a2e18 edited it to three, SHA256 a458a94329b1ad078298b042ae0bd2e25439c1f0bb5a086beb55c517d9e0aa28. Read-only Stage ledger records the latter, not the former. Preserve both historical files unchanged here. This operation is separately journaled by CI/backup receipt, not invented Prisma history. Broader baseline/history repair remains separate work.
## Execution contract
Dedicated manual workflow `auth-schema-reconcile.yml` uses DEPLOY_KEY and pinned DEPLOY_KNOWN_HOSTS, fixed SSH endpoint, isolated immutable Git checkout; it does not deploy applications. It requires REVIEWED_AUTH_SCHEMA_COMMIT matching dispatch SHA. Production additionally requires PRODUCTION_AUTH_SCHEMA_APPROVAL = SHA:SQL_SHA256:APPROVAL_ID and STAGE_AUTH_SCHEMA_ACCEPTANCE = SHA:SQL_SHA256, provisioned only after real review and Stage auth acceptance. These are explicit approval records, not automatically issued receipts. The repository's workflow/secret permissions must restrict dispatch/ref mutation to trusted maintainers; unrestricted malicious workflow edits can bypass any workflow-local gate. Review/protect this boundary before provisioning production approval.
Stage already has the DDL: Stage mode backs up production read-only, restores into isolated network-none temporary PostgreSQL, applies exact SQL there, compares resulting catalog with Stage read-only, and writes a SHA-bound structural receipt only. Production requires that receipt, repeats fresh full backup and restore/patch rehearsal, then executes only exact approved SQL. Full dumps remain mode600 host-only under /opt/backups/canina-auth-reconcile; never upload/log PII. Backup nonempty and pg_restore TOC checked. Restore errors/capacity failure stop before DDL. tmpfs is capped at 1GiB: if full restore exceeds capacity, separately review capacity rather than skipping restore. Existing read-only schema backup was 94,694 bytes / 67 tables, no PII copied; its restored PostgreSQL rehearsal is genuine, not a full data-backup recovery claim.
SQL lock_timeout 3s, statement_timeout 30s, idle transaction 60s; advisory lock plus host flock. DDL transactional; lock/contention failure aborts all. No automatic retries. Re-execution with existing outbox fails closed pending comparison, rather than silently accepting drift. Stage/production outputs compare columns/types/nullability/defaults, all outbox constraints/indexes to fresh rehearsed expected schema.
## Acceptance and rollback
1. Parent DBA review exact commit/diff plus SQL hash and workflow trusted-ref/secret permissions. Approve read-only production full backup + isolated full restore scope separately from production DDL.
2. Run Stage workflow from exact commit; inspect successful backup/restore/catalog comparison. Existing Stage backend must pass ordinary admin password login + protected dashboard, OTP flow with authorized principal, refresh, and expiry/session/generation rejection tests. Do not manufacture acceptance if no approved credential/test principal is available. Record actual test run identifiers before setting Stage acceptance approval.
3. Obtain explicit approval: execute only this commit's SQL in caninodb_prod, three nullable users columns + empty outbox and declared constraints/indexes, 3s lock/30s statement budgets, no unrelated migrations/history writes/seeds. Provision approval binding SHA, SQL digest, approval record. Trigger production workflow, inspect terminal CI log. No production execution has been performed during preparation.
4. Release separately via existing code-only develop→Stage→main CI, using actual regenerated backend client, runtime-image/schema guard, immutable image receipts and authenticated login/OTP/refresh/protected data acceptance. Keep current compatible backend until this passes. Schema approval alone does not authorize application promotion.
5. Before commit failure: transaction rollback. After successful DDL: leave additive nullable fields/table intact, roll back application to known compatible immutable image via approved CI; do not DROP columns or outbox after writes. Restore full backup only under separately approved incident/data-loss recovery process. Existing backup tables remain untouched.
## Verified preparation / limitations
Disposable network-none PG16 schema-only restore and exact SQL executed successfully (see JSON). Missing-column SELECT failed before patch and passed after; invalid status rejected; second application rejected without changes; container removed. Independent reviewer found SQL definition matched history; requested stronger gating, Stage preexisting-schema handling, capacity awareness and postcondition validation, addressed or explicit prerequisites above. Full-data restore and dedicated workflow SSH path are implemented but not yet executed in actual CI. No production writes or deployments occurred. Approval/environment gates must not be represented as production repair completion.
+14
View File
@@ -0,0 +1,14 @@
{
"schema_backup_bytes": 94694,
"schema_backup_sha256": "9c6367a06acaf97c1afe9355e4306d2a10f878b7fe156c7dbc5d58d675262c72",
"schema_tables": 67,
"pii_copied": false,
"postgres": "16.14",
"restore_and_patch_seconds": 3.669,
"pre_patch_missing_columns_failed": true,
"post_patch_queries": " auth_expires_at | stage_test_session | stage_test_generation \n-----------------+--------------------+-----------------------\n(0 rows)\n\n count \n-------\n 0\n(1 row)\n\n count \n-------\n 0\n(1 row)\n\n",
"constraints": " conname | pg_get_constraintdef \n---------------------------------------------+-----------------------------------------------------------------------------------------------------------\n prepare_workflow_outbox_attempts_check | CHECK (((attempts >= 0) AND (attempts <= 3)))\n prepare_workflow_outbox_pkey | PRIMARY KEY (id)\n prepare_workflow_outbox_reservation_id_fkey | FOREIGN KEY (reservation_id) REFERENCES orchestration_budget_ledger(id) ON DELETE RESTRICT\n prepare_workflow_outbox_run_id_fkey | FOREIGN KEY (run_id) REFERENCES orchestration_runs(id) ON DELETE RESTRICT\n prepare_workflow_outbox_status_check | CHECK (((status)::text = ANY ((ARRAY['PENDING'::character varying, 'DONE'::character varying])::text[])))\n prepare_workflow_outbox_task_id_fkey | FOREIGN KEY (task_id) REFERENCES orchestration_tasks(id) ON DELETE RESTRICT\n prepare_workflow_outbox_task_id_key | UNIQUE (task_id)\n(7 rows)\n\n",
"invalid_status_rejected": true,
"repeat_execution_rejected_without_writes": true,
"production_writes": false
}
+46
View File
@@ -0,0 +1,46 @@
-- Independently approved reconciliation; NOT a Prisma history migration.
-- Run with psql -X -v ON_ERROR_STOP=1. Do not wrap in migrate deploy/db push.
BEGIN;
SET LOCAL search_path = public, pg_catalog;
SET LOCAL lock_timeout = '3s';
SET LOCAL statement_timeout = '30s';
SET LOCAL idle_in_transaction_session_timeout = '60s';
SELECT pg_advisory_xact_lock(20261025, 1);
DO $$
DECLARE x record;
BEGIN
IF to_regclass('public.users') IS NULL THEN RAISE EXCEPTION 'users missing'; END IF;
FOR x IN SELECT * FROM (VALUES ('auth_expires_at','timestamp with time zone'),('stage_test_session','character varying(64)'),('stage_test_generation','uuid')) AS expected(name,typ) LOOP
IF EXISTS (SELECT 1 FROM pg_attribute WHERE attrelid='public.users'::regclass AND attname=x.name AND NOT attisdropped) THEN
IF NOT EXISTS (SELECT 1 FROM pg_attribute a LEFT JOIN pg_attrdef d ON d.adrelid=a.attrelid AND d.adnum=a.attnum WHERE a.attrelid='public.users'::regclass AND a.attname=x.name AND NOT a.attisdropped AND format_type(a.atttypid,a.atttypmod)=x.typ AND NOT a.attnotnull AND d.oid IS NULL) THEN
RAISE EXCEPTION 'unexpected existing users column %',x.name;
END IF;
END IF;
END LOOP;
IF to_regclass('public.prepare_workflow_outbox') IS NOT NULL THEN
RAISE EXCEPTION 'outbox already exists: inspect equivalence before any repeat execution';
END IF;
FOR x IN SELECT * FROM (VALUES ('orchestration_tasks'),('orchestration_runs'),('orchestration_budget_ledger')) AS refs(name) LOOP
IF NOT EXISTS (SELECT 1 FROM pg_attribute a WHERE a.attrelid=to_regclass('public.'||x.name) AND a.attname='id' AND a.atttypid='uuid'::regtype AND NOT a.attisdropped AND a.attnotnull) THEN
RAISE EXCEPTION 'missing UUID reference %',x.name;
END IF;
END LOOP;
END $$;
ALTER TABLE public.users ADD COLUMN IF NOT EXISTS auth_expires_at timestamptz,
ADD COLUMN IF NOT EXISTS stage_test_session varchar(64),
ADD COLUMN IF NOT EXISTS stage_test_generation uuid;
CREATE TABLE public.prepare_workflow_outbox (
id uuid PRIMARY KEY,
task_id uuid NOT NULL UNIQUE REFERENCES public.orchestration_tasks(id) ON DELETE RESTRICT,
status varchar(20) NOT NULL DEFAULT 'PENDING' CHECK (status IN ('PENDING','DONE')),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts BETWEEN 0 AND 3),
lease_owner varchar(100), lease_token uuid, lease_until timestamptz, heartbeat_at timestamptz,
run_id uuid REFERENCES public.orchestration_runs(id) ON DELETE RESTRICT,
reservation_id uuid REFERENCES public.orchestration_budget_ledger(id) ON DELETE RESTRICT,
created_at timestamptz NOT NULL DEFAULT now(),
not_before timestamptz
);
CREATE INDEX prepare_workflow_outbox_status_lease_until_idx ON public.prepare_workflow_outbox(status,lease_until);
CREATE INDEX prepare_workflow_outbox_due_idx ON public.prepare_workflow_outbox(status,not_before,created_at);
-- Deliberately no _prisma_migrations writes or unrelated DDL.
COMMIT;
+4
View File
@@ -0,0 +1,4 @@
BEGIN READ ONLY;
SET LOCAL search_path=public,pg_catalog;
SELECT jsonb_build_object('columns', (SELECT jsonb_agg(jsonb_build_array(c.relname,a.attname,format_type(a.atttypid,a.atttypmod),a.attnotnull,pg_get_expr(d.adbin,d.adrelid)) ORDER BY c.relname,a.attname) FROM pg_attribute a JOIN pg_class c ON c.oid=a.attrelid LEFT JOIN pg_attrdef d ON d.adrelid=a.attrelid AND d.adnum=a.attnum WHERE c.oid IN ('public.users'::regclass,'public.prepare_workflow_outbox'::regclass) AND a.attnum>0 AND NOT a.attisdropped AND (c.relname='prepare_workflow_outbox' OR a.attname IN ('auth_expires_at','stage_test_session','stage_test_generation'))), 'constraints',(SELECT jsonb_agg(jsonb_build_array(conname,pg_get_constraintdef(oid)) ORDER BY conname) FROM pg_constraint WHERE conrelid='public.prepare_workflow_outbox'::regclass), 'indexes',(SELECT jsonb_agg(jsonb_build_array(indexname,indexdef) ORDER BY indexname) FROM pg_indexes WHERE schemaname='public' AND tablename='prepare_workflow_outbox'));
COMMIT;
+58
View File
@@ -0,0 +1,58 @@
#!/usr/bin/env bash
# Executed on authenticated host by the dedicated CI workflow, never at app startup.
set -euo pipefail
set +x
[[ ${GITHUB_ACTIONS:-} == true && $# == 3 ]] || exit 2
environment=$1; sha=$2; approved_sql_sha=$3
[[ $sha =~ ^[0-9a-f]{40}$ && $approved_sql_sha =~ ^[0-9a-f]{64}$ ]] || exit 2
case "$environment" in stage) container=canino_db_stage;; production) container=canino_db_prod;; *) exit 2;; esac
root=$(git rev-parse --show-toplevel)
[[ $(git rev-parse HEAD) == "$sha" && -z $(git status --porcelain) ]] || exit 2
sql="$root/scripts/auth-schema-reconcile.sql"
[[ $(sha256sum "$sql" | cut -d' ' -f1) == "$approved_sql_sha" ]] || exit 2
umask 077
backup_root=/opt/backups/canina-auth-reconcile
mkdir -p "$backup_root"
exec 9>"$backup_root/operation.lock"
flock -n 9
receipt="$backup_root/stage-$approved_sql_sha.receipt"
if [[ $environment == production ]]; then
[[ -s $receipt ]] || { printf 'Missing Stage receipt\n' >&2; exit 2; }
[[ $(cat "$receipt") == "$sha" ]] || { printf 'Stage SHA mismatch\n' >&2; exit 2; }
# Auth acceptance is independently approved, never manufactured by this script.
[[ ${STAGE_AUTH_ACCEPTANCE_APPROVED:-} == "$sha:$approved_sql_sha" ]] || exit 2
[[ ${PRODUCTION_SCHEMA_APPROVED:-} == "$sha:$approved_sql_sha" ]] || exit 2
source_container=canino_db_prod
else
# Stage already contains these objects: rehearse the missing delta against a
# production backup, then compare exact Stage catalog to that rehearsed result.
source_container=canino_db_prod
fi
# Both approvals are supplied only by the reviewed protected workflow.
backup="$backup_root/$environment-$sha-$(date -u +%Y%m%dT%H%M%SZ).dump"
# Full backup remains restricted on host; never emitted to CI logs or artifacts.
docker exec "$source_container" sh -c 'PGOPTIONS="-c default_transaction_read_only=on" pg_dump -Fc --no-owner --no-privileges -U "$POSTGRES_USER" -d "$POSTGRES_DB"' > "$backup"
[[ -s $backup ]]
docker exec -i "$container" pg_restore --list < "$backup" > "$backup.toc"
[[ -s $backup.toc ]]
printf 'Backup verified: bytes=%s sha256=%s\n' "$(stat -c %s "$backup")" "$(sha256sum "$backup" | cut -d' ' -f1)"
# Restore the FULL backup into an isolated disposable database container first.
name="canina-auth-restore-${environment}-${sha:0:12}"
cleanup() { docker rm -f "$name" >/dev/null 2>&1 || true; }
trap cleanup EXIT
docker run -d --name "$name" --network none --tmpfs /var/lib/postgresql/data:rw,size=1g -e POSTGRES_HOST_AUTH_METHOD=trust postgres:16-alpine >/dev/null
ready=false
for _ in {1..60}; do if docker exec "$name" pg_isready -U postgres >/dev/null 2>&1; then ready=true; break; fi; sleep 1; done
[[ $ready == true ]]
docker exec -i "$name" pg_restore --exit-on-error --no-owner --no-privileges -U postgres -d postgres < "$backup"
docker exec -i "$name" psql -X -v ON_ERROR_STOP=1 -U postgres < "$sql"
docker exec -i "$name" psql -X -qAt -v ON_ERROR_STOP=1 -U postgres < "$root/scripts/auth-schema-verify.sql" > "$backup.expected.json"
if [[ $environment == production ]]; then
# Only the reviewed SQL is executed. No Prisma CLI, seeds or ledger edits.
docker exec -i "$container" sh -c 'psql -X -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' < "$sql"
fi
docker exec -i "$container" sh -c 'PGOPTIONS="-c default_transaction_read_only=on" psql -X -qAt -v ON_ERROR_STOP=1 -U "$POSTGRES_USER" -d "$POSTGRES_DB"' < "$root/scripts/auth-schema-verify.sql" > "$backup.actual.json"
cmp "$backup.expected.json" "$backup.actual.json"
cleanup; trap - EXIT
if [[ $environment == stage ]]; then printf '%s\n' "$sha" > "$receipt"; fi
printf 'Narrow additive reconciliation complete: environment=%s commit=%s sql_sha256=%s\n' "$environment" "$sha" "$approved_sql_sha"
+29
View File
@@ -0,0 +1,29 @@
"""Offline safety-contract tests. DB execution evidence is in docs/auth-schema-rehearsal.json."""
import pathlib
import subprocess
import unittest
ROOT = pathlib.Path(__file__).resolve().parents[1]
class Safety(unittest.TestCase):
def test_shell_syntax(self):
subprocess.run(['bash', '-n', str(ROOT/'scripts/ci-auth-schema-reconcile.sh')], check=True)
def test_narrow_sql(self):
sql=(ROOT/'scripts/auth-schema-reconcile.sql').read_text()
for forbidden in ['DROP TABLE', 'DELETE FROM', 'UPDATE public.', 'INSERT INTO', 'migrate deploy', 'db push']:
self.assertNotIn(forbidden, '\n'.join(x for x in sql.splitlines() if not x.startswith('--')))
self.assertIn("lock_timeout = '3s'", sql)
self.assertIn("statement_timeout = '30s'", sql)
self.assertIn('COMMIT;', sql)
def test_fail_closed_approvals(self):
w=(ROOT/'.gitea/workflows/auth-schema-reconcile.yml').read_text()
for expected in ['REVIEWED_AUTH_SCHEMA_COMMIT','PRODUCTION_AUTH_SCHEMA_APPROVAL','STAGE_AUTH_SCHEMA_ACCEPTANCE','StrictHostKeyChecking=yes','IdentityAgent=none']:
self.assertIn(expected,w)
s=(ROOT/'scripts/ci-auth-schema-reconcile.sh').read_text()
self.assertLess(s.index('pg_restore --exit-on-error'),s.index('if [[ $environment == production ]]; then',s.index('pg_restore --exit-on-error')))
self.assertIn('cmp "$backup.expected.json" "$backup.actual.json"',s)
def test_real_rehearsal(self):
import json
r=json.loads((ROOT/'docs/auth-schema-rehearsal.json').read_text())
self.assertTrue(r['pre_patch_missing_columns_failed'])
self.assertFalse(r['production_writes'])
self.assertGreater(r['schema_backup_bytes'],90000)
if __name__=='__main__':unittest.main()