Files
canina/infra/hermes-editorial/RECOVERY.md
T
2026-10-24 10:28:30 +00:00

6.1 KiB

Reviewed Stage editorial recovery

No live change was made by this patch. CI owns installation, parent owns authenticated recovery and any separately approved research. Do not use complete, DB writes, forged tokens or root runner invocation.

Ownership

Read-only reconnaissance: canina-hermes Docker Config.User=root; /proc/self/uid_map maps container 0 to host 100000 for 65536 IDs; passwd hermes is 10000:10000 (host 110000:110000). Migration runs inside the container namespace, never with guessed host IDs. Existing package, dedicated PMO token (0400), locks and competitor editorial-runner cache (0700) are root-owned. Profile roots and normal caches are already hermes-owned and must not be recursively changed.

provision-editorial-runtime.py runs the tracked editorial-ownership.py through the existing clean-SHA serialized Stage CI gate before installing as hermes. scan() preflights every member, symlink ancestor, UID/GID, mode and hardlink; exact package/payload/installed skills, dedicated credential, editorial state names and profile cache/editorial-runner only. Read-only scan against the existing container accepted 96 paths. General profile files, provider credentials, other caches and config are excluded. Unknown entries fail closed. Modes and contents never change. Root journal /opt/data/.editorial-ownership-v1.json is 0600, records original IDs/modes/inodes; a second migration is a no-op. A partial failure rolls back. Explicit rollback(home,journal) refuses inode/mode/path-set conflicts; use only before an installer upgrade replaces files. CI must remain serialized and runner/client inactive during migration. After installation, normal installer rollback and ownership rollback are separate operations; do not blindly restore old journal after file replacement.

Stop original RUNNING v6

After deploying/reviewing this SHA through CI, authenticated independent SUPER_ADMIN or configured owner-reviewer Admin uses the ordinary admin JWT (not a worker token). Read:

GET https://stageapi.canina.ir/api/admin/orchestration/tasks/565ace1f-8cfc-4447-b978-66c36e7b767b

Take the actual RUNNING run ID from runs (do not guess). Confirm version=6 and original process has ended. Invoke:

POST https://stageapi.canina.ir/api/admin/orchestration/tasks/565ace1f-8cfc-4447-b978-66c36e7b767b/editorial-failure
Authorization: Bearer <existing independent admin session JWT>
Content-Type: application/json

{"runId":"<actual original RUNNING run UUID>","expectedVersion":6,"idempotencyKey":"editorial-recovery:565ace1f:v6","reason":"Original supervised runner process ended before artifact ingestion; cache access failure; actual spend unknown"}

Requires exact editorial provenance, matching current run, expired lease, expected task version and unreconciled owned 20 reservation; rejects cross-run/cross-task/active lease. This does not require an expired execution approval to become valid again merely to stop work. Existing kill policy still applies: FREEZE_NEW_WORK permits stopping; STOP_AGENTS/DISABLE_MUTATIONS/EMERGENCY_HALT block as for other core operations. Owner controls switches through existing policy, not this endpoint. Identical replay is safe; different payload/key collision is rejected. Locks serialize heartbeat/result/claim/failure. Success is task FAILED v7, run FAILED, outbox DONE, token cleared, no artifact/QA/reconciliation, original unknown20 retained.

Held lease variant: canonical PMO worker may POST /api/internal/prepare-worker/editorial-failure with the same fields plus taskId and valid unexpired leaseToken; no worker can perform stale recovery. Never include actualCents or QA.

Explicit retry and budget boundary

There is intentionally no retry of this exhausted 20-cent task: preserving unknown20 and reserving another20 under a20 task cap is mathematically impossible. Reusing the original reservation as fresh spend would falsely manufacture free capacity. A repeat claim on FAILED is denied before any ledger write; duplicate failure never creates another reservation.

After predecessor is confirmed FAILED/fenced, parent can explicitly submit a new editorial proposal through existing authenticated POST /api/internal/prepare-worker/editorial-proposals, with a fresh requestId, the reviewed deployment commit SHA, same bounded topic/criteria and budgetLimitCents=20 (see editorial-execution.md contract). Obtain a new independent Gate1 via existing POST /api/admin/orchestration/tasks/<newTaskId>/gates/1/decide with APPROVED, its current expectedVersion and a new idempotencyKey. This atomically queues READY under current policy. Do not renew or mutate the FAILED predecessor. Only then execute the reviewed hermes-identity runner on the new task ID. Original unknown20 remains reserved; the new explicit approval authorizes up to20 additional, so cumulative upper bound is40. Parent must make that extra-budget authorization explicit; if the existing authorization is a cumulative20 ceiling, retry remains blocked until actual spend is measured through a separately reviewed reconciliation path. This patch does not bypass that ceiling or invent cost0.

Verification

TDD: missing failure method was observed red (3 tests), then green. Migration tests verify twice, exact ownership/modes, outside sentinel, symlink/public secret/unknown path rejection and injected partial-failure rollback. Real TCP/PostgreSQL worker test races duplicate failure, rejects wrong token/actor/extra cost field, denies heartbeat/result/reclaim after stop and verifies only original RESERVE20, no artifacts or attestations. Unit tests cover stale version, active stale recovery, wrong run/task, independent role and idempotency payload mismatch.

Disposable test database editorial_recovery_isolated only. Full migration chain has pre-existing blogs missing relation; isolated tests apply the four scoped orchestration migrations to a dedicated DB instead. Shared generated Prisma client was stale; tests/build use private generated .prisma mount, not shared mutation. Build initially failed from remapped fixture ownership and stale Prisma; private owned fixture fixed both. No Stage/Production migrations or writes, deploy, push, scheduler, runner or live research executed.