Add opt-in CI Stage credential provisioning and restricted SQLite inventory export

This commit is contained in:
devops committed 2026-10-23 16:49:09 +00:00
1 parent 4b147b0091
commit aa23c7ef91
6 files changed
+552

No files matched your search

+145
View File
@@ -0,0 +1,145 @@
# Opt-in Stage runtime provisioning (parent wiring contract)
This provisions **snapshot inventory** and credentials for the existing PREPARE
transport. It does NOT create a real LLM executor. The current deterministic worker
is still `llmPilot:false`. No provider credentials are collected, no cron installed,
no task claimed and no worker polling/service started by the provisioner.
## CI-only host invocation
Parent owns workflow/deploy.sh/AppModule; none are changed here. Run only through
the existing pinned authenticated SSH CI boundary, under its Stage environment
approval and serialized rollout lock. `CI=true` is an accident guard, not an auth
boundary. Provisioner checks clean HEAD equals the full approved commit; no branch
promotion, push, Production, RELEASE_AUTHORIZE, Docker or DB writes are performed.
Only the CI account should be able to invoke this on a root-owned trusted parent.
From the clean pinned release checkout on the host (not the dirty Stage checkout):
```sh
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000
```
Default output root is `/opt/services/canina/stage-runtime` and its parent must
already exist. First creation is staged and atomically renamed. Repeating validates
existing state without rotating credentials. Corrupt/missing/unsafe files fail
closed, not repaired. CI MUST serialize execution and prevent untrusted edits to
source/output parents; this is not a hostile-filesystem sandbox. `--root` is for
CI-reviewed test/Stage locations only, not arbitrary browser input. Secure backup
and separately approved rotation are parent/operator responsibilities.
- `secrets/` 0700, files 0400 owned by the runtime UID (1000 in the current backend
image); `hermes_username/password/signing_key` are new independent random values.
- `prepare_worker_grants` contains SHA256 only, exact separate service identities
`stage-prepare-worker`/WORKFLOW_WORKER and `stage-evidence-ci`/STAGE_CI. These are
transport service identities, **not invented DB admin users**.
- Raw `credentials/prepare_worker_token` and `credentials/stage_ci_token` stay in a
CI-owner-only 0700 directory, 0400 files. Do NOT mount either into inventory or
backend, log them, upload them as artifacts or send them to browsers. A later
reviewed one-shot executor receives only its own token through a private file.
- `artifacts/` is 0700 owned by backend UID; backend alone writes it. Persistent
local disk is not object storage or external publication evidence.
- `hermes_runtime_grants` is `{}`: deny all. No live admin query was needed/performed.
Parent must separately query actual Stage admin IDs read-only, review exact ID to
profile mapping and provision a private replacement via configuration management
before claiming panel access. Never infer identities from usernames or JWT input.
Deliberately changed grants require separate review; default provisioner refuses
nonempty runtime grants on repeat.
- Compose file-backed secrets preserve host permissions/ownership; `uid/gid/mode`
declarations do not fix bind mount ownership. Verify effective host IDs if Docker
uses userns remapping; do not chmod 0444 as a workaround.
## Consistent restricted export
After approval, the SAME CI invocation may explicitly request one profile's source
SQLite DB. Resolve its real `HERMES_HOME/state.db` through reviewed host mounts;
never guess a gateway volume path or mount the original home into inventory.
```sh
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000 \
--snapshot-db "$REVIEWED_STATE_DB" --profile default \
--snapshot-name "$UNIQUE_CI_CAPTURE_ID"
```
SQLite online `Connection.backup()` reads committed WAL frames into memory (never
`cp state.db`, never `immutable=1`). A 30-second callback deadline bounds busy
backup. Only explicit source/time/token/cost columns from `sessions` are projected.
Private session IDs become capture-local ordinal IDs; sources use an enum; no
messages, titles, prompts, user/chat IDs, paths, URLs, model config, credentials,
config.yaml, auth.json, .env, cron commands, memory or skills are exported. Unknown
schema/nonnumeric metadata fails. One profile per immutable capture; there is no
multi-DB atomicity claim. A source DB may be large: backup is memory-bound, so size
and host resources need operator review before a real capture.
Output directory is new/immutable (0500, 0400 files), never overwritten:
`inventory.json`, exact three-field `snapshot-manifest.json` and `export-report.json`.
Manifest `capturedAt` is measured immediately after successful backup, not startup,
mtime or an example. Report records actual projected row count, inventory digest,
method/exclusions; it is not a signature or proof of source authenticity. No live
capture was performed in this change. Never reuse fixture manifests as evidence.
Projected HTTP readers never consult gateway config, state DB or cron files:
model/provider are explicit `unknown`, context length is a placeholder zero,
cron arrays empty mean **not captured**, session activity false is a placeholder,
profile inventory doesn't establish live process availability. Status retains
`liveProcess:unknown`. Usage days are relative to actual capture time. Snapshot
fresh/stale provenance from the existing adapter remains mandatory. No real LLM
pilot readiness follows from this inventory responding 200.
## Exact parent compose/application wiring
`infra/hermes-runtime/compose.stage-provision.yml` is an opt-in merge fragment,
NOT called by the existing deploy script or ci-apply.sh. Parent must include it
in **every** Stage compose command after provisioning, using the existing stable
Compose project identity and existing release isolation/volume rules:
```sh
export STAGE_RUNTIME_SERVE_FILE="$PINNED_RELEASE/infra/hermes-runtime/serve.py"
export STAGE_RUNTIME_SNAPSHOT="/opt/services/canina/stage-runtime/snapshots/$UNIQUE_CI_CAPTURE_ID"
# Only inside approved CI rollout, never manually:
# docker compose -p <existing-stage-project> -f <parent-stage-compose> \
# -f "$PINNED_RELEASE/infra/hermes-runtime/compose.stage-provision.yml" \
# --profile stage-runtime-inventory <parent-approved command>
```
The fragment adds backend `PREPARE_WORKER_ENABLED=true`, fixed-path secrets
`/run/secrets/prepare_worker_grants`, `/run/secrets/hermes_runtime_grants`,
`hermes_username/password`, and artifact bind at
`/var/lib/canina/prepare-artifacts`. It adds a project-scoped **internal** network
between backend and inventory only, alias `canina-hermes-inventory` (not the live
`canina-hermes`). No host port, Docker socket or live gateway volume. Inventory is
read-only, nonroot, no capabilities, restricted tmpfs and pinned image, mounts only
its immutable projected export and script. `create_host_path:false` fails on missing
exports/storage. Original compose.readonly.yml/ci-apply.sh remain legacy review-only;
do not combine those with this provisioning fragment.
AppModule must keep AuthModule/JwtStrategy, import
`RuntimeReadModule.registerFromFile('/run/secrets/hermes_runtime_grants')` for reads,
and spread `...registerPrepareWorker()` for the opt-in worker transport (verify
existing AppModule before adding duplicates). The latter defaults to no module
unless exact env `true`; enabling compose alone cannot add missing imports. Routes
remain service-authenticated. Artifact base is fixed to
`https://stageapi.canina.ir/api/internal/prepare-worker/artifacts`; parent must verify
its actual global `/api` prefix/proxy/authenticated retrieval after CI deployment.
No URL here is asserted externally reachable by disposable tests.
Parent still must implement/review a bounded one-shot real LLM executor separately;
this commit intentionally adds neither model/provider secrets nor automatic polling.
After CI rollout verify backend UID/secret readability by names only, deny-all reads,
worker unauth denial, snapshot provenance and authenticated artifact digest readback
before a separately authorized task. Do not call RELEASE_AUTHORIZE.
## Executed disposable validation
Pinned Hermes image, `--network none`, repository mounted read-only; private
TemporaryDirectory fixtures only. RED observed missing provisioner, missing exporter,
wrong projected profile and acceptance of malformed credential; GREEN: four
provision/export/security/CI-guard unittest cases; projected HTTP auth/profile/session/
wrong-profile/model/cron assertions; existing exact-image HTTP acceptance, TCP serve
mutation denial and authenticated WebSocket 1008 denial. Compose merged config
`--quiet` passed without applying it. An initial test mount mismatch was corrected
(`/source` for repo-relative tests, `/contract` for legacy tests). No application
backend code changed; full backend suites not rerun. No host credentials, artifact
storage, snapshots, DB users, services or networks were provisioned live.