Add opt-in CI Stage credential provisioning and restricted SQLite inventory export
This commit is contained in:
1 parent
4b147b0091
commit
45b3aa7f1b
6 files changed
+552
No files matched your search
@@ -0,0 +1,145 @@
|
||||
# Opt-in Stage runtime provisioning (parent wiring contract)
|
||||
|
||||
This provisions **snapshot inventory** and credentials for the existing PREPARE
|
||||
transport. It does NOT create a real LLM executor. The current deterministic worker
|
||||
is still `llmPilot:false`. No provider credentials are collected, no cron installed,
|
||||
no task claimed and no worker polling/service started by the provisioner.
|
||||
|
||||
## CI-only host invocation
|
||||
|
||||
Parent owns workflow/deploy.sh/AppModule; none are changed here. Run only through
|
||||
the existing pinned authenticated SSH CI boundary, under its Stage environment
|
||||
approval and serialized rollout lock. `CI=true` is an accident guard, not an auth
|
||||
boundary. Provisioner checks clean HEAD equals the full approved commit; no branch
|
||||
promotion, push, Production, RELEASE_AUTHORIZE, Docker or DB writes are performed.
|
||||
Only the CI account should be able to invoke this on a root-owned trusted parent.
|
||||
|
||||
From the clean pinned release checkout on the host (not the dirty Stage checkout):
|
||||
|
||||
```sh
|
||||
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
|
||||
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000
|
||||
```
|
||||
|
||||
Default output root is `/opt/services/canina/stage-runtime` and its parent must
|
||||
already exist. First creation is staged and atomically renamed. Repeating validates
|
||||
existing state without rotating credentials. Corrupt/missing/unsafe files fail
|
||||
closed, not repaired. CI MUST serialize execution and prevent untrusted edits to
|
||||
source/output parents; this is not a hostile-filesystem sandbox. `--root` is for
|
||||
CI-reviewed test/Stage locations only, not arbitrary browser input. Secure backup
|
||||
and separately approved rotation are parent/operator responsibilities.
|
||||
|
||||
- `secrets/` 0700, files 0400 owned by the runtime UID (1000 in the current backend
|
||||
image); `hermes_username/password/signing_key` are new independent random values.
|
||||
- `prepare_worker_grants` contains SHA256 only, exact separate service identities
|
||||
`stage-prepare-worker`/WORKFLOW_WORKER and `stage-evidence-ci`/STAGE_CI. These are
|
||||
transport service identities, **not invented DB admin users**.
|
||||
- Raw `credentials/prepare_worker_token` and `credentials/stage_ci_token` stay in a
|
||||
CI-owner-only 0700 directory, 0400 files. Do NOT mount either into inventory or
|
||||
backend, log them, upload them as artifacts or send them to browsers. A later
|
||||
reviewed one-shot executor receives only its own token through a private file.
|
||||
- `artifacts/` is 0700 owned by backend UID; backend alone writes it. Persistent
|
||||
local disk is not object storage or external publication evidence.
|
||||
- `hermes_runtime_grants` is `{}`: deny all. No live admin query was needed/performed.
|
||||
Parent must separately query actual Stage admin IDs read-only, review exact ID to
|
||||
profile mapping and provision a private replacement via configuration management
|
||||
before claiming panel access. Never infer identities from usernames or JWT input.
|
||||
Deliberately changed grants require separate review; default provisioner refuses
|
||||
nonempty runtime grants on repeat.
|
||||
- Compose file-backed secrets preserve host permissions/ownership; `uid/gid/mode`
|
||||
declarations do not fix bind mount ownership. Verify effective host IDs if Docker
|
||||
uses userns remapping; do not chmod 0444 as a workaround.
|
||||
|
||||
## Consistent restricted export
|
||||
|
||||
After approval, the SAME CI invocation may explicitly request one profile's source
|
||||
SQLite DB. Resolve its real `HERMES_HOME/state.db` through reviewed host mounts;
|
||||
never guess a gateway volume path or mount the original home into inventory.
|
||||
|
||||
```sh
|
||||
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
|
||||
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000 \
|
||||
--snapshot-db "$REVIEWED_STATE_DB" --profile default \
|
||||
--snapshot-name "$UNIQUE_CI_CAPTURE_ID"
|
||||
```
|
||||
|
||||
SQLite online `Connection.backup()` reads committed WAL frames into memory (never
|
||||
`cp state.db`, never `immutable=1`). A 30-second callback deadline bounds busy
|
||||
backup. Only explicit source/time/token/cost columns from `sessions` are projected.
|
||||
Private session IDs become capture-local ordinal IDs; sources use an enum; no
|
||||
messages, titles, prompts, user/chat IDs, paths, URLs, model config, credentials,
|
||||
config.yaml, auth.json, .env, cron commands, memory or skills are exported. Unknown
|
||||
schema/nonnumeric metadata fails. One profile per immutable capture; there is no
|
||||
multi-DB atomicity claim. A source DB may be large: backup is memory-bound, so size
|
||||
and host resources need operator review before a real capture.
|
||||
|
||||
Output directory is new/immutable (0500, 0400 files), never overwritten:
|
||||
`inventory.json`, exact three-field `snapshot-manifest.json` and `export-report.json`.
|
||||
Manifest `capturedAt` is measured immediately after successful backup, not startup,
|
||||
mtime or an example. Report records actual projected row count, inventory digest,
|
||||
method/exclusions; it is not a signature or proof of source authenticity. No live
|
||||
capture was performed in this change. Never reuse fixture manifests as evidence.
|
||||
|
||||
Projected HTTP readers never consult gateway config, state DB or cron files:
|
||||
model/provider are explicit `unknown`, context length is a placeholder zero,
|
||||
cron arrays empty mean **not captured**, session activity false is a placeholder,
|
||||
profile inventory doesn't establish live process availability. Status retains
|
||||
`liveProcess:unknown`. Usage days are relative to actual capture time. Snapshot
|
||||
fresh/stale provenance from the existing adapter remains mandatory. No real LLM
|
||||
pilot readiness follows from this inventory responding 200.
|
||||
|
||||
## Exact parent compose/application wiring
|
||||
|
||||
`infra/hermes-runtime/compose.stage-provision.yml` is an opt-in merge fragment,
|
||||
NOT called by the existing deploy script or ci-apply.sh. Parent must include it
|
||||
in **every** Stage compose command after provisioning, using the existing stable
|
||||
Compose project identity and existing release isolation/volume rules:
|
||||
|
||||
```sh
|
||||
export STAGE_RUNTIME_SERVE_FILE="$PINNED_RELEASE/infra/hermes-runtime/serve.py"
|
||||
export STAGE_RUNTIME_SNAPSHOT="/opt/services/canina/stage-runtime/snapshots/$UNIQUE_CI_CAPTURE_ID"
|
||||
# Only inside approved CI rollout, never manually:
|
||||
# docker compose -p <existing-stage-project> -f <parent-stage-compose> \
|
||||
# -f "$PINNED_RELEASE/infra/hermes-runtime/compose.stage-provision.yml" \
|
||||
# --profile stage-runtime-inventory <parent-approved command>
|
||||
```
|
||||
|
||||
The fragment adds backend `PREPARE_WORKER_ENABLED=true`, fixed-path secrets
|
||||
`/run/secrets/prepare_worker_grants`, `/run/secrets/hermes_runtime_grants`,
|
||||
`hermes_username/password`, and artifact bind at
|
||||
`/var/lib/canina/prepare-artifacts`. It adds a project-scoped **internal** network
|
||||
between backend and inventory only, alias `canina-hermes-inventory` (not the live
|
||||
`canina-hermes`). No host port, Docker socket or live gateway volume. Inventory is
|
||||
read-only, nonroot, no capabilities, restricted tmpfs and pinned image, mounts only
|
||||
its immutable projected export and script. `create_host_path:false` fails on missing
|
||||
exports/storage. Original compose.readonly.yml/ci-apply.sh remain legacy review-only;
|
||||
do not combine those with this provisioning fragment.
|
||||
|
||||
AppModule must keep AuthModule/JwtStrategy, import
|
||||
`RuntimeReadModule.registerFromFile('/run/secrets/hermes_runtime_grants')` for reads,
|
||||
and spread `...registerPrepareWorker()` for the opt-in worker transport (verify
|
||||
existing AppModule before adding duplicates). The latter defaults to no module
|
||||
unless exact env `true`; enabling compose alone cannot add missing imports. Routes
|
||||
remain service-authenticated. Artifact base is fixed to
|
||||
`https://stageapi.canina.ir/api/internal/prepare-worker/artifacts`; parent must verify
|
||||
its actual global `/api` prefix/proxy/authenticated retrieval after CI deployment.
|
||||
No URL here is asserted externally reachable by disposable tests.
|
||||
|
||||
Parent still must implement/review a bounded one-shot real LLM executor separately;
|
||||
this commit intentionally adds neither model/provider secrets nor automatic polling.
|
||||
After CI rollout verify backend UID/secret readability by names only, deny-all reads,
|
||||
worker unauth denial, snapshot provenance and authenticated artifact digest readback
|
||||
before a separately authorized task. Do not call RELEASE_AUTHORIZE.
|
||||
|
||||
## Executed disposable validation
|
||||
|
||||
Pinned Hermes image, `--network none`, repository mounted read-only; private
|
||||
TemporaryDirectory fixtures only. RED observed missing provisioner, missing exporter,
|
||||
wrong projected profile and acceptance of malformed credential; GREEN: four
|
||||
provision/export/security/CI-guard unittest cases; projected HTTP auth/profile/session/
|
||||
wrong-profile/model/cron assertions; existing exact-image HTTP acceptance, TCP serve
|
||||
mutation denial and authenticated WebSocket 1008 denial. Compose merged config
|
||||
`--quiet` passed without applying it. An initial test mount mismatch was corrected
|
||||
(`/source` for repo-relative tests, `/contract` for legacy tests). No application
|
||||
backend code changed; full backend suites not rerun. No host credentials, artifact
|
||||
storage, snapshots, DB users, services or networks were provisioned live.
|
||||
Reference in new issue
Block a user