Add opt-in CI Stage credential provisioning and restricted SQLite inventory export

This commit is contained in:
devops committed 2026-10-23 16:49:09 +00:00
1 parent 4b147b0091
commit aa23c7ef91
6 files changed
+552

No files matched your search

+145
View File
@@ -0,0 +1,145 @@
# Opt-in Stage runtime provisioning (parent wiring contract)
This provisions **snapshot inventory** and credentials for the existing PREPARE
transport. It does NOT create a real LLM executor. The current deterministic worker
is still `llmPilot:false`. No provider credentials are collected, no cron installed,
no task claimed and no worker polling/service started by the provisioner.
## CI-only host invocation
Parent owns workflow/deploy.sh/AppModule; none are changed here. Run only through
the existing pinned authenticated SSH CI boundary, under its Stage environment
approval and serialized rollout lock. `CI=true` is an accident guard, not an auth
boundary. Provisioner checks clean HEAD equals the full approved commit; no branch
promotion, push, Production, RELEASE_AUTHORIZE, Docker or DB writes are performed.
Only the CI account should be able to invoke this on a root-owned trusted parent.
From the clean pinned release checkout on the host (not the dirty Stage checkout):
```sh
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000
```
Default output root is `/opt/services/canina/stage-runtime` and its parent must
already exist. First creation is staged and atomically renamed. Repeating validates
existing state without rotating credentials. Corrupt/missing/unsafe files fail
closed, not repaired. CI MUST serialize execution and prevent untrusted edits to
source/output parents; this is not a hostile-filesystem sandbox. `--root` is for
CI-reviewed test/Stage locations only, not arbitrary browser input. Secure backup
and separately approved rotation are parent/operator responsibilities.
- `secrets/` 0700, files 0400 owned by the runtime UID (1000 in the current backend
image); `hermes_username/password/signing_key` are new independent random values.
- `prepare_worker_grants` contains SHA256 only, exact separate service identities
`stage-prepare-worker`/WORKFLOW_WORKER and `stage-evidence-ci`/STAGE_CI. These are
transport service identities, **not invented DB admin users**.
- Raw `credentials/prepare_worker_token` and `credentials/stage_ci_token` stay in a
CI-owner-only 0700 directory, 0400 files. Do NOT mount either into inventory or
backend, log them, upload them as artifacts or send them to browsers. A later
reviewed one-shot executor receives only its own token through a private file.
- `artifacts/` is 0700 owned by backend UID; backend alone writes it. Persistent
local disk is not object storage or external publication evidence.
- `hermes_runtime_grants` is `{}`: deny all. No live admin query was needed/performed.
Parent must separately query actual Stage admin IDs read-only, review exact ID to
profile mapping and provision a private replacement via configuration management
before claiming panel access. Never infer identities from usernames or JWT input.
Deliberately changed grants require separate review; default provisioner refuses
nonempty runtime grants on repeat.
- Compose file-backed secrets preserve host permissions/ownership; `uid/gid/mode`
declarations do not fix bind mount ownership. Verify effective host IDs if Docker
uses userns remapping; do not chmod 0444 as a workaround.
## Consistent restricted export
After approval, the SAME CI invocation may explicitly request one profile's source
SQLite DB. Resolve its real `HERMES_HOME/state.db` through reviewed host mounts;
never guess a gateway volume path or mount the original home into inventory.
```sh
CI=true STAGE_PROVISION_ENABLED=true STAGE_PROVISION_APPROVED_SHA="$DEPLOY_SHA" \
python3 scripts/provision-stage-runtime.py --uid 1000 --gid 1000 \
--snapshot-db "$REVIEWED_STATE_DB" --profile default \
--snapshot-name "$UNIQUE_CI_CAPTURE_ID"
```
SQLite online `Connection.backup()` reads committed WAL frames into memory (never
`cp state.db`, never `immutable=1`). A 30-second callback deadline bounds busy
backup. Only explicit source/time/token/cost columns from `sessions` are projected.
Private session IDs become capture-local ordinal IDs; sources use an enum; no
messages, titles, prompts, user/chat IDs, paths, URLs, model config, credentials,
config.yaml, auth.json, .env, cron commands, memory or skills are exported. Unknown
schema/nonnumeric metadata fails. One profile per immutable capture; there is no
multi-DB atomicity claim. A source DB may be large: backup is memory-bound, so size
and host resources need operator review before a real capture.
Output directory is new/immutable (0500, 0400 files), never overwritten:
`inventory.json`, exact three-field `snapshot-manifest.json` and `export-report.json`.
Manifest `capturedAt` is measured immediately after successful backup, not startup,
mtime or an example. Report records actual projected row count, inventory digest,
method/exclusions; it is not a signature or proof of source authenticity. No live
capture was performed in this change. Never reuse fixture manifests as evidence.
Projected HTTP readers never consult gateway config, state DB or cron files:
model/provider are explicit `unknown`, context length is a placeholder zero,
cron arrays empty mean **not captured**, session activity false is a placeholder,
profile inventory doesn't establish live process availability. Status retains
`liveProcess:unknown`. Usage days are relative to actual capture time. Snapshot
fresh/stale provenance from the existing adapter remains mandatory. No real LLM
pilot readiness follows from this inventory responding 200.
## Exact parent compose/application wiring
`infra/hermes-runtime/compose.stage-provision.yml` is an opt-in merge fragment,
NOT called by the existing deploy script or ci-apply.sh. Parent must include it
in **every** Stage compose command after provisioning, using the existing stable
Compose project identity and existing release isolation/volume rules:
```sh
export STAGE_RUNTIME_SERVE_FILE="$PINNED_RELEASE/infra/hermes-runtime/serve.py"
export STAGE_RUNTIME_SNAPSHOT="/opt/services/canina/stage-runtime/snapshots/$UNIQUE_CI_CAPTURE_ID"
# Only inside approved CI rollout, never manually:
# docker compose -p <existing-stage-project> -f <parent-stage-compose> \
# -f "$PINNED_RELEASE/infra/hermes-runtime/compose.stage-provision.yml" \
# --profile stage-runtime-inventory <parent-approved command>
```
The fragment adds backend `PREPARE_WORKER_ENABLED=true`, fixed-path secrets
`/run/secrets/prepare_worker_grants`, `/run/secrets/hermes_runtime_grants`,
`hermes_username/password`, and artifact bind at
`/var/lib/canina/prepare-artifacts`. It adds a project-scoped **internal** network
between backend and inventory only, alias `canina-hermes-inventory` (not the live
`canina-hermes`). No host port, Docker socket or live gateway volume. Inventory is
read-only, nonroot, no capabilities, restricted tmpfs and pinned image, mounts only
its immutable projected export and script. `create_host_path:false` fails on missing
exports/storage. Original compose.readonly.yml/ci-apply.sh remain legacy review-only;
do not combine those with this provisioning fragment.
AppModule must keep AuthModule/JwtStrategy, import
`RuntimeReadModule.registerFromFile('/run/secrets/hermes_runtime_grants')` for reads,
and spread `...registerPrepareWorker()` for the opt-in worker transport (verify
existing AppModule before adding duplicates). The latter defaults to no module
unless exact env `true`; enabling compose alone cannot add missing imports. Routes
remain service-authenticated. Artifact base is fixed to
`https://stageapi.canina.ir/api/internal/prepare-worker/artifacts`; parent must verify
its actual global `/api` prefix/proxy/authenticated retrieval after CI deployment.
No URL here is asserted externally reachable by disposable tests.
Parent still must implement/review a bounded one-shot real LLM executor separately;
this commit intentionally adds neither model/provider secrets nor automatic polling.
After CI rollout verify backend UID/secret readability by names only, deny-all reads,
worker unauth denial, snapshot provenance and authenticated artifact digest readback
before a separately authorized task. Do not call RELEASE_AUTHORIZE.
## Executed disposable validation
Pinned Hermes image, `--network none`, repository mounted read-only; private
TemporaryDirectory fixtures only. RED observed missing provisioner, missing exporter,
wrong projected profile and acceptance of malformed credential; GREEN: four
provision/export/security/CI-guard unittest cases; projected HTTP auth/profile/session/
wrong-profile/model/cron assertions; existing exact-image HTTP acceptance, TCP serve
mutation denial and authenticated WebSocket 1008 denial. Compose merged config
`--quiet` passed without applying it. An initial test mount mismatch was corrected
(`/source` for repo-relative tests, `/contract` for legacy tests). No application
backend code changed; full backend suites not rerun. No host credentials, artifact
storage, snapshots, DB users, services or networks were provisioned live.
@@ -0,0 +1,66 @@
# Parent merges this file with its pinned Stage compose; CI opt-in only.
services:
backend_stage:
environment:
PREPARE_WORKER_ENABLED: "true"
secrets:
- hermes_username
- hermes_password
- hermes_runtime_grants
- prepare_worker_grants
volumes:
- type: bind
source: /opt/services/canina/stage-runtime/artifacts
target: /var/lib/canina/prepare-artifacts
bind:
create_host_path: false
networks:
- hermes_inventory_backend
runtime-inventory:
profiles: [stage-runtime-inventory]
image: nousresearch/hermes-agent@sha256:9774f4f39a9bb8c2f68ce728ed5e99ddbad282163be56764afacf88ed952b784
user: "1000:1000"
entrypoint: [/opt/hermes/.venv/bin/python3, /runtime/serve.py]
restart: unless-stopped
read_only: true
environment:
HERMES_HOME: /inventory
HERMES_SERVE_HEADLESS: "1"
HERMES_REQUIRE_PROJECTED_INVENTORY: "1"
PYTHONDONTWRITEBYTECODE: "1"
tmpfs: [/tmp:rw,noexec,nosuid,size=64m,mode=1777]
volumes:
- type: bind
source: ${STAGE_RUNTIME_SERVE_FILE:?absolute pinned serve.py required}
target: /runtime/serve.py
read_only: true
bind:
create_host_path: false
- type: bind
source: ${STAGE_RUNTIME_SNAPSHOT:?approved immutable projected snapshot required}
target: /inventory
read_only: true
bind:
create_host_path: false
secrets: [hermes_username, hermes_password, hermes_signing_key]
networks:
hermes_inventory_backend:
aliases: [canina-hermes-inventory]
security_opt: [no-new-privileges:true]
cap_drop: [ALL]
pids_limit: 128
mem_limit: 1g
networks:
hermes_inventory_backend:
internal: true
secrets:
hermes_username:
file: /opt/services/canina/stage-runtime/secrets/hermes_username
hermes_password:
file: /opt/services/canina/stage-runtime/secrets/hermes_password
hermes_signing_key:
file: /opt/services/canina/stage-runtime/secrets/hermes_signing_key
hermes_runtime_grants:
file: /opt/services/canina/stage-runtime/secrets/hermes_runtime_grants
prepare_worker_grants:
file: /opt/services/canina/stage-runtime/secrets/prepare_worker_grants
+42
View File
@@ -48,6 +48,48 @@ def build_app():
'liveGatewayState':'unknown','liveActiveAgents':None}
# Place before installed catch-all UI route.
web_server.app.router.routes.insert(0, web_server.app.router.routes.pop())
projected=Path(os.environ['HERMES_HOME'])/'inventory.json'
if os.environ.get('HERMES_REQUIRE_PROJECTED_INVENTORY')=='1' and not projected.is_file():
raise RuntimeError('Projected inventory required')
if projected.exists():
# Immutable exporter projection, not installed config/session/cron readers.
if projected.is_symlink() or projected.stat().st_size>32*1024*1024:
raise RuntimeError('Invalid projected inventory')
inventory=json.loads(projected.read_text())
if set(inventory)!={'schemaVersion','profile','sessions'} or inventory['schemaVersion']!=1 or not isinstance(inventory['sessions'],list):
raise RuntimeError('Invalid projected inventory')
profile=inventory['profile']
import re
if not isinstance(profile,str) or not re.fullmatch('[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}',profile):
raise RuntimeError('Invalid projected inventory')
replaced={'/api/profiles','/api/sessions','/api/analytics/usage','/api/cron/jobs','/api/model/info'}
web_server.app.router.routes[:]=[r for r in web_server.app.router.routes if getattr(r,'path',None) not in replaced]
async def projected_read(request:Request):
from fastapi import HTTPException
path=request.url.path
requested=request.query_params.get('profile',profile)
if requested!=profile: raise HTTPException(status_code=404)
if path=='/api/profiles':
return {'profiles':[{'name':profile,'is_default':profile=='default','gateway_running':False}]}
if path=='/api/model/info':
return {'model':'unknown','provider':'unknown','effective_context_length':0}
if path=='/api/cron/jobs': return [] # Cron commands deliberately not captured.
try:
limit=int(request.query_params.get('limit','20'));offset=int(request.query_params.get('offset','0'))
days=int(request.query_params.get('days','30'))
if not 1<=limit<=100 or not 0<=offset<=10000 or not 1<=days<=365: raise ValueError()
except ValueError: raise HTTPException(status_code=400)
rows=inventory['sessions']
if path=='/api/sessions':
return {'total':len(rows),'sessions':[{k:r[k] for k in ['id','profile','source','is_active']} for r in list(reversed(rows))[offset:offset+limit]],'storage':{}}
captured=datetime.fromisoformat(snapshot_provenance()['capturedAt'].replace('Z','+00:00')).timestamp()
filtered=[r for r in rows if captured-days*86400<=r['startedAt']<=captured]
return {'period_days':days,'totals':{'total_input':sum(r['inputTokens'] for r in filtered),
'total_output':sum(r['outputTokens'] for r in filtered),'total_sessions':len(filtered),
'total_estimated_cost':sum(r['estimatedCost'] for r in filtered)}}
for path in replaced:
web_server.app.add_api_route(path,projected_read,methods=['GET'])
web_server.app.router.routes.insert(0,web_server.app.router.routes.pop())
register_global_provider(BasicAuthProvider(username=secret('hermes_username'),password_hash=hash_password(secret('hermes_password')),secret=secret('hermes_signing_key').encode(),ttl_seconds=300))
web_server.app.state.auth_required=True
web_server.app.state.bound_host='0.0.0.0'
@@ -0,0 +1,22 @@
"""Projected inventory uses no gateway config or session store."""
import os, tempfile, json, importlib.util
from pathlib import Path
os.environ['HERMES_HOME'] = tempfile.mkdtemp()
os.environ['HERMES_SERVE_HEADLESS'] = '1'
root = Path(os.environ['HERMES_HOME'])
from datetime import datetime, timezone
(root/'snapshot-manifest.json').write_text(json.dumps({'schemaVersion':1,'source':'snapshot','capturedAt':datetime.now(timezone.utc).isoformat().replace('+00:00','Z')}))
(root/'inventory.json').write_text(json.dumps({'schemaVersion':1,'profile':'seo','sessions':[{'id':'snapshot-0','profile':'seo','source':'cli','startedAt':1,'inputTokens':4,'outputTokens':6,'estimatedCost':0.01,'is_active':False}]}))
spec=importlib.util.spec_from_file_location('serve',Path(__file__).parent/'serve.py');m=importlib.util.module_from_spec(spec);spec.loader.exec_module(m)
m.secret=lambda name:'projected-test-only-'+name+'x'*32
from fastapi.testclient import TestClient
c=TestClient(m.build_app(),base_url='http://localhost')
assert c.get('/api/profiles').status_code==401
r=c.post('/auth/password-login',json={'provider':'basic','username':m.secret('hermes_username'),'password':m.secret('hermes_password')});assert r.status_code==200
r=c.get('/api/profiles');assert [p['name'] for p in r.json()['profiles']]==['seo'],r.text
r=c.get('/api/sessions?profile=seo');assert r.json()['total']==1 and r.json()['sessions'][0]['id']=='snapshot-0',r.text
r=c.get('/api/sessions?profile=default');assert r.status_code==404,r.text
r=c.get('/api/model/info?profile=seo');assert r.json()['model']=='unknown',r.text
r=c.get('/api/cron/jobs?profile=seo');assert r.json()==[],r.text
assert not (root/'state.db').exists()
print('PASS projected profile/session/model/cron HTTP; no state DB/config used')
@@ -0,0 +1,75 @@
import importlib.util
import json
import os
from pathlib import Path
import sqlite3
import tempfile
import unittest
ROOT = Path(__file__).resolve().parents[2]
SPEC = importlib.util.spec_from_file_location('provision', ROOT / 'scripts/provision-stage-runtime.py')
class ProvisionTests(unittest.TestCase):
def load(self):
self.assertTrue(SPEC.origin and Path(SPEC.origin).exists(), 'Stage provisioner missing')
m = importlib.util.module_from_spec(SPEC); SPEC.loader.exec_module(m)
return m
def test_idempotent_dedicated_credentials_deny_all_runtime(self):
m = self.load()
with tempfile.TemporaryDirectory() as d:
root = Path(d) / 'target'
m.provision(root, 1000, 1000)
first = {p.name: p.read_bytes() for p in (root / 'secrets').iterdir()}
m.provision(root, 1000, 1000)
self.assertEqual(first, {p.name: p.read_bytes() for p in (root / 'secrets').iterdir()})
self.assertEqual(json.loads(first['hermes_runtime_grants']), {})
config = json.loads(first['prepare_worker_grants'])
self.assertNotEqual(config['workerIds'], config['stageCiIds'])
self.assertEqual(len({g['credentialSha256'] for g in config['grants']}), 2)
for p in (root / 'secrets').iterdir(): self.assertEqual(p.stat().st_mode & 0o777, 0o400)
self.assertEqual((root / 'artifacts').stat().st_mode & 0o777, 0o700)
def test_consistent_wal_export_projects_only_allowlisted_inventory(self):
m = self.load()
with tempfile.TemporaryDirectory() as d:
root = Path(d); db = root / 'state.db'
c = sqlite3.connect(db)
c.execute('pragma journal_mode=WAL')
c.execute('create table sessions(id text,source text,started_at real,input_tokens integer,output_tokens integer,estimated_cost_usd real,system_prompt text,title text)')
c.execute("insert into sessions values('private-person-id','cli',100,4,6,0.01,'NEVER_COPY_PROMPT','NEVER_COPY_TITLE')")
c.execute('create table messages(content text)')
c.execute("insert into messages values('NEVER_COPY_MESSAGE')"); c.commit()
(root / 'config.yaml').write_text('NEVER_COPY_KEY')
target = root / 'snapshot'
self.assertTrue(hasattr(m, 'export_snapshot'), 'Consistent inventory exporter missing')
m.export_snapshot(db, target, 'default', 1000, 1000)
text = ''.join(p.read_text() for p in target.iterdir())
for secret in ['NEVER_COPY', 'private-person-id', 'system_prompt', 'messages']:
self.assertNotIn(secret, text)
data = json.loads((target / 'inventory.json').read_text())
self.assertEqual(data['sessions'][0]['inputTokens'], 4)
self.assertEqual(data['sessions'][0]['source'], 'cli')
self.assertEqual(set(p.name for p in target.iterdir()), {'inventory.json','snapshot-manifest.json','export-report.json'})
self.assertEqual(json.loads((target / 'snapshot-manifest.json').read_text())['source'], 'snapshot')
with self.assertRaises(Exception): m.export_snapshot(db, target, 'default', 1000, 1000)
c.close()
def test_rejects_unsafe_paths_and_modified_credentials(self):
m = self.load()
with tempfile.TemporaryDirectory() as d:
root = Path(d) / 'target'; m.provision(root, 1000, 1000)
p = root / 'secrets/hermes_password'
p.chmod(0o600); p.write_text('compromised'); p.chmod(0o400)
with self.assertRaises(Exception): m.provision(root, 1000, 1000)
def test_cli_rejects_non_ci_before_mutation(self):
import subprocess
with tempfile.TemporaryDirectory() as d:
target = Path(d) / 'target'
env = {**os.environ, 'CI':'false'}
r = subprocess.run(['python3', str(SPEC.origin), '--root', str(target)], env=env, capture_output=True)
self.assertNotEqual(r.returncode, 0)
self.assertFalse(target.exists())
if __name__ == '__main__': unittest.main()
+202
View File
@@ -0,0 +1,202 @@
#!/usr/bin/env python3
"""Opt-in CI Stage provisioning. No Docker, DB writes, service start or polling."""
import argparse
import hashlib
import json
import os
from pathlib import Path
import re
import secrets
import shutil
import stat
import subprocess
import tempfile
BASE_URL = 'https://stageapi.canina.ir/api/internal/prepare-worker/artifacts'
def safe_path(path):
path = Path(path)
if not path.is_absolute() or '..' in path.parts:
raise ValueError('Absolute canonical path required')
for part in [*reversed(path.parents), path]:
if part.is_symlink():
raise ValueError('Symlink path rejected')
if part.exists() and not part.is_dir():
raise ValueError('Directory required')
return path
def write_private(path, value, uid, gid):
fd = os.open(path, os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_NOFOLLOW, 0o400)
with os.fdopen(fd, 'w') as f:
f.write(value)
f.flush(); os.fsync(f.fileno())
os.fchown(f.fileno(), uid, gid)
def provision(root, uid, gid):
root = safe_path(root)
if root.exists():
# Idempotence never silently rotates or repairs potentially compromised state.
validate(root, uid, gid)
return
if not root.parent.is_dir():
raise ValueError('Provision parent must already exist')
staging = Path(tempfile.mkdtemp(prefix='.stage-provision-', dir=root.parent))
try:
for name, mode, owner in [('secrets', 0o700, uid), ('credentials', 0o700, os.getuid()), ('artifacts', 0o700, uid), ('snapshots', 0o700, os.getuid())]:
p = staging / name; p.mkdir(mode=mode); os.chown(p, owner, gid)
values = {name: secrets.token_urlsafe(48) for name in ['hermes_username', 'hermes_password', 'hermes_signing_key', 'prepare_worker_token', 'stage_ci_token']}
config = {'grants': [
{'id': 'stage-prepare-worker', 'role': 'WORKFLOW_WORKER', 'credentialSha256': hashlib.sha256(values['prepare_worker_token'].encode()).hexdigest()},
{'id': 'stage-evidence-ci', 'role': 'STAGE_CI', 'credentialSha256': hashlib.sha256(values['stage_ci_token'].encode()).hexdigest()}],
'workerIds': ['stage-prepare-worker'], 'stageCiIds': ['stage-evidence-ci'], 'artifactBaseUrl': BASE_URL}
for name in ['hermes_username', 'hermes_password', 'hermes_signing_key']:
write_private(staging / 'secrets' / name, values[name], uid, gid)
for name in ['prepare_worker_token', 'stage_ci_token']:
write_private(staging / 'credentials' / name, values[name], os.getuid(), os.getgid())
for name, value in [('hermes_runtime_grants', {}), ('prepare_worker_grants', config)]:
write_private(staging / 'secrets' / name, json.dumps(value), uid, gid)
# Root traversal only; each child controls its own readers.
staging.chmod(0o711)
validate(staging, uid, gid)
# Serialized CI invocation required; refuse a concurrent winner.
os.rename(staging, root)
finally:
if staging.exists(): shutil.rmtree(staging)
def validate(root, uid, gid):
expected = {'secrets', 'credentials', 'artifacts', 'snapshots'}
if set(p.name for p in root.iterdir()) != expected or root.stat().st_mode & 0o077 != 0o011:
raise ValueError('Unexpected provision root')
for folder in expected:
p = safe_path(root / folder); st = p.stat()
if stat.S_IMODE(st.st_mode) != 0o700 or st.st_uid != (uid if folder in {'secrets', 'artifacts'} else os.getuid()):
raise ValueError('Unsafe provision directory')
names = {'hermes_username', 'hermes_password', 'hermes_signing_key', 'hermes_runtime_grants', 'prepare_worker_grants'}
if {p.name for p in (root / 'secrets').iterdir()} != names:
raise ValueError('Unexpected secrets')
if {p.name for p in (root / 'credentials').iterdir()} != {'prepare_worker_token', 'stage_ci_token'}:
raise ValueError('Unexpected credentials')
for directory, owner in [('secrets', uid), ('credentials', os.getuid())]:
for p in (root / directory).iterdir():
st = p.lstat()
if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1 or st.st_uid != owner or stat.S_IMODE(st.st_mode) != 0o400 or not 1 <= st.st_size <= 16384:
raise ValueError('Unsafe credential file')
for name in ['hermes_username', 'hermes_password', 'hermes_signing_key']:
if not re.fullmatch('[A-Za-z0-9_-]{64}', (root / 'secrets' / name).read_text()):
raise ValueError('Invalid dedicated credential')
config = json.loads((root / 'secrets/prepare_worker_grants').read_text())
if set(config) != {'grants', 'workerIds', 'stageCiIds', 'artifactBaseUrl'} or len(config['grants']) != 2:
raise ValueError('Unexpected worker config')
for index, name in enumerate(['prepare_worker_token', 'stage_ci_token']):
grant = config['grants'][index]
if set(grant) != {'id', 'role', 'credentialSha256'} or grant['id'] != ['stage-prepare-worker', 'stage-evidence-ci'][index]:
raise ValueError('Unexpected worker identity')
if not re.fullmatch('[A-Za-z0-9_-]{64}', (root / 'credentials' / name).read_text()):
raise ValueError('Invalid service credential')
if hashlib.sha256((root / 'credentials' / name).read_bytes()).hexdigest() != grant['credentialSha256']:
raise ValueError('Credential digest mismatch')
if config['workerIds'] != ['stage-prepare-worker'] or config['stageCiIds'] != ['stage-evidence-ci'] or config['artifactBaseUrl'] != BASE_URL or [g['role'] for g in config['grants']] != ['WORKFLOW_WORKER', 'STAGE_CI']:
raise ValueError('Unexpected grants')
if json.loads((root / 'secrets/hermes_runtime_grants').read_text()) != {}:
raise ValueError('Runtime grants require separate verified-ID review')
def export_snapshot(database, destination, profile, uid, gid):
"""SQLite online backup -> explicit metadata projection; raw backup never leaves RAM."""
import sqlite3
import time
from datetime import datetime, timezone
if not re.fullmatch('[a-zA-Z0-9][a-zA-Z0-9_-]{0,63}', profile):
raise ValueError('Invalid profile')
database = Path(database)
safe_path(database.parent)
st = database.lstat()
if not stat.S_ISREG(st.st_mode) or st.st_nlink != 1:
raise ValueError('Regular unlinked database required')
destination = safe_path(destination)
if destination.exists() or not destination.parent.is_dir():
raise ValueError('New snapshot destination required')
# mode=ro includes committed WAL frames. Never immutable=1 on a live WAL database.
src = sqlite3.connect(database.as_uri() + '?mode=ro', uri=True, timeout=5)
backup = sqlite3.connect(':memory:')
deadline = time.monotonic() + 30
def progress(status, remaining, total):
if time.monotonic() > deadline: raise ValueError('Backup deadline exceeded')
try:
src.backup(backup, pages=256, progress=progress, sleep=0.05)
captured = datetime.now(timezone.utc).isoformat().replace('+00:00', 'Z')
# Fail closed on schema drift. No SELECT *, private IDs, titles, prompts, messages,
# config, provider credentials, cron commands, filesystem paths or attachments.
rows = backup.execute('SELECT source,started_at,input_tokens,output_tokens,estimated_cost_usd FROM sessions ORDER BY started_at').fetchall()
if len(rows) > 100000: raise ValueError('Inventory exceeds bound')
allowed_sources = {'cli', 'telegram', 'discord', 'slack', 'api', 'web', 'cron'}
sessions = []
for index, (source, started, inputs, outputs, cost) in enumerate(rows):
import math
numbers = [started, inputs or 0, outputs or 0, cost or 0]
if any(not isinstance(n, (int, float)) or not math.isfinite(n) or n < 0 for n in numbers):
raise ValueError('Invalid inventory numeric metadata')
sessions.append({'id': 'snapshot-' + str(index), 'profile': profile,
'source': source if source in allowed_sources else 'other',
'startedAt': started, 'inputTokens': inputs or 0,
'outputTokens': outputs or 0, 'estimatedCost': cost or 0,
'is_active': False})
inventory = {'schemaVersion': 1, 'profile': profile, 'sessions': sessions}
payload = json.dumps(inventory, allow_nan=False, separators=(',', ':'))
if len(payload.encode()) > 32 * 1024 * 1024: raise ValueError('Inventory exceeds bound')
finally:
backup.close(); src.close()
staging = Path(tempfile.mkdtemp(prefix='.snapshot-', dir=destination.parent))
try:
manifest = {'schemaVersion': 1, 'source': 'snapshot', 'capturedAt': captured}
report = {'schemaVersion': 1, 'method': 'sqlite-online-backup-to-memory-allowlist-projection',
'capturedAt': captured, 'sessions': len(sessions),
'inventorySha256': hashlib.sha256(payload.encode()).hexdigest(),
'excluded': ['credentials', 'config', 'prompts', 'transcripts', 'private-identities', 'cron-commands'],
'liveExecutor': False}
for name, value in [('inventory.json', payload), ('snapshot-manifest.json', json.dumps(manifest)), ('export-report.json', json.dumps(report))]:
write_private(staging / name, value, uid, gid)
os.chown(staging, uid, gid)
staging.chmod(0o500)
os.rename(staging, destination)
finally:
if staging.exists():
staging.chmod(0o700); shutil.rmtree(staging)
def ci_gate():
sha = os.environ.get('STAGE_PROVISION_APPROVED_SHA', '')
if os.environ.get('CI') != 'true' or os.environ.get('STAGE_PROVISION_ENABLED') != 'true' or not re.fullmatch('[0-9a-f]{40}', sha):
raise ValueError('Approved Stage CI invocation required')
repo = Path(__file__).resolve().parents[1]
def git(*args):
return subprocess.check_output(['git', '-C', str(repo), *args], text=True).strip()
if git('rev-parse', 'HEAD') != sha or git('status', '--porcelain'):
raise ValueError('Clean approved commit required')
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument('--root', type=Path, default=Path('/opt/services/canina/stage-runtime'))
parser.add_argument('--uid', type=int, default=1000)
parser.add_argument('--gid', type=int, default=1000)
parser.add_argument('--snapshot-db', type=Path)
parser.add_argument('--snapshot-name')
parser.add_argument('--profile', default='default')
args = parser.parse_args()
ci_gate()
if args.uid <= 0 or args.gid <= 0: raise ValueError('Non-root runtime identity required')
if bool(args.snapshot_db) != bool(args.snapshot_name) or (args.snapshot_name and not re.fullmatch('[a-zA-Z0-9_-]{1,80}', args.snapshot_name)):
raise ValueError('Paired snapshot database and safe unique name required')
provision(args.root, args.uid, args.gid)
if args.snapshot_db:
export_snapshot(args.snapshot_db, args.root / 'snapshots' / args.snapshot_name, args.profile, args.uid, args.gid)
print('Stage credentials and artifact storage validated; no services started')
if __name__ == '__main__':
try: main()
except Exception: raise SystemExit('Stage provisioning rejected (details suppressed)')