Add fail-closed real Hermes Tech pilot and independent artifact QA proof
This commit is contained in:
1 parent
aa23c7ef91
commit
02badd0e25
6 files changed
+170
No files matched your search
@@ -0,0 +1,29 @@
|
||||
# Real bounded Tech/Dev Hermes pilot (experimental, NOT enabled)
|
||||
|
||||
Owned files only in `infra/hermes-pilot`; no existing core, module, tracer, profile or runtime config changed. No push/deploy/RELEASE_AUTHORIZE. Do not wire this into the existing tracer completion endpoint: that endpoint deliberately accepts only deterministic JSON.
|
||||
|
||||
## Fixed task and boundaries
|
||||
|
||||
Task `tech-dev-clamp-quantity-v1`: produce one <=2048-byte Python `clamp_quantity(quantity)` implementation, integer range 0..99. No input/browser prompt is accepted. The model has **zero tools**, no shell/browser/Git/deploy credentials. Trusted code alone writes `quantity.py`, runs separate AST-allowlisted QA (9 boundary cases), and commits an isolated, remote-free fixture repository. This is real generated-code QA, not application QA. Source commit is recorded as provenance, not represented as the fixture's base commit.
|
||||
|
||||
`infer.py` uses installed `run_agent.AIAgent`, verified from the installed signature and `agent/agent_init.py`; `enabled_toolsets=[]` is additionally enforced by asserting empty `agent.tools`. Existing approved `OPENAI_API_KEY` is read in memory only; no messaging/API-server credentials/auth pools copied. Separate temporary HERMES_HOME, no memories/context/background review/fallback. Provider/model pinned to current canina-hermes `custom`, `https://nr.parsaaghayi.ir/v1`, `Free`; no provider installation or account changes. Existing configs remain untouched. Model output is untrusted and never directly shell-executed; QA rejects all syntax except the single nested min/max return expression before isolated Python execution.
|
||||
|
||||
Installed image identity: `sha256:373ddd5e6649f525b17993e1c885cdf1352127f8e98fd329140a73237125e52f`, install stamp commit `818c13be1dc4fd28987e1e881a9408224afd4535`, v0.21.6. Discovery: `/opt/hermes/hermes chat --help` supports `--oneshot`, `--max-turns`, `--run-budget`; `/opt/hermes/.venv/bin/python` loads installed API. Official skills and https://hermes-agent.nousresearch.com/docs/llms.txt consulted. The initially guessed developer API docs URL returned404; installed source/signature is authoritative for this pinned build.
|
||||
|
||||
## PREPARE mapping — parent integration required
|
||||
|
||||
`executor.execute(lease, artifact_root, infer, heartbeat)` only accepts a fresh staging, first-attempt, 20-cent, 40hex-source lease and exact lease fields; heartbeat failure aborts before inference and before QA/commit. Trusted adapter must obtain this lease via existing authenticated claim AFTER independent Gate1/READY, verify the explicitly allowlisted task identity/scope/criteria server-side, and supply heartbeat connected to the authenticated backend. A caller-created dict is NOT an authorization. Never expose this Python interface to browser requests.
|
||||
|
||||
Budget: one model inference, max_iterations1/max_tokens160, model run budget15s, process kill18s, QA3s; outer rehearsal kill28s. The18s kill is **inside** `docker exec` so killing its client cannot leave inference running. No retries. 20-cent core reservation MUST remain reserved for unknown billing; there is no measured-dollar guarantee from this provider, so live admission must remain disabled until provider-side pricing/cost cap is independently verified. Do not reconcile to0 or claim full end-to-end success. No Stage evidence/Gate2/release calls exist.
|
||||
|
||||
`rehearse.py` is explicitly disposable: synthetic lease IDs/heartbeat local deadline, never calls core, never claims real Gate1. Run from repository root: `timeout 28s python3 infra/hermes-pilot/rehearse.py`. This is NOT a supported live adapter. Parent must preserve lease fencing and heartbeat across any longer runner and implement trusted artifact ingestion/new report verification rather than bypass tracer validation.
|
||||
|
||||
## Actual evidence and blockers
|
||||
|
||||
Evidence directory `/opt/services/canina/validation/v4-real-pilot`:
|
||||
- `red.log`: missing executor assertion failed before implementation.
|
||||
- `inference-private.log`: actual installed Hermes/provider inference exit0 under75s diagnostic ceiling; generated source saved, not fabricated. This ceiling was ONLY diagnostic, not accepted as a PREPARE lease run.
|
||||
- `model-artifact/quantity.py`: actual model artifact; independent QA passes9 cases, SHA256 `0125246b551896c5f9ee98fabeb02982a6690a9974c0eb24ef3ae676d46380d5`; isolated Git commit `95cb3d27d636ae817d6a778e8053e2f87b50e57b`.
|
||||
- `actual.log`: bounded full rehearsal failed with inner timeout exit137 at18s; no completion/QA success was assumed. Previous client-only timeout was corrected to inner process kill. **Installed model/startup cannot currently complete reliably inside30s PREPARE lease without authenticated ongoing heartbeat.** Do not extend lease locally or call the75s diagnostic a bounded lease success.
|
||||
|
||||
Contract tests: `python3 infra/hermes-pilot/test_contract.py`. Parent still owns backend integration, authenticated heartbeat orchestration, allowlist admission, billing caps, signed artifact transport and Stage CI. This commit is a real generation/independent-QA proof and fail-closed executor seam, not a live autonomous Tech department or released product.
|
||||
@@ -0,0 +1,47 @@
|
||||
"""Fixed Tech/Dev pilot. Adapter must supply a fresh authenticated PREPARE lease."""
|
||||
import datetime, hashlib, json, os, pathlib, re, subprocess, tempfile, time
|
||||
TASK='tech-dev-clamp-quantity-v1'
|
||||
IMAGE='sha256:373ddd5e6649f525b17993e1c885cdf1352127f8e98fd329140a73237125e52f'
|
||||
|
||||
def validate_lease(lease):
|
||||
if (set(lease) != {'taskId','runId','leaseToken','leaseUntil','commitSha','environment','maxCostCents','attempt'}
|
||||
or lease['environment']!='staging' or lease['maxCostCents']!=20 or lease['attempt']!=1
|
||||
or not re.fullmatch('[a-f0-9]{40}',lease['commitSha'])):
|
||||
raise ValueError('Bounded PREPARE lease required')
|
||||
for k in ('taskId','runId','leaseToken'):
|
||||
if not re.fullmatch('[a-f0-9-]{36}',lease[k]): raise ValueError('Invalid lease binding')
|
||||
expiry=datetime.datetime.fromisoformat(lease['leaseUntil'].replace('Z','+00:00')).timestamp()
|
||||
if not 5 < expiry-time.time() <= 31: raise ValueError('Fresh 30-second lease required')
|
||||
return lease
|
||||
|
||||
def execute(lease, artifact_root, infer, heartbeat):
|
||||
"""No claim/approval generation. heartbeat must call trusted server; errors abort.
|
||||
infer is the pinned runner, not a browser-supplied command or prompt.
|
||||
Completion is deliberately NOT sent: existing transport accepts tracer JSON only.
|
||||
"""
|
||||
validate_lease(lease)
|
||||
heartbeat(lease)
|
||||
root=pathlib.Path(tempfile.mkdtemp(prefix='real-pilot-',dir=artifact_root))
|
||||
def git(*args):
|
||||
return subprocess.check_output(['git','-C',str(root),*args],stderr=subprocess.DEVNULL,text=True).strip()
|
||||
git('init','-q'); git('config','user.email','pilot@invalid'); git('config','user.name','Bounded Hermes Pilot')
|
||||
(root/'README').write_text('Disposable fixture; PREPARE source '+lease['commitSha']+'\n')
|
||||
git('add','README'); git('commit','-qm','Seed isolated pilot fixture')
|
||||
base=git('rev-parse','HEAD')
|
||||
# One inference, no tools, 18s hard outer cap in supplied pinned runner.
|
||||
source=infer()
|
||||
heartbeat(lease)
|
||||
if not isinstance(source,str) or len(source.encode())>2048: raise ValueError('Invalid generated artifact')
|
||||
(root/'quantity.py').write_text(source)
|
||||
qa=subprocess.check_output(['python3','-I',str(pathlib.Path(__file__).with_name('qa.py')),str(root)],timeout=3,text=True)
|
||||
heartbeat(lease)
|
||||
git('add','quantity.py'); git('commit','-qm','Hermes generated quantity clamp; independent QA passed')
|
||||
patch=git('diff',base,'HEAD','--','quantity.py')+'\n'
|
||||
(root/'artifact.patch').write_text(patch)
|
||||
report={'kind':'REAL_HERMES_TECH_DEV_PILOT','task':TASK,'taskId':lease['taskId'],'runId':lease['runId'],
|
||||
'sourceCommit':lease['commitSha'],'artifactCommit':git('rev-parse','HEAD'),'baseCommit':base,
|
||||
'patchSha256':hashlib.sha256(patch.encode()).hexdigest(),'qa':json.loads(qa),
|
||||
'costCents':None,'costStatus':'UNKNOWN_RETAIN_FULL_20_CENT_RESERVATION',
|
||||
'gate1':'EXTERNAL_LEASE_REQUIRED','published':False,'deploymentExecuted':False}
|
||||
(root/'evidence.json').write_text(json.dumps(report,indent=2)+'\n')
|
||||
return root,report
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Executed only inside pinned canina-hermes image by controlled runner."""
|
||||
import contextlib, io, json, os, pathlib, tempfile
|
||||
# Copy ONE existing inference credential in memory; no auth pools or messaging keys.
|
||||
from dotenv import dotenv_values
|
||||
key=dotenv_values('/opt/data/.env').get('OPENAI_API_KEY')
|
||||
if not key: raise RuntimeError('Existing OPENAI_API_KEY unavailable')
|
||||
home=tempfile.TemporaryDirectory(prefix='canina-real-pilot-')
|
||||
os.environ.clear()
|
||||
os.environ.update({'PATH':'/usr/bin:/bin','HOME':home.name,'HERMES_HOME':home.name,'OPENAI_API_KEY':key,'OPENAI_BASE_URL':'https://nr.parsaaghayi.ir/v1','HERMES_DISABLE_PLUGINS':'1'})
|
||||
# Installed API verified by inspect.signature at pinned image commit 818c13be.
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
from run_agent import AIAgent
|
||||
agent=AIAgent(model='Free',provider='custom',base_url='https://nr.parsaaghayi.ir/v1',api_key=key,
|
||||
enabled_toolsets=[],max_iterations=1,max_tokens=160,run_budget_seconds=15,
|
||||
skip_context_files=True,skip_memory=True,skip_background_review=True,load_soul_identity=False,
|
||||
quiet_mode=True,fallback_model={},cwd=home.name)
|
||||
if agent.tools: raise RuntimeError('Tool-less invariant failed')
|
||||
result=agent.run_conversation('Write only Python source, no markdown: define clamp_quantity(quantity) returning integer quantity clamped to inclusive 0..99. Use one return with nested min/max calls only. No annotations, comments, imports, decorators, docstrings or other statements.')
|
||||
final=result.get('final_response')
|
||||
if not isinstance(final,str) or not final.strip(): raise RuntimeError('No model artifact; result keys='+','.join(result.keys()))
|
||||
print(json.dumps({'source':final.strip()+'\n','model':'Free','tools':0}))
|
||||
home.cleanup()
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Independent structural + behavior QA. Never executes arbitrary model Python."""
|
||||
import ast, hashlib, json, pathlib, sys
|
||||
|
||||
def verify(root):
|
||||
p=pathlib.Path(root)/'quantity.py'
|
||||
if p.is_symlink() or not p.is_file() or p.stat().st_size>2048: raise ValueError('Invalid file')
|
||||
raw=p.read_bytes(); tree=ast.parse(raw)
|
||||
if len(tree.body)!=1 or not isinstance(tree.body[0],ast.FunctionDef): raise ValueError('Single function required')
|
||||
fn=tree.body[0]
|
||||
if fn.name!='clamp_quantity' or fn.decorator_list or fn.returns or fn.args.defaults or fn.args.kw_defaults or fn.args.kwonlyargs or fn.args.posonlyargs or fn.args.vararg or fn.args.kwarg or len(fn.args.args)!=1 or fn.args.args[0].arg!='quantity' or fn.args.args[0].annotation or len(fn.body)!=1 or not isinstance(fn.body[0],ast.Return): raise ValueError('Unexpected function structure')
|
||||
def expr(n):
|
||||
if isinstance(n,ast.Constant) and type(n.value) is int and 0<=n.value<=99: return
|
||||
if isinstance(n,ast.Name) and n.id=='quantity': return
|
||||
if isinstance(n,ast.Call) and isinstance(n.func,ast.Name) and n.func.id in ('min','max') and len(n.args)==2 and not n.keywords:
|
||||
for a in n.args: expr(a)
|
||||
return
|
||||
raise ValueError('Nonallowlisted syntax')
|
||||
expr(fn.body[0].value)
|
||||
scope={'__builtins__':{},'min':min,'max':max}
|
||||
exec(compile(tree,'generated-quantity','exec'),scope)
|
||||
cases=[-1000,-1,0,1,50,98,99,100,10**30]
|
||||
for q in cases:
|
||||
actual=scope['clamp_quantity'](q)
|
||||
if type(actual) is not int or actual!=max(0,min(q,99)): raise ValueError('Behavior QA failed')
|
||||
return {'passed':True,'cases':len(cases),'sha256':hashlib.sha256(raw).hexdigest(),'scope':'quantity clamp fixture only'}
|
||||
if __name__=='__main__': print(json.dumps(verify(sys.argv[1])))
|
||||
@@ -0,0 +1,23 @@
|
||||
"""Disposable model/artifact exercise, NOT a live Gate1 approval or service completion."""
|
||||
import datetime, json, pathlib, subprocess, time, uuid
|
||||
from executor import execute, IMAGE
|
||||
|
||||
def main():
|
||||
image=subprocess.check_output(['docker','inspect','--format','{{.Image}}','canina-hermes'],text=True).strip()
|
||||
if image!=IMAGE: raise RuntimeError('Pinned installed image mismatch')
|
||||
script=pathlib.Path(__file__).with_name('infer.py').read_text()
|
||||
# stdin is fixed tracked code, not arbitrary request text; stdout contains only generated source.
|
||||
def infer():
|
||||
r=subprocess.run(['docker','exec','-i','canina-hermes','timeout','--signal=KILL','18','/opt/hermes/.venv/bin/python','-'],input=script,text=True,capture_output=True,timeout=21)
|
||||
if r.returncode: raise RuntimeError('Pinned Hermes inference failed; diagnostic kept private in runner (exit '+str(r.returncode)+')')
|
||||
return json.loads(r.stdout)['source']
|
||||
deadline=time.monotonic()+25
|
||||
lease={k:str(uuid.uuid4()) for k in ('taskId','runId','leaseToken')}
|
||||
lease.update(leaseUntil=(datetime.datetime.now(datetime.timezone.utc)+datetime.timedelta(seconds=30)).isoformat(),commitSha=subprocess.check_output(['git','rev-parse','HEAD'],text=True).strip(),environment='staging',maxCostCents=20,attempt=1)
|
||||
def fixture_heartbeat(binding):
|
||||
if time.monotonic()>deadline: raise TimeoutError('Fixture hard deadline')
|
||||
root,report=execute(lease,'/opt/services/canina/validation/v4-real-pilot',infer,fixture_heartbeat)
|
||||
report['executionContext']='DISPOSABLE_FIXTURE_NOT_LIVE_PREPARE_LEASE'
|
||||
(root/'evidence.json').write_text(json.dumps(report,indent=2)+'\n')
|
||||
print(json.dumps({'artifactDirectory':str(root),'report':report}))
|
||||
if __name__=='__main__': main()
|
||||
@@ -0,0 +1,23 @@
|
||||
import datetime, importlib.util, pathlib, tempfile, unittest, uuid
|
||||
p=pathlib.Path(__file__).with_name('executor.py')
|
||||
spec=importlib.util.spec_from_file_location('pilot',p); m=importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
|
||||
qs=importlib.util.spec_from_file_location('qa',p.with_name('qa.py')); qa=importlib.util.module_from_spec(qs); qs.loader.exec_module(qa)
|
||||
def lease():
|
||||
d={k:str(uuid.uuid4()) for k in ('taskId','runId','leaseToken')}; d.update(environment='staging',maxCostCents=20,attempt=1,commitSha='a'*40,leaseUntil=(datetime.datetime.now(datetime.timezone.utc)+datetime.timedelta(seconds=30)).isoformat()); return d
|
||||
class Contract(unittest.TestCase):
|
||||
def test_missing_lease_rejected(self):
|
||||
with self.assertRaises(ValueError): m.validate_lease({})
|
||||
def test_bounds(self):
|
||||
for k,v in [('environment','production'),('attempt',2),('maxCostCents',21),('commitSha','main'),('leaseUntil','2000-01-01T00:00:00+00:00')]:
|
||||
d=lease(); d[k]=v
|
||||
with self.assertRaises(ValueError): m.validate_lease(d)
|
||||
def test_heartbeat_denial_prevents_inference(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
def deny(_): raise PermissionError('killed')
|
||||
with self.assertRaises(PermissionError): m.execute(lease(),root,lambda:self.fail('must not infer'),deny)
|
||||
def test_qa_rejects_malicious_and_wrong_behavior(self):
|
||||
for source in ['import os\n','def clamp_quantity(quantity):\n return quantity\n','def clamp_quantity(quantity):\n return __import__("os").system("true")\n']:
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
pathlib.Path(root,'quantity.py').write_text(source)
|
||||
with self.assertRaises(ValueError): qa.verify(root)
|
||||
if __name__=='__main__': unittest.main()
|
||||
Reference in new issue
Block a user