Compare commits
2
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8cfd44cd09 | ||
|
|
c66bc58f73 |
No files matched your search
@@ -0,0 +1,78 @@
|
||||
# Real bounded Tech/Dev Hermes pilot (experimental, NOT enabled)
|
||||
|
||||
Owned files only in `infra/hermes-pilot`; no existing core, module, tracer, profile or runtime config changed. No push/deploy/RELEASE_AUTHORIZE. Do not wire this into the existing tracer completion endpoint: that endpoint deliberately accepts only deterministic JSON.
|
||||
|
||||
## Fixed task and boundaries
|
||||
|
||||
Task `tech-dev-clamp-quantity-v1`: produce one <=2048-byte Python `clamp_quantity(quantity)` implementation, integer range 0..99. No input/browser prompt is accepted. The model has **zero tools**, no shell/browser/Git/deploy credentials. Trusted code alone writes `quantity.py`, runs separate AST-allowlisted QA (9 boundary cases), and commits an isolated, remote-free fixture repository. This is real generated-code QA, not application QA. Source commit is recorded as provenance, not represented as the fixture's base commit.
|
||||
|
||||
`infer.py` uses installed `run_agent.AIAgent`, verified from the installed signature and `agent/agent_init.py`; `enabled_toolsets=[]` is additionally enforced by asserting empty `agent.tools`. Existing approved `OPENAI_API_KEY` is read in memory only; no messaging/API-server credentials/auth pools copied. Separate temporary HERMES_HOME, no memories/context/background review/fallback. Provider/model pinned to current canina-hermes `custom`, `https://nr.parsaaghayi.ir/v1`, `Free`; no provider installation or account changes. Existing configs remain untouched. Model output is untrusted and never directly shell-executed; QA rejects all syntax except the single nested min/max return expression before isolated Python execution.
|
||||
|
||||
Installed image identity: `sha256:373ddd5e6649f525b17993e1c885cdf1352127f8e98fd329140a73237125e52f`, install stamp commit `818c13be1dc4fd28987e1e881a9408224afd4535`, v0.21.6. Discovery: `/opt/hermes/hermes chat --help` supports `--oneshot`, `--max-turns`, `--run-budget`; `/opt/hermes/.venv/bin/python` loads installed API. Official skills and https://hermes-agent.nousresearch.com/docs/llms.txt consulted. The initially guessed developer API docs URL returned404; installed source/signature is authoritative for this pinned build.
|
||||
|
||||
## PREPARE mapping — parent integration required
|
||||
|
||||
`executor.execute(lease, artifact_root, infer, heartbeat)` only accepts a fresh staging, first-attempt, 20-cent, 40hex-source lease and exact lease fields; heartbeat failure aborts before inference and before QA/commit. Trusted adapter must obtain this lease via existing authenticated claim AFTER independent Gate1/READY, verify the explicitly allowlisted task identity/scope/criteria server-side, and supply heartbeat connected to the authenticated backend. A caller-created dict is NOT an authorization. Never expose this Python interface to browser requests.
|
||||
|
||||
## Heartbeat runner and parent API (this revision)
|
||||
|
||||
After independent READY/Gate1, allowlist and provider-cap admission, parent constructs
|
||||
`runner = executor.pinned_inference(executor.IMAGE)` and calls
|
||||
`executor.execute(authenticated_lease, artifact_root, runner, authenticated_heartbeat)`.
|
||||
Only trusted Python may construct runners. Callback receives an unchanged COPY of the
|
||||
original lease binding (taskId/runId/leaseToken), must use authenticated server transport,
|
||||
and must raise on any non-2xx, ownership/fencing/expiry denial or malformed response.
|
||||
Explicit `False` also denies; `None` means success for existing raise-on-error adapters.
|
||||
Callback MUST impose <=5s network timeout, no retries; a blocked callback thread cannot
|
||||
be forcibly killed by Python. It is daemonized; its result cannot restart inference or
|
||||
extend a local lease. Parent must serialize executions per server lease; the runner is
|
||||
single-use and concurrent reuse of that runner is locked out. This is not a browser API.
|
||||
|
||||
Immediate initial heartbeat; dedicated ongoing thread waits 5s between acknowledged
|
||||
calls. With required <=5s transport ceiling, heartbeat starts are <=10s apart. A stalled
|
||||
heartbeat (10s without acknowledgment), exception or denial aborts actual subprocess
|
||||
inference, not just its caller. No leaseUntil mutation/local extension occurs. Fresh
|
||||
server-authorized heartbeats retain fencing; this code cannot create authorization.
|
||||
Guard checks also fence QA/artifact commit/report return. A denial during QA prevents
|
||||
report return; any partial local fixture is not accepted completion evidence.
|
||||
|
||||
Monotonic wall ceiling 90s with cancellation reserve: work aborts at85s, remote kill
|
||||
command <=2s and local reap <=1s; bounded QA3s/Git1s. Pinned runner uses one model
|
||||
call, max_iterations1/max_tokens160, run_budget80s, tools0, no retries. An independent
|
||||
in-container timeout85s survives loss of the Docker client. Cancellation addresses
|
||||
the actual in-container process session via a random PID file plus cancellation
|
||||
marker (handles cancellation before PID publication); it kills that process group.
|
||||
Cancellation transport failure raises and prevents success, but cannot prove immediate
|
||||
remote death; independent inner timeout is the fallback. Small random marker files
|
||||
in container /tmp are intentional fencing tombstones, not configuration changes.
|
||||
|
||||
`rehearse.py` remains synthetic, never a live lease. It uses this cancellable runner
|
||||
but is NOT admission evidence. Do not run it until provider-cap approval. Existing
|
||||
20-cent reservation remains UNKNOWN, never zero, and no completion/claim/release or
|
||||
backend transport is implemented here. Existing tracer endpoint still rejects model
|
||||
reports; parent must separately implement authenticated allowlisted artifact ingestion.
|
||||
No changes to installed Hermes or runtime/profile configuration.
|
||||
|
||||
## Actual evidence and blockers
|
||||
|
||||
Evidence directory `/opt/services/canina/validation/v4-real-pilot`:
|
||||
- `red.log`: missing executor assertion failed before implementation.
|
||||
- `inference-private.log`: actual installed Hermes/provider inference exit0 under75s diagnostic ceiling; generated source saved, not fabricated. This ceiling was ONLY diagnostic, not accepted as a PREPARE lease run.
|
||||
- `model-artifact/quantity.py`: actual model artifact; independent QA passes9 cases, SHA256 `0125246b551896c5f9ee98fabeb02982a6690a9974c0eb24ef3ae676d46380d5`; isolated Git commit `95cb3d27d636ae817d6a778e8053e2f87b50e57b`.
|
||||
- `actual.log`: bounded full rehearsal failed with inner timeout exit137 at18s; no completion/QA success was assumed. Previous client-only timeout was corrected to inner process kill. **Installed model/startup cannot currently complete reliably inside30s PREPARE lease without authenticated ongoing heartbeat.** Do not extend lease locally or call the75s diagnostic a bounded lease success.
|
||||
|
||||
Contract tests: `python3 infra/hermes-pilot/test_contract.py`. Parent still owns backend integration, authenticated heartbeat orchestration, allowlist admission, billing caps, signed artifact transport and Stage CI. This commit is a real generation/independent-QA proof and fail-closed executor seam, not a live autonomous Tech department or released product.
|
||||
|
||||
## Verification of heartbeat revision
|
||||
|
||||
`python3 -m unittest discover -s infra/hermes-pilot -v`: 11 passing tests, including
|
||||
real OS child/descendant cancellation, ongoing rejection, stalled heartbeat, wall
|
||||
ceiling, concurrent runner reuse, immutable lease and unknown-cost preservation.
|
||||
The artifact success test is explicitly a deterministic process fixture, NOT model
|
||||
inference. TDD red observed for missing ongoing runner and explicit false denial;
|
||||
additional cancellation/concurrency regression cases were added and executed.
|
||||
Previously generated real model artifact was rechecked by current AST allowlist QA:
|
||||
9 cases pass, SHA256 `0125246b551896c5f9ee98fabeb02982a6690a9974c0eb24ef3ae676d46380d5`.
|
||||
No new real inference was run: README provider pricing/cap gate remains unresolved.
|
||||
Docker cancellation path is implemented but not runtime-exercised here; OS process
|
||||
group cancellation is exercised. No live backend/claim/release, push or deploy.
|
||||
@@ -0,0 +1,64 @@
|
||||
"""Fixed Tech/Dev pilot. Adapter must supply a fresh authenticated PREPARE lease."""
|
||||
import datetime, hashlib, json, os, pathlib, re, subprocess, tempfile, time
|
||||
TASK='tech-dev-clamp-quantity-v1'
|
||||
IMAGE='sha256:373ddd5e6649f525b17993e1c885cdf1352127f8e98fd329140a73237125e52f'
|
||||
|
||||
def validate_lease(lease):
|
||||
if (set(lease) != {'taskId','runId','leaseToken','leaseUntil','commitSha','environment','maxCostCents','attempt'}
|
||||
or lease['environment']!='staging' or lease['maxCostCents']!=20 or lease['attempt']!=1
|
||||
or not re.fullmatch('[a-f0-9]{40}',lease['commitSha'])):
|
||||
raise ValueError('Bounded PREPARE lease required')
|
||||
for k in ('taskId','runId','leaseToken'):
|
||||
if not re.fullmatch('[a-f0-9-]{36}',lease[k]): raise ValueError('Invalid lease binding')
|
||||
expiry=datetime.datetime.fromisoformat(lease['leaseUntil'].replace('Z','+00:00')).timestamp()
|
||||
if not 5 < expiry-time.time() <= 31: raise ValueError('Fresh 30-second lease required')
|
||||
return lease
|
||||
|
||||
def _execute(lease, artifact_root, infer, heartbeat):
|
||||
"""No claim/approval generation. heartbeat must call trusted server; errors abort.
|
||||
infer is the pinned runner, not a browser-supplied command or prompt.
|
||||
Completion is deliberately NOT sent: existing transport accepts tracer JSON only.
|
||||
"""
|
||||
validate_lease(lease)
|
||||
heartbeat(lease)
|
||||
root=pathlib.Path(tempfile.mkdtemp(prefix='real-pilot-',dir=artifact_root))
|
||||
def git(*args):
|
||||
return subprocess.check_output(['git','-C',str(root),*args],stderr=subprocess.DEVNULL,text=True,timeout=1).strip()
|
||||
git('init','-q'); git('config','user.email','pilot@invalid'); git('config','user.name','Bounded Hermes Pilot')
|
||||
(root/'README').write_text('Disposable fixture; PREPARE source '+lease['commitSha']+'\n')
|
||||
git('add','README'); git('commit','-qm','Seed isolated pilot fixture')
|
||||
base=git('rev-parse','HEAD')
|
||||
# One cancellable inference, no tools; ongoing authenticated guard.
|
||||
source=infer()
|
||||
heartbeat(lease)
|
||||
if not isinstance(source,str) or len(source.encode())>2048: raise ValueError('Invalid generated artifact')
|
||||
(root/'quantity.py').write_text(source)
|
||||
qa=subprocess.check_output(['python3','-I',str(pathlib.Path(__file__).with_name('qa.py')),str(root)],timeout=3,text=True)
|
||||
heartbeat(lease)
|
||||
git('add','quantity.py'); git('commit','-qm','Hermes generated quantity clamp; independent QA passed')
|
||||
patch=git('diff',base,'HEAD','--','quantity.py')+'\n'
|
||||
(root/'artifact.patch').write_text(patch)
|
||||
report={'kind':'REAL_HERMES_TECH_DEV_PILOT','task':TASK,'taskId':lease['taskId'],'runId':lease['runId'],
|
||||
'sourceCommit':lease['commitSha'],'artifactCommit':git('rev-parse','HEAD'),'baseCommit':base,
|
||||
'patchSha256':hashlib.sha256(patch.encode()).hexdigest(),'qa':json.loads(qa),
|
||||
'costCents':None,'costStatus':'UNKNOWN_RETAIN_FULL_20_CENT_RESERVATION',
|
||||
'gate1':'EXTERNAL_LEASE_REQUIRED','published':False,'deploymentExecuted':False}
|
||||
(root/'evidence.json').write_text(json.dumps(report,indent=2)+'\n')
|
||||
return root,report
|
||||
|
||||
from runtime import LeaseGuard, ProcessInference, pinned_inference
|
||||
|
||||
def execute(lease, artifact_root, infer, heartbeat):
|
||||
validate_lease(lease)
|
||||
with LeaseGuard(lease, heartbeat) as guard:
|
||||
guard.check()
|
||||
if not isinstance(infer, ProcessInference):
|
||||
raise TypeError("Cancellable ProcessInference required")
|
||||
def generate():
|
||||
response = json.loads(infer.run(guard))
|
||||
if not isinstance(response,dict) or response.get("tools") != 0 or response.get("model") != "Free":
|
||||
raise ValueError("Pinned tool-less model required")
|
||||
return response["source"]
|
||||
result = _execute(lease, artifact_root, generate, lambda _: guard.check())
|
||||
guard.check()
|
||||
return result
|
||||
@@ -0,0 +1,22 @@
|
||||
"""Executed only inside pinned canina-hermes image by controlled runner."""
|
||||
import contextlib, io, json, os, pathlib, tempfile
|
||||
# Copy ONE existing inference credential in memory; no auth pools or messaging keys.
|
||||
from dotenv import dotenv_values
|
||||
key=dotenv_values('/opt/data/.env').get('OPENAI_API_KEY')
|
||||
if not key: raise RuntimeError('Existing OPENAI_API_KEY unavailable')
|
||||
home=tempfile.TemporaryDirectory(prefix='canina-real-pilot-')
|
||||
os.environ.clear()
|
||||
os.environ.update({'PATH':'/usr/bin:/bin','HOME':home.name,'HERMES_HOME':home.name,'OPENAI_API_KEY':key,'OPENAI_BASE_URL':'https://nr.parsaaghayi.ir/v1','HERMES_DISABLE_PLUGINS':'1'})
|
||||
# Installed API verified by inspect.signature at pinned image commit 818c13be.
|
||||
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
|
||||
from run_agent import AIAgent
|
||||
agent=AIAgent(model='Free',provider='custom',base_url='https://nr.parsaaghayi.ir/v1',api_key=key,
|
||||
enabled_toolsets=[],max_iterations=1,max_tokens=160,run_budget_seconds=80,
|
||||
skip_context_files=True,skip_memory=True,skip_background_review=True,load_soul_identity=False,
|
||||
quiet_mode=True,fallback_model={},cwd=home.name)
|
||||
if agent.tools: raise RuntimeError('Tool-less invariant failed')
|
||||
result=agent.run_conversation('Write only Python source, no markdown: define clamp_quantity(quantity) returning integer quantity clamped to inclusive 0..99. Use one return with nested min/max calls only. No annotations, comments, imports, decorators, docstrings or other statements.')
|
||||
final=result.get('final_response')
|
||||
if not isinstance(final,str) or not final.strip(): raise RuntimeError('No model artifact; result keys='+','.join(result.keys()))
|
||||
print(json.dumps({'source':final.strip()+'\n','model':'Free','tools':0}))
|
||||
home.cleanup()
|
||||
@@ -0,0 +1,26 @@
|
||||
"""Independent structural + behavior QA. Never executes arbitrary model Python."""
|
||||
import ast, hashlib, json, pathlib, sys
|
||||
|
||||
def verify(root):
|
||||
p=pathlib.Path(root)/'quantity.py'
|
||||
if p.is_symlink() or not p.is_file() or p.stat().st_size>2048: raise ValueError('Invalid file')
|
||||
raw=p.read_bytes(); tree=ast.parse(raw)
|
||||
if len(tree.body)!=1 or not isinstance(tree.body[0],ast.FunctionDef): raise ValueError('Single function required')
|
||||
fn=tree.body[0]
|
||||
if fn.name!='clamp_quantity' or fn.decorator_list or fn.returns or fn.args.defaults or fn.args.kw_defaults or fn.args.kwonlyargs or fn.args.posonlyargs or fn.args.vararg or fn.args.kwarg or len(fn.args.args)!=1 or fn.args.args[0].arg!='quantity' or fn.args.args[0].annotation or len(fn.body)!=1 or not isinstance(fn.body[0],ast.Return): raise ValueError('Unexpected function structure')
|
||||
def expr(n):
|
||||
if isinstance(n,ast.Constant) and type(n.value) is int and 0<=n.value<=99: return
|
||||
if isinstance(n,ast.Name) and n.id=='quantity': return
|
||||
if isinstance(n,ast.Call) and isinstance(n.func,ast.Name) and n.func.id in ('min','max') and len(n.args)==2 and not n.keywords:
|
||||
for a in n.args: expr(a)
|
||||
return
|
||||
raise ValueError('Nonallowlisted syntax')
|
||||
expr(fn.body[0].value)
|
||||
scope={'__builtins__':{},'min':min,'max':max}
|
||||
exec(compile(tree,'generated-quantity','exec'),scope)
|
||||
cases=[-1000,-1,0,1,50,98,99,100,10**30]
|
||||
for q in cases:
|
||||
actual=scope['clamp_quantity'](q)
|
||||
if type(actual) is not int or actual!=max(0,min(q,99)): raise ValueError('Behavior QA failed')
|
||||
return {'passed':True,'cases':len(cases),'sha256':hashlib.sha256(raw).hexdigest(),'scope':'quantity clamp fixture only'}
|
||||
if __name__=='__main__': print(json.dumps(verify(sys.argv[1])))
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Disposable model/artifact exercise, NOT a live Gate1 approval or service completion."""
|
||||
import datetime, json, pathlib, subprocess, time, uuid
|
||||
from executor import execute, IMAGE, pinned_inference
|
||||
|
||||
def main():
|
||||
image=subprocess.check_output(['docker','inspect','--format','{{.Image}}','canina-hermes'],text=True).strip()
|
||||
if image!=IMAGE: raise RuntimeError('Pinned installed image mismatch')
|
||||
infer=pinned_inference(IMAGE)
|
||||
deadline=time.monotonic()+90
|
||||
lease={k:str(uuid.uuid4()) for k in ('taskId','runId','leaseToken')}
|
||||
lease.update(leaseUntil=(datetime.datetime.now(datetime.timezone.utc)+datetime.timedelta(seconds=30)).isoformat(),commitSha=subprocess.check_output(['git','rev-parse','HEAD'],text=True).strip(),environment='staging',maxCostCents=20,attempt=1)
|
||||
def fixture_heartbeat(binding):
|
||||
if time.monotonic()>deadline: raise TimeoutError('Fixture hard deadline')
|
||||
root,report=execute(lease,'/opt/services/canina/validation/v4-real-pilot',infer,fixture_heartbeat)
|
||||
report['executionContext']='DISPOSABLE_FIXTURE_NOT_LIVE_PREPARE_LEASE'
|
||||
(root/'evidence.json').write_text(json.dumps(report,indent=2)+'\n')
|
||||
print(json.dumps({'artifactDirectory':str(root),'report':report}))
|
||||
if __name__=='__main__': main()
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Trusted subprocess boundary; no claim, lease extension or completion transport."""
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import signal
|
||||
import subprocess
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
|
||||
HEARTBEAT_INTERVAL = 5.0
|
||||
HEARTBEAT_TIMEOUT = 5.0
|
||||
MAX_WALL_SECONDS = 90.0
|
||||
|
||||
class LeaseGuard:
|
||||
def __init__(self, lease, heartbeat):
|
||||
self.lease, self.heartbeat = lease, heartbeat
|
||||
self.stop = threading.Event()
|
||||
self.ready = threading.Event()
|
||||
self.error = None
|
||||
self.started = time.monotonic()
|
||||
self.last = self.started
|
||||
self.thread = threading.Thread(target=self._run, daemon=True)
|
||||
|
||||
def _run(self):
|
||||
while not self.stop.is_set():
|
||||
try:
|
||||
if self.heartbeat(dict(self.lease)) is False:
|
||||
raise PermissionError("Authenticated heartbeat denied")
|
||||
self.last = time.monotonic()
|
||||
self.ready.set()
|
||||
except BaseException as exc:
|
||||
self.error = exc
|
||||
self.ready.set()
|
||||
return
|
||||
if self.stop.wait(HEARTBEAT_INTERVAL):
|
||||
return
|
||||
|
||||
def check(self):
|
||||
if self.error is not None:
|
||||
raise self.error
|
||||
now = time.monotonic()
|
||||
if now - self.started >= MAX_WALL_SECONDS - 5:
|
||||
raise TimeoutError('Pilot wall deadline (cancellation reserve included)')
|
||||
if now - self.last >= HEARTBEAT_INTERVAL + HEARTBEAT_TIMEOUT:
|
||||
raise TimeoutError('Authenticated heartbeat stalled')
|
||||
|
||||
def __enter__(self):
|
||||
self.thread.start()
|
||||
if not self.ready.wait(HEARTBEAT_TIMEOUT):
|
||||
self.stop.set()
|
||||
raise TimeoutError('Initial authenticated heartbeat stalled')
|
||||
try:
|
||||
self.check()
|
||||
except BaseException:
|
||||
self.stop.set()
|
||||
raise
|
||||
return self
|
||||
|
||||
def __exit__(self, *args):
|
||||
self.stop.set()
|
||||
self.thread.join(timeout=.05)
|
||||
|
||||
class ProcessInference:
|
||||
"""Trusted argv only. cancel_remote MUST kill the in-container process group."""
|
||||
def __init__(self, argv, input_text=None, cancel_remote=None):
|
||||
self.argv, self.input_text, self.cancel_remote = argv, input_text, cancel_remote
|
||||
self.used = False
|
||||
self.lock = threading.Lock()
|
||||
|
||||
def run(self, guard):
|
||||
with self.lock:
|
||||
if self.used:
|
||||
raise RuntimeError('Exactly one inference allowed; no retry')
|
||||
self.used = True
|
||||
guard.check()
|
||||
# Files avoid pipe deadlocks and unbounded captured output in memory.
|
||||
with tempfile.TemporaryFile(mode='w+') as incoming, tempfile.TemporaryFile(mode='w+') as out, tempfile.TemporaryFile(mode='w+') as err:
|
||||
incoming.write(self.input_text or '')
|
||||
incoming.seek(0)
|
||||
child = subprocess.Popen(self.argv, stdin=incoming, stdout=out, stderr=err, start_new_session=True)
|
||||
try:
|
||||
while child.poll() is None:
|
||||
guard.check()
|
||||
time.sleep(.02)
|
||||
guard.check()
|
||||
if child.returncode:
|
||||
raise RuntimeError('Inference failed (exit %d)' % child.returncode)
|
||||
out.seek(0)
|
||||
result = out.read(4097)
|
||||
if len(result) > 4096:
|
||||
raise ValueError('Oversized inference response')
|
||||
return result
|
||||
finally:
|
||||
# Always cancel remote, even when docker exec itself failed.
|
||||
try:
|
||||
if self.cancel_remote:
|
||||
self.cancel_remote()
|
||||
finally:
|
||||
try:
|
||||
os.killpg(child.pid, signal.SIGKILL)
|
||||
except ProcessLookupError:
|
||||
pass
|
||||
child.wait(timeout=1)
|
||||
|
||||
|
||||
def pinned_inference(image):
|
||||
actual = subprocess.check_output(['docker','inspect','--format','{{.Image}}','canina-hermes'], text=True, timeout=2).strip()
|
||||
if actual != image:
|
||||
raise RuntimeError('Pinned installed image mismatch')
|
||||
token = uuid.uuid4().hex
|
||||
pidfile = '/tmp/canina-pilot-' + token + '.pid'
|
||||
script = pathlib.Path(__file__).with_name('infer.py').read_text()
|
||||
# Session leader is created INSIDE the container. Inner kill is independent of client.
|
||||
wrapper = ('import os,signal,subprocess,sys; '
|
||||
'p=subprocess.Popen(["timeout","--signal=KILL","85","/opt/hermes/.venv/bin/python","-"],stdin=sys.stdin,start_new_session=True); '
|
||||
'open(%r,"w").write(str(p.pid)); '
|
||||
'os.path.exists(%r) and os.killpg(p.pid,signal.SIGKILL); sys.exit(p.wait())') % (pidfile, pidfile+'.cancel')
|
||||
cancel = ('import os,signal; p=%r; '
|
||||
'\nopen(p+".cancel","w").close()\ntry:\n pid=int(open(p).read()); os.killpg(pid,signal.SIGKILL)\n'
|
||||
'except (FileNotFoundError,ProcessLookupError): pass\n'
|
||||
'finally:\n'
|
||||
' try: os.unlink(p)\n except FileNotFoundError: pass\n') % pidfile
|
||||
def cancel_remote():
|
||||
subprocess.run(['docker','exec','canina-hermes','/opt/hermes/.venv/bin/python','-c',cancel], check=True, timeout=2, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
return ProcessInference(['docker','exec','-i','canina-hermes','/opt/hermes/.venv/bin/python','-c',wrapper], script, cancel_remote)
|
||||
@@ -0,0 +1,68 @@
|
||||
import json, pathlib, sys, tempfile, threading, time, unittest
|
||||
from unittest.mock import patch
|
||||
sys.path.insert(0,str(pathlib.Path(__file__).parent))
|
||||
import executor as m
|
||||
import runtime
|
||||
from test_contract import lease
|
||||
|
||||
class Cancellation(unittest.TestCase):
|
||||
def test_wall_deadline_kills_descendant(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
marker=pathlib.Path(root,'late')
|
||||
child=f'import pathlib,time; time.sleep(.7); pathlib.Path({str(marker)!r}).touch()'
|
||||
code=f'import subprocess,sys,time; subprocess.Popen([sys.executable,"-c",{child!r}]); time.sleep(5)'
|
||||
with patch.object(runtime,'MAX_WALL_SECONDS',5.2):
|
||||
with self.assertRaises(TimeoutError): m.execute(lease(),root,m.ProcessInference([sys.executable,'-c',code]),lambda _: None)
|
||||
time.sleep(.8)
|
||||
self.assertFalse(marker.exists())
|
||||
|
||||
def test_stalled_heartbeat_cancels_child(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
marker=pathlib.Path(root,'late'); release=threading.Event(); calls=[]
|
||||
def heartbeat(_):
|
||||
calls.append(time.monotonic())
|
||||
if len(calls)>1: release.wait(3)
|
||||
code=f'import time,pathlib; time.sleep(.7); pathlib.Path({str(marker)!r}).touch()'
|
||||
try:
|
||||
with patch.object(runtime,'HEARTBEAT_INTERVAL',.05),patch.object(runtime,'HEARTBEAT_TIMEOUT',.1):
|
||||
with self.assertRaises(TimeoutError): m.execute(lease(),root,m.ProcessInference([sys.executable,'-c',code]),heartbeat)
|
||||
finally: release.set()
|
||||
time.sleep(.8); self.assertFalse(marker.exists())
|
||||
|
||||
def test_fixture_artifact_qa_and_cost_reservation(self):
|
||||
# Explicit deterministic PROCESS fixture, never reported as real model inference.
|
||||
source='def clamp_quantity(quantity):\n return min(99, max(0, quantity))\n'
|
||||
response=json.dumps(dict(source=source,model='Free',tools=0))
|
||||
beats=[]; original=lease(); snapshot=dict(original)
|
||||
def heartbeat(binding):
|
||||
beats.append(time.monotonic()); binding['leaseUntil']='untrusted mutation'
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
infer=m.ProcessInference([sys.executable,'-c',f'import time; time.sleep(.25); print({response!r})'])
|
||||
with patch.object(runtime,'HEARTBEAT_INTERVAL',.05):
|
||||
_,report=m.execute(original,root,infer,heartbeat)
|
||||
self.assertEqual(original,snapshot)
|
||||
self.assertGreater(len(beats),2)
|
||||
self.assertIsNone(report['costCents'])
|
||||
self.assertEqual(report['costStatus'],'UNKNOWN_RETAIN_FULL_20_CENT_RESERVATION')
|
||||
with self.assertRaises(RuntimeError):
|
||||
with runtime.LeaseGuard(lease(),lambda _:None) as guard: infer.run(guard)
|
||||
|
||||
def test_concurrent_reuse_rejected(self):
|
||||
infer=m.ProcessInference([sys.executable,"-c","import time; time.sleep(.3); print(1)"])
|
||||
errors=[]
|
||||
with runtime.LeaseGuard(lease(),lambda _:None) as guard:
|
||||
t=threading.Thread(target=lambda:infer.run(guard)); t.start()
|
||||
time.sleep(.05)
|
||||
with self.assertRaises(RuntimeError): infer.run(guard)
|
||||
t.join()
|
||||
|
||||
def test_explicit_false_heartbeat_denied(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
with self.assertRaises(PermissionError):
|
||||
m.execute(lease(),root,m.ProcessInference([sys.executable,"-c","print(1)"]),lambda _:False)
|
||||
|
||||
def test_callable_not_cancellable_rejected(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
with self.assertRaises(TypeError): m.execute(lease(),root,lambda:'',lambda _:None)
|
||||
|
||||
if __name__=='__main__': unittest.main()
|
||||
@@ -0,0 +1,23 @@
|
||||
import datetime, importlib.util, pathlib, tempfile, unittest, uuid
|
||||
p=pathlib.Path(__file__).with_name('executor.py')
|
||||
spec=importlib.util.spec_from_file_location('pilot',p); m=importlib.util.module_from_spec(spec); spec.loader.exec_module(m)
|
||||
qs=importlib.util.spec_from_file_location('qa',p.with_name('qa.py')); qa=importlib.util.module_from_spec(qs); qs.loader.exec_module(qa)
|
||||
def lease():
|
||||
d={k:str(uuid.uuid4()) for k in ('taskId','runId','leaseToken')}; d.update(environment='staging',maxCostCents=20,attempt=1,commitSha='a'*40,leaseUntil=(datetime.datetime.now(datetime.timezone.utc)+datetime.timedelta(seconds=30)).isoformat()); return d
|
||||
class Contract(unittest.TestCase):
|
||||
def test_missing_lease_rejected(self):
|
||||
with self.assertRaises(ValueError): m.validate_lease({})
|
||||
def test_bounds(self):
|
||||
for k,v in [('environment','production'),('attempt',2),('maxCostCents',21),('commitSha','main'),('leaseUntil','2000-01-01T00:00:00+00:00')]:
|
||||
d=lease(); d[k]=v
|
||||
with self.assertRaises(ValueError): m.validate_lease(d)
|
||||
def test_heartbeat_denial_prevents_inference(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
def deny(_): raise PermissionError('killed')
|
||||
with self.assertRaises(PermissionError): m.execute(lease(),root,lambda:self.fail('must not infer'),deny)
|
||||
def test_qa_rejects_malicious_and_wrong_behavior(self):
|
||||
for source in ['import os\n','def clamp_quantity(quantity):\n return quantity\n','def clamp_quantity(quantity):\n return __import__("os").system("true")\n']:
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
pathlib.Path(root,'quantity.py').write_text(source)
|
||||
with self.assertRaises(ValueError): qa.verify(root)
|
||||
if __name__=='__main__': unittest.main()
|
||||
@@ -0,0 +1,24 @@
|
||||
import datetime, pathlib, sys, tempfile, threading, time, unittest
|
||||
from unittest.mock import patch
|
||||
sys.path.insert(0, str(pathlib.Path(__file__).parent))
|
||||
import executor as m
|
||||
import runtime
|
||||
from test_contract import lease
|
||||
|
||||
class Heartbeats(unittest.TestCase):
|
||||
def test_ongoing_denial_kills_real_child(self):
|
||||
with tempfile.TemporaryDirectory() as root:
|
||||
marker=pathlib.Path(root,'late')
|
||||
code=f'import time,pathlib; time.sleep(1); pathlib.Path({str(marker)!r}).touch()'
|
||||
beats=[]
|
||||
def beat(binding):
|
||||
beats.append(time.monotonic())
|
||||
if len(beats)>1: raise PermissionError('revoked')
|
||||
with patch.object(runtime,'HEARTBEAT_INTERVAL',.1):
|
||||
with self.assertRaises(PermissionError):
|
||||
m.execute(lease(),root,m.ProcessInference([sys.executable,'-c',code]),beat)
|
||||
time.sleep(1.1)
|
||||
self.assertFalse(marker.exists())
|
||||
self.assertGreaterEqual(len(beats),2)
|
||||
|
||||
if __name__=='__main__': unittest.main()
|
||||
Reference in new issue
Block a user