Files
canina/infra/hermes-editorial/TIMEOUT_ROOTCAUSE.md
T
2026-10-24 15:38:04 +00:00

4.9 KiB

Editorial timeout investigation (no inference replay)

Outcome and evidence

Investigated task 4a109290-3722-4e0d-acd5-118da297b959 context, latest canonical competitor session 20261024_132234_a57b57. Read container source, logs and SQLite using actual hermes UID10000; SQLite opened mode=ro. No API requests, task claims, paid inference, live code modifications, DB writes, deploys or Production operations performed.

Sources in canina-hermes:

  • /opt/data/profiles/canina-competitor-worker/state.db: latest session has 7 messages (user1/assistant3/tool3), 3 completed API calls, 3 tools; no end timestamp/reason. Session start 1792848171.040386.
  • /opt/data/profiles/canina-competitor-worker/logs/agent.log: startup plugins at 13:22:29.711; turn begins13:22:45.529; API1 latency6.2s completes13:22:57.313; search5.82s completes13:23:03.259; API2 latency4.5s completes13:23:07.772; extraction1 2.01s completes13:23:09.797; API3 latency21.7s completes13:23:31.528; extraction2 .97s completes13:23:32.508.
  • SQLite last_activity_at1792848225.5890484 =13:23:45.589 UTC, description receiving stream response, provenance unknown. This is positive evidence that final model streaming had begun after the tools, not a hung extraction. It does NOT reveal first-token/last-token timing or prove whether upstream subsequently stalled/failed.
  • Total logged tool duration8.80s. Completed API latency32.4s. Startup to turn ~15.8s. Last extraction ~62.8s after first plugin log; model stream active ~75.9s after first plugin log. Exact Popen launch time was not persisted.

Confirmed contract mismatch

infra/hermes-editorial/runner.py previously supplied CLI --run-budget120 and killed its process group at120s from Popen, discarding stdout on timeout. Installed CLI help confirms advisory wrapup semantics, not a subprocess deadline. Installed /opt/hermes/agent/turn_context.py:647 starts budget only during turn preparation (after startup); /opt/hermes/agent/turn_iteration_prep.py:183 checks80% wrapup only between iterations. /opt/hermes/agent/chat_completion_helpers.py:679-688 caps implicit stale waits to min(timeout,max(60,remaining*.5)); explicit timeouts do not yield to budget. Thus run-budget can exceed remaining outer time and cannot force a final envelope before the runner kill. Session evidence was available in stream init but discarded with TemporaryFile on failure.

PROFILE_TIMEOUT proves the runner deadline fired. No logged provider503/error for this session; a previous agent's provider503 is NOT evidence about this run. Underlying reason final response did not finish in available time remains upstream latency/stall versus other unlogged processing; do not label confirmed provider outage. Most recent activity narrows bottleneck to final model response, not search/extraction or account provisioning.

Narrow mitigation (not acceptance claim)

Keep120s/profile,540s/task and8turns unchanged. Supply advisory90s budget (or less for remaining task time), reserving30s startup/terminalization headroom. This is not a hard upstream guarantee: installed CLI minimum60s stale floor still requires the outer hard kill.

Persist atomic mode0600 per-invocation/per-profile diagnostics in runner-state every heartbeat and after cancellation. Allowlist only session ID, elapsed time, event type/counts and constant outcome. Never retain provider text/errors, prompts, tool outputs, tokens/lease secrets or raw stderr. Report timeout as timeout even when a partial text stream exists. Evidence is diagnostic, not retrieval proof or QA.

Tests: deterministic real child process timeout with partial init/tools/text; safe evidence and mode; malformed stream injection; deadline headroom; actual installed StreamJsonEmitter callbacks without agent/model. Existing heartbeat/process-group kill and protocol tests retained. Python discovery added to manual no-deploy runtime-contract workflow (stage rollout already discovers this directory).

Verification

RED: new timeout evidence/budget tests failed because execute lacked evidence_path and cli_budget was absent. GREEN: local suite with source copied read-only from the installed container emitter:25 discovered,24 passed,1 UID-drop fixture skipped (local runtime nonroot). Remote repository suite:25 discovered,23 passed,2 installed-emitter tests skipped (host has no Hermes installation); actual root UID-drop fixture passed. No inference or research acceptance/QA success claimed. Unknown reservations remain unchanged.

Limits and follow-up

Do not blindly rerun one-time CI activation or paid task. Parent can integrate this commit through normal CI and arrange a separately authorized Stage acceptance window. Headroom/evidence mitigation is tested; successful live research remains unverified. An exact provider error cannot be recovered from already discarded stream. Later diagnostics preserve session correlation and partial event counts without leaking content.