Files
hermes-agent/evals/gateway_completion
Teknium 9ee3d129b4 fix: wake idle process watches on their owning profile transport
Salvage #82561's idle watch consumer through the existing drain, retaining
notify-off suppression and retrying failed injections. Preserve #102647's
secondary-profile ownership for reconstructed sources, preflight, synthetic
completion turns, and raw process progress/final notices. Unavailable
transports defer rather than borrowing the primary bot or losing the event.

Keep retained transport provenance and alias-aware relay resolution. Add
three invariant tests and an offline real-process/Discord lifecycle eval.

Co-authored-by: trwpang <134544126+trwpang@users.noreply.github.com>
Co-authored-by: holny <holny@foxmail.com>
2026-09-07 14:16:57 -07:00
..
…

Completion delivery lifecycle probe

Run with a fresh HOME and HERMES_HOME using the repository virtualenv:

probe=$(mktemp -d)
HOME="$probe" HERMES_HOME="$probe/state" .venv/bin/python evals/gateway_completion/delivery_lifecycle.py

No remote model or Discord credentials are used. The probe starts actual terminal subprocesses, process-reader threads and async delegation workers; it uses the actual DiscordAdapter and runner lifecycle with only outbound transport and model boundary replaced locally. It checks notify completion, idle watch matching, raw notice-only behavior, async completion delivery, and unavailable raw API-owner retention. HERMES_EVAL_REPO selects another checkout for A/B.

On base 03f3b09222: terminal notify control succeeds, idle watch stays queued with zero turns and the probe fails. Fixed: idle watch wakes once, the later post-turn hook adds no duplicate, normal async Discord completion is delivered, and missing API transport remains pending at zero attempts. Durable restart evidence additionally used ten fresh subprocesses sharing SQLite: base exhausted eight attempts and dropped; fixed stayed pending/zero then delivered both batch siblings at one attempt, with one API delivery row and no model turn.