Item 2 of #112548: a Desktop/dashboard build that predates server→client
requests has no response path, so every clarify/approval/sudo/secret/vault/
connection/bridge request sat for the full deadline (clarify: 300s). Only the
tour probed. Clients now advertise once per connection
(`client.capabilities {server_requests: true}`, sent by the shared TypeScript
channel on `gateway.ready`); `send()` / `send_async()` return the
error-response shape (None) at once when every WebSocket peer of the session
is a build that never advertised. Sessions with no client attached still wait
so the reconnect replay (`open_requests`) keeps working; the stdio TUI ships
with the backend and is not gated. The advertisement is dropped on disconnect.
Reviewer minors from #113227:
- tools/approval_gateway_wait.py: the verdict is the choice committed under
the approval lock while leaving the queue, so an /approve that lands after
the deadline check but before the entry is dropped is an answer, not a
timeout (the client was already acked "ok").
- tests/tui_gateway/test_protocol.py: the error-fails-fast test that only
restated pre-existing behaviour is replaced by the two capability
invariants (never advertised → fails fast; advertised → frame written,
waits, forgotten on disconnect).
- server_requests.send try/finally around event.wait already landed on main
(4371ed34a9); nothing to change.
Docs: programmatic-integration.md (advertise once per connection; method
list), tui_gateway/AGENTS.md; contracts regenerated.
The Desktop pool caps local `hermes serve` children and holds each child's
slot lease for its lifetime. Its renderer refreshes lastActiveAt every 60s
for every open socket, so a bot-tile-pinned resident is keepalive-fresh
forever: occupied is not busy, and the pool cannot tell the difference. The
renderer's own turn bookkeeping cannot see cron fires (HERMES_DESKTOP=1 runs
the in-process ticker), messaging turns served by a pooled backend, or a
session blocked on an approval, so it is not a safe proof either.
Add `GET /api/health/idle` (token-gated; whether a turn is running is
activity recon, unlike the public liveness route) returning
`idle: true|false|null` from `hermes_cli/web_server_idle_proof.py`. It
reuses the SSH idle-exit primitive `turn_in_flight()` (running gateway
sessions + running cron jobs) and adds the human-input ledgers: open
server->client requests (`server_requests.open_request_count`) and queued
gateway approvals (`approval.pending_gateway_approval_count`). Any ledger
that cannot be read yields `null`, which the Desktop treats as busy.
Tests: unit invariants over the fail-closed table and the real ledgers, and
a live test that boots three desktop-shaped children (HERMES_DESKTOP=1,
per-child HERMES_HOME, port 0), holds one busy in the cron running-job
ledger, and probes all three over HTTP. RED on base: every child 404s.
The settlement rewrite dropped main's try/finally around event.wait():
a KeyboardInterrupt/SystemExit/injected error during the wait left the
request in _open forever — replayed to every reconnecting client as a
zombie card and reported by pending_kind() as a human still being
waited on. Pop it under _lock and emit request.cancel(interrupted)
before re-raising. Also fold the two settlement-wins tests into one
parametrized test (same invariant, two settlement paths).
`send()` decided `timed_out` from `event.wait()`'s return value and only then
withdrew the request from `_open`. A response frame accepted by
`resolve_response` in that gap was committed on the request and then thrown
away: `send()` returned None and emitted `request.cancel reason=timeout`, so
the clarify/approval card the user had just answered was torn down and the
answer evaporated — with `resolve_response` having reported True to the frame
router. Clarify and approval waits are exactly where a human answers at the
last second.
The verdict now comes from the state committed under `_lock`: `send()` pops
its own entry and reads `answered`/`result`/`locked` in one critical section;
an answered request returns its result whatever the wait said, and only a
request that was still open when `send()` withdrew it counts as a timeout. A
response for an id that is no longer open is logged at debug level instead
of dropped in silence.
Live A/B (commenter's scenarios, real module, threads): base A 20/20 lost,
B 32/1500 lost; #112559 alone A 0/20, B 30/1500; with this commit A 0/20,
B 0/3000; C (normal answer), D (plain timeout cancels once) and E (-32601
error frame fails fast) unchanged.
Second race window found by aron-intframe on #112548; the same pop-under-lock
verdict is in #112563 by @fangliquanflq.
Part of #112548
Co-authored-by: fangliquan <fangliquan@qq.com>
Handlers own their documented domain codes (4006 missing session_id, 4015 bad
url, 4009 orphan claim); the contract's job on the way in is the one check no
handler performs — an unknown key (4000 with the key path). Missing/mistyped
fields are re-checked AFTER a successful handler answer under the strict
test policy, so a contract narrower than the wire still fails the suite.
Two models widened from the suite: SeedMessage (clients forward stored rows
verbatim), tool.complete.args (mirrored child rows omit it). Tests that
drove session.activate with prompt params (and vice versa) or stubbed
_live_session_payload with a bare {session_id} now send the real shapes.
tui_gateway/contracts/ is the single source of truth for the JSON-RPC wire:
Params/Result/Payload bases, a registry of METHODS / SERVER_REQUESTS /
EVENTS, runtime validation (unknown/mistyped params answer 4000 with the
field path; a result or payload that violates its model raises under the
test suite and logs once in production), and the server-request contracts
+ shared value shapes as the authoring template. scripts/gen_gateway_contracts.py
renders the tables through a small JSON-Schema-subset walker into
apps/shared/src/gateway-contract.generated.ts and
gateway-contract.openrpc.json (unsupported constructs raise at generation).
Method contracts for the 213 remaining handlers follow in the next commits.
The gateway asked the user questions (approval, clarify, sudo, secret,
vault, MCP setup, the desktop read/act bridges) by emitting a
`<x>.request` EVENT carrying a hand-minted request_id, blocking the
agent thread on a module dict keyed by that id, and exposing a paired
`<x>.respond` METHOD per kind — thirteen pairs, four registries
(`_pending`, `_answers`, `_batch_clarify`, `_EXPIRING_REQUESTS`) and a
per-kind reconnect snapshot (`pending_clarify` / `pending_approval`)
that only two of the thirteen kinds ever got. JSON-RPC already has the
primitive: the server sends a request frame with an id and the client
answers with a response frame bearing the same id.
`tui_gateway/server_requests.py` owns the one mechanism:
send() block the agent thread until the response frame
(`srq-<n>` ids; ints belong to the client)
send_async() fire-and-callback variant (bot relay)
cancel*() withdraw with ONE `request.cancel {id, method, reason}`
event (timeout / interrupt / process exit /
answered elsewhere) instead of per-kind *.expire
open_requests() the still-open frames, replayed by session.resume,
session.activate and session.events.since so a
reconnecting client re-renders every kind, not two
clarify.lock stays a real client→server RPC (locks one batch
answer early); locked answers merge into the final
set even when the closing response carries only the
tail the user answered last
A client that does not implement a method answers -32601 and the agent
fails fast (the old fixed-timeout "unavailable" probes for tour/preview
still work — a wire error IS an answer). Approval: the queue entry's
settle hook withdraws the request when `/approve` from another surface,
a timeout or an interrupt resolves it first, so no window keeps a dead
card. Compute-host children own their waits; the parent mirrors their
open frames for replay and relays `clarify.lock` + response frames.
Clients: `JsonRpcRequestChannel` gains `onRequest` (unhandled → -32601,
dedup by id) and `JsonRpcGatewayClient` re-delivers `open_requests`
from the replay result. Desktop gets `gateway-event/server-requests.ts`
(one handler per method, replacing the request branches of
`input-requests.ts` / `desktop-bridge.ts`) and a `store/server-requests`
registry so every answer site calls `respondToServerRequest(id, result)`
synchronously; the TUI gets `createServerRequestHandler.ts` +
`serverRequestStore.ts`. `gateway-events.json` now pins both halves
(events + server request methods); the two contract tests check both.
Live (real stdio gateway, real `clarify_callback` on the agent thread):
before, `clarify.request` event + `clarify.respond` RPC, batch final
answers lost ('' returned); after, `{"id":"srq-…","method":"clarify"}`
frame, `session.events.since.open_requests` replays it, response frame
`{"answer":"yes"}` reaches the agent, batch lock + final response
merge to `{"q0":"1","q1":"free text"}`.
The backend never sent a JSON-RPC request; when it needed an answer from the
renderer it hand-correlated a `*.request` notification with a later `*.respond`
method through four module-level dicts, a timeout thread and 13 derived
`*.expire` names, plus a separate reconnect snapshot per prompt kind. That is a
second request/response layer built on a protocol that already has one.
`tui_gateway/server_requests.py` sends `{id: "srq-…", method, params}` and
blocks on the response frame with that id (string ids never collide with the
clients' integer ids). One `request.cancel {id, method, reason}` notification
withdraws a request on timeout / interrupt / session close. `open_requests` on
`session.resume` / `session.activate` / `session.events.since` re-delivers
unanswered requests after a reconnect; the shared TypeScript channel does that
itself before the caller sees the result. Batch clarify keeps its per-question
locks as a normal `clarify.lock` RPC (the last lock resolves the request).
Approvals stay queue-backed (`tools.approval` owns the timeout, `/approve all`,
coalescing): the request resolves the queue entry and the entry's own
resolution withdraws the request through `register_gateway_settle`.
Deleted: `_block`, `_respond`, `_pending`, `_answers`,
`_pending_prompt_payloads`, `_batch_clarify`, `_EXPIRING_REQUESTS`, the
`*.respond` methods, every `*.request` / `*.expire` event, `pending_clarify`.
Compute-host (turn isolation) mirrors the child's open request and relays the
response frame / lock to it. Desktop, TUI and shared clients register
`onRequest` handlers where they used to switch on `*.request` events; answers
are response frames over the socket the request arrived on, so #91684's
owner-routing class cannot recur for prompts.
