Files
hermes-agent/tests/gateway/test_relay_teardown_drain.py
victor-kyriazakos 6a7cf19302 fix(gateway,relay): stop frozen-preview finals and dropped idle-session delegation callbacks (#82592)
* fix(gateway): stop frozen-preview finals and dropped idle-session delegation callbacks

Two relay-plane delivery losses from the 2026-08-09 staging incident:

1. stream_consumer: the skip-redundant-finalize branch recorded _accumulated
   as the delivered turn-final payload even when the last ACKED edit was an
   earlier throttled preview snapshot, so delivered_final_matches reconciled
   True and the gateway suppressed the corrective final send — the user was
   left with a cut-off message ending in the streaming cursor. Extracted
   _mark_skip_redundant_finalize(): records the last acked wire payload
   (cursor-stripped), so a preview/final mismatch now returns False and the
   normal final send fires.

2. run.py: _classify_completion_target classified every ended parent session
   terminal unless it ended by compression. Idle/timeout session ends are the
   norm on scale-to-zero relay deployments and the chat route remains valid;
   completed async delegation results were terminally dropped. Ended parents
   now classify deliver unless the end was an explicit user boundary
   (session_reset / user_exit / session_switch).

* fix(relay): drain in-flight outbound frames before transport teardown

disconnect() failed every pending outbound future immediately with
'relay transport closed', so a trailing finalize edit racing turn
teardown was lost even though the connector socket could still serve
it. Bounded drain grace (5s) lets in-flight requests resolve; silent
connectors still tear down promptly. asyncio.wait (not gather+wait_for)
so a timeout doesn't cancel futures owned by the fail-remaining loop.

* fix(gateway): route completion injection through the alias-aware transport resolver

Third relay-plane delivery loss from the 2026-08-09 staging incidents: a
delegation batch completed while the gateway was up, the watcher drained
the event, and delivery vanished with no log line. _inject_watch_notification
resolved its adapter with a literal p.value == platform_name scan of
self.adapters — a relay-fronted gateway registers ONE adapter under
Platform.RELAY fronting N logical platforms, so 'slack' never matched and
the injection returned None ('no gateway route'), silently dropping the
completion. The handoff path already documents this exact trap and uses
resolve_delivery_transport; the injection path now does the same (native
wins; relay eligible only when it fronts the logical platform), with the
literal scan kept as fallback for stub runners and exotic platforms.

* fix(relay): clamp disconnect drain grace to the runner's adapter-disconnect budget

Review finding (JoaoMarcos44, #82592): a fixed 5.0s drain in front of the
three 1.0s sequential teardown awaits gives an 8.0s worst case inside the
runner's 5.0s asyncio.wait_for(adapter.disconnect()) — tripping it cancels
teardown mid-drain, skips the fail-pending loop, and leaves outbound
callers blocked until _OUTBOUND_TIMEOUT_S (30s). The effective grace is
now budget - 3*TEARDOWN - margin (env-aware via the same
HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT the runner reads), so the drain
can never push teardown past its caller's budget; a budget too small for
any drain disables it cleanly.

* test(gateway): pin the final-send suppression contract across a behaviour matrix

The gateway skips its own final send when the stream consumer claims the turn
final already reached the user. Every incident in that family — #71643 (stale
finalize snapshot), #78541 (payload-less multi-message split), #82656 (frozen
preview left with a visible cursor) — is the same failure: the consumer claimed
delivery for text the platform never rendered, so the corrective send was
suppressed and the answer was lost with no retry.

Each was fixed with a scenario test pinned to one branch of
GatewayStreamConsumer.run(). The got_done handler now has five sibling branches
that each set the suppression flags and record a turn-final payload, and nothing
checks them as a group: a new branch, or a new early `return True` in
_send_or_edit, can reintroduce the class without failing a test.

Pin the invariant instead of the branch — if the consumer offers the gateway any
signal it would trust, the complete final text must have reached the wire — and
assert it across {edit always / dies / never / lies} x {send always / never} x
{fresh-final on / off} x {clean / interrupted stream}.

The adapter records only frames that actually rendered, so an ACK the platform
drops does not count as delivery. 24 honest-transport scenarios hold the
invariant as a hard assertion. The 16 lying-transport scenarios are checked too;
the single combination that still violates it is reported as an expected
failure documenting the open exposure rather than asserting it away.

Refs #82656

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(gateway,relay): prime relay egress routing for synthetic injections + cap stale completion replay

Defect #4 from the 2026-08-09 staging incidents (upgrade-robustness):
after every gateway restart the durable async-delegation replay injected
completions correctly (post-741663cf1) but their replies bounced at the
connector — 'slack egress declined: target not routed to an onboarded
tenant'. The relay adapter re-attaches tenant discriminators
(metadata.scope_id / metadata.user_id) from per-chat caches warmed ONLY by
inbound traffic; synthetic turns race those cold caches on every deploy,
scale-to-zero wake, and crash recovery.

- relay adapter: prime_routing_cache() — feeds a synthetic event's
  session-store origin through the same _capture_scope used for real
  inbound (never raises).
- run.py injection path: prime the resolved adapter before handle_message
  (duck-typed; native adapters unaffected).
- async_delegation: 48h staleness cap in restore_undelivered_completions —
  a pending completion older than the cap is terminally dropped (payload
  stays queryable) instead of re-run as a fresh full-context turn; the
  post-restart replay of a July session burned a 102K-token context.

Also carried: JoaoMarcos44's suppression behaviour-matrix harness
(cherry-picked from #82676, authorship preserved) — 39 passed + 1 xfail
(the documented ACK-then-drop transport-honesty residue).

* test: use recent timestamps in restored-ownership fixtures

test_restore_stamps_restored_flag persisted its completion with epoch-era
toy timestamps (dispatched_at=1.0), which the new 48h replay staleness cap
correctly classifies as stale — the fixture then exercised the cap instead
of the restored-flag contract (CI slice 4 failure). Timestamps are now
now-relative; the staleness behavior itself is pinned separately in
test_relay_injection_egress_priming.py.

* fix(gateway,relay): close four review findings on the relay delivery fixes

Review follow-ups on this branch (NousResearch#82592):

1. HIGH — classifier/resolver mismatch (falsely-acknowledged loss).
   _classify_completion_target now returns "deliver" for idle-ended
   parents, but _resolve_async_delegation_session still dropped every
   non-compression-ended pin: the durable row was acked at adapter
   acceptance, then the injection died inside the pipeline with no
   retry — strictly worse than the honest terminal drop on main, and
   the delivery leg defect #2's fix depends on did not exist. The
   resolver now retargets non-user-boundary ends (idle/timeout/
   lifecycle) to the chat's current session — session_entry already IS
   the routing key's current session for the same chat — while user
   boundaries (session_reset / new_session / user_exit /
   session_switch) stay fail-closed. Both sides share one module-level
   _USER_BOUNDARY_END_REASONS so the verdict and the routing decision
   cannot drift again; a coherence test asserts deliver-verdicts
   resolve non-None across representative end reasons.

2. HIGH — drain clamp missed adapter-level spend. The effective drain
   grace budgeted drain + 3x teardown, but RelayAdapter.disconnect
   spends revocation-monitor teardown + go_idle time BEFORE the
   transport drain inside the same runner wait_for; worst case still
   blew the budget and cancelled teardown mid-drain (skipping the
   fail-pending loop). The adapter now measures its own elapsed time
   and threads the REMAINING budget into
   transport.disconnect(budget_s=...); legacy/stub transports without
   the keyword fall back to the no-arg signature.

3. P1 — _request_response racing disconnect() could register a future
   after the fail-pending loop already ran, stranding the caller for
   the full _OUTBOUND_TIMEOUT_S (30s). Fail fast with the same
   "relay transport closed" error once _closing is set.

4. P1 — _build_process_event_source's last-resort reconstruction
   dropped scope_id, so a scoped relay completion whose session-store
   origin was unavailable primed no tenant discriminator and could
   still bounce off the connector's fail-closed egress guard.
   scope_id now threads through the reconstructed SessionSource, with
   a warning when a scoped chat reconstructs without one.

All four: RED reproduced with the fix reverted, GREEN after; relay/
delegation delivery families pass (43 + 71 + 179 across the touched
suites); full tests/gateway run shows only failures already failing
identically on merge base 2446c8bb6 (env/dep issues).

* fix(gateway,relay): make pending-frame failure cancellation-safe; persist completion routing origin

Two remaining review findings on this branch (NousResearch#82592):

1. Cancellation could strand outbound waiters past the fail-pending
   loop. transport.disconnect() failed pending futures only at the END
   of the drain + three teardown awaits; a cancellation landing
   mid-drain (the runner's wait_for budget, an outer cleanup deadline)
   skipped the loop entirely and left registered futures unresolved —
   their callers blocked until _OUTBOUND_TIMEOUT_S (30s). The budget
   threading added earlier shrinks the window but is not a hard
   guarantee. The fail-pending loop (and the going_idle ack failure)
   now run in a `finally`, so no exit path — normal, error, or
   cancelled — can leave a registered future unresolved. Idempotent:
   done futures are skipped, a second disconnect() pass is a no-op.

2. Durable completions did not persist their routing origin, so the
   scope_id threading in the fallback SessionSource reconstruction had
   nothing to carry on the exact path it exists for (restart replay
   with session store + source cache gone): the async-delegation event
   producers never populated scope_id and the durable rows never
   stored it. Dispatch now snapshots the originating turn's
   scope_id/user_id/user_name from the session context
   (_capture_routing_origin — a new HERMES_SESSION_SCOPE_ID contextvar
   bound by the gateway at session-bind time alongside the existing
   vars), stores them in the existing task_json payload (no schema
   migration), and re-attaches them to all three completion-event
   shapes (live single, live batch, crash-recovery rebuild). The
   gateway's fallback reconstruction then primes both discriminators
   after a restart.

Tests: cancellation mid-drain -> every pending future resolves with
"relay transport closed" (mutation: moving the loop out of the finally
goes RED); second-pass disconnect idempotence; end-to-end
dispatch -> owner-death recovery -> event carries scope_id -> fallback
SessionSource primes it (mutations: dropping the dispatch capture or
the task_json persistence both go RED); live completion event carries
the origin. 94 passed + 1 xfailed across the delivery/delegation
suites; tests/tools delegation family 73 passed (2 collection errors
pre-existing on merge base 2446c8bb6).

---------

Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ben Barclay <ben@nousresearch.com>
2026-08-11 09:56:13 +10:00

102 lines
3.7 KiB
Python

"""Regression: relay transport teardown must drain in-flight outbound frames.
Staging incident 2026-08-09 (frozen preview "It launched but ▉"): a trailing
finalize edit was in flight — awaiting its ``outbound_result`` — when the
gateway tore down the relay transport. ``disconnect()`` failed the pending
future immediately with "relay transport closed", so the finalize edit never
reached the platform even though the connector socket was still perfectly
able to serve it.
Contract under test: ``disconnect()`` gives in-flight outbound requests a
bounded grace window to resolve before failing them; requests that resolve
inside the window succeed, and the teardown still completes promptly when
the connector never answers.
"""
import asyncio
import pytest
from gateway.relay.ws_transport import WebSocketRelayTransport
def _transport():
t = object.__new__(WebSocketRelayTransport)
t._closing = False
t._supervisor = None
t._reader = None
t._ws = None
t._pending = {}
t._going_idle_ack = None
return t
@pytest.mark.asyncio
async def test_disconnect_waits_for_inflight_outbound():
"""An outbound frame whose result arrives during teardown must resolve
successfully, not be failed with 'relay transport closed'."""
t = _transport()
fut = asyncio.get_running_loop().create_future()
t._pending["req-1"] = fut
async def _connector_answers_soon():
await asyncio.sleep(0.05)
if not fut.done():
fut.set_result({"success": True, "message_id": "m9"})
answer_task = asyncio.create_task(_connector_answers_soon())
await t.disconnect()
await answer_task
assert fut.done()
assert fut.exception() is None, (
"in-flight finalize edit was failed at teardown instead of being "
"allowed to complete within the drain grace window"
)
assert fut.result()["success"] is True
@pytest.mark.asyncio
async def test_disconnect_still_bounded_when_connector_silent():
"""No answer ever arrives: the drain must give up within the grace
bound and fail the future so callers don't hang."""
t = _transport()
fut = asyncio.get_running_loop().create_future()
t._pending["req-2"] = fut
started = asyncio.get_running_loop().time()
await t.disconnect()
elapsed = asyncio.get_running_loop().time() - started
assert fut.done() and fut.exception() is not None
assert elapsed < 10.0, f"teardown took {elapsed:.1f}s — grace must be bounded"
def test_drain_grace_clamps_to_disconnect_budget(monkeypatch):
"""Drain + the three sequential teardown awaits must fit the runner's
adapter-disconnect budget (default 5s): grace = budget - 3*1.0s - margin."""
from gateway.relay import ws_transport as wt
monkeypatch.delenv("HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT", raising=False)
grace = wt._disconnect_drain_grace_s()
budget = 5.0
reserved = 3 * wt._TEARDOWN_AWAIT_TIMEOUT_S
assert grace + reserved < budget, (
f"grace {grace}s + teardown awaits {reserved}s exceeds the runner's "
f"{budget}s disconnect budget — wait_for would cancel teardown "
"mid-drain and skip the fail-pending loop"
)
assert grace > 0, "budget default must still leave usable drain time"
def test_drain_grace_honors_env_budget(monkeypatch):
from gateway.relay import ws_transport as wt
monkeypatch.setenv("HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT", "12")
assert wt._disconnect_drain_grace_s() == wt._DISCONNECT_DRAIN_GRACE_S
monkeypatch.setenv("HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT", "3.5")
grace = wt._disconnect_drain_grace_s()
assert 0 <= grace <= 3.5 - 3 * wt._TEARDOWN_AWAIT_TIMEOUT_S
monkeypatch.setenv("HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT", "1")
assert wt._disconnect_drain_grace_s() == 0.0