Files
hermes-agent/gateway
0xbyt4 c2b9a517e0 fix(gateway): harden proxy mode SSE streaming — 3 resilience bugs
Proxy mode forwards platform messages to a remote Hermes API server via
SSE. The streaming loop introduced in 90c98345 had three robustness
gaps that could hang the gateway or truncate responses on imperfect
upstream behaviour.

1. `[DONE]` marker didn't break the outer chunk loop
   ---------------------------------------------------
   The `break` on `[DONE]` only exited the inner line-parse `while`,
   leaving the outer `async for chunk in resp.content.iter_any():` to
   keep reading. If the upstream held the connection open after
   `[DONE]` (buggy proxy, crashed server, network hang), the client
   waited up to sock_read=1800 seconds (30 min) for the next chunk.

   Fix: set a `done` flag when `[DONE]` is seen and check it at the
   top of the outer loop.

2. No TCP connect timeout
   -----------------------
   `ClientTimeout(total=0, sock_read=1800)` left `sock_connect` at
   the default `None` (no timeout). An unreachable proxy host (DNS
   fail, firewall, remote down) would hang on TCP connect for the OS
   default (minutes) before surfacing an error to the user.

   Fix: add `sock_connect=30` so connect failures surface within 30s.

3. SSE JSON parse exception handling was too narrow
   -------------------------------------------------
   The inner parse caught only `json.JSONDecodeError`. A response like
   `{"choices": [null]}` parsed successfully, then
   `choices[0].get("delta", {})` raised `AttributeError: 'NoneType'
   object has no attribute 'get'`. That bubbled up to the outer
   `except Exception`, aborting the entire stream — any further chunks
   were lost, and the user saw the accumulated partial response
   without knowing why.

   Fix: add type guards (`isinstance(choices, list)`, `isinstance(first,
   dict)`, `isinstance(delta, dict)`) and extend the caught exceptions
   to `(json.JSONDecodeError, TypeError, AttributeError)`. One bad
   chunk now skips, the stream keeps parsing.

New tests in `tests/gateway/test_proxy_mode.py::TestStreamingResilience`:

- `test_done_marker_stops_reading_trailing_chunks` — verifies trailing
  chunks after `[DONE]` are dropped (not appended to `full_response`)
- `test_client_timeout_sets_sock_connect` — captures the ClientTimeout
  kwargs and asserts `sock_connect` is set to a reasonable bound
- `test_malformed_chunk_is_skipped_not_fatal` — streams good/bad/good
  chunks and verifies both good chunks are captured, bad ones skipped
2026-09-12 21:18:24 -07:00
..