Independent-review minors on #126157 (#123989 class):
- session.create is a plain @method, so tui_gateway/session_workdir.py::_register_session_cwd
wrote the cwd record under the RAW session key while the scoped turn read
profile:<p>:<key> and missed it until the first `cd`. The writer now binds the
session's own profile_home around register_task_env_overrides.
- tools/terminal_tool.py::_task_env_overrides stayed raw-keyed, so two profiles
registering the same task id (docker_image / cwd) still collided. Writer, clear
and both readers (_has_isolation_overrides, resolve_task_overrides) now use
_qualify_task_key; the isolation-override branch of _resolve_container_task_id
returns the qualified key so the env cache and the override record agree.
- gateway/platforms/api_server.py::_derive_chat_session_id prefixed the seed for
the NAMED LAUNCH profile addressed via /p/<launch>/, so prefixed and
un-prefixed requests on one profile derived different ids. The prefix is now
skipped when the routed name is the process's launch profile
(_names_launch_profile vs get_routing_process_hermes_home()).
Tests (red on 3720519199c, green here):
tests/tui_gateway/test_session_cwd_profile_key.py::test_secondary_profile_session_cwd_is_found_inside_its_scope
tests/gateway/test_api_server.py::TestDeriveChatSessionId::test_launch_profile_prefix_keeps_the_unprefixed_id
Fold the two salvaged fixes to the class and the salvage bar:
- tools/terminal_tool.py: `_qualify_task_key` reuses `_routed_home_task_key`
instead of a duplicate qualifier; branch 2 (session-isolated sandboxes) AND
branch 3's `session:<key>` (SSH-style backends) carry the routed profile, so a
session id two profiles share (header-less API fingerprint, a DM chat id served
by two bots) never resolves to one `_active_environments` slot. Cwd records
keep the symmetric qualification. No routed home → raw key, unchanged.
- gateway/platforms/api_server.py: `_derive_chat_session_id` namespaces only
named routed profiles, so default/standalone ids are byte-identical and live
Open WebUI conversations survive the upgrade.
- Tests trimmed to two invariants: A→B→A across two temp profile homes under
multiplex (distinct sandbox keys, alias follows the qualified parent, cwd
isolated per profile), and derived-id namespacing without moving default.
Both fail on 5912ed81ed, pass here.
_derive_chat_session_id hashes only the system prompt + first user
message, so under gateway.multiplex_profiles two header-less
conversations that open with identical text on different profiles
derive the same api-<digest> id whichever /p/<profile>/ prefix they
arrive on. The id keys process-global resources — the session store
and the persistent Docker sandbox labeled hermes-task-id (= the session
id on this surface) — so the second profile's turns attach to the
first profile's container, under its image, volumes and labels,
breaking the documented per-profile container isolation (#123989).
Fold the request's profile (the same _api_request_profile the
idempotency-scope key already namespaces by) into the fingerprint
seed. Standalone listeners are unaffected semantically: no profile
prefix means the 'default' namespace, and ids stay stable across the
turns of one conversation.
The API adapter keeps its own per-profile-home SessionDB cache (`_session_dbs`)
for routed requests served under a profile's runtime scope. `_unserve_profile`'s
`close_all_under(home)` tore that generation down without telling the cache, so
the next routed request kept the dead handle (StateDbReplacedError after a
same-name recreate). Same class as the store/runner caches fixed in #123619:
treat a cleared `_shared_registry_owned` as absent and reopen through the
registry.
Trim #123619's two regression tests into one store+runner invariant, add one
for the API adapter cache.
The browse picker swallowed SessionActiveWriteGuardError into a generic
"Delete failed.", and the dashboard bulk delete only reported the deleted
count, silently keeping rows a live turn owns. Surface both: the picker
flashes that the session is active, and SessionsPage shows a toast with the
skipped_active count (new en key; other locales fall back to English via
defineLocale). Also move the api_server import into its sorted slot.
delete_sessions(exclude_active_write_guards=True) dropped guarded rows
silently: the web bulk-delete endpoint returned only a count and the
dashboard removed every selected row optimistically, so refused rows
reappeared on the next reload with no explanation.
The store now appends refused ids to an optional skipped_ids list inside
the same write transaction, the endpoint returns them as skipped_active,
and SessionsPage keeps those rows listed. Also hoists the
SessionActiveWriteGuardError imports to module top (hermes_state_errors
is stdlib-only) and drops the assertion-less lineage comment in the test.
Refactor entry-side deletion refusal to execute in-transaction via
`_write_guards_reject(conn, sid)` (#123583), per maintainer review:
- Underlying `delete_session` and `delete_sessions` now accept an opt-in
kwarg `exclude_active_write_guards=True` running inside `_do` write
transaction, eliminating the race condition where a turn acquires the lease
between check and delete.
- Raises `SessionActiveWriteGuardError` when refusing single delete, leaving
the row untouched; `delete_sessions` atomically skips active rows.
- Checks both active turn leases and compression locks via the existing
reclaim-aware `_write_guards_reject` helper.
- Covers all user-facing delete sinks:
* Web `DELETE /api/sessions/{id}` -> 409 Conflict
* Web `POST /api/sessions/bulk-delete` -> skips active rows
* Web / CLI `prune` -> passes `exclude_active_write_guards=True` so lineage
parents of active conversations are not pruned
* API Server `DELETE /api/sessions/{id}` -> 409 session_active_turn
* CLI `hermes sessions delete` & `export --delete-after-verified` -> exits 1
* CLI browse picker -> refuses active delete
* TUI Gateway `session.delete` -> 4023 error
- Conforms to rubric with 2 targeted invariant tests in
`tests/hermes_state/test_delete_session_write_guards.py`.
- Updates user guide and web dashboard docs for 409 / exit 1.
(cherry picked from commit 2c037a7a79dc211b49bacc72e3140951ccf900cf)
DELETE /api/sessions/<id> removed only the state.db rows: the durable
channel->session routing index (gateway_routing table + sessions.json
mirror) survived, so the next Discord/Telegram message routed to the SAME
id and resurrected the deleted row, and the on-disk .json/.jsonl
transcripts plus request_dump files were never scrubbed because
sessions_dir was not passed to delete_session (#42422).
- SessionStore.remove_by_session_id: drop every entry pointing at the id
(one channel can hold several) and persist the drop to both durable
copies; the index is written back by the gateway process, so removing
only DB rows elsewhere is undone by the next whole-index save.
- The API delete handler now passes the request-scoped sessions_dir and
clears the routing entries through the runner's SessionStore.
- Deletes made out of the gateway process self-heal at routing time via the
stale-route guard once a missing row counts as ended.
Fixes https://github.com/NousResearch/hermes-agent/issues/42422
The api_server platform wrote runtime status exactly once at bind (the
_connected mark) and never again: last_heartbeat stayed at boot time and
metrics_today froze at zero, so the dashboard showed stale API Server
activity until a full app restart (#52323).
The adapter now keeps daily request/message/token counters and a bounded
latency sample, publishes a metrics-bearing snapshot at bind, records
metrics after each completed _run_agent turn and /v1/runs run, and a
30-second heartbeat loop re-publishes while connected. gateway.status
gains a platform_metrics field on the platform payload, and
/health/detailed serves the live adapter metrics alongside the
persisted platform map.
Salvaged from #52345 by @itsflownium (Flownium) — reworked onto current
main (run-worker submission, bind-retry loop, readiness work counts).
Fixes#52323
Co-authored-by: Flownium <157689911+itsflownium@users.noreply.github.com>
Routes session chat stream, the live-bot-chat SSE sibling, /v1/runs/{id}/events
and the OpenAI-compat stream through a shared helper so CORS reaches the wire
before prepare() flushes the head (#72892, #6358).
Co-authored-by: kawanoii <61106070+kawanoii@users.noreply.github.com>
The CORS middleware cannot inject headers into StreamResponse after
prepare() flushes them, so streaming endpoints must resolve CORS headers
up front. _handle_session_chat_stream was the only SSE handler that did
not, leaving Access-Control-Allow-Origin absent on the streamed response
even when API_SERVER_CORS_ORIGINS was configured and the origin was
allowed. The sibling endpoints /v1/chat/completions and /v1/responses
already applied the same pattern; this applies it to the session stream.
(cherry picked from commit a2355e12a646a7baf32ff011a8bea358fdccc2c9)
- drop _run_stream_subscribers; the sweep reads _RunStream.subscribers
- per-write timeout via asyncio.timeout (no Task per token), force_close kept
- _sse_frame(id=) instead of hand-prepended id: lines
- reconnect queue gets headroom for the replay length
Strict OpenAI clients reject the named hermes.tool.progress SSE frames. Setting
tool_progress_events: false under platforms.api_server (loaded into
PlatformConfig.extra by from_dict) now drops them; default stays on.
Reimplements the intent of #42640 against the adapter config actually read in
production. Overlaps #49069 (erikerosev).
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Stamp owner + running status for the completion id so POST /v1/runs/{id}/approval
reaches the mapping instead of 404ing in _load_owned_run; retire it (terminal status via
terminal_run_status, owner release) from the single existing EOS done-callback through a
new _spawn_stream_agent on_done hook. Extract _approval_request_event (redact + choices +
_run_event envelope) for the runs bridge, session stream and chat completions; both new
surfaces park waiting_for_approval like /v1/runs. _run_agent reuses
_unregister_approval_notify.
Session chat streams already register the agent under the run id (stop works),
but guarded tools had no approval notifier and failed closed. Register a
run-id-keyed approval session, emit approval.request on the SSE stream, and drop
the mapping when the turn ends.
Co-authored-by: baleian <baleian90@gmail.com>
Hand-grafted from #51878 onto the split api_server_openai_routes.py; approvals keyed
by completion id and resolved via POST /v1/runs/{id}/approval (#51871).
Partial salvage of #119405: kept the four api_server.py hunks (per-profile
single-flight asyncio.Lock dict, _artifact_store_for_async, awaited
to_thread for store.store and store.load), dropped the 392-line test file
because it is mostly change-detector/timing assertions.
(cherry picked from commit cdfcc39181)
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
A multiplexed API server mirrors every route under /p/<profile>/ and
authenticates each mirror with that profile's key, but it kept one
ResponseStore at the home it was constructed in. Conversation names are
client-chosen strings ("main", "my-project"), so another profile's key
could post `conversation: <name>`, receive that profile's transcript,
instructions and session id as its agent's context, become the
conversation's tip (the owner's next turn replayed the intruder's
messages), and GET or DELETE the owner's responses by id.
The adapter now resolves the store from the request's profile home, as
the SessionDB cache already does: the construction home keeps
self._response_store (and its response_store.db), every other routed
home gets its own <home>/response_store.db, opened on first use and
closed on disconnect. The stream state captures its store when the
request starts, so a snapshot written after the scope ends (disconnect)
still lands in the right one.
Rows a secondary profile wrote into the shared store before this change
stay there, visible to the construction home's profile only; they are
not migrated.
The api_server platform builds a fresh AIAgent per request (per-request
callbacks, model route, ephemeral prompt), so the memory provider was
re-initialised on every request. External providers deliver recall as the
PREVIOUS turn's background prefetch held on the provider instance, so a
continued session (X-Hermes-Session-Id, previous_response_id, declared
session key) never received automatic recall, and for hindsight
local_embedded each init also restarted the embedded daemon, killing the
retain still in flight. Pre-existing: the same probe fails on main before
the hindsight catalog migration (526d135a96, bundled provider).
ApiServerMemorySessions parks the session's initialised MemoryManager
between requests (exclusive check-out/check-in, keyed by profile home +
session id, LRU/idle eviction under the owning profile's scope) and
AIAgent(memory_manager=...) adopts it instead of loading and initialising
the provider again. /v1/chat/completions, /v1/responses, session chat and
/v1/runs all go through the same two seams (_create_agent, turn finally).
Local message_agent, the Desktop relay, `hermes peer dm` and `hermes peer run`
all hand a turn to the live Bot Chat owner's mailbox and then need the same
thing: poll the receipt until the owner settles it, the budget lapses, or the
caller wants out. Each lane carried (or lacked) its own copy of that loop,
which is how the class drifted — the relay answered with a receipt sentence
(#115316), the two peer lanes never asked the owner at all (#114959, #115174).
`tools/bot_live_delivery.py::await_delivery` (+ `await_delivery_async` for the
aiohttp handlers, which must not park a worker thread for 300 s) is now the one
loop; `_wait_live_dm`, `bot_relay.deliver`, `_answer_through_live_bot_chat`
and `_execute_run_via_live_owner` call it. `should_stop` carries the peer-run
`/stop` check the runs lane needs.
The count was incremented on the submitting thread and released only in the
worker's finally; when loop.run_in_executor itself raised (default executor
already shut down during quiesce -> RuntimeError) no worker ever ran and
_API_WORKER_LIVE stayed elevated for the process lifetime, making the shutdown
close gate skip the SessionDB close forever. _submit_api_worker now owns both
sides of the submission.
Handler-side _inflight_agent_runs drops in the handler finally on
cancellation while the run_in_executor thread lives on, letting the
SessionDB close gate observe zero live runs under a live writer.
Count the worker lifetime itself (increment before run_in_executor,
decrement in the worker finally) and gate
_stop_quiesce_and_close_session_dbs on it alongside ctx.api_live.
Fixes#116535
POST /api/sessions/{id}/chat/stream is the SSE sibling of the chat route and
went _prepare_session_chat -> _run_agent with no owner check, so it stayed a
second writer into a live-owned canonical Bot Chat (#114959, "Remaining
siblings"). It now admits through the same _admit_to_live_bot_chat door; the
owner's settled receipt is streamed as the run's single assistant.completed
event, a receipt still open at the budget as run.queued (the 202 shape), a
failed one as an error event carrying the reason. The receipt wait is shared
with the JSON route (_await_live_bot_chat_receipt) and sends SSE keepalives
while it waits.
Docs: the peer dm paragraph now carries both the read-timeout wording
(#116885) and the open-chat wording in one paragraph so either landing order
resolves to this text.
`hermes peer run` posts POST /v1/runs with the peer's canonical Bot Chat as
session_id. Like the /chat transport before #114959, the run executed here
while a Desktop session held that chat's lease — a second writer the open chat
never showed, with the two transcripts interleaved in state.db.
The admission that /api/sessions/{id}/chat now performs moves onto the adapter
as one helper both peer transports call, so the two lanes cannot drift. When
the selected session is the live-held canonical Bot Chat, /v1/runs admits the
message to the owner's mailbox and drives the run from the owner's receipt
instead of an executor: `settled` completes it with the reply, a failed
receipt fails it with the owner's classified reason, and the run retires the
way an executor-backed one does. `peer run` keeps its run_id and `peer status`
keeps working; the status carries the delivery_id.
/stop cannot reach the owner's turn — the mailbox has no recall once a record
is claimed — so a stop ends this run as cancelled while the chat finishes on
its own; the stop handler already reports a run without an in-process agent as
not interruptible here.
`hermes peer dm` posts POST /api/sessions/{id}/chat. When the target is the
profile's canonical Bot Chat and a Desktop holds it live, that Desktop session
owns the chat's single-writer lease, and every other writer is refused
SESSION_NOT_OWNED — per-session exclusivity is correctness, enforced
unconditionally (hermes_cli/active_sessions.py). The API server neither takes
that lease nor checks it, so the turn ran beside the owner: the open chat never
showed the message or the reply, the live session's context never learned of
them, and two writers appended to one transcript in state.db.
Hand the message to the owner's mailbox instead, as local DMs
(tools/bot_mode_dm.py) and relayed DMs (tui_gateway/methods_bot_relay.py,
budget so the peer still gets the reply on the same call. A turn still running
at that deadline answers 202 with the delivery id, and `peer dm` reports the
message as queued in that chat instead of printing "(no reply)".
Only the canonical Bot Chat's own compression lineage is handed off: a peer turn
into any other session, or into a Bot Chat nobody holds, runs here as before.
Tests: the four-row table is the whole discriminator (owner answers, owner still
running, another session, nobody holding the chat) and the client row pins that a
queued answer reads as delivered.
`hermes peer dm` posts to POST /api/sessions/{id}/chat, the third Bot-DM
transport. The local (`tools.bot_mode_dm`) and relayed
(`tui_gateway.methods_bot_relay`) lanes both re-run a transiently failed turn
once under the shared policy (`tools.bot_failure_reasons.retry_action`) and
resume the row the failed attempt left as the transcript's unanswered tail;
this lane ran the turn once and handed the provider's 429 paragraph to the
sender as the reply (#115325).
The policy is asked about a result dict now, not two streams: `result_retry_action`
joins `error` + `failure_reason` (the turn loop's own typed verdict) so one
classifier serves every lane, and the server-error rule accepts the providers'
`server_error` / `overloaded_error` spellings — the codes the in-process lanes
key on instead of a status number.
The resume half is the CLI lane's rule extracted to `agent.session_persistence.
adopt_unanswered_turn`, which `quiet_single_query` (env-gated dispatcher re-run)
and the API lane (in-process re-run, on the agent it just built) now share.
The regression drives the real route and the real `_run_agent` over a real
store: a 429 re-runs the same DM once with the persisted row adopted as this
turn's user message (so no second copy), a 401 still reports one attempt.
(cherry picked from commit 8fe6d46ada5b8064bc7132ee956fda998244c10a)
The OpenAI-compatible SSE writers carried tool progress and reasoning but no
lifecycle status, so an API client waiting through a provider outage (now the
auto-recovery ladder) saw a silent socket with no way to tell "waiting on the
provider" from "hung". _spawn_stream_agent wires status_callback into the
agent and both writers (/v1/chat/completions, /v1/responses) emit
`event: hermes.status` with {kind, text}, redacted like every other API-bound
error text. The Responses writer's tag dispatch moves to a table so the new tag
does not grow an if/elif ladder.
Rate-limit now wins over the auth pattern in the gateway's provider-error
reply table, `401` only matches as a standalone status token (a bare
`\b401\b` also hit timestamp fragments like `05:14:15,401`), and the
rate-limit reply names the reset window when the envelope carries
`resets_in_seconds` or `retry after Ns` instead of "wait a moment" for a
weekly quota. The pre-turn credential-resolution failure on chat surfaces
and the api_server `_ProviderAuthResolutionError` label follow the same
rule via `is_rate_limited_auth_error` on the cause chain.
WHY: a quota/429 envelope often also carries an auth-shaped preamble and
"Credentials are still valid"; classifying it as auth sent operators to
re-login working credentials across profiles (#89401).
OpenAI-compatible clients (Open WebUI, opencode, LibreChat, the Vercel AI SDK)
saw only answer text from the API server: the streaming writers never wired the
agent's structured ``reasoning_callback``, so reasoning deltas that every native
surface already renders were dropped at the transport boundary (#99552).
- `_spawn_stream_agent` passes a `reasoning_callback` (via `_run_agent` /
`_create_agent`) that tags deltas `("__reasoning__", text)` on the stream queue,
keeping them distinct from answer text. The lossy 500-char `reasoning.available`
progress preview is deliberately not used.
- `/v1/chat/completions`: reasoning rides `choices[0].delta.reasoning_content`
(the DeepSeek-style field those clients render as a thinking block).
- `/v1/responses`: each thinking burst is a spec-native `reasoning` output item
(`output_item.added`, `reasoning_summary_part.added`,
`reasoning_summary_text.delta/done`, `reasoning_summary_part.done`,
`output_item.done`), closed before the next message/function_call item opens
and echoed in `response.completed` output; `sequence_number` stays monotonic.
- `GET /v1/capabilities` advertises `features.reasoning_streaming: true`.
- Docs: api-server page documents the wire fields and the capability flag.
Gating is unchanged: nothing is emitted unless the model produces reasoning under
the resolved `reasoning_config` (`model_options.reasoning.enabled: false` opts out).
api_server.py keeps both interim_assistant_callback (#115903) and reasoning_callback (#115797) in every _create_agent/_run_agent slot.
api_server_openai_routes.py keeps both _ResponsesStream method sets and both __commentary__/__reasoning__ tag branches; emit_commentary closes any open reasoning item first.
`_session_key_for_source` reads the pinned identity before falling back to
`source.profile` / the active profile. The remaining `get_active_profile_name()`
reads in gateway/ run before any event exists (adapter boot, cron ticker homes,
startup log, advertised model name) and are marked `# launch profile, pre-identity`.
POST /v1/runs + GET /v1/runs/{id}/events now carries mid-turn assistant
commentary as `message.interim` {text, already_streamed} — the same
contract the TUI gateway emits — so Runs clients can tell an active,
tool-heavy Codex turn from a stalled one instead of seeing only tool.*
events until run.completed (#67580).
APIServerAdapter._create_agent applies the same
`display.interim_assistant_messages` gate the messaging gateway and the
TUI apply (resolve_display_setting, per-platform override honoured):
when off, no callback is installed and nothing leaves the agent on any
of the three streaming surfaces. Dedup of repeated commentary stays in
agent/stream_delivery.py, where every surface already relies on it.
Tests: /v1/runs event contract; session SSE + /v1/responses item shape
plus the display gate (both red on the base commit). Docs: event
payloads on all three surfaces.
Co-authored-by: RoySRose <sungwook0115.kim@gmail.com>
APIServerAdapter._create_agent / _run_agent never accepted an
interim_assistant_callback, so Codex phase="commentary" preambles and
other mid-turn assistant text were produced by the agent and dropped at
the API-server boundary; streaming clients saw only tool events and the
final answer (#67580).
Forward the callback into AIAgent and emit a typed `assistant.commentary`
event ({message_id, text, already_streamed}) on
POST /api/sessions/{id}/chat/stream, separate from assistant.delta and
never concatenated into assistant.completed.
Hand-reapplied onto the current facade layout from #67613.
The persisted conversation_history snapshot embeds the cumulative transcript
with every tool output verbatim, so a few large tool outputs made a single
response_store.db write ~677 KB (2.7x the configured rotation threshold)
while the response.completed payload for the same turn was already trimmed.
Add gateway.api_server.history_tool_output_max_chars (default 0 = store
verbatim, current behaviour). When set, tool rows and string tool-call
arguments longer than N chars are cut to the head plus the same
"...[K more chars]" marker _trim_tool_items uses, in a copied row, before the
snapshot is stored; user/assistant text and the agent's own in-memory
transcript are untouched. Opt-in rather than default because the stored
history is exactly what the model is replayed on the next chained turn.
Live (200 KB tool output, temp HERMES_HOME): stored row 200,659 B off ->
4,681 B with the cap at 4000.
When the primary provider fails with AuthError during gateway credential
resolution (before AIAgent is constructed), the fallback provider is
silently used with no user-visible notice. The existing
_pending_fallback_notice mechanism only covers in-conversation-loop
fallback activation, not the pre-agent gateway path.
Fix by:
1. Capturing primary provider/model from config in
_resolve_runtime_agent_kwargs() before the AuthError try block
2. Adding _fallback_notice metadata to the returned fallback dict
3. Popping it in TurnRunner.run_sync() before forwarding kwargs
4. Setting agent._pending_fallback_notice after agent creation/cache
The existing _emit_pending_fallback_notice() mechanism then surfaces
the notice on the next turn, deduplicating naturally.
Fixes#74349
(cherry picked from commit 7d66877ecb4718dbfebc7729e7a0ea96d8eab174)
OpenAI-compatible clients (Open WebUI, opencode, LibreChat, the Vercel AI SDK)
saw only answer text from the API server: the streaming writers never wired the
agent's structured ``reasoning_callback``, so reasoning deltas that every native
surface already renders were dropped at the transport boundary (#99552).
- `_spawn_stream_agent` passes a `reasoning_callback` (via `_run_agent` /
`_create_agent`) that tags deltas `("__reasoning__", text)` on the stream queue,
keeping them distinct from answer text. The lossy 500-char `reasoning.available`
progress preview is deliberately not used.
- `/v1/chat/completions`: reasoning rides `choices[0].delta.reasoning_content`
(the DeepSeek-style field those clients render as a thinking block).
- `/v1/responses`: each thinking burst is a spec-native `reasoning` output item
(`output_item.added`, `reasoning_summary_part.added`,
`reasoning_summary_text.delta/done`, `reasoning_summary_part.done`,
`output_item.done`), closed before the next message/function_call item opens
and echoed in `response.completed` output; `sequence_number` stays monotonic.
- `GET /v1/capabilities` advertises `features.reasoning_streaming: true`.
- Docs: api-server page documents the wire fields and the capability flag.
Gating is unchanged: nothing is emitted unless the model produces reasoning under
the resolved `reasoning_config` (`model_options.reasoning.enabled: false` opts out).
On macOS, an exclusive bind (SO_REUSEADDR disabled) refuses a port that is
still held only by a server-side TIME_WAIT socket for 2*MSL (~30s) after a
previous connection close — the gateway's own shutdown, or any
``Connection: close`` request. A ``/restart`` issued within that window
failed to bind, even though nobody was actually listening.
For an explicit host, a refused connect probe proves nobody is listening:
the kernel still rejects an exact duplicate bind even with SO_REUSEADDR, and
a foreign wildcard listener would answer the probe. So one retry with
SO_REUSEADDR is safe in that case. A wildcard host keeps the strict
exclusive path, since a foreign listener on a non-loopback interface can't
be probed this way. A live listener still wins either way.
The bind, probe, and retry logic is shared by webhook.py and api_server.py
through the new gateway/platforms/tcp_site.py, since both adapters had the
same exposure — api_server additionally marked a spurious TIME_WAIT bind
failure as a non-retryable port conflict.
(cherry picked from commit 289fd06148c447829f1e38b66ff78b9e4bebdeb4)
Gate review on the stack:
- `hermes gateway restart` under systemd only accepted `gateway_state == "running"`
as proof of the replacement; a boot with a parked platform stamps `degraded` for
its whole life, so every restart/update on such a host waited out the 60 s+
budget and reported a false failure while the gateway was serving. The verifier
now accepts `degraded` as restarted and prints one DEGRADED warning.
- Drain release and scale-to-zero wake re-stamped `running` unconditionally, wiping
the parked-platform signal after the first `.drain_request.json` cycle. Every
"we are serving" stamp now goes through `_serving_state()`; the mixed fatal +
retryable boot path sets the flag too (it fell through as a plain run before).
`_startup_parked_platforms` is a bool — the joined error text was only ever logged.
- `_wait_for_tcp_port_free`: an unresolvable or unreachable configured host raised
a non-refused OSError on every probe and burned the full 10 s wait; only a
connect timeout means "listener alive", anything else means nothing to wait for.
- Windows `restart()` replaces its blind `time.sleep(1.0)` "let Windows release the
port" with the same configured-address wait.
- The api_server bind retry rebuilds the AppRunner per EADDRINUSE attempt instead of
calling aiohttp's private `_unreg_site`; attempt count is a named constant.
Test: a `degraded` replacement is reported as restarted (red on the previous head).
E2E re-run: predecessor releases the port 0.5 s after the first bind → bound on the
third attempt (+0.61 s), connect() True.
