Refactor entry-side deletion refusal to execute in-transaction via
`_write_guards_reject(conn, sid)` (#123583), per maintainer review:
- Underlying `delete_session` and `delete_sessions` now accept an opt-in
kwarg `exclude_active_write_guards=True` running inside `_do` write
transaction, eliminating the race condition where a turn acquires the lease
between check and delete.
- Raises `SessionActiveWriteGuardError` when refusing single delete, leaving
the row untouched; `delete_sessions` atomically skips active rows.
- Checks both active turn leases and compression locks via the existing
reclaim-aware `_write_guards_reject` helper.
- Covers all user-facing delete sinks:
* Web `DELETE /api/sessions/{id}` -> 409 Conflict
* Web `POST /api/sessions/bulk-delete` -> skips active rows
* Web / CLI `prune` -> passes `exclude_active_write_guards=True` so lineage
parents of active conversations are not pruned
* API Server `DELETE /api/sessions/{id}` -> 409 session_active_turn
* CLI `hermes sessions delete` & `export --delete-after-verified` -> exits 1
* CLI browse picker -> refuses active delete
* TUI Gateway `session.delete` -> 4023 error
- Conforms to rubric with 2 targeted invariant tests in
`tests/hermes_state/test_delete_session_write_guards.py`.
- Updates user guide and web dashboard docs for 409 / exit 1.
(cherry picked from commit 2c037a7a79dc211b49bacc72e3140951ccf900cf)
DELETE /api/sessions/<id> removed only the state.db rows: the durable
channel->session routing index (gateway_routing table + sessions.json
mirror) survived, so the next Discord/Telegram message routed to the SAME
id and resurrected the deleted row, and the on-disk .json/.jsonl
transcripts plus request_dump files were never scrubbed because
sessions_dir was not passed to delete_session (#42422).
- SessionStore.remove_by_session_id: drop every entry pointing at the id
(one channel can hold several) and persist the drop to both durable
copies; the index is written back by the gateway process, so removing
only DB rows elsewhere is undone by the next whole-index save.
- The API delete handler now passes the request-scoped sessions_dir and
clears the routing entries through the runner's SessionStore.
- Deletes made out of the gateway process self-heal at routing time via the
stale-route guard once a missing row counts as ended.
Fixes https://github.com/NousResearch/hermes-agent/issues/42422
The api_server platform wrote runtime status exactly once at bind (the
_connected mark) and never again: last_heartbeat stayed at boot time and
metrics_today froze at zero, so the dashboard showed stale API Server
activity until a full app restart (#52323).
The adapter now keeps daily request/message/token counters and a bounded
latency sample, publishes a metrics-bearing snapshot at bind, records
metrics after each completed _run_agent turn and /v1/runs run, and a
30-second heartbeat loop re-publishes while connected. gateway.status
gains a platform_metrics field on the platform payload, and
/health/detailed serves the live adapter metrics alongside the
persisted platform map.
Salvaged from #52345 by @itsflownium (Flownium) — reworked onto current
main (run-worker submission, bind-retry loop, readiness work counts).
Fixes#52323
Co-authored-by: Flownium <157689911+itsflownium@users.noreply.github.com>
Routes session chat stream, the live-bot-chat SSE sibling, /v1/runs/{id}/events
and the OpenAI-compat stream through a shared helper so CORS reaches the wire
before prepare() flushes the head (#72892, #6358).
Co-authored-by: kawanoii <61106070+kawanoii@users.noreply.github.com>
The CORS middleware cannot inject headers into StreamResponse after
prepare() flushes them, so streaming endpoints must resolve CORS headers
up front. _handle_session_chat_stream was the only SSE handler that did
not, leaving Access-Control-Allow-Origin absent on the streamed response
even when API_SERVER_CORS_ORIGINS was configured and the origin was
allowed. The sibling endpoints /v1/chat/completions and /v1/responses
already applied the same pattern; this applies it to the session stream.
(cherry picked from commit a2355e12a646a7baf32ff011a8bea358fdccc2c9)
- drop _run_stream_subscribers; the sweep reads _RunStream.subscribers
- per-write timeout via asyncio.timeout (no Task per token), force_close kept
- _sse_frame(id=) instead of hand-prepended id: lines
- reconnect queue gets headroom for the replay length
Strict OpenAI clients reject the named hermes.tool.progress SSE frames. Setting
tool_progress_events: false under platforms.api_server (loaded into
PlatformConfig.extra by from_dict) now drops them; default stays on.
Reimplements the intent of #42640 against the adapter config actually read in
production. Overlaps #49069 (erikerosev).
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Stamp owner + running status for the completion id so POST /v1/runs/{id}/approval
reaches the mapping instead of 404ing in _load_owned_run; retire it (terminal status via
terminal_run_status, owner release) from the single existing EOS done-callback through a
new _spawn_stream_agent on_done hook. Extract _approval_request_event (redact + choices +
_run_event envelope) for the runs bridge, session stream and chat completions; both new
surfaces park waiting_for_approval like /v1/runs. _run_agent reuses
_unregister_approval_notify.
Session chat streams already register the agent under the run id (stop works),
but guarded tools had no approval notifier and failed closed. Register a
run-id-keyed approval session, emit approval.request on the SSE stream, and drop
the mapping when the turn ends.
Co-authored-by: baleian <baleian90@gmail.com>
Hand-grafted from #51878 onto the split api_server_openai_routes.py; approvals keyed
by completion id and resolved via POST /v1/runs/{id}/approval (#51871).
Partial salvage of #119405: kept the four api_server.py hunks (per-profile
single-flight asyncio.Lock dict, _artifact_store_for_async, awaited
to_thread for store.store and store.load), dropped the 392-line test file
because it is mostly change-detector/timing assertions.
(cherry picked from commit cdfcc39181)
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
A multiplexed API server mirrors every route under /p/<profile>/ and
authenticates each mirror with that profile's key, but it kept one
ResponseStore at the home it was constructed in. Conversation names are
client-chosen strings ("main", "my-project"), so another profile's key
could post `conversation: <name>`, receive that profile's transcript,
instructions and session id as its agent's context, become the
conversation's tip (the owner's next turn replayed the intruder's
messages), and GET or DELETE the owner's responses by id.
The adapter now resolves the store from the request's profile home, as
the SessionDB cache already does: the construction home keeps
self._response_store (and its response_store.db), every other routed
home gets its own <home>/response_store.db, opened on first use and
closed on disconnect. The stream state captures its store when the
request starts, so a snapshot written after the scope ends (disconnect)
still lands in the right one.
Rows a secondary profile wrote into the shared store before this change
stay there, visible to the construction home's profile only; they are
not migrated.
The api_server platform builds a fresh AIAgent per request (per-request
callbacks, model route, ephemeral prompt), so the memory provider was
re-initialised on every request. External providers deliver recall as the
PREVIOUS turn's background prefetch held on the provider instance, so a
continued session (X-Hermes-Session-Id, previous_response_id, declared
session key) never received automatic recall, and for hindsight
local_embedded each init also restarted the embedded daemon, killing the
retain still in flight. Pre-existing: the same probe fails on main before
the hindsight catalog migration (526d135a96, bundled provider).
ApiServerMemorySessions parks the session's initialised MemoryManager
between requests (exclusive check-out/check-in, keyed by profile home +
session id, LRU/idle eviction under the owning profile's scope) and
AIAgent(memory_manager=...) adopts it instead of loading and initialising
the provider again. /v1/chat/completions, /v1/responses, session chat and
/v1/runs all go through the same two seams (_create_agent, turn finally).
Local message_agent, the Desktop relay, `hermes peer dm` and `hermes peer run`
all hand a turn to the live Bot Chat owner's mailbox and then need the same
thing: poll the receipt until the owner settles it, the budget lapses, or the
caller wants out. Each lane carried (or lacked) its own copy of that loop,
which is how the class drifted — the relay answered with a receipt sentence
(#115316), the two peer lanes never asked the owner at all (#114959, #115174).
`tools/bot_live_delivery.py::await_delivery` (+ `await_delivery_async` for the
aiohttp handlers, which must not park a worker thread for 300 s) is now the one
loop; `_wait_live_dm`, `bot_relay.deliver`, `_answer_through_live_bot_chat`
and `_execute_run_via_live_owner` call it. `should_stop` carries the peer-run
`/stop` check the runs lane needs.
The count was incremented on the submitting thread and released only in the
worker's finally; when loop.run_in_executor itself raised (default executor
already shut down during quiesce -> RuntimeError) no worker ever ran and
_API_WORKER_LIVE stayed elevated for the process lifetime, making the shutdown
close gate skip the SessionDB close forever. _submit_api_worker now owns both
sides of the submission.
Handler-side _inflight_agent_runs drops in the handler finally on
cancellation while the run_in_executor thread lives on, letting the
SessionDB close gate observe zero live runs under a live writer.
Count the worker lifetime itself (increment before run_in_executor,
decrement in the worker finally) and gate
_stop_quiesce_and_close_session_dbs on it alongside ctx.api_live.
Fixes#116535
POST /api/sessions/{id}/chat/stream is the SSE sibling of the chat route and
went _prepare_session_chat -> _run_agent with no owner check, so it stayed a
second writer into a live-owned canonical Bot Chat (#114959, "Remaining
siblings"). It now admits through the same _admit_to_live_bot_chat door; the
owner's settled receipt is streamed as the run's single assistant.completed
event, a receipt still open at the budget as run.queued (the 202 shape), a
failed one as an error event carrying the reason. The receipt wait is shared
with the JSON route (_await_live_bot_chat_receipt) and sends SSE keepalives
while it waits.
Docs: the peer dm paragraph now carries both the read-timeout wording
(#116885) and the open-chat wording in one paragraph so either landing order
resolves to this text.
`hermes peer run` posts POST /v1/runs with the peer's canonical Bot Chat as
session_id. Like the /chat transport before #114959, the run executed here
while a Desktop session held that chat's lease — a second writer the open chat
never showed, with the two transcripts interleaved in state.db.
The admission that /api/sessions/{id}/chat now performs moves onto the adapter
as one helper both peer transports call, so the two lanes cannot drift. When
the selected session is the live-held canonical Bot Chat, /v1/runs admits the
message to the owner's mailbox and drives the run from the owner's receipt
instead of an executor: `settled` completes it with the reply, a failed
receipt fails it with the owner's classified reason, and the run retires the
way an executor-backed one does. `peer run` keeps its run_id and `peer status`
keeps working; the status carries the delivery_id.
/stop cannot reach the owner's turn — the mailbox has no recall once a record
is claimed — so a stop ends this run as cancelled while the chat finishes on
its own; the stop handler already reports a run without an in-process agent as
not interruptible here.
`hermes peer dm` posts POST /api/sessions/{id}/chat. When the target is the
profile's canonical Bot Chat and a Desktop holds it live, that Desktop session
owns the chat's single-writer lease, and every other writer is refused
SESSION_NOT_OWNED — per-session exclusivity is correctness, enforced
unconditionally (hermes_cli/active_sessions.py). The API server neither takes
that lease nor checks it, so the turn ran beside the owner: the open chat never
showed the message or the reply, the live session's context never learned of
them, and two writers appended to one transcript in state.db.
Hand the message to the owner's mailbox instead, as local DMs
(tools/bot_mode_dm.py) and relayed DMs (tui_gateway/methods_bot_relay.py,
budget so the peer still gets the reply on the same call. A turn still running
at that deadline answers 202 with the delivery id, and `peer dm` reports the
message as queued in that chat instead of printing "(no reply)".
Only the canonical Bot Chat's own compression lineage is handed off: a peer turn
into any other session, or into a Bot Chat nobody holds, runs here as before.
Tests: the four-row table is the whole discriminator (owner answers, owner still
running, another session, nobody holding the chat) and the client row pins that a
queued answer reads as delivered.
`hermes peer dm` posts to POST /api/sessions/{id}/chat, the third Bot-DM
transport. The local (`tools.bot_mode_dm`) and relayed
(`tui_gateway.methods_bot_relay`) lanes both re-run a transiently failed turn
once under the shared policy (`tools.bot_failure_reasons.retry_action`) and
resume the row the failed attempt left as the transcript's unanswered tail;
this lane ran the turn once and handed the provider's 429 paragraph to the
sender as the reply (#115325).
The policy is asked about a result dict now, not two streams: `result_retry_action`
joins `error` + `failure_reason` (the turn loop's own typed verdict) so one
classifier serves every lane, and the server-error rule accepts the providers'
`server_error` / `overloaded_error` spellings — the codes the in-process lanes
key on instead of a status number.
The resume half is the CLI lane's rule extracted to `agent.session_persistence.
adopt_unanswered_turn`, which `quiet_single_query` (env-gated dispatcher re-run)
and the API lane (in-process re-run, on the agent it just built) now share.
The regression drives the real route and the real `_run_agent` over a real
store: a 429 re-runs the same DM once with the persisted row adopted as this
turn's user message (so no second copy), a 401 still reports one attempt.
(cherry picked from commit 8fe6d46ada5b8064bc7132ee956fda998244c10a)
The OpenAI-compatible SSE writers carried tool progress and reasoning but no
lifecycle status, so an API client waiting through a provider outage (now the
auto-recovery ladder) saw a silent socket with no way to tell "waiting on the
provider" from "hung". _spawn_stream_agent wires status_callback into the
agent and both writers (/v1/chat/completions, /v1/responses) emit
`event: hermes.status` with {kind, text}, redacted like every other API-bound
error text. The Responses writer's tag dispatch moves to a table so the new tag
does not grow an if/elif ladder.
Rate-limit now wins over the auth pattern in the gateway's provider-error
reply table, `401` only matches as a standalone status token (a bare
`\b401\b` also hit timestamp fragments like `05:14:15,401`), and the
rate-limit reply names the reset window when the envelope carries
`resets_in_seconds` or `retry after Ns` instead of "wait a moment" for a
weekly quota. The pre-turn credential-resolution failure on chat surfaces
and the api_server `_ProviderAuthResolutionError` label follow the same
rule via `is_rate_limited_auth_error` on the cause chain.
WHY: a quota/429 envelope often also carries an auth-shaped preamble and
"Credentials are still valid"; classifying it as auth sent operators to
re-login working credentials across profiles (#89401).
OpenAI-compatible clients (Open WebUI, opencode, LibreChat, the Vercel AI SDK)
saw only answer text from the API server: the streaming writers never wired the
agent's structured ``reasoning_callback``, so reasoning deltas that every native
surface already renders were dropped at the transport boundary (#99552).
- `_spawn_stream_agent` passes a `reasoning_callback` (via `_run_agent` /
`_create_agent`) that tags deltas `("__reasoning__", text)` on the stream queue,
keeping them distinct from answer text. The lossy 500-char `reasoning.available`
progress preview is deliberately not used.
- `/v1/chat/completions`: reasoning rides `choices[0].delta.reasoning_content`
(the DeepSeek-style field those clients render as a thinking block).
- `/v1/responses`: each thinking burst is a spec-native `reasoning` output item
(`output_item.added`, `reasoning_summary_part.added`,
`reasoning_summary_text.delta/done`, `reasoning_summary_part.done`,
`output_item.done`), closed before the next message/function_call item opens
and echoed in `response.completed` output; `sequence_number` stays monotonic.
- `GET /v1/capabilities` advertises `features.reasoning_streaming: true`.
- Docs: api-server page documents the wire fields and the capability flag.
Gating is unchanged: nothing is emitted unless the model produces reasoning under
the resolved `reasoning_config` (`model_options.reasoning.enabled: false` opts out).
api_server.py keeps both interim_assistant_callback (#115903) and reasoning_callback (#115797) in every _create_agent/_run_agent slot.
api_server_openai_routes.py keeps both _ResponsesStream method sets and both __commentary__/__reasoning__ tag branches; emit_commentary closes any open reasoning item first.
`_session_key_for_source` reads the pinned identity before falling back to
`source.profile` / the active profile. The remaining `get_active_profile_name()`
reads in gateway/ run before any event exists (adapter boot, cron ticker homes,
startup log, advertised model name) and are marked `# launch profile, pre-identity`.
POST /v1/runs + GET /v1/runs/{id}/events now carries mid-turn assistant
commentary as `message.interim` {text, already_streamed} — the same
contract the TUI gateway emits — so Runs clients can tell an active,
tool-heavy Codex turn from a stalled one instead of seeing only tool.*
events until run.completed (#67580).
APIServerAdapter._create_agent applies the same
`display.interim_assistant_messages` gate the messaging gateway and the
TUI apply (resolve_display_setting, per-platform override honoured):
when off, no callback is installed and nothing leaves the agent on any
of the three streaming surfaces. Dedup of repeated commentary stays in
agent/stream_delivery.py, where every surface already relies on it.
Tests: /v1/runs event contract; session SSE + /v1/responses item shape
plus the display gate (both red on the base commit). Docs: event
payloads on all three surfaces.
Co-authored-by: RoySRose <sungwook0115.kim@gmail.com>
APIServerAdapter._create_agent / _run_agent never accepted an
interim_assistant_callback, so Codex phase="commentary" preambles and
other mid-turn assistant text were produced by the agent and dropped at
the API-server boundary; streaming clients saw only tool events and the
final answer (#67580).
Forward the callback into AIAgent and emit a typed `assistant.commentary`
event ({message_id, text, already_streamed}) on
POST /api/sessions/{id}/chat/stream, separate from assistant.delta and
never concatenated into assistant.completed.
Hand-reapplied onto the current facade layout from #67613.
The persisted conversation_history snapshot embeds the cumulative transcript
with every tool output verbatim, so a few large tool outputs made a single
response_store.db write ~677 KB (2.7x the configured rotation threshold)
while the response.completed payload for the same turn was already trimmed.
Add gateway.api_server.history_tool_output_max_chars (default 0 = store
verbatim, current behaviour). When set, tool rows and string tool-call
arguments longer than N chars are cut to the head plus the same
"...[K more chars]" marker _trim_tool_items uses, in a copied row, before the
snapshot is stored; user/assistant text and the agent's own in-memory
transcript are untouched. Opt-in rather than default because the stored
history is exactly what the model is replayed on the next chained turn.
Live (200 KB tool output, temp HERMES_HOME): stored row 200,659 B off ->
4,681 B with the cap at 4000.
When the primary provider fails with AuthError during gateway credential
resolution (before AIAgent is constructed), the fallback provider is
silently used with no user-visible notice. The existing
_pending_fallback_notice mechanism only covers in-conversation-loop
fallback activation, not the pre-agent gateway path.
Fix by:
1. Capturing primary provider/model from config in
_resolve_runtime_agent_kwargs() before the AuthError try block
2. Adding _fallback_notice metadata to the returned fallback dict
3. Popping it in TurnRunner.run_sync() before forwarding kwargs
4. Setting agent._pending_fallback_notice after agent creation/cache
The existing _emit_pending_fallback_notice() mechanism then surfaces
the notice on the next turn, deduplicating naturally.
Fixes#74349
(cherry picked from commit 7d66877ecb4718dbfebc7729e7a0ea96d8eab174)
OpenAI-compatible clients (Open WebUI, opencode, LibreChat, the Vercel AI SDK)
saw only answer text from the API server: the streaming writers never wired the
agent's structured ``reasoning_callback``, so reasoning deltas that every native
surface already renders were dropped at the transport boundary (#99552).
- `_spawn_stream_agent` passes a `reasoning_callback` (via `_run_agent` /
`_create_agent`) that tags deltas `("__reasoning__", text)` on the stream queue,
keeping them distinct from answer text. The lossy 500-char `reasoning.available`
progress preview is deliberately not used.
- `/v1/chat/completions`: reasoning rides `choices[0].delta.reasoning_content`
(the DeepSeek-style field those clients render as a thinking block).
- `/v1/responses`: each thinking burst is a spec-native `reasoning` output item
(`output_item.added`, `reasoning_summary_part.added`,
`reasoning_summary_text.delta/done`, `reasoning_summary_part.done`,
`output_item.done`), closed before the next message/function_call item opens
and echoed in `response.completed` output; `sequence_number` stays monotonic.
- `GET /v1/capabilities` advertises `features.reasoning_streaming: true`.
- Docs: api-server page documents the wire fields and the capability flag.
Gating is unchanged: nothing is emitted unless the model produces reasoning under
the resolved `reasoning_config` (`model_options.reasoning.enabled: false` opts out).
On macOS, an exclusive bind (SO_REUSEADDR disabled) refuses a port that is
still held only by a server-side TIME_WAIT socket for 2*MSL (~30s) after a
previous connection close — the gateway's own shutdown, or any
``Connection: close`` request. A ``/restart`` issued within that window
failed to bind, even though nobody was actually listening.
For an explicit host, a refused connect probe proves nobody is listening:
the kernel still rejects an exact duplicate bind even with SO_REUSEADDR, and
a foreign wildcard listener would answer the probe. So one retry with
SO_REUSEADDR is safe in that case. A wildcard host keeps the strict
exclusive path, since a foreign listener on a non-loopback interface can't
be probed this way. A live listener still wins either way.
The bind, probe, and retry logic is shared by webhook.py and api_server.py
through the new gateway/platforms/tcp_site.py, since both adapters had the
same exposure — api_server additionally marked a spurious TIME_WAIT bind
failure as a non-retryable port conflict.
(cherry picked from commit 289fd06148c447829f1e38b66ff78b9e4bebdeb4)
Gate review on the stack:
- `hermes gateway restart` under systemd only accepted `gateway_state == "running"`
as proof of the replacement; a boot with a parked platform stamps `degraded` for
its whole life, so every restart/update on such a host waited out the 60 s+
budget and reported a false failure while the gateway was serving. The verifier
now accepts `degraded` as restarted and prints one DEGRADED warning.
- Drain release and scale-to-zero wake re-stamped `running` unconditionally, wiping
the parked-platform signal after the first `.drain_request.json` cycle. Every
"we are serving" stamp now goes through `_serving_state()`; the mixed fatal +
retryable boot path sets the flag too (it fell through as a plain run before).
`_startup_parked_platforms` is a bool — the joined error text was only ever logged.
- `_wait_for_tcp_port_free`: an unresolvable or unreachable configured host raised
a non-refused OSError on every probe and burned the full 10 s wait; only a
connect timeout means "listener alive", anything else means nothing to wait for.
- Windows `restart()` replaces its blind `time.sleep(1.0)` "let Windows release the
port" with the same configured-address wait.
- The api_server bind retry rebuilds the AppRunner per EADDRINUSE attempt instead of
calling aiohttp's private `_unreg_site`; attempt count is a named constant.
Test: a `degraded` replacement is reported as restarted (red on the previous head).
E2E re-run: predecessor releases the port 0.5 s after the first bind → bound on the
third attempt (+0.61 s), connect() True.
Folds on the salvaged #92060 (@eliasburlison):
- `_wait_for_api_server_port_free` read only API_SERVER_HOST/PORT from the
environment, so an api_server port set in config.yaml (`platforms.api_server.port`)
was never waited on. The adapter's host/port resolution is now the shared
`gateway.platforms.api_server.listen_address()`, used by both the adapter and the
restart path, and the wait is skipped when api_server is disabled (a foreign
listener on the default port is nobody's race).
- Only ECONNREFUSED means the listener is gone. A timed-out connect (full accept
queue on a draining predecessor) was reported as "free", which would have let the
replacement start straight into EADDRINUSE again.
- The bind retry in `APIServerAdapter.connect` creates a fresh TCPSite per attempt
and unregisters the failed one: aiohttp registers the site before binding, so
re-starting the same object raises "already registered".
- The injected clock/sleeper/connect knobs are gone; the tests bind a real listener.
Tests: a real listener closed 300 ms in → wait returns True; a busy port taken from
config.yaml (env unset) → wait reports busy while enabled and is skipped when disabled.
E2E: with the predecessor releasing the port 0.5 s after the first bind, origin/main
logs `Errno 48 ... address already in use` and connect() returns False; this head
binds on the third attempt (+0.61 s) and connect() returns True.
On macOS api_server cannot SO_REUSEADDR, so replacing the process as
soon as the old PID exits still hits EADDRINUSE. The new process then
stays up with no API. Wait for the listen port to refuse connections,
and retry the bind a few times before treating the conflict as fatal.
Fixes#91547
(cherry picked from commit 5cb9545023646277b2fb59ebfb5e89b0fd161aba)
run_internal_session_turn belongs with the run machinery that already owns
_resolve_live_session_id, not in the api_server.py facade. Pure move: the
adapter keeps a thin delegating method, behaviour unchanged.
Follow-up to the salvaged #114680 (@phoebsie):
- `APIServerAdapter.run_internal_session_turn` takes the owning profile as a required
argument and raises without it. The dropped fallback (ContextVar, then the process-active
profile) had no caller and would have let a missing ownership proof silently bind a
wake to whatever profile the process happened to run as — the exact class #114679
reports.
- A draining gateway refuses the in-process wake immediately, mirroring the HTTP
self-post's 503 (>= 400 → raise → the notifier rewinds the claim and the next tick
retries). The concurrent-run cap keeps the 429-style backoff.
- Contributor attribution mapping for builder@phoebie.local -> @phoebsie.
Independent review findings addressed:
- adopt the live continuation tip (api_server_runs._resolve_live_session_id)
before the in-process wake, exactly as the HTTP self-post does: a rotated
(compressed) origin must be woken on the transcript that is actually live,
not the retired parent slice (CompressionSessionClosedError otherwise);
- retry a saturated concurrent-run cap with the HTTP path's own backoff
(gateway.wake._RETRY_DELAYS_SECONDS) instead of failing on the first check,
so a busy listener no longer burns one notifier failure per tick toward the
12-strike drop of a durable subscription;
- keep the historical HTTP self-post on a standalone (non-multiplex) gateway:
the in-process route is now chosen by a served-profile helper that requires
multiplex_profiles, and the adapter is told the authorized profile explicitly
instead of re-deriving it from HERMES_HOME;
- skip the ownership store read for the default profile (the fallthrough already
authorizes it) and run the delivery-time recheck off the event loop;
- tests: standalone named-profile gateway, missing store, failed-connect
boundary, adapter without in-process delivery, platform-wide api_server route
vs the default profile's destinations, cap retry/exhaustion, compression tip,
and the profile handed to the adapter (8 -> 15 cases).
A Kanban notify+wake subscription whose destination is a raw session id on the
shared api_server (platform=api_server, chat_id=<session id>, notifier_profile=
served secondary) was never delivered under gateway.multiplex_profiles: no
profile_routes entry can anchor a session id, so _adapter_for_subscription()
returned None and _claim_for_sub skipped the row before claiming (cursor frozen,
no delivery attempt). A platform-wide api_server route is not the fix — it
matches every api_server destination, so the default profile's own api_server
subscriptions would fail closed instead.
The wake leg had the same shape: _self_post_chat_completion() always POSTs to
the unprefixed /v1/chat/completions with the PRIMARY adapter's key, so a served
profile's wake turn would resume the session in the default profile's store.
/p/<profile>/v1/... requires that profile's own API_SERVER_KEY, which a
route-only profile legitimately does not have.
Authorize the shared adapter only when the subscription's session id is a row in
the served profile's own state.db stamped for that profile (NULL legacy stamp =
the store's own profile, the same rule the dashboard session routes apply) and
run the wake in-process under that profile's runtime scope through a new
APIServerAdapter.run_internal_session_turn(). Unknown session, foreign stamp,
unserved/deleted profile and unreadable store all fail closed; the concurrent-run
cap defers the wake (cursor rewind, retry) instead of bypassing it.
