Keep the two invariants (cross-profile isolation, same-namespace dedup) and a short
docstring; the composite key already covers route and principal rotation.
Review on this PR (tracked issue #84256) found the profile-only fix left
two more dimensions of the shared OpenAI-compat Idempotency-Key cache
unscoped:
1. No endpoint/route discriminator. _run_idempotent() is shared by both
POST /v1/chat/completions and POST /v1/responses, but _IdempotencyCache
._store keeps the fingerprint only as a value inside the key's slot —
not part of the key itself. A client reusing one Idempotency-Key across
both routes would have the second route's settled response silently
overwrite the first route's storage slot, so a legitimate retry of the
first request would recompute instead of replaying its own cached
result. Each call site now passes a stable logical route discriminator
("chat_completions" / "responses") folded directly into the cache key,
rather than relying on request.path (which would wrongly split /v1/...
and its /p/<profile>/v1/... alias into different namespaces).
2. No authenticated principal. The sibling durable /v1/runs admission API's
_run_idempotency_scope() already composes
sha256(profile \0 expected-api-key-or-sentinel) for its non-room case.
_run_idempotent() now delegates directly to
self._run_idempotency_scope(request) for its principal/profile scope
instead of re-deriving the profile alone, so both endpoints share one
definition of "who is this request for" and a caller whose
API_SERVER_KEY changes mid-cache-lifetime can no longer replay a
previous principal's cached response.
Adds regression tests proving: route isolation (including the exact
chat -> responses -> chat retry scenario from the review), concurrent
cross-profile inflight isolation, principal/key-rotation isolation, and
that same-route/same-profile/same-principal dedupe is unchanged.
Mutation-verified with three independent negative controls: reverting the
whole diff (TypeError on the now-required route kwarg), keeping route as a
parameter but dropping it from the key (only the two route-isolation tests
fail), and keeping the shared-scope call but reverting it to a bare
profile lookup (only the two principal-isolation tests fail).
_run_idempotent() (chat completions / responses) keyed the shared
_idem_cache purely on the client-supplied Idempotency-Key header plus a
body fingerprint, with no profile identity anywhere in the key. Under
gateway.multiplex_profiles, both the native routes and their /p/<profile>/
mirrors share the same process-global cache, so two different profiles'
clients sending the same Idempotency-Key with a matching fingerprint (a
realistic case for automation/SDKs that derive the key deterministically
from request content) silently share a cached response instead of each
profile running its own turn.
The sibling /v1/runs admission API already scopes its own idempotency
namespace this way (_run_idempotency_scope() folds in
_api_request_profile.get() or "default"); this mirrors that exact
expression into _run_idempotent()'s cache key.
An explicit X-Hermes-Session-Id client re-sending a pre-rotation id read
the closed parent's stale history slice, bound the turn and the wake
target to the closed parent, and — once the detached delivery writer
adopted the live tip — could no longer consume a late delegation delivery
row persisted on the tip, while still being granted wake_capable="1".
Resolve the client-addressed id through the same canonical
resolve_resume_session_id resolution the delivery writer (gateway/wake.py)
and /v1/runs use, before loading history: the read, the turn, and the wake
binding all select the live continuation tip. Response headers (SSE and
JSON) keep echoing the client-provided id, so the external identity stays
stable across rotations regardless of when a turn rotates relative to
prepared response headers.
Compression-tip read/write routing and the stable-echo identity contract
originate from #13437 (manuelschipper), ported onto the post-refactor
route owner with real-SessionDB regressions (JSON and streaming).
Reported-and-investigated-by: shojikumaru (Sho + Alpha) via #98619
Co-authored-by: manuelschipper <11737902+manuelschipper@users.noreply.github.com>
Rebuilt against the post-refactor owners: the chat-completions route now
lives in gateway/platforms/api_server_openai_routes.py, the wake gate in
tools/delegate_tool_dispatch.py, and session_context.py was reshaped —
the original patch aimed at code main no longer has.
Header-less OpenAI-compatible clients get a fingerprint-derived session
id bound as the api_server chat_id. delegate_task's background gate
treated ANY bound session id as wake-capable and dispatched detached
subagents, but for derived ids the wake self-post lands in a session
whose history the client never reloads — the result is undeliverable by
construction.
Bind a wake_capable provenance flag at session-bind time, DEFAULT-DENY
at the central boundary (set_session_vars and _bind_api_server_session
both treat an omitted declaration as denied; a binder that says nothing
grants no wake authority): "1" only from audited producers whose client
can address the id again (explicit X-Hermes-Session-Id — 403-gated on
API_SERVER_KEY — native /api/sessions/{id} routes, /v1/runs); "" for
fingerprint-derived ids. The delegate gate requires the flag (fail-closed,
captured pre-child-construction alongside origin_wake_sid) and keeps the
forced-sync fallback with its honest note.
Reported-and-investigated-by: shojikumaru (Sho + Alpha) via #98619
Port the run-ownership invariants from PR #93747 onto main's `_run_owners`
model in gateway/platforms/api_server_runs.py:
- `_request_owns_run` no longer admits run state that exists without an
owner stamp. Under gateway.multiplex_profiles every served profile holds
a valid key, so the "backward compatibility" branch made the boundary
allow-all whenever provenance was missing. Unstamped state now fails
closed; only an in-memory owner match or a durable idempotency record
under the caller's own scope admits a run.
- POST /api/sessions/{id}/chat/stream claims `_run_owners` at the run mint,
inside the request's profile scope, so its run is confined to the
creating profile like /v1/runs.
- Owner release is tied to "no run-keyed state survives"
(`_release_run_owner_if_forgotten`) and runs at every retirement point
(task finally, SSE stream close, both sweep loops, chat-stream finally),
not only the terminal-status sweep — no stranded entries, no stateful id
ever left unowned.
Docs: note that runs are per-profile scoped (replaces the now-false
visibility admonition proposed in PR #92822).
Fixes#93689Fixes#90415
Supersedes #93747, #93704, #92822
Co-authored-by: RickyYii <237135932+RickyYii@users.noreply.github.com>
Co-authored-by: liuhao1024 <11816344+liuhao1024@users.noreply.github.com>
Three cases in TestCreateAgentModelRecovery: the alias is never
cached, a poisoned cache is never served, and legitimate
last-known-good recovery still works.
Salvaged from #72739 (net diff onto current main; the read-side legacy-row
guard now flows through the shared _stored_session_model() helper that
landed in #88751, keeping one resolver for both chat sites).
POST /api/sessions persisted the advertised virtual alias (hermes-agent)
whenever the request omitted model or echoed the alias back; later turns
replayed it upstream as a real model id and every turn on the session
failed. Null the alias in _session_runtime_request_from_body() — shared by
session create, chat, chat-stream, and model-lock — and stop the
create-handler's raw-body fallback from bypassing that normalization.
Two bugs found by a REAL two-gateway live test (two isolated HERMES_HOMEs,
bravo running the api_server platform, alpha's agent autonomously running
`hermes peer dm` from its Bot Chat protocol; reply relayed correctly and
persisted in bravo's canonical Bot Chat):
1. peer dm parsed the session-create response flat, but api_server wraps
the row: {"object": "hermes.session", "session": {...}} — every first DM
to a fresh peer failed with "Peer did not return a session id" (and the
orphaned Bot Chat then 400'd retries with duplicate-title). Parse the
wrapped shape; the test fake now mirrors the real response shape so this
class can't pass green again.
2. api_server: a session created with no model persists the advertised
virtual model ("hermes-agent") on the row; session chat then replayed it
as a REAL model id and the provider 400'd ("hermes-agent is not a valid
model ID"). _request_agent_overrides already filters the virtual model
for per-request bodies — apply the same filter to the stored session
model at both chat sites (sync + stream), so it means "gateway default"
exactly like the request-body path.
Live E2E transcript (bravo's Bot Chat, via /api/sessions/{id}/messages):
user: Message from 🤖 alpha (@alpha): What is your callsign?
assistant: CALLSIGN-BRAVO-7
peer cmd unit suite 10/10 with the corrected fake.
e81d18dfb collapsed six per-surface copies of reasoning resolution onto
resolve_reasoning_config() and, in its own words, "fixes the gateway
resolving reasoning against config model.default instead of the session's
effective model". It did not touch gateway/platforms/api_server.py, which
kept that defect.
_create_agent() called GatewayRunner._load_reasoning_config() with no
model on its first line — before the model precedence chain (browser lock
-> session /model -> session row -> route -> per-request -> defaults) has
run. Per-model agent.reasoning_overrides therefore keyed off model.default
on the one surface where every request names its own model: a request for
a model with an override silently got the global effort instead.
Resolve after the chain settles, so the override follows the model the
request actually runs. An explicit per-request reasoning parameter still
takes precedence over config.
The existing test stub for _load_reasoning_config took no arguments (it
mirrored the old call); it now matches the real signature, as the sibling
stub in the same file already did.
The non-streaming /v1/responses path built function_call and
function_call_output output items with no status field (and no item id),
while the SSE streaming path correctly emits status in_progress ->
completed. Spec-strict OpenAI clients reading the non-streaming output
array could interpret the status-less function_call items as pending
calls the CLIENT must execute — but these tools were already executed
server-side by the Hermes agent and are replayed for structured tool UI
only. Reported by a community user whose GPT-5.6 client concluded 'a
server should not tell an OpenAI client to execute a tool the server
already executed itself'.
- _extract_output_items now stamps status: completed and spec-shaped
item ids (fc_/fco_) on replayed items, matching the streaming path
- test updated to pin status + id shape
- docs example updated + explicit note that output tool calls are
replayed, never pending
CI caught a missed caller-shape update. Both PRODUCTION callers of
_write_sse_chat_completion / _write_sse_responses were converted to
ThreadSafeAsyncQueue, but two pre-existing tests in
tests/gateway/test_api_server.py construct the writer's queue
themselves and still passed a stdlib queue.Queue.
The consumer now does 'await asyncio.wait_for(stream_q.get(), ...)',
which on a queue.Queue blocks the thread forever:
test_stream_cancelled_persists_incomplete_snapshot hung until
pytest-timeout killed it (CI reported the whole file as 'no tests
ran (timeout before collection)'). The sibling disconnect test only
survived because it pre-fills before the first await.
tests/gateway/test_api_server.py: 99 passed (was 1 failed + a 60s
hang); with the SSE/api_server suites: 147 passed.
The API server intentionally lets concurrent runs share a client-provided
session_id (= process task_id), so the SSE-disconnect reap could kill a
process a still-live concurrent run spawned after the disconnecting run's
baseline — the same stale-reaper bug class the gateway path gates via
run_generation.
- Per-task-id run epochs (monotonic counter): each run claims the epoch at
publish; a reaper holding a superseded epoch declines to kill. A missing
entry (the run's own clear pruned it) still reaps, so the leak fix isn't
silently disabled.
- _publish_turn_process_ownership / _clear_turn_process_ownership helpers
replace the copy-pasted marker set/clear blocks, so attribute names and
epoch bookkeeping can't drift between surfaces.
- /v1/runs — the third own-lifecycle surface — now records ownership and
reaps on POST /v1/runs/{id}/stop and on server-side SSE cancellation,
closing the remaining sibling paths of #76115.
dbbb10d39 shipped without direct test coverage for its own new logic
— the same gap teknium's review flagged on the competing PR. Close it:
- _reap_gateway_turn_processes: skips when is_still_current() is
False, proceeds when True, fails open (reaps) if the check itself
raises rather than silently disabling the leak fix.
- _abandon_timed_out_gateway_turn: still marks the turn abandoned
(interrupt fires) even when the reap itself is skipped.
- api_server._reap_disconnected_agent_processes: reaps the
baseline-diff for an owned turn, no-ops when the agent never
recorded ownership markers.
- APIServerAdapter._run_agent: markers are populated with the right
task_id/baseline during the turn and cleared once it completes,
closing the same race window fixed in gateway/run.py for this
separate agent-lifecycle surface.
Hardening on top of the salvaged #51642: free-text fields
(preview/goal/summary/output_tail) pass redact_sensitive_text(force=True)
before leaving on the public /v1/runs SSE stream — same treatment the API
already applies to error text — and child_session_id survives the
allowlist so clients can correlate the child's session. Both were flagged
in the sweeper review of #51642 and unaddressed.
Three parity fixes between the API server and the native gateway's
agent-runtime resolution, integrated with the provider-aware request
routing that landed in #70853:
- Session-persisted model is honored: POST /api/sessions {"model": ...}
stores a model that the chat handlers previously fetched and threw
away. A stored value that matches a model_routes alias goes through
the route path (route provider/credentials apply); a raw model string
threads through as session_model, pinning the session's turns ahead
of per-request body values but below an explicit session /model
override.
- Empty-model recovery: provider-catalog default when config has no
model.default but a provider resolved, plus last-known-good model
recovery (#35314) keyed on gateway_session_key only (never ephemeral
session_id — no unbounded growth from one-off requests).
- Provider auth failures surface as controlled responses: RuntimeError
from _resolve_runtime_agent_kwargs() is re-raised as a dedicated
_ProviderAuthResolutionError at the call site, caught narrowly in
_run_agent() and the /v1/runs executor to return run.py's response
shape instead of an undifferentiated 500 (session-chat endpoints
previously returned a raw aiohttp 500 with no JSON body).
Salvaged from PR #57947 by @FvanW; session-model route-alias resolution
from PR #59941 by @kaishi00.
Co-authored-by: kaishi00 <kaishi00@users.noreply.github.com>
Add authenticated GET /api/model/options to the gateway API server,
sharing the dashboard/TUI picker payload builder so external clients
can sync to the user's configured Hermes provider catalog instead of
scraping the single OpenAI-compatible /v1/models alias.
- new shared hermes_cli.inventory.build_model_options_payload() wraps
build_models_payload with the stable picker shape and safe
custom-provider probe policy (probe current only on normal open,
probe all + cache bust on explicit refresh)
- dashboard web_server and TUI gateway model.options refactored onto
the shared builder; dashboard build moved off the event loop via
run_in_threadpool
- capabilities endpoint advertises model_options
- docs for both API server and programmatic integration
Salvaged from PR #54689 by @abundantbeing.
Follow-ups on the salvaged #54426 routing contract:
- Bare `model` without `provider` on the OpenAI-compatible endpoints
(/v1/chat/completions, /v1/responses) is now opt-in via
gateway.platforms.api_server.direct_model_requests (default off) —
generic OpenAI clients hardcode model names ('gpt-4o', ...) and
existing deployments rely on those falling back to the gateway
default. Explicit `provider` requests and the Hermes-native
session-chat + /v1/runs surfaces are always honored.
Idea credit: PR #22825 by @mssteuer.
- A model_routes alias with no `model` key can no longer leak the
alias string as the executing model name (defensive; parse-time
validation already drops such routes).
- Fix mis-indented _run_agent call args in _handle_session_chat_stream.
- Docs: document the opt-in flag.
Carry model, provider, and model_options through the API server's
execution surfaces (session chat, Chat Completions, Responses, /v1/runs)
without mutating global configuration. Precedence: session /model
override -> model_routes alias -> direct request selection -> global
defaults. Conflicting route/provider mixes fail closed with 400.
model_options stays request-scoped regardless of which selection wins.
Salvaged from PR #54426 by @abundantbeing.
When _api_key_passes_startup_guard() rejects the key (missing,
placeholder/too short, or fail-closed unverifiable strength), connect()
returned a bare False with no fatal-error info. gateway.run's reconnect
watcher treats that as transient and re-queues with backoff forever —
each retry re-instantiating the adapter and its ResponseStore sqlite
connection. Observed in production (#37011): ~501 leaked connections
(1002 fds) over ~2.5 days until EMFILE made the whole gateway
unresponsive.
Set a non-retryable fatal error (api_server_key_invalid) in connect()
when the guard rejects, covering all three rejection branches, so the
platform drops from the reconnect queue; recover with
`/platform resume api_server` after fixing the key. Same treatment as
the port-conflict guard (api_server_port_in_use, #65665 / bda8bd76a8).
Tests mirror the port-conflict precedent: each rejection path asserts
connect() is False, has_fatal_error True, fatal_error_retryable False,
and fatal_error_code api_server_key_invalid, plus a strong-key control.
Re-implementation of #38803 by @cifangyiquan against current main —
their patch targeted the old inline guard in connect() which was since
extracted to _api_key_passes_startup_guard() (and gained the
fail-closed branch in 683059feb5), so the original diff no longer
applies. Their production diagnosis and non-retryable direction
preserved.
Refs: #38803, #37011
Salvaged from PR #36180 (commits 68dfeb4b16 and 86f437509b by arimu1),
re-applied onto current main with the incidental black-reformat churn
stripped out (~1,700 lines -> the semantic change + tests).
Previously gateway/config.py enrolled the api_server platform on
`api_server_enabled or api_server_key`, so API_SERVER_ENABLED=true with
no key (or a weak/placeholder key) still loaded the platform: the
adapter is instantiated (ResponseStore/SQLite opened in __init__), the
reconnect watcher spins, and the startup guard refuses at connect() —
logging errors forever. Now the platform is enrolled only when
API_SERVER_KEY passes the same strength bar as the adapter's startup
guard (has_usable_secret, min_length=16), via a shared
_has_usable_api_server_key() helper.
The no-op `lambda cfg: True` connected-checker for API_SERVER is also
replaced with the same key check, so get_connected_platforms() only
reports the platform "up" when it could actually start.
Known limitation (intentionally out of scope): a YAML config with
`platforms.api_server.enabled: true` and no key still loads the
platform; this gate covers the env-override path only.
Dropped from the original PR: EMAIL/SMS checker additions (scope creep
beyond the PR title; absent on current main) and the wholesale black
reformat of gateway/config.py and tests.
Fixes#36111
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`_api_key_passes_startup_guard` refuses to start the API server on a weak
`API_SERVER_KEY`, and its own log says why:
This endpoint dispatches terminal-capable agent work — a guessable key
is remote code execution.
But the check is wrapped so that a failure to import it starts the server
anyway:
try:
from hermes_cli.auth import has_usable_secret
if not has_usable_secret(self._api_key, min_length=16):
... return False
except ImportError:
pass
return True
`hermes_cli.auth` imports httpx at module scope and pulls in a large slice of
the CLI, so an import failure is not hypothetical — a trimmed image, a partial
install, or a circular import during gateway startup all produce one. When it
happens the strength check silently disappears and only the presence check
above it remains, so a placeholder key passes.
Reproduced against the real guard with the import blocked:
weak key, normal : False
weak key, ImportError : True <-- starts on a 4-char key
strong key, normal : True
Fail closed instead: an unverifiable key does not get to expose the endpoint,
and the log names the actual problem so the operator can repair the install.
This is the posture tools/credential_files.py already takes — it refuses a
mount when its deny-list cannot be consulted rather than risking it. The catch
also widens from ImportError to Exception, so an AttributeError or an error
raised inside the check cannot reopen the same hole.
Both happy paths are untouched: a strong key still starts, a weak or missing
key is still refused with the existing messages.
Unrelated to #38803, which fixes the retry behaviour after this guard rejects
and assumes the guard ran.
tests/gateway/test_api_server.py: new TestApiKeyStartupGuardFailsClosed — a
weak key is refused when the check is unavailable, a strong key is refused too
(fail-closed), plus three controls pinning the unchanged normal paths. The two
fail-open tests fail on main; the three controls pass there. 222 passed in the
api_server suites; 1475 passed across every suite touching api_server, with
the same 8 pre-existing failures on clean main.
The gateway /compress path can force a user-leading layout that leaves
the compaction summary after a retained system head, so scanning only
the leading block misses it. Preserve marker-carrying messages wherever
they sit, filling the remaining budget with the most recent other
messages, and cover the non-leading position with a test.
Address review feedback (PR #58133): the original test mocked _run_agent
with _compressed=True directly, bypassing the detection logic.
New tests mock _create_agent instead, so _run_agent's detection path
runs naturally and reads agent.session_id / _last_compaction_in_place:
1. test_rotation_compression_exercises_detection_and_persists_rotated_session_id
- Fake agent with rotated session_id -> verifies _compressed is set,
compressed history is stored, and rotated session_id propagates to
both response_store and X-Hermes-Session-Id header.
2. test_inplace_compression_exercises_detection_and_persists_compressed_history
- Fake agent with _last_compaction_in_place=True, session_id unchanged
-> verifies _compressed is set, compressed history is stored, and
session_id does NOT rotate.
3. test_chained_rotation_propagates_effective_session_id
- Two-request chain: first request triggers rotation, second request
loads history using the rotated session_id stored by the first.
Asserts the compressed transcript is loaded correctly for chaining.
_readiness_work_counts()'s active_api_runs set is {"queued", "running",
"waiting_for_approval"} — it excludes "stopping", the status
_handle_stop_run() sets while a run is being interrupted. Since the stop
is fully cooperative (the run stays "stopping" — doing real
executor-thread work — until the agent actually notices the interrupt and
the task settles to "cancelled", an unbounded window, not a fixed
timeout), /health/detailed's background_queues.active_api_runs
undercounts real active work for that whole duration.
Fix: add "stopping" to the active-status set. background_queues.status
itself is hardcoded "ok" (gateway/readiness.py), so this doesn't change
overall readiness — it only corrects the count value external monitoring
tooling reads from this endpoint.
/health and /health/detailed resolve the version via importlib.metadata
first, falling back to hermes_cli.__version__. On editable/source
checkouts — including the standard git-based install that hermes-setup
performs — hermes_agent-*.dist-info can survive a source update
unchanged, so the health endpoints keep reporting the previous release
even though the running code (CLI, dashboard, release tags) is newer.
Stale metadata does not raise, so the source fallback never fires.
Flip the preference: use hermes_cli.__version__ (the runtime source of
truth shared by the CLI and dashboard) first, and fall back to
distribution metadata only when the source import fails. The
never-raise contract of the version probe is unchanged.
Observed live: CLI, dashboard, and pyproject all reported 0.18.2 while
/health returned 0.18.0 from a stale hermes_agent-0.18.0.dist-info left
behind by a source update.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Rebased onto current main, where _ensure_session_db grew a per-profile
cache (get_hermes_home()-keyed) for /p/<profile>/ multiplex — after this
PR's base. The PR's async rewrite assumed the old single-self._session_db
model, so on current main `session_db=self._session_db` in _create_agent
would pass None in production (the real DB lives in the per-home cache).
Split the concern: keep a SYNC _ensure_session_db (per-profile, used by
_create_agent + the many sync-patching create_agent tests) and add an
async _ensure_session_db_async that captures the profile home on the loop
thread then offloads only the SQLite open via to_thread (single-flight).
Both share _open_and_cache_session_db. Request handlers use the async
variant; _create_agent reverts to the sync call. Updated the first-request
test's FakeDB to accept db_path to match main's SessionDB(db_path=...).
Co-authored-by: necoweb3 <sswdarius@gmail.com>
- Make create sequence (check + insert + title) atomic via single
_execute_write call with BEGIN IMMEDIATE, closing the TOCTOU window
where two concurrent same-ID creates could both return 201.
- Offload _ensure_session_db() to asyncio.to_thread with single-flight
lock so first-request SQLite init doesn't block the event loop.
- Add concurrent same-ID create test (one 201, one 409) and
first-request path test covering the initialization.
_check_auth gates every OpenAI-compatible API server endpoint. It compared
the client's raw bearer token against the configured key with
hmac.compare_digest on two str values. compare_digest raises TypeError on
a str containing non-ASCII characters, and the token comes straight from
the Authorization header — so a request with a single non-ASCII byte in
the key (a stray unicode char, a smart quote, a pasted BOM) crashed the
handler with an unhandled TypeError. Every endpoint calls _check_auth
without a try/except, so the framework turned that into a 500 Internal
Server Error instead of the intended 401 Invalid API key.
Compare as bytes, matching web_server.py's dashboard-token check
(hmac.compare_digest(auth.encode(), expected.encode())). Encoding both
sides keeps the timing-safe comparison and its semantics identical for
valid keys while making a non-ASCII token fail closed with a clean 401.
Adds regression tests: a non-ASCII bearer token returns 401 (no raise),
and a non-ASCII configured key still authenticates against its exact
value.
Two test mocks stubbed the old zero-arg signature; the chokepoint refactor
added an optional model param that call sites now pass. Swept the full test
tree for other stale stubs of the changed functions — the rest use
MagicMock/patch(return_value=...), which tolerate the new arg.
Adds a no-code routing layer to the OpenAI-compatible API server so one
Hermes deployment can map different API clients to different
model/provider backends. Clients pick a backend by sending a configured
alias as the OpenAI 'model' field; unmatched values fall back to the
global model. Configured aliases are listed by GET /v1/models.
Precedence (highest first): session /model override > model_routes
route > global config. Route provider credentials resolve through
_resolve_runtime_agent_kwargs_for_provider (same seam as
channel_overrides); per-route api_key/base_url are upstream provider
credential overrides — never caller auth, never logged.
Salvaged and rebased from PR #3176 by @Mibayy onto current main.
/health/detailed leaked runtime state (gateway state, connected
platforms, active-agent counts, PID, exit reason) with no auth. Gate it
behind the same Bearer auth as other API routes; plain /health stays
open for liveness probes.
Also refuse to start on a placeholder/too-short (<16 char) API_SERVER_KEY
regardless of bind address — a guessable key on a terminal-capable
endpoint is RCE-adjacent even on loopback, since any local process can
reach it. The required-key check was already unconditional; this extends
the strength floor to loopback binds too. Startup guards are hoisted
above app/background-task creation so a rejected start leaves no partial
state.
Salvaged from #44073 (external-surface hardening), split into a focused
PR per maintainer request.
Co-authored-by: Hermes Agent <agent@nousresearch.com>
When the primary provider's auth fails (expired token / 429 quota cap),
_resolve_runtime_agent_kwargs() falls through to the fallback provider
chain, whose runtime dict carries its own 'model' key. api_server's
_create_agent then did AIAgent(model=model, **runtime_kwargs), colliding
on 'model' and 500ing every /v1/chat/completions request while a fallback
was active. Pop the runtime model and let it override the config model,
mirroring the native gateway path (_resolve_session_agent_runtime).
Salvaged from #35716 by @ryo-solo (earliest submitter); the PR's second
half (Mistral reasoning_content strip) is already handled on main and
dropped.
Co-authored-by: Hermes Agent <noreply@nousresearch.com>
Defense-in-depth on top of _safe_session_filename_component (#5958):
Sink (makes the bad write impossible regardless of entry point):
- run_agent._save_session_log: sanitize session_id before building the
session_{sid}.json snapshot path.
- agent_runtime_helpers.dump_api_request_debug: sanitize before building
the request_dump_{sid}_{ts}.json path.
Boundary (clean 400 instead of a silently-hashed filename):
- api_server rejects path-traversal-shaped X-Hermes-Session-Id on the
session-continuation path and the explicit /api/sessions create path,
reusing gateway.session._is_path_unsafe (mirrors the native gateway's
entry-boundary guard). Also enforces the session-header length cap on
the continuation path.
Tests: traversal session_id stays contained at the write site; sanitizer
always yields a traversal-free segment; the API header rejects
../, absolute, and Windows-traversal IDs with 400.
Follow-up to the salvaged #37733 fix. The contributor centralized
redaction at _openai_error and the chat/responses failure paths, which
covers the OpenAI-compatible envelopes transitively. Two sibling classes
crossed the same authenticated HTTP boundary unredacted:
- 8x cron-management endpoints returning {"error": str(e)} on 500
- the session-chat SSE error event ({"message": str(exc)})
Route both through the same _redact_api_error_text(force=True) helper.
Add AUTHOR_MAP entry for coygeek and a TestRedactApiErrorText guard
covering mask/force/limit/passthrough behavior.
Force API-server error text through the existing secret redactor before returning OpenAI-compatible errors, response fallback text, response snapshots, and run failure events. This prevents credential-shaped provider failure text from crossing the API-server boundary while preserving debuggable sanitized messages.
