Adds platforms.slack.extra.native_task_cards: when enabled, live tool
calls render as Slack-native plan/task cards via chat.startStream /
chat.appendStream (task_display_mode: plan, task_update chunks) instead
of text/edit progress bubbles. ID-bearing tool_start/tool_complete
callbacks correlate concurrent same-name tool calls correctly; any
native API failure falls back to one continuously edited text update.
The stream is stopped exactly once when the turn finalizes.
Salvaged from PR #29496 onto current main (TurnRunner/TurnContext seam);
closes#29483.
The NS-570 epoch stamp clears a drain marker that survives a machine
restart — but it assumes every drain-gated action ends in a restart. When
a maintenance action completes WITHOUT recreating the container and the
writer never cancels the drain, the orphaned marker still carries the
current epoch, so the 1s drain watcher honours it forever and the gateway
bounces every inbound message with the 'draining for a maintenance
action' text (observed in the field: a Hermes Cloud instance refused all
Telegram turns for ~3 days).
The marker already records requested_at; now the readers check it. A
marker older than DRAIN_REQUEST_MAX_AGE_SECONDS (1h) reads as stale in
drain_requested() and drain_notification_suppressed(), with a loud
warning log. Leniency mirrors the epoch check: a missing or unparseable
timestamp still reads as drain-active (fail-safe toward quiescing), and
a legitimately long drain keeps a sanctioned keep-alive — re-calling
write_drain_request() refreshes requested_at.
Fixes#85433
Salvage of #37865 by @verybigdog. Adds delivery_mode (notify / notify+wake / wake)
on kanban notify subscriptions, persists chat_type + user_id_alt so a woken turn
reconstructs the creator's real session key, inherits the return path to child
tasks, and keeps wake out of the model-exposed send_message schema.
Original commits were authored under a local placeholder identity
(hermes-agent@users.noreply.local); re-attributed to the contributor's
public email.
Third lane of the same contract (found in live staging validation):
/sethome run as a top-level relay-fronted Slack DM message captured the
adapter's session-keying thread stamp (the /sethome message's own id)
into the persisted HomeChannel.thread_id and its legacy env mirror.
Every bare-platform delivery (deliver="slack") then resolved home chat +
home thread and landed inside the ephemeral thread around the old
/sethome message. Extracted _home_thread_from_source with the same
synthetic-stamp recognition as cron origin capture; a /sethome run
inside a genuine thread keeps that thread as the home target. Users
repair an already-poisoned home target by rerunning /sethome.
Bug 1: relay-fronted Slack in thread-per-message mode stamps each top-level
message's own id as source.thread_id (session KEYING, native thread_ts
parity). Cron origin capture persisted that stamp as durable routing, so
every delivery landed inside the ephemeral thread spawned around the
creation message instead of the top-level conversation. Fix at the source:
_origin_from_env drops a Slack thread id equal to the creation message's
own id (genuine in-thread creations keep theirs). Fire-time repair for
already-persisted jobs: deliver=origin and the explicit-target Slack
re-attach treat an origin thread as stale when the origin chat is the
configured Slack home chat — top-level (or the home target's configured
thread) wins; non-home working threads are preserved.
Bug 2: _preflight_check_delivery and cron_delivery_targets validated
deliver prefixes against get_connected_platforms(), which only sees
natively configured platforms — a relay-only deployment ({relay}) rejected
'slack:CHAT' with 'no gateway credentials configured' although fire-time
routing (resolve_delivery_transport + RelayAdapter.fronts_platform)
delivers it. New gateway.relay.relay_fronted_platforms() (env-derived from
GATEWAY_RELAY_PLATFORMS — the same source that seeds the live adapter's
identity set, so validation and routing cannot disagree) is unioned into
the connected set when the relay is connected. Native topologies keep the
strict credential check unchanged.
The session chat stream registered its wrapper task in _active_run_tasks,
but that turn is already counted by active_agent_work_count() via
_inflight_agent_runs (_run_agent) — the drain saw 2 for one turn
(test_session_chat_sse_turn_is_interrupted). Keep only the agent-ref
registration; run-scoped steer control doesn't need the task entry.
Adds POST /v1/runs/{run_id}/steer and bridges Browser-Extension/WebUI
session chat streams into the active run registry so live runs on those
surfaces are steerable too.
- steer accepted only while run status is exactly 'running'; stop/stopping/
terminal states return 409 run_not_accepting_steer even while cooperative
shutdown retains the agent reference
- session SSE disconnect/cancellation interrupts and drains the executor-
backed run instead of cancelling only the async wrapper; control refs stay
registered until the turn actually exits
- undelivered steer text (accepted after the final response) is preserved as
pending_steer on the terminal run.completed event/status so clients can
replay it as the next user turn instead of losing it
- docs for the endpoint, next-tool-boundary delivery, acceptance-vs-delivery
semantics
Salvaged from PR #54466 by @abundantbeing.
Webhook agent runs default to the constrained hermes-webhook toolset
(web/vision/clarify) because payloads can carry untrusted third-party
content. That default is right for public webhooks but wrong for trusted
local pushes (e.g. an OOM monitor daemon that needs the agent to run
ps/free/py-spy): the only workaround was widening platform_toolsets.webhook,
which elevates EVERY webhook route at once.
This adds a 'toolsets' key on individual webhook route configs (static
routes in config.yaml and dynamic subscriptions in
webhook_subscriptions.json) that replaces the platform-level resolution
for that route only:
- BasePlatformAdapter.toolsets_for_source(): per-source override hook,
default None (no behavior change for any other platform).
- WebhookAdapter.toolsets_for_source(): maps the session chat_id
(webhook:{route}:{delivery_id}) back to its route config and returns
the route's toolsets list.
- GatewayRunner._resolve_enabled_toolsets_for_source(): shared resolver
used by both agent-run call sites; validates the override through the
SAME _get_platform_tools path as platform config, so unknown names and
platform-restricted toolsets (e.g. discord_admin) are dropped rather
than trusted.
Deliberately NOT exposed via 'hermes webhook subscribe': granting elevated
tools is a manual config edit only, so an agent-created subscription
cannot self-grant terminal at runtime.
Cache-safe: the toolset list is resolved before agent construction and is
constant for a route, so the per-session agent signature and frozen system
prompt are unaffected mid-conversation.
- Collapse the duplicated discord LoginFailure/PrivilegedIntentsRequired
classification (name-match + isinstance blocks repeated the same code/
message tuples) into a single _is() helper — one message per failure.
- Replace the user-facing HERMES_RECONNECT_ATTENTION_AFTER_SECONDS env var
with agent.reconnect_attention_after in config.yaml (default 7200, 0
disables), bridged internally like gateway_timeout. .env is for secrets.
- Use _float_env for robust parsing instead of bare int(os.getenv(...)).
- Document terminal classification + needs_attention escalation in
website/docs/user-guide/configuration.md.
Fleet triage after the 2026-08-11 storm resolution found agents whose sole
platform had been silently 'retrying' for weeks: revoked Telegram tokens,
Discord privileged-intent rejections, and Photon sidecars that can never
start were all funnelled into the indefinite reconnect queue with no owner
signal (OOF-151/152/153, epic OOF-156).
Two-part fix:
1. Per-adapter classification — by exception TYPE only, never message text:
- telegram: InvalidToken/Forbidden -> telegram_auth_error, retryable=False
(new _looks_like_auth_error, mirrors _looks_like_network_error)
- discord: LoginFailure -> discord_auth_error, PrivilegedIntentsRequired
-> discord_intents_required (both retryable=False); every other path now
sets an explicit code (previously the generic branch set NO fatal info,
which the gateway read as 'probably transient')
- photon: new typed PhotonSidecarStartupError; deps-install failure ->
SIDECAR_DEPS_MISSING and missing node binary -> SIDECAR_NODE_MISSING
(retryable=False); ambiguous startup crashes stay retryable
- email: IMAP/SMTP failures now always set a fatal code;
SMTPAuthenticationError -> email_auth_error, retryable=False (IMAP4.error
is type-ambiguous between bad creds and transient NOs, so IMAP stays
retryable)
2. Gateway escalation — platforms continuously in the reconnect queue past
HERMES_RECONNECT_ATTENTION_AFTER_SECONDS (default 2h, 0 disables) get
needs_attention=true + retrying_since stamped into runtime status, once
per episode, cleared on successful reconnect.
Deliberately NOT a circuit breaker: retries never stop. The auto-pause
mechanism was removed for good reason (transient DNS outages left bots
silently dead); this preserves that and only adds visibility. No new
platform_state enum values — NAS's status schema is strict — only additive
fields.
Unknown exception types always stay retryable: a false terminal recreates
the silently-dead-bot problem, and the escalation path covers
misclassified permanent failures.
Adds a 'council' synthesis style to MoA (per preset via synthesis_style,
one-shot via the new /council command on CLI + gateway). Reference models
answer independently; the aggregator chairs the deliberation and produces
a user-facing report of consensus, per-model disagreements (with the
differing assumptions behind them), unique contributions, and a
recommendation with an explicit confidence level.
Inspired by Perplexity's Model Council rollout to Perplexity Computer
(changelog 08/04/26): pick a board of 2-8 models, run them independently,
synthesize where they agree/disagree and what each uniquely surfaces.
Part A — pre_command observer hook (observer-first per #64182 ground rule 3):
- New VALID_HOOKS event `pre_command`: fires when a recognized slash command
is about to be dispatched, BEFORE the handler runs, on both surfaces:
- CLI: cli.py process_command (right after alias resolution)
- Gateway: gateway/run.py _handle_message cold-path canonical dispatch
- Payload: surface ('cli'|'gateway'), command (canonical), alias_used,
args_raw, session_key, platform. Return values IGNORED in v1; a plugin
returning a directive-shaped dict gets a debug log so future
block/rewrite adopters are discoverable (#64231 taxonomy).
- Deliberately NOT fired on the gateway running-agent intercept path
(/stop, /approve, busy_policy dispatch during an active run): those are
control-plane escape hatches on an in-flight run and must stay outside
plugin observation/veto reach.
- fire_pre_command_hook() helper never raises, so broken plugin infra can
never break command dispatch.
Part B — ctx.call_mcp (capability-gated, default-off, ground rule 4):
- PluginContext.call_mcp(server, tool, arguments, timeout=30): synchronous,
callable from plugin hooks/tools, routes through the EXISTING native MCP
client machinery (tools.mcp_tool._make_tool_handler: background loop,
trust-tier gates, circuit breaker, reconnect) — never a parallel client.
- Gate: plugins.entries.<id>.mcp_allowlist (list of server names).
Absent key / unreadable config / non-list value => default-deny.
Unlisted server raises PermissionError naming the exact config key.
TODO seam left for the #64228 declared-capability model.
- Bounded: timeout clamped to 1-600s and forwarded to the MCP loop call;
results capped at 64KB with truncation marker; stable
{ok, result|error, structuredContent?, truncated?} envelope.
Tests (transport mocked, no live MCP servers):
- tests/hermes_cli/test_pre_command_hook.py: both surfaces fire, canonical
alias reporting (/exit->quit, /q->queue), hook-before-handler ordering,
control-plane exclusion, hook failure non-fatal, observer-only directive
handling.
- tests/hermes_cli/test_plugin_call_mcp.py: default-deny (absent entry,
unreadable config, non-list, '*'), allowlist enforced per-server,
denied calls never touch transport, timeout forwarding/clamping,
result truncation, error/structuredContent envelopes.
Docs: hooks.md shipped-catalog row for pre_command; plugins.md
"Calling MCP servers from plugins" section with the security note.
Closes#64204
Adds a pre_transcription transform hook (prompt/language/model mutable,
file_path read-only, last-writer-wins per the transform_* convention)
fired before any STT backend, threads prompt to faster-whisper
(initial_prompt) and OpenAI/Groq/Mistral/DeepInfra (prompt), adds an
optional stt.prompt config key on the same plumbing, and keeps the
no-hook dispatch path byte-identical. Fixes#64168.
Documents the new surface for users: a "Transcription prompt
(vocabulary hints)" subsection in the configuration guide (composition
order, per-provider support matrix, length contract, privacy warning),
a pre_transcription entry in the hooks reference, and the mirrored row
in the plugins hook table.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01AG6LyYMvHC2o6HbVUozmVR
Builds on Paolo Antinori's #68431 salvage for #64176. Move plugin dispatch behind the profile-scoped runner authorization boundary, fail closed on malformed reaction identities, preserve observer registration across Telegram app rebuilds, and document the deliberately observer-only contract.
Co-authored-by: Paolo Antinori <pantinor@redhat.com>
First slice of #64176's observer-hook half — a normalized-envelope inbound
event hook, replacing raw-SDK handler args with a stable contract (per #64176's
"normalized versioned envelopes only; raw SDK gated behind a capability" rule).
- VALID_HOOKS: register the four gateway_* names from #64176
(gateway_platform_event fires today; gateway_session_titled /
gateway_message_delivered / gateway_thread_created reserved pending #64176's
fire-sites).
- BasePlatformAdapter._fire_gateway_hook: reusable, has_hook-guarded,
per-call-isolated fire helper (the no-subscriber common case short-circuits).
- TelegramAdapter: a group-99 catch-all TypeHandler normalizes inbound updates
into gateway_platform_event envelopes. message_reaction -> {platform,
event_type:"reaction", payload{emojis, custom_emoji_ids, chat_id, message_id,
thread_id}} (custom-emoji reactions captured via custom_emoji_id; standard via
.emoji — no None in consumer-facing lists). Other update types return None
pending #64176's taxonomy (#64231). Normalization is wrapped so a malformed
update can't raise into PTB dispatch.
Observer-only — zero behavioral change to core dispatch. Supersedes the raw
inbound half of #62584 (telegram:update -> normalized gateway_platform_event).
Tests: VALID_HOOKS registration; _fire_gateway_hook routing/has_hook/isolation;
_normalize for standard, custom-emoji, and mixed reactions + non-reaction;
_on_platform_update firing + normalize-error isolation.
Ran code-review (high) + simplify before pushing.
Route plugin target parsing, validation, and host-driven delivery through PlatformEntry across CLI and cron while preserving the host-only send_message policy.
* fix(gateway): pass live adapters to cron fire webhook's fire_due
The Chronos fire webhook (/api/cron/fire) called
provider.fire_due(job_id, adapters=None, loop=loop), so every
externally-triggered fire delivered through the standalone path even
with a live gateway in-process. E2EE platforms and relay-fronted
logical platforms (whose ONLY send path is the live relay adapter — no
native credential exists on the box) failed every external fire with
"platform 'X' not configured/enabled", while the same job delivered
fine under the built-in ticker (gateway/run.py passes runner.adapters).
Resolve the runner (self.gateway_runner → app['gateway_runner'] →
_gateway_runner_ref(), the same chain the drain check uses) and forward
its adapters. No runner → adapters=None, preserving the historical
standalone path byte-identically.
Note: does not by itself fix Fly-hosted scale-to-zero deployments where
NAS's callback lands on the DASHBOARD process (internal_port 9119) —
_fire_cron_job_for_profile there has no gateway runner in-process. That
topology needs a separate fire handoff (design pending).
* fix(cron): dashboard forwards Chronos fires to the gateway (503 when unreachable)
The dashboard's /api/cron/fire executed cron jobs in the DASHBOARD
process via _fire_cron_job_for_profile with adapters=None. On hosted
deployments (Fly proxy exposes only the dashboard's port) that made
every managed-cron fire deliver through the standalone send path, which
cannot serve relay-fronted logical platforms (their only sender is the
live relay adapter in the gateway process — no native credential exists
on the box) or E2EE rooms. It also ran the whole agent turn inside the
dashboard: wrong process for memory/session ownership and fire-claim
attribution.
Restore the invariant that the GATEWAY owns cron execution:
- Dashboard route: after verifying the NAS JWT and resolving the job's
profile, FORWARD the fire to the gateway api_server's own
/api/cron/fire on loopback, NAS bearer preserved (the gateway
re-verifies the JWT — defense in depth, no new trust link), and pass
the gateway's response through. Gateway unreachable → 503 so NAS
retries per the Chronos contract (non-2xx = retryable; the store CAS
de-dupes the eventual double fire). Deliberately NO local-execution
fallback.
- Endpoint resolution mirrors gateway/config.py's api_server load order
per target profile (config.yaml extra.port → API_SERVER_PORT from
process env or the profile's .env → 8642), with /p/<profile>/ prefix
routing under multiplex.
- docker/stage2-hook.sh: generate a strong API_SERVER_KEY into .env on
first boot when absent (never overwrites an operator value), so the
loopback api_server passes its startup guard on hosted images. The
fire route itself is NAS-JWT-authed; the key gates the rest of the
api_server surface. The listener binds 127.0.0.1 by default and the
Fly service exposes only the dashboard port.
- _fire_cron_job_for_profile kept but deprecated (late-binding seam
compatibility); no route calls it.
- docs/chronos-managed-cron-contract.md: document the two-hop inbound
topology and the 503-retry semantics.
Depends on the previous commit (fire webhook passes live adapters to
fire_due) — together they make NAS→dashboard→gateway fires deliver over
relay end to end.
* fix(cron): read the profile api_server port via the canonical config loader
CI guard test_config_read_guard flagged the new _gateway_fire_endpoint
for a raw yaml.safe_load of the profile's config.yaml — the exact drift
class the guard exists to kill (raw reads miss the managed-scope
overlay, ${ENV_VAR} expansion, and root-model normalization).
Read through load_config() under a HERMES_HOME override scoped to the
target profile instead (the same pattern the deprecated
_fire_cron_job_for_profile uses for its store scope), and pull the port
with cfg_get. Test updated to stub load_config rather than write a raw
config.yaml.
* fix(gateway): only messaging platforms count for the scale-to-zero arm gate
The stage2 hook now generates API_SERVER_KEY for every Docker container,
and key presence force-enables the api_server platform. The scale-to-zero
arm gate counted every enabled platform, so the loopback api_server
listener made messaging_is_relay_only_or_absent False on every hosted
instance — silently disarming the feature (the not-armed log would show
enabled platforms=['relay','api_server']).
The arm gate and the not-armed logger now share one helper that filters
to enabled MESSAGING platforms, excluding LOCAL/API_SERVER/WEBHOOK —
the same non-messaging exclusion set _connect_platforms already uses.
A genuinely enabled direct-socket platform (Discord/Telegram) still
disarms. Two of the three new tests fail without this fix.
The relay interactions passthrough lane (_discord_interaction_to_event)
built its SessionSource with platform=Platform.RELAY and no
delivered_via_upstream_relay marker — unlike the relay text lane
(ws_transport._event_from_wire), which maps the connector's platform to
the logical enum and stamps the authenticated-upstream flag.
Consequences of the mismatch:
- /sethome sent as a Discord slash command persisted the home channel
under platforms.relay.home_channel (invisible to cron delivery, which
looks up the logical platform) and mirrored it into the dead
RELAY_HOME_CHANNEL env var — so cron jobs with deliver='discord' kept
falling back to local-only even after the resolution/delivery fixes.
The absent trust marker also meant via_relay=False, so the handler's
'Relay does not authenticate this logical home target' guard —
designed to reject exactly this misfiled shape — never engaged.
- Session keys forked: the connector binds the interaction's follow-up
capability under buildSessionKey with platform 'discord' and
chat_type 'group' (interactionSessionSource), while the gateway keyed
the same interaction as relay/channel.
- _capture_scope skipped recording _platform_by_chat (it ignores the
generic 'relay'), losing the egress sender hint for the chat.
Stamp Platform.DISCORD (the lane statically parses Discord interaction
wire payloads), chat_type 'group' for guild channels (native-adapter and
connector parity), and delivered_via_upstream_relay=True (parity with
the text lane; set locally, never read off the wire).
With this, slash-command /sethome files under platforms.discord and
passes the via_relay guard legitimately, and cron delivery over relay
works end to end with the #84300 resolution fixes.
_scale_to_zero_has_live_background_work() counted every task in
_background_tasks — but _spawn_supervised parks all permanent watchers
there (session-expiry, kanban, reconnect, the scale-to-zero watcher
itself, ...). An armed gateway therefore considered itself busy forever
and never went dormant or suspended. Verified live on staging
(hermes-agent-stg-test-6698, 2026-08-12): armed at 05:25, fully idle for
25+ minutes, zero 'going dormant' lines. Fly's coarse proxy autostop used
to mask the bug; once the gateway took ownership of the suspend (#84295)
it became load-bearing.
_spawn_supervised now tags its tasks and the busy check skips them.
Transient tasks (startup-resume events, delegation, tracked processes)
still block suspend. New tests exercise the REAL _spawn_supervised path
rather than a stubbed _background_tasks set — the stubbing is exactly why
the earlier tests missed this (same call-site trap as the F25 arm bug);
the key test fails on main and passes with the fix.
Fly Proxy autostop judges idle exclusively on inbound proxied connections.
It cannot see an in-flight agent turn (outbound-only LLM traffic), and since
Fly's mid-2026 proxy change an open outbound socket (the relay WS) no longer
holds a machine awake. With autostop:"suspend", Fly suspended machines while
they were still processing long-running jobs, and could suspend before the
gateway flipped the relay destination (the buffered-event black hole).
The scale-to-zero watcher now owns the suspend: after the idle predicate
holds (no running agents, no live background work, inbound-quiet) and the
go_dormant() quiesce completes (relay drained + flipped), it POSTs
/v1/apps/{app}/machines/{id}/suspend on the local /.fly/api flaps socket.
Suspend is skipped when the quiesce fails or inbound lands mid-quiesce
(flip-before-freeze), and off-Fly the step is a no-op (fail-awake).
Pairs with the NAS change that provisions scale-to-zero machines with
autostop:"off" (gateway-owned suspend); wake is unchanged (Fly-proxied
wakeUrl poke + autostart).
* feat(relay): ambient token endpoint mode for gateway.idp.token_url
When gateway.idp.token_url is configured WITHOUT client_id/client_secret,
treat the URL as a metadata-server-style ambient credential endpoint:
plain GET, response body is the token (raw JWT or {"access_token": ...}
JSON envelope). Covers workload-identity proxies such as Domino's
$DOMINO_API_PROXY/access-token, which mint short-lived user-scoped OIDC
tokens with no client registration.
Previously this configuration was a hard error (client_id/client_secret
missing), so no working deployment changes behaviour: creds present keeps
the OAuth2 client_credentials POST, no token_url keeps Nous Portal. The
misconfig error now self-diagnoses (names the ambient fallback and how to
select the client_credentials grant instead).
* fix(relay): reject short plain-text bodies in ambient token shape gate
Review finding: the shape gate accepted any base64url-alphabet word, so an
IdP answering the ambient GET with a terse error body ('unauthorized',
'error', 'null') had that word returned as a bearer token instead of the
fail-closed misconfiguration error. Tighten the gate to JWT-like dotted
tokens (3+ segments) or long opaque tokens (>= 32 chars); short bare words
now raise the self-diagnosing ambient error.
* fix(relay): partial IdP client credentials keep the loud error, never select ambient GET
The ambient-endpoint dispatch used 'not client_id or not client_secret',
so configuring exactly one credential (a mistyped client_credentials
setup) silently issued a GET at the IdP token endpoint and then raised
'no client_id/client_secret configured' — factually wrong for that
operator, and a stray request the old hard error never made.
Ambient mode now requires NEITHER credential; a partial pair raises
immediately, names the missing key, and issues no HTTP request (tests
assert urlopen is never called). Docstring and relay.md now say
'neither' instead of 'without'.
* fix(relay): ambient JSON envelope requires a string access_token, no coercion
Review finding (P2): the JSON-envelope branch accepted any truthy
access_token via str() coercion — a number became '12345…', a boolean
became 'True', an object became its Python repr — bypassing the fail-
closed contract and deferring the failure to the connector, where it
hides the real endpoint problem.
The envelope value must now be a non-empty string, the same contract the
client_credentials path enforces on its token response. Deliberately NO
shape gate on envelope values: an envelope is an intentional token
response (mode-1 symmetry), and opaque tokens may use the standard-base64
alphabet the raw-body gate rejects. Mutation check: reverting the branch
to str() coercion sends the 3 coercion tests red (3 failed, 15 passed).
---------
Co-authored-by: Ben Barclay <ben@nousresearch.com>
* fix(relay): stop sibling gateways answering another instance's button press
A Discord button press arrives on the passthrough plane, and the connector
fans a passthrough forward out to EVERY live gateway session of the tenant
(relayServer.routeBusMessage delivers `passthrough` via sessionsByTenant),
unlike a message, which it narrows to the admitted instance set. The prompt
went out from exactly one instance and _pending_prompts is process-local, so
every sibling gateway saw an answer for a prompt it never minted, could not
tell that from its own prompt expiring, and fell through to chat dispatch --
where the option-shaped text ("/c1") is not a real command and run.py replied
"Unknown command `/c1`". One copy per sibling, under the single real ack.
Prompt ids are now minted as `<per-process nonce>.<8 hex>`, so an answer can
be attributed to the process that minted it. A prompt answer is always
consumed, never re-dispatched as chat: a sibling's prompt and a repeat answer
are both dropped silently, and an expired prompt of our own gets a short
"no longer waiting" notice from the owning gateway only.
Ids stay inside the connector codec's contract ([A-Za-z0-9_.-], <=32 chars,
64-byte callback budget -- verified against promptCodec.ts: 52 bytes worst
case with a full-length option id). An id with no nonce segment (a prompt in
flight across an in-place upgrade) is still treated as ours.
Tests: 4 added, each verified to fail without the fix. Full relay suite green
(160 tests).
* style(tests): ruff-format the added relay prompt tests
Completes the bug class from #83906 — the same blocking fsync-on-event-loop
pattern existed in two more async gateway paths:
- slash_commands.py _handle_restart_command: two atomic_json_write calls
for .restart_notify.json and .restart_last_processed.json were blocking
on fsync inside an async function. Now offloaded via asyncio.to_thread.
- run.py _clear_restart_failure_count: called from
_handle_message_with_agent (async, per-turn path) after a successful
agent turn. Made the method async and offloaded the atomic_json_write
call via asyncio.to_thread. Caller updated to await.
Shutdown-path calls in _stop_impl_body (_increment_restart_failure_counts,
planned restart notification marker) are intentionally left synchronous —
the event loop is draining/stopping and offloading adds complexity for no
benefit.
atomic_json_write() calls os.fsync(), which blocks until the write
reaches stable storage. build_channel_directory() already offloads its
builders with asyncio.to_thread (#60794) but still called the persist
step directly on the loop, so the Discord heartbeat waited on a disk
flush.
* fix(gateway): stop frozen-preview finals and dropped idle-session delegation callbacks
Two relay-plane delivery losses from the 2026-08-09 staging incident:
1. stream_consumer: the skip-redundant-finalize branch recorded _accumulated
as the delivered turn-final payload even when the last ACKED edit was an
earlier throttled preview snapshot, so delivered_final_matches reconciled
True and the gateway suppressed the corrective final send — the user was
left with a cut-off message ending in the streaming cursor. Extracted
_mark_skip_redundant_finalize(): records the last acked wire payload
(cursor-stripped), so a preview/final mismatch now returns False and the
normal final send fires.
2. run.py: _classify_completion_target classified every ended parent session
terminal unless it ended by compression. Idle/timeout session ends are the
norm on scale-to-zero relay deployments and the chat route remains valid;
completed async delegation results were terminally dropped. Ended parents
now classify deliver unless the end was an explicit user boundary
(session_reset / user_exit / session_switch).
* fix(relay): drain in-flight outbound frames before transport teardown
disconnect() failed every pending outbound future immediately with
'relay transport closed', so a trailing finalize edit racing turn
teardown was lost even though the connector socket could still serve
it. Bounded drain grace (5s) lets in-flight requests resolve; silent
connectors still tear down promptly. asyncio.wait (not gather+wait_for)
so a timeout doesn't cancel futures owned by the fail-remaining loop.
* fix(gateway): route completion injection through the alias-aware transport resolver
Third relay-plane delivery loss from the 2026-08-09 staging incidents: a
delegation batch completed while the gateway was up, the watcher drained
the event, and delivery vanished with no log line. _inject_watch_notification
resolved its adapter with a literal p.value == platform_name scan of
self.adapters — a relay-fronted gateway registers ONE adapter under
Platform.RELAY fronting N logical platforms, so 'slack' never matched and
the injection returned None ('no gateway route'), silently dropping the
completion. The handoff path already documents this exact trap and uses
resolve_delivery_transport; the injection path now does the same (native
wins; relay eligible only when it fronts the logical platform), with the
literal scan kept as fallback for stub runners and exotic platforms.
* fix(relay): clamp disconnect drain grace to the runner's adapter-disconnect budget
Review finding (JoaoMarcos44, #82592): a fixed 5.0s drain in front of the
three 1.0s sequential teardown awaits gives an 8.0s worst case inside the
runner's 5.0s asyncio.wait_for(adapter.disconnect()) — tripping it cancels
teardown mid-drain, skips the fail-pending loop, and leaves outbound
callers blocked until _OUTBOUND_TIMEOUT_S (30s). The effective grace is
now budget - 3*TEARDOWN - margin (env-aware via the same
HERMES_GATEWAY_ADAPTER_DISCONNECT_TIMEOUT the runner reads), so the drain
can never push teardown past its caller's budget; a budget too small for
any drain disables it cleanly.
* test(gateway): pin the final-send suppression contract across a behaviour matrix
The gateway skips its own final send when the stream consumer claims the turn
final already reached the user. Every incident in that family — #71643 (stale
finalize snapshot), #78541 (payload-less multi-message split), #82656 (frozen
preview left with a visible cursor) — is the same failure: the consumer claimed
delivery for text the platform never rendered, so the corrective send was
suppressed and the answer was lost with no retry.
Each was fixed with a scenario test pinned to one branch of
GatewayStreamConsumer.run(). The got_done handler now has five sibling branches
that each set the suppression flags and record a turn-final payload, and nothing
checks them as a group: a new branch, or a new early `return True` in
_send_or_edit, can reintroduce the class without failing a test.
Pin the invariant instead of the branch — if the consumer offers the gateway any
signal it would trust, the complete final text must have reached the wire — and
assert it across {edit always / dies / never / lies} x {send always / never} x
{fresh-final on / off} x {clean / interrupted stream}.
The adapter records only frames that actually rendered, so an ACK the platform
drops does not count as delivery. 24 honest-transport scenarios hold the
invariant as a hard assertion. The 16 lying-transport scenarios are checked too;
the single combination that still violates it is reported as an expected
failure documenting the open exposure rather than asserting it away.
Refs #82656
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(gateway,relay): prime relay egress routing for synthetic injections + cap stale completion replay
Defect #4 from the 2026-08-09 staging incidents (upgrade-robustness):
after every gateway restart the durable async-delegation replay injected
completions correctly (post-741663cf1) but their replies bounced at the
connector — 'slack egress declined: target not routed to an onboarded
tenant'. The relay adapter re-attaches tenant discriminators
(metadata.scope_id / metadata.user_id) from per-chat caches warmed ONLY by
inbound traffic; synthetic turns race those cold caches on every deploy,
scale-to-zero wake, and crash recovery.
- relay adapter: prime_routing_cache() — feeds a synthetic event's
session-store origin through the same _capture_scope used for real
inbound (never raises).
- run.py injection path: prime the resolved adapter before handle_message
(duck-typed; native adapters unaffected).
- async_delegation: 48h staleness cap in restore_undelivered_completions —
a pending completion older than the cap is terminally dropped (payload
stays queryable) instead of re-run as a fresh full-context turn; the
post-restart replay of a July session burned a 102K-token context.
Also carried: JoaoMarcos44's suppression behaviour-matrix harness
(cherry-picked from #82676, authorship preserved) — 39 passed + 1 xfail
(the documented ACK-then-drop transport-honesty residue).
* test: use recent timestamps in restored-ownership fixtures
test_restore_stamps_restored_flag persisted its completion with epoch-era
toy timestamps (dispatched_at=1.0), which the new 48h replay staleness cap
correctly classifies as stale — the fixture then exercised the cap instead
of the restored-flag contract (CI slice 4 failure). Timestamps are now
now-relative; the staleness behavior itself is pinned separately in
test_relay_injection_egress_priming.py.
* fix(gateway,relay): close four review findings on the relay delivery fixes
Review follow-ups on this branch (NousResearch#82592):
1. HIGH — classifier/resolver mismatch (falsely-acknowledged loss).
_classify_completion_target now returns "deliver" for idle-ended
parents, but _resolve_async_delegation_session still dropped every
non-compression-ended pin: the durable row was acked at adapter
acceptance, then the injection died inside the pipeline with no
retry — strictly worse than the honest terminal drop on main, and
the delivery leg defect #2's fix depends on did not exist. The
resolver now retargets non-user-boundary ends (idle/timeout/
lifecycle) to the chat's current session — session_entry already IS
the routing key's current session for the same chat — while user
boundaries (session_reset / new_session / user_exit /
session_switch) stay fail-closed. Both sides share one module-level
_USER_BOUNDARY_END_REASONS so the verdict and the routing decision
cannot drift again; a coherence test asserts deliver-verdicts
resolve non-None across representative end reasons.
2. HIGH — drain clamp missed adapter-level spend. The effective drain
grace budgeted drain + 3x teardown, but RelayAdapter.disconnect
spends revocation-monitor teardown + go_idle time BEFORE the
transport drain inside the same runner wait_for; worst case still
blew the budget and cancelled teardown mid-drain (skipping the
fail-pending loop). The adapter now measures its own elapsed time
and threads the REMAINING budget into
transport.disconnect(budget_s=...); legacy/stub transports without
the keyword fall back to the no-arg signature.
3. P1 — _request_response racing disconnect() could register a future
after the fail-pending loop already ran, stranding the caller for
the full _OUTBOUND_TIMEOUT_S (30s). Fail fast with the same
"relay transport closed" error once _closing is set.
4. P1 — _build_process_event_source's last-resort reconstruction
dropped scope_id, so a scoped relay completion whose session-store
origin was unavailable primed no tenant discriminator and could
still bounce off the connector's fail-closed egress guard.
scope_id now threads through the reconstructed SessionSource, with
a warning when a scoped chat reconstructs without one.
All four: RED reproduced with the fix reverted, GREEN after; relay/
delegation delivery families pass (43 + 71 + 179 across the touched
suites); full tests/gateway run shows only failures already failing
identically on merge base 2446c8bb6 (env/dep issues).
* fix(gateway,relay): make pending-frame failure cancellation-safe; persist completion routing origin
Two remaining review findings on this branch (NousResearch#82592):
1. Cancellation could strand outbound waiters past the fail-pending
loop. transport.disconnect() failed pending futures only at the END
of the drain + three teardown awaits; a cancellation landing
mid-drain (the runner's wait_for budget, an outer cleanup deadline)
skipped the loop entirely and left registered futures unresolved —
their callers blocked until _OUTBOUND_TIMEOUT_S (30s). The budget
threading added earlier shrinks the window but is not a hard
guarantee. The fail-pending loop (and the going_idle ack failure)
now run in a `finally`, so no exit path — normal, error, or
cancelled — can leave a registered future unresolved. Idempotent:
done futures are skipped, a second disconnect() pass is a no-op.
2. Durable completions did not persist their routing origin, so the
scope_id threading in the fallback SessionSource reconstruction had
nothing to carry on the exact path it exists for (restart replay
with session store + source cache gone): the async-delegation event
producers never populated scope_id and the durable rows never
stored it. Dispatch now snapshots the originating turn's
scope_id/user_id/user_name from the session context
(_capture_routing_origin — a new HERMES_SESSION_SCOPE_ID contextvar
bound by the gateway at session-bind time alongside the existing
vars), stores them in the existing task_json payload (no schema
migration), and re-attaches them to all three completion-event
shapes (live single, live batch, crash-recovery rebuild). The
gateway's fallback reconstruction then primes both discriminators
after a restart.
Tests: cancellation mid-drain -> every pending future resolves with
"relay transport closed" (mutation: moving the loop out of the finally
goes RED); second-pass disconnect idempotence; end-to-end
dispatch -> owner-death recovery -> event carries scope_id -> fallback
SessionSource primes it (mutations: dropping the dispatch capture or
the task_json persistence both go RED); live completion event carries
the origin. 94 passed + 1 xfailed across the delivery/delegation
suites; tests/tools delegation family 73 passed (2 collection errors
pre-existing on merge base 2446c8bb6).
---------
Co-authored-by: joaomarcos <joaomarcosdias444@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Ben Barclay <ben@nousresearch.com>
evaluate_after_turn() calls judge_goal() which makes a synchronous
HTTP request to the auxiliary LLM. Running it on the event-loop
thread blocks Discord heartbeats for 10-40s, causing connection
flaps and gateway instability.
Offload to the default thread-pool executor so the event loop
stays responsive during evaluation.
Add a non-terminal "review" status so a worker that finished implementation
can hand off for human review without abusing kanban_block. The old
kanban_block(reason="review-required: ...") convention routed the handoff
through the unblock-loop breaker, so a normal review -> changes -> review
cycle was falsely escalated to triage.
- kanban_db: request_review (running/ready -> review, non-block, emits
review_requested), reopen_review_task (review -> ready/todo, review_reopened),
complete_task accepts review -> done, and a review_dispatch gate (default off,
shared by the dispatcher loop and the gateway health probe).
- kanban_request_review worker tool + `request-review` / `reopen-review` CLI
verbs; tool wired through toolsets, EXPOSED_TOOLS, _POLISHED_TOOLS.
- Gateway notifier wakes the origin subscriber on review_requested and
block_loop_detected; the subscription survives until done/archived, so every
review cycle re-notifies.
- Dashboard PATCH + bulk route the review transitions (request_review /
reopen_review_task) and render the review column.
- goals.py goal-loop and KANBAN_GUIDANCE recognize review as a terminator.
- Docs (reference tables, user guide, AGENTS.md, zh-Hans mirrors) + tests.
needs_input / failed are unchanged: they still route through kanban_block,
still count toward block_recurrences, and still escalate to triage.
The Hugging Face agent-harness registry matches standard-var values
EXACTLY against the harness id. Our registry id is 'hermes-agent'
(huggingface.js agent-harnesses.ts), so AI_AGENT=hermes was counted as
'unknown' — fixed at both entry points.
Remote terminal backends (Docker/SSH/Modal/Daytona/Singularity/Vercel)
never inherit the Hermes process env, and the cross-session leak guard
deliberately strips HERMES_SESSION_* from subprocess envs in engaged
multi-session hosts — so hf/huggingface_hub traffic from those shells was
unattributable. _wrap_command now exports AI_AGENT/HERMES_AGENT inside
every wrapped command with ${VAR:-default} semantics (outer harness is
never clobbered), and the snapshot dump excludes both names so a baked
value can never shadow a later outer harness.
E2E: verified against real huggingface_hub 1.27.0 detect_agent() with a
cached registry — 'hermes-agent' detected via AI_AGENT and via
HERMES_SESSION_ID; old 'hermes' value reproduced the 'unknown' bug.
CLI and gateway entry points now set AI_AGENT=hermes (the emerging
cross-agent standard read by e.g. huggingface_hub agent detection) and
HERMES_AGENT=true, via setdefault so an outer harness is never
clobbered.
Both session recovery paths (the startup stale-entry repoint and the
lazy in-message recovery) rebuilt the routing entry with updated_at=now
and never consulted _should_reset, so an opt-in idle/daily session_reset
policy was silently dead across any gateway restart: a recovered session
always looked freshly active, and since every subsequent message bumps
updated_at, a session recovered stale could then never age out at all.
Fix in three parts:
- _create_entry_from_recovered_row derives updated_at from the durable
last_activity_at the finder already returns on the row (no extra DB
round-trip; the original PR added SessionDB.get_last_activity for
this, unnecessary post-#82633), falling back to created_at. An
invalid or missing started_at now maps to epoch 0 instead of now — an
invalid durable timestamp must look old, never freshly active.
reset_had_activity is set from the row's durable activity/message
signals so the continuity hint stays accurate.
- _recover_session_from_db evaluates _should_reset on the rebuilt entry:
an overdue session is durably promoted to a reset boundary
(promote_to_session_reset, falling back to end_session) and the stale
mapping is dropped instead of repointed.
- _query_recoverable_session no longer reopens the row; the
get_or_create_session recovery phase evaluates _should_reset first and
either feeds the normal auto-reset create path (reset notice,
prev_session_id continuity, durable promotion) or reopens and
publishes the recovered entry exactly as before.
Behavior is unchanged under the default session_reset mode "none":
_should_reset returns None there, so recovery still resumes every
recoverable row — only users who opted into idle/daily resets see the
policy actually applied across restarts.
Cherry-picked from #78618 and adapted to the #82633 finder.
(cherry picked from commit 31c71f762961638c199287fc6ffe836115c4892b)
Complete the /branch routing-identity fix (salvaged from PR #62278 by
@jcjc81): in addition to user_id/session_key/chat_id/chat_type/thread_id,
forward origin_json and display_name at create_session() time, matching
the reset-path db_create_kwargs pattern (#82633) so the branch row is
born with full identity — no backfill gap for state.db consumers
(mcp_serve, mirror, channel directory) if a crash lands before
switch_session().
The obsolete compression-rotation half of #62278 was dropped: rotation
now goes exclusively through publish_compression_child, which already
copies all identity columns in-transaction.
The sweeper flagged two gaps in the routing-columns fix:
1. /branch create_session() omitted user_id and session_key — the
fallback lookup path (find_latest_gateway_session_for_peer) requires
user_id to match the complete peer tuple when session_key lookup fails,
and /resume IDOR guards reject sessions without matching user_id.
2. Compression-rotation create_session() omitted agent._user_id — same
problem: rotated child cannot satisfy persisted /resume ownership proof
before the later gateway backfill.
Forward user_id and session_key at CREATE time in both call sites so
the child row is immediately fully routable with zero backfill gap.
Extended tests: compression rotation asserts user_id is carried (and None
for CLI sessions). Branch routing asserts both user_id and session_key on
the child row before switch_session runs.
Same defect as the compression-rotation fix in the prior commit, found
during a full-audit of every create_session() call site per the repo's
'fix the whole bug class, sibling call paths included' contribution
guidance.
_handle_branch_command() (gateway/slash_commands.py) creates the branched
child session via create_session() without chat_id/chat_type/thread_id.
The routing columns are only backfilled later, when switch_session() runs
at the end of the function and calls _record_gateway_session_peer(). In
between, the function copies the parent's conversation history to the new
session_id one message at a time, with each append_message() call
independently try/excepted (best-effort) — a crash/kill anywhere in that
window leaves the branched session permanently unroutable, same failure
mode as the compression bug: NULL chat_id/thread_id can never be found by
find_latest_gateway_session_for_peer, AND unreachable via /resume's IDOR
guard (which requires the row's chat_id/thread_id to match the caller's).
Fix: forward source.chat_id/chat_type/thread_id at create_session() time,
mirroring the existing correct pattern already used by /title's
auto-create path a few hundred lines up in the same file (which has an
explicit IDOR-scoping comment justifying it).
Tests: tests/gateway/test_branch_routing_columns.py drives the real
_handle_branch_command against a real SessionStore + SessionDB (SQLite in
tmp_path, no DB/session-store mocks). Patches switch_session to simulate a
crash landing before it runs (the exact gap the routing columns need to
survive), then asserts the branched child's chat_id/chat_type/thread_id
are already correct in state.db at that point. RED verified against
unpatched code (assert None == '170829464'), GREEN after the fix.
Regression: 102/102 across the new test + pre-existing /branch, session
boundary, compression rotation, DM thread seeding, session API, and
resume-command suites. Broader tests/gateway/ -k "branch or session_api or
resume or topic_mode or session_boundary" sweep: 255/255 passed, 1
(unrelated) skip.
When the per-session pending transcript queue hits _MAX_PENDING_PER_SESSION
(200) while the session DB is broken, the gateway previously popped the
oldest message and discarded it permanently — silent user data loss during
live operation (#78182). The on-disk pending spool only ran at shutdown via
flush_pending_to_file.
Extend that existing spool machinery for runtime drops:
- gateway/shutdown_flush.py: add spool_dropped_transcript_message() and
drain_transcript_spool(), reusing _get_flush_dir/_write_payload (same
atomic-JSON pending_messages/ spool format). recover_pending_to_db()
now also replays transcript_cap_drop payloads left over across restarts.
- gateway/session.py: on cap eviction, spool the dropped message and log a
WARNING that includes the spool path; if spooling fails, degrade to the
previous drop-and-warn behavior. On the next fully successful transcript
flush for that session, drain and replay spooled messages in drop order;
replay failures keep the spool files for the next attempt.
- tests/gateway/test_pending_queue_spool.py: drop→spool→drain roundtrip,
per-session drain isolation, spool-failure degradation, replay-failure
retention, and spool primitive ordering/reason filtering.
No new config; extends existing flush_pending_to_file infrastructure per
AGENTS.md guidance.
Refs #82616, #78182
