`gateway/mirror.py` captured `sessions.json` under the home that was live at
import. Under the multiplexed gateway one process serves every profile, so the
pre-migration fallback in `_find_session_id()` resolved every profile's lookup
against the launch profile's index — a `platform`+`chat_id` pair present in both
indexes returns the launch profile's session id, mirroring a delivery into a
session that belongs to another profile.
Only the fallback is affected: the primary `state.db` query goes through a bare
`acquire()`, which follows `get_hermes_home()` at call time and is already
profile-correct. The access is read-only, so nothing was written to the wrong
profile.
Applies the resolve pattern the rest of the repo already uses (gateway/hooks.py,
gateway/sticker_cache.py, tools/process_registry.py, tools/checkpoint_manager.py):
keep the module constant as the monkeypatch surface, snapshot it at import, and
re-resolve only when unchanged.
The added test reloads the module under a launch home and then serves a lookup
for a second profile; without the fix it returns `sess_launch` instead of
`sess_active`.
Fixes#112844
`hermes profile delete` removes the profile directory and tears its runtime down, but the name is
also baked into durable identity the delete path never touches — `agent:<name>:*` routing keys,
`gateway_heartbeats.profile` and `delivery_obligations`. An inbound event on a chat keyed to the
dead name then enters the routing index, resolves a profile whose directory is gone, and logs
`Profile '<name>' does not exist` on every event for the life of the store (the #111926 flood,
reached from a *deleted* rather than a renamed profile). The delete side is now symmetric with the
rename rekey (`rekey_profile_state` / `rekey_profile_routing` / `migrate-profile-identity`), with
the same ownership rule:
- `SessionDB.purge_profile_state(name)` — the mirror of `rekey_profile_state`, in one
`_execute_write` transaction. Routing keys, heartbeat rows and the telegram topic rows the rekey
also owns are hard-deleted (a binding is matched by `profile_name` OR its `session_key`
namespace, because the rename rewrites both); `delivery_obligations` rows are terminalized
(`state='abandoned'`) rather than dropped, so pending delivery state is not lost silently.
- `SessionStore.purge_profile_routing(name)` — the mirror of `rekey_profile_routing`: drops the
in-memory entries and persists the drop. Mandatory, not belt-and-braces — the owning process
writes its in-memory copy back, so a durable delete made elsewhere is undone by its next save.
- A delete-only control verb `purge-profile-identity`, deliberately NOT inside
`_unserve_profile()`: that hook also unserves a rename's old name, whose identity the rekey still
has to migrate. `hermes profile delete` requires the owner's `{"ok": true}` answer and reports a
partial settlement (naming the retry) instead of a clean success.
- The retry is the new `hermes profile purge-identity <name>`. It refuses a name that is a live
profile again: the purge keys off the name alone, so `delete foo` (settlement pending) →
`create foo` → `purge-identity foo` would otherwise delete the NEW incarnation's identity. The
delete path tombstones the directory before it purges, so the guard never blocks the delete.
- `sessions` rows are not deleted by the purge: it settles identity, not history. What a delete
leaves of a profile's conversation record is `delete_profile`'s business — it removes the
profile's own home, `state.db` included.
Tests (`scripts/run_tests.sh`, red on base → green): `tests/hermes_state/test_purge_profile_state.py`,
`tests/gateway/test_purge_profile_routing.py`, `tests/gateway/test_profile_identity_purge.py`,
`tests/hermes_cli/test_profile_identity_purge_cmd.py` and `TestDeleteProfile` in
`tests/hermes_cli/test_profiles.py` — 95 passed, 0 failed across those five files.
A native hosted room running a second profile calls
tui_gateway.launch_profile_policy.activate_multi_profile_hosting() inside the
messaging gateway process, so get_secret() fails closed for every unscoped read
afterwards. A gateway with gateway.multiplex_profiles: false never bound a scope
(the config flag was the only gate), so its next ordinary turn died in
_resolve_session_agent_runtime with UnscopedSecretError / "Hermes could not read
this profile's API key" until restart (#112878).
Builds on the predicate from #112884: instead of re-entering the routed-profile
scope (which rebuilds credentials from .env alone and would drop a key injected
by systemd / `op run`), the standalone branch binds the launch profile's OWN
scope — launch_secret_scope() (.env + external sources over the env frozen at
activation) plus its terminal policy — exactly what the tui_gateway already
binds for launch-profile RPC bodies. One predicate, GatewayTurnMixin
._standalone_launch_scope(), no-op while the process is single-profile.
Whole class: every standalone entry point now runs under it — the foreground /
background turn wrappers, busy, goals and heartbeat-restore paths already
routed through _profile_scope_for_source, and the primary adapter's message,
busy-session and platform-event handlers (slash commands such as /model,
/status and /compress run inside _handle_message and were equally stranded).
Cron already binds its own per-fire scope. The guard is not weakened: reads
outside any scope still fail closed and a secondary never sees the launch env.
Tests: tests/gateway/test_standalone_gateway_launch_scope.py — real activation
helper, real server-bound secondary scope entered and left, then the standalone
turn and handler resolve both the .env key and the env-injected key (red on
origin/main with UnscopedSecretError); control: no activation → nullcontext and
no scope in the handler. The #112884 test moves here in trimmed form.
Co-authored-by: Lyti4 <205342405+Lyti4@users.noreply.github.com>
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
* fix(launchd): park EX_CONFIG token conflicts instead of KeepAlive-looping
systemd already stops restarting on exit 78; launchd KeepAlive=true
respawned the same token/port collision every 30s. Map 78 to a clean
stop under SuccessfulExit=false so the job stays down until the holder
releases the lock.
* test(launchd): pin EX_CONFIG park under SuccessfulExit KeepAlive
The plist must not use unconditional KeepAlive, and the stderr wrapper
must turn gateway exit 78 into a clean stop without swallowing exit 75
or a non-gateway child's 78.
The boot guard logged its refusal once and nothing else knew. A user whose default says multiplex
but whose gateway serves one profile would read `hermes gateway status` and see a healthy gateway.
The verdict now lands in gateway_state.json (multiplex_standalone_reason) and status prints it with
both remedies; a boot that does multiplex clears it.
DEFAULT_CONFIG now ships gateway.multiplex_profiles: true. GatewayConfig keeps an UNSET flag as
None so the boot can tell "the operator chose" from "the default applies"; every reader tests
truthiness, so an undecided flag never multiplexes by accident.
hermes_cli/gateway_multiplex_mode.py settles the unset default once per boot (called from
load_gateway_config_for_runner and `gateway run --config`): the same preflight `hermes gateway
migrate --multiplex` runs — default profile, >= 2 profiles, no secondary running its own gateway
(live pid or installed unit), no duplicate-credential / port-binder blocker, migratable host. A
refusal is a logged warning naming the blocker and the migrate one-liner; the gateway comes up
standalone exactly as before. Explicit values (config.yaml, GATEWAY_MULTIPLEX_PROFILES) pass through
verbatim; `--standalone` already pins false.
Other processes stop guessing the verdict from the merged default: named_profile_served_by_running
_multiplexer, the enroll warning, the dashboard listener guard, the cron-fire port resolver, container
boot and the migration plan (_read_multiplex_flag) read the live gateway's served_profiles record
first and the EXPLICIT flag second — so a per-profile fleet with the flag unset still reads as
"not yet multiplexed" and the fold proceeds.
Docs: multi-profile-gateways.md, multiplexing-gateway.md, hermes_cli/AGENTS.md.
gateway/run_notifications.py::_send_session_db_warning_notifications already
computed profile_arg but left `hermes doctor` (fts_index branch) and
`hermes gateway restart` (default branch) bare; agent/turn_finalizer.py had a
bare `hermes doctor` in the error fallback used when the explainer produced
no text. Same class as the previous commit. Also reflows the explainer
strings so continuation lines break at clause boundaries.
Same bug class as the warm-up: `_start_free_tier_bootstrap` ran on a bare executor thread, and
`anon_auth.ensure_portal_identity` resolves the Portal through `_nous_portal_env_override`, so
under multiplex a non-production HERMES_PORTAL_BASE_URL read as absent and the identity would be
minted against the production Portal. Route the hop through `_run_boot_probe_in_launch_scope`.
Gated behind guest onboarding, so latent on today's fleet; one test pins the binding.
Hoist the "enter the launch profile's scope under multiplex, hop to the executor with
copy_context" into GatewayStartupMixin._run_boot_probe_in_launch_scope so the warm-up has one
call site instead of two executor branches, and the sibling boot probes can share it.
_scoped_operator_override now takes the candidate names: _nous_portal_env_override used to call
it twice (HERMES_PORTAL_BASE_URL, then the NOUS_PORTAL_BASE_URL alias), so one lost-scope event
logged two identical WARNINGs; one try around the list logs once and names both.
Tests: drop the dead UnscopedSecretError arm in the probe (the helper swallows it), fold the
no-scope-left-behind assertions into the multiplex test, keep the environ-semantics contract for
single-profile. Docstrings keep the WHY; the incident chain lives in the PR.
Scope entry that cannot be built falls back to the unscoped executor call (the
load_gateway_config_for_runner shape) instead of aborting startup. The no-scope-left-behind
assertions run in the warm-up's own task — asyncio.run copies the context, so asserting from the
caller proved nothing.
_warm_turn_prerequisites ran _warm_turn_machinery_sync on a bare executor thread with no
profile scope. get_tool_definitions runs every check_fn; the vision probe resolves live Nous
runtime credentials (resolve_vision_provider_client -> _resolve_nous_runtime_api ->
resolve_nous_runtime_credentials -> _nous_effective_routing). Under GATEWAY_MULTIPLEX_PROFILES
get_secret fails closed there, so HERMES_PORTAL_BASE_URL / NOUS_INFERENCE_BASE_URL read as
absent, the Portal allowlist heals to production, and a staging deployment's refresh token is
POSTed to portal.nousresearch.com: invalid_grant, "Nous OAuth state quarantined", ~10 s after
every boot and before any inbound turn. Reproduced live on hermes-agent-stg-gg-probe-test-0062
with the override present in the profile .env (four NAS re-seeds, four boot-time deaths); the
unscoped call stack was captured on the box.
The warm-up now enters the launch profile's _profile_runtime_scope and carries the context
into the executor thread with copy_context().run — the same shape _discover_gateway_mcp_tools
uses (#95518). Single-profile gateways are unchanged.
_scoped_operator_override logs which override was unreadable and why; the downstream
"ignoring invalid portal_base_url" warning never named the caller that lost its scope.
Tests: scoped warm-up resolves the .env override on the executor thread (red on main), no
scope leaks past the warm-up, single-profile keeps environ semantics.
A long-lived serve process keeps a deleted profile as the context home of threads
that outlive the delete. A bare `mkdir(parents=True)` right before an atomic write
brings `profiles/<name>/` back after `hermes profile delete` has written the
tombstone and removed the tree.
The writers in `utils` and the seven callers named in #112592 are guarded by the
preceding commits; this one applies the same `mkdir_under_hermes_home` idiom to the
other pre-write directory creations found by the same mechanical rule (auth,
personality, plugin catalog, skills sync, tool discovery cache, platform adapters,
memory plugins, local runtime supervisor, process identity, breadcrumbs). The two
sites that pass `mode=` keep their mkdir behind `assert_named_profile_home_live`.
The guard is a no-op unless the target has a provable `profiles/<name>` ancestor.
Salvaged from #112596 (30-file sweep) on top of #112594 / #112601; the overlapping
files were resolved to the already-landed versions.
- hermes_cli/profiles.py was already past the 2,000-line gate; the rename identity
migration (migrate_profile_identity, _migrate_profile_identity, control-answer helpers)
now lives in hermes_cli/profile_identity.py, imported late from rename_profile and the
profile subcommand.
- The migrate-profile-identity control-verb handler moves out of gateway/run.py into
gateway/run_profile_reconcile.py (migrate_profile_identity_verb), beside the other
hot-serve control-verb logic.
- tests/utils/ is not a mirrored source dir: the atomic-writer tests move to
tests/test_utils_atomic_writers_deleted_profile.py (root-module test placement).
- Drop duplicate no-op/idempotent/raw-answer tests so each fix carries invariant tests only.
Behaviour unchanged; authored commits from @xielevi, @KoNit-K and @kokhlo are preserved.
Background writers that still carry a tombstoned profile as their Hermes
home (reasoning-caps warm thread, models cache, models.dev ETag, gateway
lifecycle ledger, MCP OAuth tokens, memory store) re-created
profiles/<name>/ with a bare mkdir right before an atomic write. Route the
parent-dir creation through mkdir_under_hermes_home so a deleted named
profile raises FileNotFoundError and stays gone, matching the tombstone
contract already enforced for logging and state.
Renaming a profile moved profiles/<old>/ to profiles/<new>/, so the row DATA
travelled with the directory, but the profile name is also baked into
keys/values the move left untouched: session keys (agent:<old>:* namespace),
sessions.profile_name (fail-closed owner ladder / Desktop sidebar scope /
@session: deep links), sessions.origin_json.profile,
gateway_heartbeats.profile, delivery_obligations (session_key +
adapter_profile), telegram_dm_topic_* profile_name bindings, and the
gateway_routing index. Left stale, every inbound event on a chat keyed to the
old name resolved to a profile that no longer exists — flooding errors.log
with "Profile <old> does not exist ... falling back to global HERMES_HOME"
every few seconds — and renamed sessions dropped out of the sidebar / broke
their deep links.
The routing index is held in memory by a live multiplexer and written back
periodically, so a CLI-side DB rewrite alone is clobbered. Fix in layers:
- SessionDB.rekey_profile_state: atomic durable rewrite of the state.db
tables, matching the agent:<name>: namespace by exact prefix (substr, not
LIKE — '_' is a legal profile-name character and a LIKE wildcard), rewriting
the profile inside routing/origin JSON, and REFUSING on a target collision
(routing rows or telegram bindings) instead of silently merging.
- SessionStore.rekey_profile_routing: rekey the in-memory routing index
(keys + origin.profile) then persist — the half a DB write cannot reach.
Raises on a target-key collision before mutating.
- Control verb migrate-profile-identity (params-carrying; the socket passes
params only to handlers that declare them, bare handlers unchanged) so a
live gateway rekeys its in-memory copy AND both durable stores (routing home
+ the renamed profile's own state.db).
- rename_profile calls the verb when a multiplexer is live and, if it fails,
does NOT fall back to a racing CLI-side write: it prints a warning telling
the operator to restart the gateway and retry. With no live gateway it
performs the durable rewrite itself (safe: nothing else holds the store
open).
Checkpoints keyed by the profile's workdir path are a known related gap,
tracked separately, not addressed here.
Tests: rekey_profile_state (all tables, routing/origin JSON, collisions,
idempotent, no-op), rekey_profile_routing (namespace + origin, no-op, no
overwrite), control verb param passing, and rename end-to-end for both the
live-gateway (delegates, refuses unsafe fallback) and no-gateway (durable
rewrite) paths.
`POST /api/sessions/{id}/chat/stream` mints a run_id and writes run status,
but its terminal write omitted `output=` — the reply text went only onto the
SSE queue. `POST /v1/runs` has always recorded it.
The asymmetry costs a caller its answer. A client whose socket dies mid-turn —
a sleeping laptop, a dropped WiFi link, a peer DM over a flaky LAN — sees the
run reach "completed" through GET /v1/runs/{run_id} and has no way to learn
what the agent said. Worse, the disconnect path interrupts the agent, which
unwinds and *returns* a partial result, so that partial answer is recorded as
a clean "completed" and then discarded: indistinguishable from a run that
produced nothing, and equally unrecoverable.
Record `output` on this route too. Terminal statuses are already retained for
_RUN_STATUS_TTL (3600s), so the existing GET /v1/runs/{run_id} becomes a
recovery path for any client that loses its stream, without a new endpoint,
without touching the run registries, and with no change to the streaming
contract.
Deliberately nothing else: the detached turn stays out of _active_run_tasks
(it is already counted via _inflight_agent_runs, and a task entry would
double-count it in the shutdown drain), and the run stays out of
_run_streams_created, so the orphan sweeper's 300s reap still cannot see it.
Tests: a disconnect mid-stream now leaves the reply readable both in the run
record and through _handle_get_run; a structural guard asserts BOTH routes
still pass output=, anchored on the status write rather than the SSE payload's
"completed": True key, so the asymmetry cannot quietly return.
Review finding on #112260: the hosted-room member relabel regex was a third
hand-copied opener list that missed the frames context_compressor and
title_generator already treat as harness input, so a member reply starting
with "[System: ..." or "[IMPORTANT: 2 background processes ..." reached peers
in its exact trusted shape. agent.prompt_builder.CONTROL_FRAME_OPENERS is now
the single source; the gateway regex is built from it and the desktop TS
literal mirrors it byte-for-byte.
Follow-up to the two cherry-picked contributor commits (#111571 gateway, #111576 Desktop),
which neutralized the same class with two different mechanisms: an invisible U+200B after
the `[` on the gateway path and a phrase replacement ("RESERVED CONTROL MARKER NEUTRALIZED")
on the Desktop path.
Both room-transcript builders now share one frame set (the mid-turn steer marker open/close,
the compaction handoff, runtime/system notes, planning-state and async-delegation frames —
the same openers agent/title_generator and agent/context_compressor already classify as
harness-authored, case-insensitive) and one visible relabel: the opener `[` becomes
`[member-quoted `. The peer still reads what the member wrote, but the exact trusted shape
the system prompt tells the model to honour is gone and the label says who authored it.
A zero-width space is invisible to a human reading the transcript and easy for a model to
skip over; the visible label is not.
Genuine user lines, stored room events and the displayed message are untouched on both
surfaces (probed live: stored member event byte-identical, `User (user):` line verbatim).
Tests trimmed to one invariant per surface, built from agent.prompt_builder's real marker
constants on the Python side.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Co-authored-by: Hukla <129692708+huklaa@users.noreply.github.com>
_resolve_hermes_bin preferred `shutil.which("hermes")` over the running
interpreter's module argv. On Windows a hermes.exe planted earlier in PATH is
therefore the argv /update and /restart re-exec, hijacking the update process
(#111569). Flip the order: python -m hermes_cli.main (exactly this install)
wins whenever hermes_cli is importable; PATH stays as the fallback, then None.
`terminal(background=true, notify_on_complete=true)` appended its watcher descriptor to
`process_registry.pending_watchers`, which only the post-turn hooks drain. A process that
finished while the turn that launched it was still running (an agent sleep-polling for
hours) had no watcher task at all: the completion_queue entry sat inert, nothing was
injected, and the chat stayed mute until that turn ended (#112033).
- `_register_completion_watcher` arms the watcher on the live gateway loop at registration
(`GatewayRunner.arm_process_watcher`, via the existing `_gateway_runner_ref` /
`_gateway_loop` seam that send_message and cron already use); `pending_watchers` stays
the fallback while the gateway is not serving (checkpoint recovery at startup, shutdown).
- The agent-notify branch of `_run_process_watcher` keeps its design (the agent's next turn
is the user-facing report) but, when the launching turn is still active at process exit,
the injection only queues a follow-up — so the concise receipt is sent to the chat right
away instead of never. The busy check is taken before injection because the injected turn
itself installs the adapter's session guard.
Live probe (real process, real GatewayRunner loop, fake telegram adapter, busy session):
before — pending_watchers=1 after exit, 0 watcher tasks, 0 injections, 0 receipts;
after — pending_watchers=0, watcher task armed at launch, 1 injection, 1 concise receipt.
Control (idle session): 1 injection, 0 receipts, unchanged.
Slimmer redo of #112038 by @KoNit-K: same two gaps closed, without a second scheduler
registry / loop attribute on ProcessRegistry and GatewayRunner.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
A route without its own base_url (persisted route / SessionDB row / plain config) no longer
replaces the _resolve_runtime_agent_kwargs read in _resolve_gateway_model_context, so the
custom endpoint is still probed and the matching model.context_length pin survives; only a
/model switch carrying its own endpoint bypasses the default runtime read.
Two diagnostics diverged after a session-only `/model` switch (#111436).
/status: `_status_model_route` only took `context_total` from a live/cached
compressor or the raw `model.context_length` pin, so between turns (no
compressor yet) it fell to the occupancy-only line ("Context: ~79,455
tokens") while /context resolved the 1M window for the same session. /status
now runs the same resolver /context uses (`_resolve_gateway_model_context`,
off the event loop — it can probe /models), fed the WINNING route's
provider/base_url/api_key so the lookup targets the endpoint that serves the
displayed model, never a losing route's endpoint. The raw config pin moves
into the resolver, which already drops it when the route no longer matches
the configured one — a session switch must not inherit the default model's
pin. A window the resolver merely invented (unknown model →
DEFAULT_FALLBACK_CONTEXT) is grounded via a catalog match: `context_source`
is "default" only when no catalog entry matches, and /status keeps the honest
occupancy-only line for that case (catalog-listed 256K models still display).
Validator: `_validate_anthropic_messages` used one soft-accept message for
both "listing unreachable" and "listing answered 200 but lacks the slug", so
a reachable endpoint was described as one that "does not implement GET
/v1/models". The two cases now get distinct wording; the reachable case
matches case-insensitively and surfaces alias candidates at similarity 0.4
(`kimi-k3` vs `k3` ≈ 0.44 sits below the default 0.5 cutoff).
Slimmer redo of #111458 by @KoNit-K, which resolved only the override route
(not persisted/DB routes) and displayed the fallback window unconditionally.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Slim the salvaged #112111 mechanism (issue #112109) while keeping its
behaviour: the boot pass and the reconnect hook both run
_replay_pending_planned_restart_notification, which sends to every home
channel still owed an online notice, records delivered targets in
.restart_pending.json and unlinks the marker only when every owed target
(configured home with gateway_restart_notification=true) has been reached.
Dropped from the contributor diff:
- the per-target on_delivered checkpoint callback and pending_targets
field: delivered targets are written once after the pass. Residual is a
benign duplicate notice only if the process dies mid send-loop.
- getattr-based lazy lock -> class attribute default, same idiom as
run_profile_reconcile._reconcile_lock.
- _clear_planned_restart_notification in gateway/run.py: no production
caller remained; the roundtrip test unlinks the path directly.
- tests trimmed to two invariants: offline-at-boot is replayed once on
reconnect (with live-at-boot control), and partial delivery is persisted
so a fresh process does not re-notify and an opted-out home never keeps
the marker alive.
Live probe (temp HERMES_HOME, Discord home, adapter absent at boot then
reconnected): base consumed the marker with 0 sends; fixed head retains it
and sends the online notice exactly once on reconnect, then clears it.
A /handoff into a Telegram forum supergroup created a topic and bound the CLI session under
telegram🧵<chat>:<topic>, while the Telegram adapter keys every topic reply
telegram:group:<chat>:<topic> — the same restart-orphaning shape as the Slack case. Non-private
Telegram homes now use chat_type group; private-chat DM topics are unchanged.
`/handoff slack` bound the CLI session under `slack🧵<channel>:<ts>` while every inbound
reply in that thread resolves (via the Slack adapter's source shape) to
`slack:dm|group:<team>:<channel>:<ts>`. After a gateway restart the reply key found no
binding and the gateway opened a fresh empty session, orphaning the handed-off one (#111896).
The handoff destination now mirrors the adapter: chat_type `dm` for a D… home channel, else
`group`, plus the workspace scope_id (home channel provenance, falling back to the adapter's
channel→team map). Channel handoffs were equally affected (`thread` vs `group`), so the fix
covers both, not only DMs. Discord and Telegram destinations are unchanged.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
The installer drops uv in $HERMES_HOME/bin without exporting it, so the
bare `uv pip install --python ...` tip failed with `uv: command not found`
for installer-only users. The four copies of the tip (QQ Bot, Feishu, WeCom,
managed Telegram bot) now render through one helper, managed_uv.pip_install_hint,
which names the managed binary when present and falls back to `uv` otherwise.
The standard Hermes install is a `uv venv`, which ships no `pip` module:
`<venv>/bin/python -m pip install qrcode` fails with "No module named pip"
(the exact console output in #111695). Switch all four QR-fallback tips
(Feishu, WeCom, QQ onboarding, Telegram managed bot) to
`uv pip install --python <sys.executable> qrcode`, the form the in-tree
plugin install hints already use (hindsight, mem0), so the printed command
works as-is and still targets the active profile's interpreter.
Adds the Feishu-surface invariant test from #111696 and tightens the
Telegram test to the working command form.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
The Feishu, WeCom, QQ onboarding and Telegram managed-bot flows printed a
hard-coded 'pip install qrcode' tip when the qrcode package was missing. In
Hermes' isolated venv the bare pip either doesn't exist or targets an
unrelated system Python. Print '{sys.executable} -m pip install qrcode'
instead, matching the existing codebase convention for install hints.
Fixes#111695
Slims the salvaged preview helper (drop the try/except around json.dumps —
default=str cannot raise on tool results) and documents the tool.completed
SSE shape. Adds the control test that the multimodal envelope dict is still
classified as a success, so the dict passthrough only widens the failure
detection to real structured results.
The idea of carrying the tool result on tool.completed for /v1/runs
consumers was first proposed in #22362; that PR's executor half
(result=function_result) is already on main, and its wire half is landed
here in redacted, bounded form instead of the raw payload.
Salvages #111821 (@KoNit-K), part of #111815.
Co-authored-by: kidrauhl123 <105764349+kidrauhl123@users.noreply.github.com>
`hermes kanban dispatch` (plain output), the standalone daemon's stuck warning
and the gateway's embedded dispatcher stuck warning all reported a bare
`Spawned: 0` / "0 workers spawned" while the respawn guard held every ready
card — the reason existed only as a `respawn_guarded` task event visible via
`hermes kanban tail`. Operators watching the gateway health warning for 73+
ticks (#111910) had nothing to act on.
- `kanban_db_dispatch.describe_suppression()` renders the guard reasons per
task plus rate_limited / skipped_locked / memory_pressure for one or more
DispatchResults, so the CLI daemon and gateway warnings share one wording:
`Last tick held back: active_pr=1, memory_pressure=elevated.`
- plain `dispatch` output prints `Guarded (<reason>): <task id>` and the
tick-level holds, mirroring the JSON fields.
- kanban docs: how to see why a ready card is not spawning.
Co-authored-by: Steven Saehrig <trac3r726@users.noreply.github.com>
Part of #111910
An interrupted turn left `finalize_turn` with a diagnostic `final_response`
("Operation interrupted: waiting for model response") and no failure, so the
result said `completed=True` — the only producer that did; `turn_recovery`
and `codex_runtime` already return `completed=False` for an interrupt and the
gateway stream gate documents that contract. `completed` now also requires
`not interrupted`.
The API server then hard-coded the terminal status: the session chat stream
emitted `assistant.completed {completed: true, interrupted: false}` and
`run.completed` for every turn that did not raise, and `/v1/runs` booked any
non-`failed` result as `completed` — including an interrupt that did not come
through `/stop` and a turn that ran out of iteration budget. Automation that
reads the run status or the terminal event saw unfinished work as delivered,
and `partial: true` could sit next to `completed: true` in one payload.
`api_server_runs.terminal_run_status()` is now the single mapping for both
surfaces: interrupted -> `cancelled`, failed/partial/`completed=False` ->
`failed` (with `turn_exit_reason` and the fallback text as `output`),
otherwise `completed`; the terminal event is always `run.<status>` and a
late `pending_steer` rides on every terminal status instead of only on
`completed`.
CLI exit codes (`-q` quiet mode, `-z` one-shot) are deliberately unchanged
here: `hermes -z` returning 0 whenever text was produced was a stated design
choice (093f567f0d) and scripts depend on it, so that flip needs a
maintainer decision.
Fixes the gateway/producer half of #111770; slimmer redo of #111785 by
@KoNit-K (same mapping idea, one helper instead of three ladders).
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Salvage follow-up to the cherry-picked #112055 hunk (@Finn763):
- tests: fold the landed-commit control into the turn-hold-miss test and drop the
two extra end-to-end tests (sub-limit identity is already pinned by the pure
contract test) so the fix carries exactly two invariant tests.
- gateway/run_turn.py::bound_model_input_without_hygiene: drop the isinstance /
limit<=0 defences — the caller always passes the transcript list and a
_knob()-validated positive int, so the checks could never fire.
- docs: configuration.md now says hygiene_hard_message_limit also bounds the
model payload on any turn where hygiene did not land (payload-only, disk
untouched, landed summaries adopted as-is).
Co-authored-by: Noor-Sol <187156657+Noor-Sol@users.noreply.github.com>
Co-authored-by: Kevin Rajan <7121943+kvnloo@users.noreply.github.com>
A long-lived gateway DM rots because a turn where hygiene did not commit
(turn-hold expiry, idle/ceiling timeout, cooldown, or one already in flight)
fed the model the FULL uncompressed transcript — watermark-fenced summaries
can land late, but a turn with none had no fail-closed bound at all.
bound_model_input_without_hygiene() keeps the leading system/session_meta
setup rows plus the newest tail, total <= hygiene_hard_message_limit, and
never starts the kept tail on an orphaned tool result. It is deterministic
and payload-only: the stored transcript is untouched, so the agent's durable
prefix slice (history_offset) and the disk rows are unchanged, and the
hygiene compressor still sees the full transcript.
Applied at the three exits of _hmwa_run_session_hygiene where hygiene did
not land. A landed compression publishes a NEW list on attempt.history, so
object identity (`attempt.history is history`) decides — an adopted summary
is passed through byte-identical, and below the limit the bound returns the
same list object (no copy, no behaviour change).
The durable-run events stream (`GET /v1/runs/{id}/events`) hardcoded its own 30 s idle
keepalive, so it still fell outside the ~20 s idle deadline common to remote
OpenAI-compatible clients even after CHAT_COMPLETIONS_SSE_KEEPALIVE_SECONDS dropped to
10 s. Point it at the shared constant so every API-server SSE writer (chat completions,
Responses, native session stream, run events) keeps the wire alive on the same cadence.
Rewrite the salvaged regression test as two invariants: an idle OpenAI stream writes a
`: keepalive` comment inside the remote-client window (module-local fake clock, no global
`time.monotonic`/`asyncio.wait_for` patching, which would also steer asyncio), and the run
events stream follows the shared constant end-to-end through aiohttp. Both are red on
origin/main.
Root AGENTS.md § Code Shape Rules replaces "module-level constants are fine — they cache after
_apply_profile_override() sets HERMES_HOME" (true for `hermes -p x <cmd>`, inverted under the
multiplex gateway and the Desktop/dashboard `serve` backend, where os.environ holds the LAUNCH
profile) with the invariant: a profile = home + secret scope + terminal scope, bound per profile
ACTIVITY, and every execution point with no turn on the stack binds it explicitly. Names the real
seams: gateway/run.py::_profile_runtime_scope, tui_gateway @_profile_scoped +
_session_profile_runtime_scope (+ _profile_runtime_scope_tokens, launch_profile_policy ->
set_multiplex_active), cron/scheduler_provider.py::_profile_cron_scope,
gateway/run_agent_cache.py::_run_release_in_profile_scope, tools/environments/local.py::
served_profile_child_env, agent/memory_provider.py::spawn_context_thread. Adds a routing-table row
for profiles / multiplex / secret scope.
Area AGENTS.md paragraphs, one per seam, for gateway/ (activity-not-turn binding, hooks per
profile, adapter YAML never reaches os.environ, unserved shared-ingress reported via
_note_unserved_secondary_platform + needs_attention at the single writer), tui_gateway/ (RPC
binding is home AND secret AND terminal; HOME-only is half-bound; teardown chokepoint), cron/
(per-home tick lock, ticker scope incl. pre-loop code, kanban notifier routing, worker liveness by
(pid, worker_started_at) fingerprint, descendant fence as a path), hermes_cli/ (DEFAULT_CONFIG
key <-> reader parity, service-install matrix, -p vs multiplex home binding), tools/ (check_fn
reads through get_secret and is cached per hermes_home_key, one env builder per spawn, MCP trust
per profile), plugins/ (lifecycle hooks are bound by the caller; never cache the home from
initialize()), apps/desktop/src/ (pooled serve per (connection, profile); remote topologies),
agent/ (end-of-session flush is caller-bound; set_multiplex_active gates fail-closed).
Corrects the statements the multiplex model made wrong, in the same PR: root module-constant
sentence; hermes_cli "sets HERMES_HOME before any import" (+ cli-internals.md);
ADDING_A_PLATFORM.md §2 raw os.getenv loader (now an _ENV_STEPS row through config.py::_getenv)
and §4 platform_env_map in gateway/run.py (now _PLATFORM_ALLOWLIST_ENV in pairing.py + registry
allowed_users_env); platform_registry.py "may set os.environ (guard with not os.getenv)";
cron/AGENTS.md hardcoded ~/.hermes/cron/.tick.lock; gateway-internals.md agent:main as THE key
format, ~/.hermes/hooks/, single-profile `gateway stop`, plus a new "Multiplexed profiles"
section; tools/AGENTS.md os.getenv check_fn sample; "installed per turn" wording; "one temp
HERMES_HOME" E2E wording; multi-profile-gateways.md intro lists system units, Windows tasks, s6
and the Desktop backend.
Symptom: `hermes update` sat for ~40s after "Refreshing cua-driver" and ended
with "Fleet version check returned no rows" (exit 1); the restarted gateway
took 26s to reach "Starting Hermes Gateway" instead of the usual 3s. The
gateway constructor was running `maybe_auto_prune_checkpoints` synchronously,
before the control socket, adapters and the code_sha stamp, and on a 1.2 GB
store its `git gc --prune=now` (a full repack) takes 20-28s — twice, because
the size-cap shrink gc'd again even when it could drop nothing.
The same defect sat on the tool-call path: `CheckpointManager._take` ran
`_enforce_size_cap`, whose `_shrink_store_to_cap` returned True without
dropping anything and triggered a 20-28s gc on the first file-mutating tool
call of every turn once the store was over the cap. That loop also re-measured
a pack size that cannot move without a gc, so a single over-cap checkpoint
dropped 20 rounds of history and flattened every project to one snapshot.
- `_take` never gcs: `_prune` and `_enforce_size_cap` rewrite refs (cheap),
drop at most one snapshot round, and mark the store `.gc-pending`.
- `prune_checkpoints` gcs only when a ref moved (project deleted, or the
pending marker), and its cap loop is drop -> gc -> re-measure.
- `maybe_auto_prune_checkpoints` claims the interval marker before the run
so a failing prune costs one day, not a gc per housekeeping tick.
- `auto_prune_from_config` is the one config-driven entry point; the gateway
calls it from the housekeeping tick (last chore), the CLI from a daemon
thread. Nothing on either startup path waits for git.
Live A/B on a copy of a real 1.2 GB / 224-ref store: checkpoint 20.5s ->
1.2-1.6s (0 inline gc); the single repack (19.6s) now runs in the prune.
The gateway's idle-command path resolved skill slash commands inline on the
loop: a cold skill scan, skill file loads and the unavailable-skill rglob over
every skills dir. On a 1.5k-skill install that held the loop ~2 minutes, the
loop-liveness watchdog fired and the gateway exited mid-session (#111091).
_hm_skill_slash_rewrite now runs through _run_in_executor_with_context so the
profile contextvars the scan is scoped to survive the hop. Known commands still
short-circuit before any I/O (previous commit).
Same class in the Telegram inline picker: build_inline_results rebuilds the
command/skill catalog per keystroke via _collect_gateway_skill_entries, the
same path-resolution pass #110707 traced in the command menu.
One invariant test: a known command triggers no scan; an unknown command's scan
runs while the loop keeps ticking. Red on origin/main and on the reorder-only
tree, green here.
Reorder _hm_unknown_slash_reply so the I/O-free GATEWAY_KNOWN_COMMANDS
membership test short-circuits before _check_unavailable_skill walks the
skills tree (#111091). Taken from #111099 by @KoNit-K; the index cache half of
that PR is superseded by running the scan off the loop.
The cherry-picked commit extends the config/profile/MCP/managed-scope/completer/
OAuth/skills-manifest signatures. This commit finishes the class and trims it:
- `file_signature()` lives in `utils.py` next to the other stat/metadata helpers
instead of `hermes_cli.managed_scope` (gateway/ and agent/ callers no longer
reach into the managed-scope module for a generic stat helper).
- `hermes_cli/config_effective.py` was left comparing 2-/4-wide prefixes against
the widened `_RAW_CONFIG_CACHE` / `_load_config_cache_sig` records, so
`load_user_config_effective()` re-parsed on every call (3 parses for 3 calls on
an unchanged file, 1 before); index by the new widths.
- Sibling caches keyed on the same (mtime, size) shape and reading the SAME files
now use the helper: `load_env()` memo, `agent/skill_utils` raw-config and
external-dirs caches, `hermes_cli/model_switch` alias identity, `agent/moa_loop`
preset stamp, `hermes_cli/auth` global auth-store memo.
- Tests trimmed to one invariant each (pinned-mtime replacement invalidates; an
unchanged file still hits), both red on origin/main.
Left alone on purpose: `tools/registry.py`, `tools/skills_tool_dedup.py`,
`gateway/status.py`, `hermes_cli/banner.py`, `hermes_cli/main.py`,
`hermes_cli/session_recovery.py` — those fingerprint source files, PID/lock files
or write to persisted on-disk caches shared across processes, where an inode/ctime
key would churn on every checkout/copy rather than catch a replaced config.
Config caches, the profile re-scan watcher, the MCP reconciler, managed-scope
reads, the completer memo, OAuth heal marks, and the skill-manifest snapshot
keyed change detection on (st_mtime_ns, st_size) only, so a replacement that
preserved both (cp -p, rsync -t, timestamp-pinning scripts, sync clients) was
treated as unchanged and stale values were served until restart.
Add st_ino (fresh inode on atomic replace) and st_ctime_ns (cannot be
backdated via os.utime) to every signature via a shared
hermes_cli.managed_scope.file_signature() helper.
Fixes#111105
