One statement of the ownership rule (module doc) instead of three; the
lock-refusal docstring keeps only why --replace is not offered. The
record-less-holder note names the real cause (a holder this process cannot
interrogate), and the start-beside INFO line and the lock refusal take the
migrate command from MIGRATE_COMMAND / _migrate_command() like their
siblings.
decide() runs twice per start (the CLI guard in hermes_cli.gateway and again
from start_gateway -> _host_attach_or_none), and the lock-losing path in
_claim_host_gateway_role re-derives the same fact via _owner_is_standalone(),
so every boot of a generated unit beside another profile's standalone gateway
logged the "starting beside it" WARNING three times. decide() is a verdict
function; reporting belongs at the action site. Demote host_attach's copy to
INFO and keep run.py's lock-claim WARNING, which fires exactly once on both the
--replace and plain paths and carries the `gateway migrate --multiplex` hint.
The lifecycle test still asserts the converge hint is logged by decide(); it
now captures at INFO.
decide() grants REPLACE_HOST for an owner whose served set is unknown, but
_replace_target_belongs_to_other_profile can then only prove ownership from
THIS home's gateway.pid record (owner.serves() is False when served_known is
False, so the cross-home served-set shortcut never fires). "Whichever home
launched it" is therefore true only for the serves-this-profile leg; a
served-unknown owner launched from another home is refused (exit 1, one
supervisor retry). Say so at the host_attach module doc, the decide()
REPLACE_HOST comment and the _host_attach_or_none comment, and qualify the
`--replace` remedy in _unknown_served_message the same way.
_refuse_message is reached for an owner known NOT to serve this profile, so
its "take the host over: --replace" line pointed at a path that lands on the
same refusal (decide() only returns REPLACE_HOST for an owner that serves us
or has an unknown served set). Drop it in favour of a note, and interpolate
MIGRATE_COMMAND like the sibling refusal in run.py instead of hard-coding the
command. Text/comment changes only; no behaviour change.
The host_attach module doc, the REPLACE_HOST consumer comment in
_host_attach_or_none and the _refuse_second_host_gateway refusal still
promised the old semantics: --replace takes the host over from whichever
home launched it. Since decide() only returns REPLACE_HOST for an owner
that serves this profile (or has not published its served set), and the
lock claim no longer treats --replace as --force, that advice is untrue
on exactly the path where it was printed.
_refuse_second_host_gateway is reached after losing the host lock to a
live process -- including one with no readable record (publish_record
failed after the claim, or a different HOST_PROTOCOL_VERSION during a
rolling upgrade). Offering --replace there sends the operator around a
loop that ends in the same exit 75. The message now says to stop the
other gateway or use --force, and states that --replace does not skip
the lock check. Text/comment changes only; no behaviour change.
Pending async-delegation completions were replayed only at process start
(restore_undelivered_completions from ProcessRegistry.__init__ and the gateway's
secondary-profile restore). A result persisted by a process that then died (a
desktop reload) sat at delivery_state='pending' until some process restarted.
The gateway async-delegation watcher and the TUI notification poller now sweep
each profile home they serve every 30s: abandoned in-flight rows are classified
by recover_abandoned_delegations, and terminal pending rows whose owner fails
the shared start-time liveness check, idle for 60s and not under a live claim,
are offered once to this process's completion queue. Delivery still goes
through claim_completion_delivery, so two processes offering the same row never
both deliver it. Rows past _MAX_DELIVERY_ATTEMPTS or the replay age converge to
dropped.
Fixes#97202
Move retention, orphan pruning, status and clear operations into a topical sibling and route callers and tests directly to their defining module. Keep the shared store paths and git execution in checkpoint_manager; shorten the local child-env WHAT docstring.
Conflict resolutions and semantic fixups:
- tools/environments/base.py: main's hard-exit kill fence (kill a spawn the
fence missed, deregister from _live_foreground in a finally) wrapped around
pm-clean's output collector.
- pyproject.toml: pm-clean's marker list plus main's new `live` marker.
- hermes_cli/main.py: pm-clean runs startup recovery from hermes_bootstrap, so
the old early-recovery block stays gone; main's interrupted-pull restore
(auto-merged above it) runs right after bootstrap, as on main.
- hermes_cli/update_cmd.py: main's interrupted-pull marker now guards
pm-clean's first tree mutation (release-tag detach, ff-only, or reconcile)
and is cleared once git is done. The marker's target is the ref git actually
moves to (a release tag, not always origin/<branch>), since the restore
compares against it.
- hermes_cli/_early_recovery.py: restore `import subprocess`, which pm-clean
had dropped and main's auto-merged restore needs (NameError on the first
launch after a killed update; test_update_interrupted_pull red -> green).
- apps/desktop/src/i18n/{de,es,fr}.ts: main's new locales carry the full
settings.about block; trim it to `updates` as pm-clean's type and the other
overlays do (tsc: 27 errors -> 0).
- main's new e2e tests: `import yaml` -> hermes_yaml; wake-word import table
names pyopen_wakeword (pm-clean's wake-openwakeword extra); the anthropic
key-leak switch leg needs the SDK, and the api_server two-tenant test needs
aiohttp, both PM runtime extras the test env does not carry.
Follow-up to the salvaged mirror commit.
- Opt-in (`mirror_to_session: true`, or `hermes webhook subscribe --mirror-to-session`),
default off like cron's `mirror_delivery`. The mirrored text lands with user authority in
the target chat, and on `deliver_only` routes it is the raw rendered payload, so a route
author has to ask for it. Only a real boolean `true` opts in (a YAML string "false" no
longer does).
- The mirror runs inside the routed profile's scope, so a `/p/<profile>/` delivery writes
into THAT profile's state.db. A Telegram DM chat_id is the user's id on every bot; an
unscoped mirror landed in the default profile's DM with the same person (live-probed).
- Tests trimmed to two invariants against a real state.db: an opted-in delivery lands in the
routed profile's chat session and not the default profile's; a route without the opt-in
(including `mirror_to_session: "false"`) never touches the target transcript.
- Docs: webhooks guide section "Replying to a delivery" (default, profile scope, trust note),
CLI reference row, bundled hermes-agent skill reference.
A webhook route that delivers to telegram/discord/... runs in an ephemeral
webhook:<route>:<delivery_id> session. The delivered text never reaches
the target chat's own session, so when the user replies there ("so he's
out?") the agent has no idea it just sent them anything and asks what
they mean.
After a successful cross-platform send, mirror the delivered text into the
target chat's transcript as a labelled user turn ("[Webhook delivery:
<route>]\n..."), the same path and role convention cron briefs use
(cron.scheduler_delivery._maybe_mirror_cron_delivery, #2221). Best-effort:
never fails the delivery, skipped when the chat has no session yet.
Route opt-out via mirror_to_session: false. delivery_info now carries the
route name and the flag for both agent-run and deliver_only routes.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CZ15jNgPUHdPsMw4wWon6v
The SIGTERM handler arms a 1s os._exit timer, then runs _shutdown_sessions: a flush of up to
5s, then _stop_turns_before_exit, whose kill was the graceful TERM, wait 1s, KILL. A command
that ignores SIGTERM was still alive when the timer fired, and os._exit left it reparented to
init (live: `trap '' TERM; sleep 3600` survived a SIGTERM to `python -m tui_gateway.entry`).
- kill_live_foreground_processes(now=True): SIGKILL each in-flight foreground tree at once,
no TERM grace, no wait (BaseEnvironment._force_kill_process; LocalEnvironment kills the
recorded process group, never our own).
- The grace timer's exit (entry._hard_exit) runs it before os._exit.
- _stop_turns_before_exit SIGKILLs whatever is still alive halfway through its settle budget
(it ignored the interrupt's TERM), so the tool call still ends with a result the teardown
persists instead of a dangling tool_call in state.db.
- The other hard exits that skip cleanup do the same before os._exit: the serve parent-death
watchdog, the CLI exit watchdog, the kanban worker's SIGTERM path, and the messaging
gateway's shutdown and loop-liveness watchdogs.
- Deflake test_shutdown_mid_tool_kills_the_command_and_keeps_its_result: the 0.5s settle
budget was too tight under -n 40 (1 red in 9 runs); the join returns when the turn ends.
- tools/browser_tool_install.py: keep pm-clean's frozen old-updater stub; main's
UTF-8 decode fix touched only the npx prefetch body it replaces.
- tests/hermes_cli/test_update_scoped_reconciliation.py: keep pm-clean's test
subset (catch-up rides the PM completion owner) and take main's gateway-less
host evidence (#120740): the updated seed that holds the host at a running
gateway, and the two gateway-less matrices for the source change that merged
cleanly into update_cmd_fleet.py.
8f0322da5b closes a failed turn with a Hermes-authored assistant row
(FAILED_TURN_NOTICE / PARTIAL_FAILED_TURN_NOTICE). It carried no marker, so
the only way a client could tell it from a real answer was matching the
English copy.
Both writers (agent/conversation_loop.py::_close_durable_failed_turn and
gateway/run_turn.py::_hmwa_close_failed_turn) now stamp
display_kind="failed_turn" (agent/turn_failure_copy.py::FAILED_TURN_DISPLAY_KIND).
display_kind is a DB/display column already stripped from every provider
request (agent/turn_context.py), so the wire bytes and prompt cache are
unchanged; the ACP loopback test asserts the replayed row is exactly
{"role": "assistant", "content": FAILED_TURN_NOTICE}.
session.resume (tui_gateway/session_history.py::_legacy_display_kind) also
types untyped rows already on disk from the last five days, matching the
Python constants in-process, so clients key on the type alone.
gateway.run bridges config.yaml into os.environ at import, keyed on
get_hermes_home(). The Desktop backend (hermes serve) first imports it lazily
from a session's agent build (tui_gateway.agent_callbacks._wire_callbacks),
under that session's routed profile override, so whichever secondary profile
built first latched its terminal.* (TERMINAL_CWD, backend...) and bridged
settings into the launch process env for every later launch-profile turn and
cron job.
Found by tests/e2e/core/tenancy/test_two_tenant_desktop_backend.py (C7 canary):
the default profile's cron env snapshot carried alpha's/beta's TERMINAL_CWD.
Use get_process_hermes_home(); identical for a standalone gateway.
Conflict resolutions and semantic fixups:
- utils.py / hermes_yaml.py: main widened ruamel's round-trip emitter so a long
double-quoted scalar is never folded after an escaped backslash. pm-clean builds
every rt emitter through hermes_yaml.roundtrip_yaml(), so the width lives there
(ROUNDTRIP_YAML_WIDTH moves with it); xai_retirement imports it from hermes_yaml.
- hermes_cli/banner.py: keep pm-clean's removal of the banner update check. Main's
GIT_NO_LAZY_FETCH fix for it applies to its replacement, source_check: every
read-only probe (source_git_env) now refuses promisor lazy fetches, and the
partial-clone test targets that probe (red without the flag).
- .github/workflows/tests.yml: keep setup-pm; main's uv pin bump does not apply.
Main's WAL-capable SQLite gates are kept, run against $HERMES_PYTHON (the
PM-pinned interpreter, SQLite 3.53.1). The e2e step takes main's
--include-integration invocation.
- apps/desktop: package.json has no build block here, so main's macOS locale-marker
restore joins the darwin branch of the existing after-pack.mjs, and its test
loads the hook from electron-builder.config.cjs and imports PlatformPackager
from app-builder-lib's root (electron-builder 27 exports no ./out paths). The
win32 row is dropped: this hook sanitizes and signs PE trees on win32 by design.
- reconciliation.ts: main's rowId hydration (#119326) was merged into the first of
pm-clean's split helpers only; the resolver is now one helper both halves use.
- en.ts: both sides' keys kept. tests/tools/test_lazy_deps.py stays deleted.
- Tests main added with `import yaml` use hermes_yaml, like the rest of the tree.
After an overflow split the heads are sealed and the message id is cleared,
so the tail goes out as a first send in the tick that carries the tool
boundary. When that send failed, _end_segment skipped the unseen-tail flush
(it required a truthy _message_id) and _reset_segment_state cleared
_accumulated: ~1.1k chars vanished. The same loss existed for a plain failed
first send at a segment break.
The flush now runs whenever the boundary update did not land, except for the
__no_edit__ sentinel (its continuation still goes out once via the fallback
final). With a real message id the condition is unchanged, so no new send
there. The flush also honours the egress guard like the other new-message
fallbacks, since it can now run on the path where a draft was declined.
Tests: the oversized-leftover file now pins one invariant, parametrized over
none / commentary / segment_break / tail_send_fails: every tail line reaches
the channel, never twice as a new message, and post-boundary text is not
glued onto the pre-boundary preview. tail_send_fails is red on f361b39
(24 lines lost), green here.
Review finding on the seal re-split: `continue` after `_split_first_send`
skipped the rest of the tick, so a commentary drained with the oversized
burst was dropped and a tool boundary was lost (post-tool text then
extended the pre-tool preview). The initial-overflow gate had the same
`continue`.
Seal first (a no-op without a message id), split once, then push the
tail through `_push_update` in the same tick so commentary, segment end
and the flush barrier run normally after it. `continue` is kept only when
a head send failed and the full text must stay for the fallback final.
When the stream consumer's first send failed (e.g. a Telegram timeout that
never reached the platform), _first_send disabled edits but left the
message id unset, so the next tick sent another first send: a partial
preview that could never be edited. The final reply then went out as a new
message and the truncated preview stayed on screen next to it (on every
first-send timeout, not only under load). With edits disabled, skip
non-final first sends; the final is delivered once, complete.
Found by the C12 exactly-once suite (fault_matrix[stream_timeout_first_send-
fk_tg|fk_dc]).
Two follow-ups to the re-split after _seal_overflow_heads (previous commit),
found by the exactly-once E2E suite driving the real GatewayRunner on
platforms whose limit a streamed reply overflows (Discord 2000):
1. _split_first_send kept the adapter's " (n/n)" chunk indicator on the
tail chunk it reuses as the live preview, so later deltas were appended
after it: the user saw "...wo (3/3)rd0575..." embedded mid-reply and the
visible text no longer matched the transcript. Strip the indicator from
the kept tail.
2. The re-split after a seal now `continue`s like the gate above it when the
turn is not finished: the tail is still unsent, and falling through into
the segment-break reset would clear it.
tests/gateway/test_stream_consumer.py::...fence_aware_split asserted the old
behaviour (the tail starts with the full indicator-suffixed chunk); it now
asserts the indicator is dropped from the tail and never appears in an edit.
_seal_overflow_heads clears the edit target mid-iteration, so the overflow gate
evaluated at the top of run() is already stale when the leftover is pushed. The
seal loop exits after ONE successful seal (its condition includes
_message_id is not None), so a buffer far past the limit leaves most of itself
behind. That leftover then reaches _push_update -> _first_send with no message to
edit, on a NON-final tick, and adapter.send() chunks and caps it itself, numbering
the pieces (i/n). The turn-final lane later publishes the same text again with its
own denominator, because the two lanes use different budgets and different payload
shapes.
Observed in production on Discord: one inbound message, one API call, no tool
turns, a 36642-char answer, and two interleaved sequences on screen - (i/10) and
(i/9) - whose first chunks were byte-identical once the indicator was stripped and
whose concatenations shared a prefix. The gateway's normal final send was
correctly suppressed (streamed=True, content_delivered=True), which places the
duplicate inside the consumer rather than in the gateway's delivery ledger.
Fix: re-check the overflow gate AFTER the seal and hand a still-oversized leftover
to _split_first_send, which is the consumer's own cap-aware, ledger-aware splitter
and already owns sealing. This removes the same-iteration invalidation rather than
compensating for it downstream, and it does not add a fourth delivery flag: the
existing flags behaved correctly here.
Negative control on this branch: with the change reverted, both new tests fail,
including the one asserting that no line is published twice.
Every fail-open config reader turned a read error on an intact file into
a stand-in ({} from read_raw_config and the TUI's _load_cfg_raw, defaults
or a possibly stale last-known-good from load_config). Writers then
mutated that stand-in and saved it; the writer re-reads the file, finds it
fine, and merges by deletion, so one EMFILE/EIO during a TUI config.set,
a dashboard save, a migration step or any load_config()->save_config()
caller replaced the whole file with the stand-in plus one key.
- Fallbacks from a failed read are now FailedConfigRead (a dict subclass
carrying the error). Readers are unchanged; save_config() and
atomic_config_write() refuse to persist one, so the error reaches the
caller and the file stays byte-identical. The subclass rides the
load->mutate->save round trip, so every writer is covered without
touching each of them.
- A read error's fallback is no longer cached (nor recorded as the next
last-known-good), so the retry reads the file again.
- PUT /api/config merges into a new dict, so it reads strictly instead.
- Gateway /verbose and /footer wrote the effective view back (fail-open
to {} on error, ${VAR} values expanded); they now round-trip the raw
file strictly.
Review follow-ups on the crash-left reply adoption:
- A persisted reply is now judged the way live delivery would have judged
it. A bare silence marker ([SILENT] / SILENT / NO_REPLY ...) on an
internal turn, or the reply to a diagnostic wake whose chat policy mutes
diagnostics (read in the routed profile's scope, as the adapter does), is
owed nothing: the marker is cleared and nothing is sent or resumed. A
human turn's bare silence marker becomes the same "returned only a silence
marker" notice the live path sends. Before, the raw "NO_REPLY" reached the
user as a "Recovered reply".
- The active-turn marker start is written as aware UTC and compared as
epoch seconds (startup adoption and recover_interrupted_turns). A naive
local wall clock read by a process in another zone (DST, container vs
unit TZ) was hours off: a fresh in-flight turn was dropped as stale, or a
previous turn's reply could be adopted as this one's. updated_at stays
naive local for the older binary's recency heuristic; a pre-upgrade naive
marker still reads as local time.
- Reply timestamps go through coerce_epoch instead of float(), so one odd
transcript row cannot abort the whole recovery pass.
On an unclean start, suspend_recently_active(120) marked every session
touched in the last 120 s resume_pending/restart_interrupted, so startup
auto-resume ran a fresh model turn for chats whose turn had already
finished and been delivered: one kill re-answered 52 chats in the C12
delivery suite. The durable active-turn markers already name the exact
in-flight turns, so the recency sweep is removed.
That sweep also hid a real window: _handle_message cleared the turn
marker in its finally BEFORE the adapter recorded the delivery
obligation, so a kill in between left neither marker nor ledger row and
the persisted reply was never sent. The adapter now owns the marker for
turns it delivers and clears it right after record_delivery_obligation
(or once nothing more is owed). At unclean startup a marked turn whose
final reply is already in the transcript has that reply adopted into the
delivery ledger (unowned, 'attempting': sent once, marked as a possible
duplicate) instead of being regenerated; a marked turn with no reply
resumes once, as before.
Every entry point wrapped `import hermes_bootstrap` in
`except ModuleNotFoundError: pass` for a partial update that left the
bootstrap unregistered. It also swallowed a module the bootstrap itself
failed to import, and since the bootstrap now owns PM activation that
silently ran the tree on stale dependencies: exactly how a pre-PM
editable venv hid its unreachable `pm` until it crashed on ruamel.
Re-raise unless the missing module is hermes_bootstrap, at all six entry
points. The stale "only Windows UTF-8 stdio suffers" comments go with it.
The boot sweep now moves every claimed row to attempting. If the platform went
fatal between the claim and the send (restart notification, flood sleep),
_obligation_adapter skipped the row without releasing it, so it sat in
attempting owned by this live process: the reconnect sweep only takes failed
rows and the boot sweep skips live owners, so the reply waited for the next
restart (and then carried a false duplicate marker). Release any undispatched
claim, not only runtime ones.
Found by independent review of #120450.
Two invariants on the boot outbox sweep, both red on the previous ledger:
- a pending row whose boot redelivery was interrupted after the platform
accepted it (or that an older build claimed) is redelivered with the
recovered marker, never a second plain copy;
- a boot-claimed failed row is not re-claimable by the runtime reconnect
sweep while the boot send is in flight (it used to stay 'failed' and
owned by this process, so a reconnect could send it a second time).
Also corrects the startup-claim comment in _obligation_adapter and the
messaging docs bullet on mid-send recovery.
sweep_recoverable claimed a dead-owner 'pending' row by re-stamping the
owner and spending an attempt but left state='pending', and the boot
redelivery path never marked it attempting before adapter.send. A boot
killed after the platform accepted that plain resend (before
mark_delivered) left the row 'pending', so the next boot resent it
UNMARKED: a silent duplicate reply.
The claim UPDATE now moves every claimed row to 'attempting' in the same
owner-stamp CAS; needs_marker still reads the pre-claim state, so a
never-claimed pending row is redelivered plainly once and any later copy
carries RECOVERED_MARKER. A pending row that an older build already
claimed (attempts > 0) is also marked, since it may have been sent.
The reconnect watcher replaces a failed adapter with a NEW instance, and
every adapter keeps its inbound MessageDeduplicator on the instance. The
rebuilt adapter started with an empty cache, so a platform re-delivering a
recent inbound ID right after the reconnect (websocket resume replay,
webhook retry, unacked poll batch) got it processed and answered again.
The reconnect queue entry now holds the retired adapter's
MessageDeduplicator attributes by reference, and the rebuilt adapter
absorbs their live IDs before it connects. The multiplex secondary-profile
reconnect path gets the same handover. Any adapter using the shared helper
is covered without per-adapter code.
In-place compaction is the default. It soft-archives every earlier row of
a session under the same id (active = 0, compacted = 1), and Desktop and
the dashboard still show those turns. The md/qmd export read the session
through export_session -> get_messages with the default live-only clause,
so it wrote only the compaction summary and the carried tail.
verify_export_file then compared the file with that same dict, and
delete_session removed every row of the session, including the archived
turns that never reached the file.
The md/qmd export now reads the display history (include_compacted), for
a single session and for --lineage logical. export_session and
export_session_lineage take include_compacted, off by default:
import_sessions inserts every message as live context, so the JSON export
and stranded-session adoption keep reading live rows only.
The other transcripts people read had the same hole without the delete:
`/save md` and `/save html` (CLI and gateway), `sessions export --format
html` (one session or all of them) and `--only user-prompts` each held 1
of 6 answers on a six-turn session after one compaction. They now read
the display history too (SAVE_TRANSCRIPT_FORMATS; export_all gains
include_compacted and reads per session then, since the display read
dedupes per session). `/save json`, JSONL and the dashboard's JSON export
stay live-only for the import reason above.
Before deleting, the verify step also re-counts the store's display rows
for every session the file covers and refuses on a mismatch. A message
that lands while the files are written, or a later export change that
reads a narrower view, now refuses the delete instead of being removed
unseen. Like the adoption retire loop, the re-count runs just before
delete_session, not inside its transaction. Rewind rows (undone turns,
the superseded originals of a carried tail) are still deleted without
being exported, as `hermes sessions delete` does: they are not part of
the history the session shows.
Measured through the real CLI on a session with 6 turns and one default
in-place compaction (15 rows, 13 shown): before, 3 messages were exported
and all 15 rows deleted, with answers 1-5 missing from the file; after,
13 messages are exported in display order, then deleted.
(cherry picked from commit adeaff1e33f1ae2b8a066ef2374bb29ade50b3b8)
The "No session model override" debug line eagerly built a list over every
session's conversation.model_override as a logger.debug argument, 2-4 times
per inbound message, even when DEBUG logging was off. Gate the whole call
on logger.isEnabledFor(logging.DEBUG) so the scan costs nothing in normal
operation.
Re-applied by hand from #77810 (targeted gateway/run.py; the code now lives
in gateway/run_turn.py, logic unchanged). Dropped the PR's routing test.
(cherry picked from commit c3c2bf3801)
Partial salvage of #119405: kept the four api_server.py hunks (per-profile
single-flight asyncio.Lock dict, _artifact_store_for_async, awaited
to_thread for store.store and store.load), dropped the 392-line test file
because it is mostly change-detector/timing assertions.
(cherry picked from commit cdfcc39181)
_poll_loop called _save_sync_buf inline after every getUpdates response.
It ends in atomic_json_write (mkstemp + fsync + os.replace), so each poll
cycle blocked the gateway's event loop for the duration of an fsync, and
it did so even when nothing changed: an empty long-poll, and the timeout
sentinel from _get_updates, echo the current buffer back, so the same
value was rewritten every cycle.
Write only when the buffer differs from the one in memory, and dispatch
the write with asyncio.to_thread. The loop awaits it before the next
poll, so writes stay serialized.
(cherry picked from commit 316b775941ffcc7e1bdd17c044e879cbca1eb14f)
The authz gate calls PairingStore.is_approved on every inbound message,
and each call re-read and re-parsed {platform}-approved.json from disk
plus rebuilt per-user alias sets. Approvals change only through
explicit pairing writes (this process or the pairing CLI), which always
bump the file mtime, so a (mtime_ns, size)-keyed cache is an exact
invalidator. A stat per message replaces a read + parse per message.
The loud PermissionError warning in _load_json still runs on every
real read, and same-process writes through _save_json invalidate
automatically via the mtime bump.
(cherry picked from commit 0a0d2296bdf36ae58ca4443c2d26b8fc577c88dc)
_detect_macos_system_proxy() forks `scutil --proxy` with no cache, and
resolve_proxy_url() calls it on the SEND path — inside the per-chunk sender in
_send_chunks and again per media attachment, not once per adapter. It is also
reached whenever no explicit proxy is configured, which is the common case.
Measured on macOS: a bare `scutil --proxy` fork is 11.05 ms, and
resolve_proxy_url("DISCORD_PROXY") is 11.209 ms — essentially all fork+exec, to
read an OS setting that changes when someone edits Network Settings or joins a
VPN. A three-chunk reply paid ~34 ms of it; a message with two attachments
paid more again. macOS-only, so it never shows up in CI.
Memoise the answer for 60 s. The TTL is the staleness a proxy change can
suffer; a send that goes out on a stale answer fails and is retried, the same
outcome as any transient proxy error. A failing scutil is cached too, so a
broken or slow one cannot re-fork per chunk either. reset_macos_proxy_cache()
forces a re-read.
resolve_proxy_url: 11.209 -> 0.742 ms (-93%), same value returned.
(cherry picked from commit 635b06c1007e5806bf83e9d1aef2db0be1a486c2)
Same class as the config/manifest loader swap earlier in this stack. The
/ready probe re-read and pure-Python-parsed config.yaml on every poll, inside
the aiohttp handler (110 KB seeded config: 2332 -> 487 ms per probe on a
loaded runner). It now uses utils.load_yaml_file_readonly: the C loader, and
repeat probes reuse the parse until the file signature changes. Parse errors
are not cached, so an edit that breaks or fixes the file shows on the next
probe. The bundled-platform manifest reader was the one startup manifest
reader left on yaml.safe_load. managed_scope imports fast_safe_load at module
level next to file_signature instead of lazily.
_slack_tools_loaded/_discord_tools_loaded run per turn via _ephemeral_change_key and
_get_platform_tools only reads the config, so the deepcopy in load_config() is pure
overhead on the event loop. Use load_config_readonly().
Partial salvage of #117993: kept the two gateway/session.py hunks, dropped
TestCapabilityProbeConfigLoader because it monkeypatched cfg.load_config to raise —
a change-detector on the symbol read, not an invariant.
(cherry picked from commit a487903180090c5a2fccd4749b05d302cf2740b1)
utils.fast_safe_load already exists, is pinned by tests/test_fast_safe_load.py, and
its comment names exactly these payers: 'startup parses config.yaml and every plugin
manifest, so the slow path cost ~0.9 s of cold start'. The migration was started —
hermes_cli/config.py uses it eight times, hermes_cli/main.py and hermes_cli/plugins.py
too — but the file-level loaders it was written for were never converted.
The cost is config SIZE, and the size is the installer's doing: it seeds config.yaml
by copying cli-config.yaml.example, 120,897 bytes of mostly comments. Nothing caches
load_gateway_config() and it has 238 production call sites.
Profiled before assuming a cause — reader.forward 34 ms, scanner.scan_to_next_token
28 ms, reader.peek 13 ms: the pure-Python PyYAML scanner, nothing else.
Measured A/B on one realistic pass (gateway config + every bundled plugin
description), medians of five runs, __pycache__ cleared between arms:
48.8 ms -> 2.46 ms. Per path: load_gateway_config 48.3 -> 1.84 ms, managed
config.yaml 44.4 -> 1.03 ms, 105 plugin.yaml manifests 59.3 -> 5.9 ms.
Same parse, same restricted tag set, same result — only the loader changes. Drops the
three 'import yaml' statements the swap orphaned.
(cherry picked from commit a4efd6a4506043159310eba02c32b673c455b88b)
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
gateway.standalone wins over the parked marker: profile_lifecycle() returns
False for an opted-out profile, so `-p X gateway stop|start` keeps addressing
X's own gateway process and never writes gateway.parked, even while a stale
host record still lists X. Every installed-roster caller now threads both
kwargs (`include_standalone=True, include_parked=True`): the host-attach peer
walk and the standalone boot notice were reading the roster without parked
profiles.
Dashboard twin (#119886 class): `/api/gateway/stop?profile=X` on a served
profile no longer answers 409 — the spawned `hermes -p X gateway stop` parks
it; `/api/gateway/start` on a parked profile is allowed while a host
multiplexer is live (the child unparks it) and still refused when nothing can
serve it. Tests trimmed to the invariants: the marker-appearing case was a
subset of the boot-and-reconcile test.
Under the host multiplexer every installed named profile came online because
its directory existed, and the only way to stop one profile's bots was to stop
the host, which stopped everyone's. Both are fleet-operator blockers.
`hermes -p X gateway stop` on a profile served by the host now parks it: it
writes `profiles/X/gateway.parked` first (so the 30s reconcile cannot re-add X
between the verb and the marker) and sends the new `unserve-profile` control
verb, which tears down X's adapters, reconnects and cron inside X's own scope
(the teardown `_unserve_profile` already used for deleted profiles). `start`
removes the marker and sends `serve-profile`, which runs the same add-path the
reconcile loop uses. `restart` cycles both without parking. The default
profile keeps today's whole-host meaning.
`profiles_to_serve()` skips parked profiles, so adapters, cron, ingress
membership and the served record follow from one chokepoint; roster callers
that mean "every installed profile" (plugin deps, Windows update, dashboard
listing and topology, migration inventory) pass `include_parked=True`.
Provisioning may pre-create the marker: an installed profile stays offline
until an operator starts it. Host boot logs one INFO per parked profile;
`gateway status` shows `parked (hermes -p X gateway start)`.
Tests: profiles_to_serve contract, control verbs through the runner
(round-trip and every refusal), CLI marker-before-socket ordering, reconcile
honouring the marker both ways, migration inventory retaining parked
profiles, two-home E2E through the real loaders.
Symptom: a host that serves several profiles from one process and mirrors
the active turn's profile into `os.environ["HERMES_HOME"]` for legacy
readers (Hermes WebUI does this on every chat turn, next to the
context-local override) makes every launch-home decision see the served
profile as the launch profile. Two profiles that both configure `atlassian`
with different credentials share whichever MCP connection came first: a
READ_ONLY_MODE=false profile ends up calling a read-only server
(nesquena/hermes-webui#7721). The same misjudgement leaves the launch
residue in the served profile's child env, seeds the launch profile's
bridged allow-all grant into the served profile's secret scope, and lets
the served profile's `terminal.*` config bridge into the shared process env.
Cause: four launch-home checks compare the task's override with
`get_process_hermes_home()`, which reads `HERMES_HOME` live:
`agent.secret_scope.serves_routed_profile` (keys the MCP ledger via
`_mcp_registry_scope`, #108352 / #111481, and the check_fn cache, #111151),
`agent.secret_scope._is_process_home`, `tools.environments.local._is_routed_home`
and `hermes_cli.env_loader._process_hermes_home`. Under the mirror the two
sides are equal for every turn.
Change: `hermes_constants.pin_process_hermes_home(path | None)` lets the
host record the home it serves as its own; `get_routing_process_hermes_home()`
returns the pin when set, else `get_process_hermes_home()`; the four checks
compare against it. The pin is deliberately NOT folded into
`get_process_hermes_home()`: `get_hermes_home()` falls back to it for tasks
carrying no override (MCP loop, spawners), and the host's mirror exists
precisely so those readers see the served profile. Only "is this task
routed / is this the launch home" changes. Unpinned, behaviour is
byte-for-byte the old one; hosts that never mutate `HERMES_HOME` need not
call it. `activate_multi_profile_hosting()` is not the seam for this: it
flips `get_secret` fail-closed process-wide and freezes the launch env,
which an embedding host cannot adopt as a bug fix.
Tests (2 invariants, parametrized over the four checks plus the MCP ledger
key; red on main, green here): pinned + mirrored env -> the served home is
routed and the launch home is not, the MCP key is `(home_key, name)`,
`get_process_hermes_home()` still follows the env var; never pinned or
pinned-then-cleared -> old semantics, including "a mirrored env var IS the
launch home". `tests/conftest.py` resets the pin per test so the
module-global cannot leak between files.
Live repro (WebUI + a stdio FastMCP server named `atlassian` in two
profiles, one gated by READ_ONLY_MODE): base -> one ledger key
`'atlassian'`, the write profile lists only the read-only tools; fixed ->
`(<read_home_key>, 'atlassian')` and `(<write_home_key>, 'atlassian')`,
each profile lists its own tools.
Docs: `gateway/AGENTS.md` § Profile scope (one launch-home identity) and the
isolation table in `website/docs/user-guide/multi-profile-gateways.md`.
Also maps the author e-mail under contributors/emails/ (attribution check).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
A multiplexed API server mirrors every route under /p/<profile>/ and
authenticates each mirror with that profile's key, but it kept one
ResponseStore at the home it was constructed in. Conversation names are
client-chosen strings ("main", "my-project"), so another profile's key
could post `conversation: <name>`, receive that profile's transcript,
instructions and session id as its agent's context, become the
conversation's tip (the owner's next turn replayed the intruder's
messages), and GET or DELETE the owner's responses by id.
The adapter now resolves the store from the request's profile home, as
the SessionDB cache already does: the construction home keeps
self._response_store (and its response_store.db), every other routed
home gets its own <home>/response_store.db, opened on first use and
closed on disconnect. The stream state captures its store when the
request starts, so a snapshot written after the scope ends (disconnect)
still lands in the right one.
Rows a secondary profile wrote into the shared store before this change
stay there, visible to the construction home's profile only; they are
not migrated.
A multiplexed gateway defers two things for a platform that is offline at boot: the
planned-restart online notice (the marker keeps the target owed "for its reconnect", #112109)
and restart-interrupted sessions (left resume_pending "for the reconnect watcher"). The owed
set spans every served profile's home channel, and a served profile's session never falls
back to the default bot, so both wait on that profile's own reconnect.
Only the primary reconnect (_install_reconnected_adapter) acted on either.
_run_secondary_profile_reconnect published the adapter, redelivered failed obligations and
returned: the notice never went out and the marker outlived the outage, and the deferred
sessions stayed stranded until the freshness window aged them out.
The secondary reconnect now does what the primary does, through one shared helper for the
notice replay (still lock-serialized, delivered targets recorded, so a concurrent primary
replay cannot double-send).
A multiplexed gateway connects each served profile's adapter inside
_profile_runtime_scope(<profile home>). The receive loop an adapter starts
while connecting inherits that home override, so every final reply the bot
sends is recorded from it. The ledger resolved its path through
get_hermes_home(), which follows the override, and the rows landed in
profiles/<name>/state.db. The boot sweep (sweep_recoverable) and the boot
flood-timer arming (pending_retries) run in the launch context and open the
launch state.db, so they never saw those rows. A served bot's reply cut off by
a crash or SIGKILL between finalize and platform ACK was never redelivered, a
flood-refused reply that spanned a restart was never retried, and
resume_pending was not cleared for a session whose answer sat in the ledger.
The ledger is meant to be one shared store: the boot sweep already scopes rows
by (platform, adapter_profile), and the profile purge terminalizes rows in the
shared store. _db_path now resolves from get_process_hermes_home(), as the
gateway's other process-level files do (gateway.status). It deliberately skips
the get_hermes_home() fallback that lifecycle_ledger uses when HERMES_HOME is
unset: a default gateway started in the foreground has no HERMES_HOME, and that
fallback would follow the override again.
Rows an earlier build already wrote to a profile's state.db stay where they are.
The salvaged #119607 taught only the drain cap (launchd_service_label) to read
HERMES_LAUNCHD_LABEL. The gateway grandchild under the generated plist also
reads XPC_SERVICE_NAME=0 in is_gateway_supervisor_process (exit-75 restart
route) and control_socket._detect_supervisor (identify payload), so one
process was "launchd" to the drain cap and "manual" to the restart route.
One seam: gateway.restart.launchd_job_label applies the ai.hermes predicate to
XPC_SERVICE_NAME then HERMES_LAUNCHD_LABEL; the drain cap, the restart route,
the control socket and the wrapper's export all call it. launchd_service_label
and read_launchd_exit_timeout_s take `platform` as data so the mapping is
tested on Linux without patching sys.platform (AGENTS.md: don't fake the host
OS); the salvaged tests are trimmed to two invariants each and lose their
monkeypatch of sys.platform.
Not live-run: this host is Linux, launchd is code-path + wrapper-subprocess
proof only.
