An IPv6 literal followed by :port is only a valid URL authority inside
brackets; http://::1:9119 is unparseable by browsers and urllib. Add
hermes_cli/url_utils.format_url_host and use it at every site that
builds an http URL from a (possibly IPv6) bind host: the named-profile
dashboard attach URL, the browser auto-open URL, the startup banner,
the plugin-activation loopback request, the proxy listening banner, and
the gateway-migration listener URLs. Zone identifiers (%eth0) are
percent-encoded per RFC 6874.
Fixes#67367
Co-authored-by: sjungwon03 <sjungwon03@gmail.com>
app.add_middleware(CORSMiddleware, ...) was registered before the
@app.middleware(http) auth decorators, making CORS the innermost
layer in Starlette's onion model. A cross-origin OPTIONS preflight
hit auth first → 401, and the CORS headers were never sent.
Moving the add_middleware call after all @app.middleware registrations
makes CORS outermost, so preflights are answered before auth runs.
Fixes#59052
A prior backend whose session died (dead spawner, PID re-parented to
init) stays alive, so it keeps holding the host flock and its record
passes every staleness check. A fresh backend hit HELD_BY_OTHER and
returned observe-only forever; every attach retried against the same
orphan and its 10s probe timed out indefinitely.
On HELD_BY_OTHER, look the conflicting owner up in the spawn ledger
and apply the established spawner_is_dead discipline (previously wired
only into the mcp-helper reap): a provably dead-spawned owner — or a
null-spawner owner reparented to init, old past the lock-write grace,
and claimed by no Desktop SSH lock — is terminated (SIGTERM, 2s,
SIGKILL), its record discarded, and the host lock re-claimed.
Live-spawned, lock-claimed, young, interactive, self, and unprovable
owners are never touched.
The pre-auth POST routes under /auth/ buffered the whole JSON body before validation, with no size limit, so anonymous clients could hold large uploads in memory. A pure-ASGI middleware now answers 413 when Content-Length exceeds 64 KiB, and counts the bytes actually received so chunked or understated bodies are cut off at the same limit. Authenticated routes and WebSockets are unchanged.
Reported-by: Tenable Research
Scoping a supervised dashboard/serve backend's background processes moved them out of the
backend's service cgroup, so they survived a backend restart, but nothing in dashboard/serve
startup recovers the process checkpoint: they kept running while process list/poll/kill could
not find them, and the next spawn dropped their checkpoint record. Recovering them in serve
would conflict with the gateway owning background processes (#106742). Revert to main's
behaviour for dashboard/serve (workers stay in the backend's cgroup and die with it) and leave
escapee cleanup there to the serve-to-gateway cutover.
Tests: the escapee in the hang test printed before the kill deadline, which ended the reader's
blocked read and let the old close finish in time, so the test passed without the fix. The hang
test's escapee is now silent; late-output rejection has its own test. Removing the reader-close
guard now fails the hang test.
Only the supervised messaging gateway ran background processes in their own
systemd scope (_is_supervised_gateway_process requires _HERMES_GATEWAY and the
gateway PID file). A dashboard or serve backend run by systemd hosts the same
kind of agent turns, but its background processes shared its cgroup, so a
descendant that setsid()s past the PTY/process-group kill survived it: the kill
reported success while the escapee kept running (and kept the PTY slave open).
web_server.start_server now marks the process as a backend host (process-local,
so terminal children and CLIs that inherit the supervisor env don't count), and
_scope_argv scopes when that host was launched by a service manager. The kill
already stops the scope, which reaps the whole cgroup. The self-kill guards that
also read _is_supervised_gateway_process are unchanged.
A Desktop reconnect that cannot prove the recorded pid is its own (dead-looking
pid, FOREIGN argv proof) drops desktop-ssh/<ownershipId>/backend.lock.json
without signalling that pid and spawns a new nonce. The superseded serve had no
signal that it lost ownership: it lived until the idle watchdog fired (900s
grace, gated on zero WebSockets, which a half-open tunnel keeps non-zero for up
to the 600s ping timeout), so a reconnect storm stacked detached backends, each
an extra state.db writer (#132034).
web_server_owner_exit mirrors web_server_skew_exit: a pure decision (retire only
when a VALID lock names another spawnNonce and the process is past the reaper's
180s settle window; missing, unreadable or skewed locks keep it up) and a daemon
watchdog that needs two consecutive observations, then exits through the
retirement fence so it never tears down mid-turn and admission is closed first.
The lock is read through dashboard_procs' existing validator (split out as
read_valid_backend_lock, shared with _lock_owned_serve_pids) rather than a
second parser, and its path is the validated --ssh-session-token-file's
ownership directory. Server-side only: cleanupStale is untouched, so this
composes with #126035.
Root-cause analysis and watchdog design from the #132034 report by @VexterZ;
lock-ownership watchdog first drafted in #89452 by @pmwanje2 (prior art).
Refs #132034 (POSIX SSH hosts; Windows-host locks are not validated yet)
One answer per profile, asked once, with the Desktop strip's three equal
choices (Send to Nous / Local only / No thanks) now offered by:
- `hermes setup`: at the end of every flow (Quick, Full, Blank Slate,
Portal, --quick), via the setup-completed record they all funnel into.
- `hermes` / `hermes --tui`: once before an interactive chat on a profile
that never answered. Skipped for -q, piped/JSON output, spawned actions
(HERMES_NONINTERACTIVE), Desktop-hosted panes and managed installs.
- Web dashboard: a banner for the profile being managed, backed by
GET/PUT /api/shared-metrics/consent.
"No thanks" is the terminal default so Enter never opts anyone in; Esc and
the banner's X leave the question open.
Also fixes the answer that could not stick: a CLI "no" equals the shipped
defaults, so save_config stripped it and every surface (Desktop strip
included) kept asking. The answer is now written to config.yaml directly,
and `decided` has one definition shared by the RPC, the CLI and the router.
The code-skew watchdog only covered SSH-isolated backends; Desktop-owned
local serves share the same 'updater never restarts me' property, so they
served stale modules until the app happened to restart them (R1). Extend
start_code_skew_watchdog wiring to is_desktop_owned_backend() backends.
Centralize the skew refusal: POST /api/model/set and the tui_gateway
model.options / model.save_key RPCs now refuse with the same restart
message the picker and the gateway /model switch already return (R2).
uvicorn's default timeout_graceful_shutdown=None makes Server.shutdown()
wait forever on a lingering ASGI task; Desktop sends a single SIGTERM and
escalates to SIGKILL ~5s later, so a mid-turn request orphans the backend
(#76244). Set timeout_graceful_shutdown=3, under the kill budget, with a
behavior-level regression test that cancels the lingerer and returns.
The fail-open built-in path gated ticks via can_dispatch, so a skipped
tick still recorded a successful heartbeat and cleared the tick error on
the store a live gateway owns. Tick this backend's own store through the
multiplex loop with the same profile_gate instead: a gated-out profile is
neither ticked nor heartbeated.
One _gateway_owns_cron(name, home) predicate now backs every path, so the
fail-open and external-provider paths also stand down for a satellite
served by the live default multiplexer. The external-provider wait probes
once per interval.
Tests: split the external-provider stand-down test and drop the iterator
that hid over-probing; pin the fail-open start kwargs exactly; cover the
satellite gate and the heartbeat through the real built-in loop.
(cherry picked from commit b9250805e7eb951632e9f56e1ce6321e9cbcb395)
The multiplex ticker already yields to a live gateway through its per-tick
profile_gate. The fail-open paths (profile enumeration failure, empty served
set, external provider) still took one ownership probe at backend start and
returned for good, so Desktop cron stayed dead after that gateway stopped.
The built-in single-store ticker now gets the same probe as a per-tick
can_dispatch gate: it stands down while a gateway owns this home, fires once
it stops, and stands down again if it returns. An external provider takes no
per-tick gate, so its start waits, re-probing every interval and honoring
stop_event, until the gateway is gone.
Fixes#126822
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
(cherry picked from commit 4ffe4b3fd68b96fd48c4e8847344a69de7c4f0c2)
29ca3f2770 bailed out of _start_desktop_cron_ticker whenever a gateway was
live on the home at backend start. The ticker is started once per backend
lifetime, so when that gateway later stopped (Desktop's own gateway stop,
a crash) nothing ticked cron until Desktop restarted. The multiplex
ticker's per-tick profile_gate already stands down while a gateway runs,
so the startup stand-down now applies only to the ungated fail-open paths
it was written for.
(cherry picked from commit ad59becf079f388667620481276f55b8421edcb3)
(cherry picked from commit f2240b871d045f85af100b343bb4a053663fff86)
Two product questions shared metrics could not answer: how long each surface
takes from launch to usable (so startup regressions show per release), and how
current, on which channel and on what class of machine installs run.
hermes.startup.latency {surface, latency_bucket} records once per process start:
cli (process creation -> first rendered prompt, or -q dispatch; Kanban workers
excluded), gateway_boot (-> GatewayRunner.start done), serve_boot (-> hermes
serve listening), and tui / desktop_attach reported by the clients through the
new shared_metrics.startup_latency RPC. The clients declare their surface
because a Desktop on a URL/cloud backend has no HERMES_DESKTOP there; env
detection is the fallback for older clients. In-process surfaces measure from
psutil's process create time, the earliest timestamp available, and hand the
runtime start to a daemon thread under the caller's context so no event loop
waits on it. Everything goes through _emit, so disabled profiles record nothing.
The install snapshot gains release_channel, version_age_bucket, behind_bucket,
ram_bucket, gpu_class and local_model_provider_used. All are read offline:
the installed commit's own date, the channel record / packaged channel / checkout
branch (never the remote URL or branch name), and the update check's existing
cache for this exact revision (never a network call). Rows counted before these
fields existed stay valid as a legacy field set.
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
The Desktop backend's `_spawn_gateway_restart` reaped "orphan" gateways
BEFORE spawning `hermes gateway restart`; the child then applied its
multiplexer refusal (`_guard_named_profile_under_multiplexer`, exit 78) and
started nothing, so a refused restart still cost the profile its gateway
(the reporter's 'refused restart, heartbeat stopped between two attempts').
Move the #77276 reap into the child, after the refusal guard, on the
service-manager branch (the manual fallback already reaps inside
stop_profile_gateway(); `--all` and the externally-supervised hand-back do
their own stop). The backend now stops nothing itself.
Test changes: `test_reap_failure_does_not_block_spawn` asserted the reap ran
in the backend (the bug); it is replaced by the two invariants: the backend
spawns without reaping, and the CLI reaps only after the guard.
Review cleanups on the orphan-reap startup grace:
- Drop the docstring claim that the Desktop boot sweep is "the one caller
that races a launch": web_server._spawn_gateway_restart also reaps
(grace-less) before its coalesce check, so the claim was wrong.
- Cut the 7-line lifespan comment to one line; the full reasoning lives in
the _reap_unsupervised_gateway_orphans docstring, so the two can't drift.
- Drop the never-asserted seen["extra_exclude"] and the constant-only
`_REAP_MIN_AGE_SECONDS > 0` assert; a grace-less revert already fails the
`seen["min_age_s"] == _REAP_MIN_AGE_SECONDS` check.
Co-authored-by: Halldrix <halldrix@users.noreply.github.com>
The desktop ticker's per-tick profile_gate only arms in the multiplex path;
the fail-open paths (profile enumeration failure, empty served set, external
provider) start an ungated single-store ticker that races a live gateway's
tick-lock on the same HERMES_HOME. When the desktop wins, delivery has no
live platform adapter and the cold send hangs until script_timeout (600s).
Bail out before resolving the provider when a live gateway is running on this
backend's HERMES_HOME; on probe failure fall through to the existing per-tick
gating instead of standing down blindly.
Fixes#52202
A headless `hermes serve` emitted only HERMES_BACKEND_READY. A packaged
Desktop artifact whose readiness parser predates the neutral token
(the CLI-only `hermes update` path never rebuilds the packaged app)
matched nothing, then killed the healthy backend after the
port-announcement timeout — the artifact-skew signature of #60772.
Announce the neutral token first and the legacy HERMES_DASHBOARD_READY
one after it; current parsers match either, and stale packaged parsers
match the legacy line. The legacy `dashboard` backend keeps its token.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
The serve a remote Desktop spawns over SSH is never restarted by the
host's updater (only its client holds the token and owner nonce), and
the idle watchdog stays quiet while that client is connected. After an
update it therefore kept running the pre-update code against the new
tree until the client happened to reconnect.
Poll the checkout sha against the sha this process loaded. On two
consecutive mismatches, with no update in flight, retire through the
existing retirement fence (which proves idle and closes admission), and
exit cleanly so the client respawns the backend on the new code. Unknown
shas, busy or unreadable ledgers and a live update all keep it up.
hermes serve --isolated (the backend another machine's Desktop spawns
over SSH) opts out of the host singleton on the CLI side, but its spawn
ledger row carried no structured marker, so the local Desktop's
attach-first discovery adopted it. Nothing on this host owns that
process, so a Desktop-driven update left the local app on stale code.
Record isolated=True in the ledger row and skip such rows in
parseSpawnLedger. An ordinary serve is still attached even when an
isolated record is newer.
Ink TUI and desktop never registered an inject host, and sharing
set_gateway_message_injector with a live messaging gateway would let
the last writer win. A separate host queues the reported session_key
onto that session's prompt queue and leaves other keys for the gateway.
Fixes#87412
A bare skipif never ran anywhere: the Windows job selects -m windows_only.
Use the marker, drop the issue number from the filename, and let an old
uvicorn without get_loop_factory fall through to _run_serve's existing
selector-policy fallback instead of a second try/except.
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
Review follow-up on the host-rendezvous isolation:
* Excluding Desktop-owned children from ROLE_SERVE broke #119644's terminal
path: `hermes plugins install` found "the running Desktop backend" only via
read_record(ROLE_SERVE), so on a Desktop-only box open chats no longer lit
up. A Desktop child now publishes ROLE_DESKTOP_SERVE (own lock, record and
0600 token). The attach/refuse ladder (_host_backend_attachment) still reads
ROLE_SERVE only, so a supervised public dashboard is never blocked by it;
notify_serve_backend prefers the host owner and falls back to the Desktop
record. /api/host/identity reports the role actually published.
* is_desktop_owned_backend() missed Desktop's SSH spawn — `env HERMES_DESKTOP=1
hermes serve --isolated --ssh-session-token-file F` carries NO token env var
(remote-lifecycle tests assert the var name never appears) — so the SSH child
still claimed ROLE_SERVE on the remote host and its MCP discovery flipped to
deferred. The predicate now accepts the token-FILE argv shape; the argv half
lives in _startup_fast.is_desktop_ssh_backend_argv (stdlib-only, importable
before main.py's import wall) and replaces the two duplicate substring checks
in main.py and dashboard_procs.py.
* web_server.py's remaining three bare HERMES_DESKTOP reads (cron ticker,
managed-gateway teardown, orphan serve reap) route through the predicate;
the tests that modelled a Desktop child with the bare flag set the spawn
token, as a real pool child does.
Widen the two salvaged fixes to the whole class and make the refusal
supervisor-safe:
- `process_identity.is_desktop_owned_backend()` is the single discriminator
(HERMES_DESKTOP=1 AND the per-spawn HERMES_DASHBOARD_SESSION_TOKEN). The
attach bypass, the named-profile reroute, the env sanitizer, the MCP
discovery timing and the host-rendezvous publish all key on it now; a shell
that merely inherited the flag from Desktop is treated as a normal launch
(#119210). The publish skip from #119832 keyed on the bare flag, which
would have hidden a supervised service launched from a Desktop shell.
- `_attach_to_host_backend` refuses with exit 78 (EX_CONFIG) instead of 1.
Exit 1 under `Restart=always` was an infinite restart loop with nothing
listening on the ingress port (2035 restarts, #119824); 78 is the
deliberate-refusal code `RestartPreventExitStatus=78` parks on, the same
contract the gateway unit already uses.
- Docs: the systemd example carries RestartPreventExitStatus=78 and the
user-side remedy (stop the owner, or `--isolated`).
- Tests trimmed to ≤2 invariants per fix; the control case in the desktop
tests sets the token so it models a real pool child.
A fresh dashboard /chat always spawned the TUI in the dashboard process's
launch directory, so from a phone or any browser there was no way to aim
a new session at a specific repository. Amp's runners now serve many
directories and the web composer offers a picker of the runner's projects
and discovered git checkouts; this ports that mechanism onto the surface
Hermes already has: the dashboard is the phone/web front, the host's
projects.db + repo-discovery cache are the served directories.
- GET /api/chat/workspaces: the profile's projects (with folders) and
discovered repos (session-derived + scanned), default_cwd, home;
?scan=1 rescans desktop.repo_scan_roots on the host, so headless
installs (no Desktop to populate the cache) discover repos too.
- /api/pty?cwd=<dir>: validated (existing directory, fail-closed 400
via the PTY error path) and forwarded to the TUI child as HERMES_CWD
(self-spawned gateway cwd) + HERMES_TUI_CWD (explicit cwd on
session.create for the dashboard's in-memory gateway, whose own cwd
is the launch dir). Resumed sessions ignore it.
- ui-tui: session.create carries cwd when HERMES_TUI_CWD is set, so
/new inside a dashboard chat stays in the picked workspace too.
- web: workspace selector in the Chat rail above "New chat" (projects,
repos by recency, Other path…, rescan), remembered per profile in
localStorage; 17 locales.
- docs: web-dashboard.md rail + REST sections.
Live E2E: real `hermes dashboard` under a scratch HOME with two git
repos under desktop.repo_scan_roots -> /api/chat/workspaces?scan=1
lists both; /api/pty?cwd=repo-a -> TUI status bar shows ~/code/repo-a;
/api/pty?cwd=<missing> -> "Working directory does not exist" + close.
AGENTS.md showed three stacked platforms() decorators as the pattern —
exactly what the conftest rejects at collection. Show one marker per
test with the any-of form instead, and describe the lane selector as
resolving specs (posix reaches the macOS lane) rather than grepping for
the literal word, which is no longer how it works.
Remove the three MERGE-CHECK notes (AGENTS.md, hermes_cli/web_server.py)
left over from the origin/main merge; they were reviewer prompts, not
documentation of the code.
Round-2 review fixes for the multiplex-only convergence.
BLOCKER 1 — a CONVERGED host reported itself half-migrated and every `hermes update`
SIGTERMed the only gateway it had. The plan derived `has_gateway` from
`live_gateway_pid_for_home`, which deliberately answers with the HOST multiplexer's pid for
every home it serves (topology reporting, working as designed). Ownership is now proved
against the host gateway's own launch home: a profile the host process merely SERVES owns
nothing.
BLOCKER 2 — the compensator and the flipped host lock contradicted each other. It restored
N per-profile gateways, which the lock now refuses by construction (it clears the served
record first, so each secondary is told to start normally and the flock loser exits 75 into
a respawn loop). It now restores exactly ONE gateway — the default's, on the recorded flag —
names the profiles it did not rebuild, verifies the gateway is actually live, and never
returns True after a start that produced nothing.
BLOCKER 3 — a stranded host could never recover: the resume branch sat behind the preflight
gate, so any new blocker made the only documented recovery ("re-run, it resumes") refuse
forever. The manifest is now read first and outranks the gate; the checks still run and are
printed as findings (SHOULD 5), because a resume compensates an already-destructive state
instead of initiating one.
BLOCKER 4 — an s6 container could boot with ZERO gateways: named slots are registered down
unconditionally, so an image only ever driven as `hermes -p X gateway start` started nothing
while every action reported "registered". The root slot now inherits any named slot's
autostart intent and the boot names the folded profiles.
Also: the SIGTERM disclosure is derived from what the apply actually signals (the default's
own gateway and unit-supervised secondaries with no readable pid were both undisclosed);
`_multiplex_profiles_enabled` / `default_gateway_multiplexes` no longer answer from the
retired `false`, which made CLI/dashboard report "standalone" while the runtime multiplexed;
and the ingress + secret-scope carve-outs of the retirement are documented.
main folded the auto-archive housekeeping into per-profile state.db maintenance; the plugin
update-check chore stays. The consolidated-away TestWorkdirParallelPool stays removed. Two new
utf-8 reads in gateway/host_rendezvous.py read utf-8-sig.
Three defects in the eager activation gate:
- profiles_to_serve() takes the multiplex flag as an ARGUMENT and never reads config, so
any host with >=2 profile dirs armed the fail-closed guard at boot — including hosts
that deliberately pinned gateway.multiplex_profiles: false. The gate now reads
GATEWAY_MULTIPLEX_PROFILES then config.yaml and skips activation when it is explicitly
false; the lazy backstop is unchanged.
- The snapshot was taken at the TOP of start_server, before keepalive / auth gate /
uvicorn build. It is the ONLY source for launch keys with no .env to rebuild from
(systemd Environment=, op run, Compose), so a credential injected or rotated by a later
boot step was invisible for the process lifetime. Activation moves to the last boot step.
- A failing probe returned False silently: the guard never armed and the caller's
diagnostic was dead code. It now logs at WARNING with exc_info and fails CLOSED — an
unreadable profiles/ cannot prove the host is single-profile.
Also: a profile dir holding only an EMPTY .env is what a crashed `hermes profile create`
leaves behind, and counting it flipped a single-profile host fail-closed at its next
boot. The gate counts named_profile_has_servable_identity instead.
One host process serves every profile, so every execution point must bind the
profile it is acting FOR. These six ran unscoped (or bound only part of a
scope) and resolved get_hermes_home()/credentials against the LAUNCH profile:
- tui_gateway/session_reaper: the idle-reaper and exit-flush transcript writes
now enter the SESSION's profile scope, the same chokepoint _finalize_session
already binds. A served profile's transcript was landing in the launch home.
- gateway/run: MCP shutdown tears down per served profile inside that
profile's scope (mirrors startup discovery and the reconcile chore), with a
trailing wildcard pass under the launch profile's own scope.
- gateway/run_profile_reconcile: _unserve_profile's adapter teardown, agent
eviction and state/memory handle release now run inside the deleted
profile's scope.
- gateway/run_adapters + run_goals + run_notifications: a body with no routed
profile no longer means "no scope". launch_profile_scope_if_multiplexed()
binds the launch profile once the process multiplexes; before activation it
is still literally a nullcontext, so single-profile hosts are unchanged.
- hermes_cli/kanban_db_dispatch: one _worker_profile_scope helper binds the
assignee's secret AND terminal scope for toolset resolution and the spawn-env
build, unconditionally instead of only under multiplex.
- hermes_cli/web_server: `hermes serve` activates multi-profile hosting at boot
when the host has more than one servable profile home, instead of lazily on
the first ?profile= request after earlier work already ran unscoped.
Secret scope is never widened: a non-launch home resolves from its own .env and
sources only; the launch home keeps its existing env-over-.env precedence.
Reviewer findings on the host rendezvous record + serve attach.
- Attach now PROVES the owner before exiting 0: a bounded TCP connect to the
recorded endpoint plus a token-authenticated GET /api/host/identity that must
answer as the recorded pid+role. A supervisor or `hermes update` relaunch
landing in the old process's graceful-shutdown window (socket closed, atexit
not yet run) previously exited 0 with NOTHING listening, so the service
reported success for a dead backend. Anything short of a proven owner falls
through to the bind.
- Unprovable liveness (no psutil, an unexpected psutil error) is a CANDIDATE,
not an owner: it goes to the same probe instead of exiting 0, which is what
turned a record for a long-dead pid into a permanent silent outage.
- An explicitly typed --port/--host the owner cannot serve is a non-zero refusal
naming the owner, never a silent loopback redirect; and a `hermes dashboard`
user is never routed to a headless `serve` backend (servesSpa in the identity
answer).
- SIGTERM — the NORMAL stop (systemd stop, docker stop, the update relaunch) —
now clears the record and the 0600 token and releases the host lock. atexit
does not run on it: uvicorn's capture_signals re-raises into the default
disposition, so a live session token outlived its process indefinitely. The
handler only prepends cleanup and hands off to the previous handler, leaving
the shutdown sequence unchanged.
- Host lock claim is tri-state (acquired / held-by-other / could-not-open) and
logs the OSError: an unwritable lock dir used to be reported as "another
gateway owns this host", sending operators hunting a process that never
existed. Its handle cache is keyed by (role, resolved lock path), so a changed
lock dir can no longer make owns_host_lock() lie.
- Windows: the token is written through the SSH runtime's protected
owner+SYSTEM DACL writer (os.open(0o600) sets no ACLs there, and os.replace
fails against an open reader); read_token's docstring no longer claims the
mode bits prove same-OS-user.
- gateway/status: a RELATIVE $XDG_STATE_HOME is ignored per the XDG spec — it
made the host lock dir CWD-relative, so two serves started from different
directories shared no singleton.
Live A/B (real processes): kill -TERM of a fully started `hermes serve` left
host-serve.json + host-serve.token on disk on base, removes both on head (exit
status -15 unchanged). A record with a LIVE pid and a closed port made base exit
0 with no bind; head binds. `serve --port 8899` against an owner on another port
exits 1 naming the owner.
Multiplex-only (Teknium ruling): exactly ONE `hermes serve` and ONE
`hermes gateway run` per host, each multiplexing every profile. The existing
gateway lock/PID files are anchored to HERMES_HOME, so N profiles were N
independently acquirable locks and `hermes serve` had no singleton at all.
- `gateway/host_rendezvous.py`: a host lock + a published record (pid,
creation time, port, protocolVersion, tokenFingerprint, served-profile set)
under the one cross-profile lock root already in the tree. A record whose PID
is dead, or alive with a different creation time (PID reuse), is stale and is
never attached to. Removed on clean exit.
- Gateway: takes the host lock ALONGSIDE the per-home lock and publishes the
record. Observe-only — a second gateway still starts, and logs the owner.
- Serve: publishes the record plus a 0600 token file on bind, and a second
`hermes serve` for ANY profile discovers it, prints the live pid/port and
exits 0 instead of binding a second port. The bare-TCP `_dashboard_listening`
probe (which proved only that *something* answers) is replaced by
record-based discovery with creation-time proof; the unified re-exec stays as
the no-record fallback. `--isolated` and HERMES_DESKTOP=1 opt out.
`spawn-ledger.json` keeps being written unchanged, so the merged Desktop attach
ladder (#117983) keeps working.
Follow-up to the cherry-picked #116014:
- Pin per the dependency policy (pre-1.0: `<0.(minor+2)`): httptools
`>=0.6.3,<0.9` (floor = uvicorn[standard]'s own floor), uvloop
`>=0.15.1,<0.24`. `watchfiles>=0.20,<2` already complied.
- Copy uvicorn's own uvloop marker (win32, cygwin, PyPy) plus
`sys_platform != 'android'` so `pip install '.[all]'` on those
hosts does not fail on the extra either.
- `tools/lazy_deps.py` mirrors the `web` extra for the lazy dashboard
install: it also requested `uvicorn[standard]`, so a Termux user
opening the dashboard would have hit the same uvloop build at first
use. The web_server install hint follows.
- `uv lock` regenerated; the lock delta is exactly the pyproject delta.
- Two invariant tests: no Termux-reachable extra (or core, or the lazy
dashboard feature) requests uvloop; `[all]` still does, off Android.
- Docs: troubleshooting entry in the Termux guide.
An unpinned cron job used to snapshot the global provider/model at creation and treat that
snapshot as its effective pin (#44585), so `hermes model` / `/model` never moved the fleet and
`hermes cron resnap` existed to catch jobs up. New ruling: jobs run on whatever the main agent
model is when they fire. Resolution is per-job pin > cron.model / cron.model_provider (the cron
fleet default) > model.default.
`pinned` replaces the implicit snapshot with an explicit lock: create/update with pinned=true
writes the CURRENT main provider+model onto the job as an ordinary per-job pin; pinned=false
releases both. The cronjob tool exposes it (schema: only when the user asks; it can only lock
the main model, never point spend at a different one) and reports `pinned` per job; the CLI
gets `--pin` / `--unpin`. Legacy records that still carry *_snapshot keys follow the main model.
Removed with the snapshot: `hermes cron resnap`, the tool's resnap action + `all` param, the
"N unpinned jobs keep running on ..." notice in `hermes model` / `hermes config set` / the
dashboard model assignment, and the Desktop cron-model-impact card (setMainModelAssignment
keeps the expensive-model confirm flow in store/model-assignment.ts).
Live A/B (real store + run_job against a temp HERMES_HOME): main-model X -> Y, unpinned job
fires on X before, Y after; pinned job stays on X; unpin -> Y; legacy snapshot record -> Y.
