Review feedback (@kvnloo): when the post-restart fleet probe has no row for a
planned gateway's profile, the incarnation fallback silently does nothing and
the tripwire output reads identically to a genuinely missed restart. Log the
missing evidence at debug so the next report is diagnosable:
No post-restart gateway evidence for profile 'coder' (planned pid 76508 via
launchd) — reconciling on restart bookkeeping alone
Also make the call-site comment name the snapshot dependency the evidence
rests on (the same note #110343 carries at its wiring point).
`match_runtime_outcomes()` can only bridge a planned gateway runtime and the
restart bookkeeping by SERVICE NAME, and the two disagree whenever a service
serves a profile its name does not encode:
- macOS: the root-home LaunchAgent `ai.hermes.gateway` honours the install's
sticky active profile (`hermes profile use <name>`), so the process declares
`profile: coder` on its control socket while its label encodes the install
root. `_gateway_service_matches_profile("coder", "ai.hermes.gateway")` is
False for every naming rule, so the planned runtime is reported
`unaccounted` → `restart.incomplete` → exit 1 after a restart that verified
clean (fleet row: new PID, new SHA). The leftover `fleet_restart_pending`
marker then warns on every later invocation.
- Linux: a hash-suffixed unit for a custom HERMES_HOME has the same shape of
problem (#110238), which the open name-matching PRs only paper over for the
default profile.
Reconcile gateways on incarnation evidence too — the rule serve/dashboard
runtimes already use via `stale_serve_pids`: the post-restart fleet snapshot
(control-socket identity per profile) is passed in as `live_gateway_pids`, and
a planned PID that is gone while a gateway answers for the same profile counts
as `restarted`. Requiring a successor keeps the tripwire honest: no successor,
or the planned PID still answering, stays `unaccounted` and still exits 1.
`down` rows are excluded from the evidence — they report the pre-restart PID.
Fixes#110333
A profiles/<name> symlinked to a separate Hermes install is inventoried by the
update plan, left alone by the restart phase, and already classified
'external' by the post-restart fleet matrix. Reconciliation had no such
outcome, so the gateway ended 'unaccounted' and every hermes update exited 1.
Feed the fleet's verified external gateway pids into match_runtime_outcomes:
an untouched gateway in that set is 'external', reported informationally and
excluded from the exit-1 tripwire. No fleet evidence keeps it unaccounted.
Fixes#120240
Review minors on #126179:
(a) update_cmd_fleet._marker_only_restart_obsolete: a fleet whose every
row is EXTERNAL (served from another checkout root) fell through the
row loop and cleared the marker with zero evidence about this checkout.
For the SHA-less, inventory-less record it is now treated like an empty
fleet and stays armed.
(b) update_cmd_windows._hermes_holder_subcommand stripped only double
quotes before the inline-bootstrap check, so a shlex.join'ed
runtime_command (`python -I -c '<bootstrap>' dashboard`, the shape
launchd ProgramArguments take through main_dashboard) was never
inventoried. Strip both quote kinds, as _is_entry already does.
Follow-up trim of the #125964 salvage:
- Armer falls back to _current_checkout_sha() (the identity the reader compares the fleet
against) instead of a new receipt helper; at arm time the receipt has no post_update yet.
- Reader: the SHA-less, inventory-less record follows the existing inventory-less rule
(live fleet all current on the checkout) with no extra pid-liveness gate, matching the
invariant the SHA-bearing inventory-less record already has. The gatewayless branch stays
fail-closed for an SHA-less record (checkout_contains("") is not evidence).
- Drop the obligation_fields pid plumbing and update_receipt helper; keep 2 tests:
discharge on live-fleet evidence, and the armer recording the checkout SHA.
A no-op `hermes update` (pre == post SHA) whose head capture failed armed the
host update-restart record with `expected_sha: ""` and no inventory. The reader
bailed out on the empty SHA before the live-fleet reconciliation, so the
"gateways may still be serving pre-update modules" warning could never clear
even when every live gateway was current on the checkout.
Reader (hermes_cli/update_cmd_fleet.py):
- An inventoried obligation without its SHA stays fail-closed.
- An inventory-less, SHA-less record now falls through to the existing
live-fleet check once the process that armed it is gone (pid liveness via
pid_exists_stdlib), so a status call cannot discharge a restart phase that
is still running.
Armer (hermes_cli/update_cmd.py):
- When the completion request carries no SHA, fall back to the receipt's
post_update.sha (then pre_update.sha) instead of arming with "".
obligation_fields() now reports the arming pid alongside expected_sha.
Tests: discharge on all-current fleet, kept when the armer pid is alive, kept
on a stale fleet, kept with an inventory but no SHA; the discharge test fails
on the pre-fix reader with the issue's exact warning.
When the post-update dashboard respawn raised, the receipt derived the
runtime outcome from the incarnation liveness probe: the pre-update pid
being gone is exactly what a failed respawn looks like from the outside,
so the row read "restarted" and the overall receipt read "success".
The truth already existed on the producer side (the respawn path returns
the argvs that failed to spawn) and was dropped at the call site.
Thread it through: _restart_killed_backends now returns the PIDs whose
argv respawn was attempted and failed, _finish_dashboard_update_cleanup
returns them to both update paths, and match_runtime_outcomes checks
failed_respawn_pids before the incarnation probe — "we watched the spawn
fail" outranks "gone". Reuses the existing "failed" vocabulary (the unit
path already emits it) and escalates the receipt to "partial" like the
unaccounted-tripwire class. See #109290.
The systemd unit pass only listed and restarted hermes-gateway*/hermes-serve*
units, never hermes-dashboard*, although the runtime-outcome reconciliation
already credits hermes-dashboard{,-<profile>} unit restarts. A systemd-
supervised dashboard therefore stayed on pre-update code after every update:
the receipt recorded outcome 'deferred', nothing ever restarted it, and the
update still exited 0 (success). Same blind spot in the abort-recovery serve
pass, which enumerated hermes-serve* only.
Extend the unit list glob, the name gate and the recovery pass regex to the
hermes-dashboard* family (exact base unit or hyphenated profile, same strict
shape that rejects hermes-server.service). SIGUSR1 drain eligibility is
unchanged: dashboard units do not run gateway/run.py and keep going straight
to the blunt restart, like hermes-serve*.
Fixes#125297
Follow-up to the salvaged restart-wait commits:
- A drain or cron timeout of .inf now means "wait indefinitely" instead of
an OverflowError from the integer stop envelope, which crashed
`hermes gateway restart` and made `hermes update` silently fall back to
its 45s floor. The fleet "draining (up to Ns)" lines and the drain
progress report format the budget instead of int()-ing it, so an
unbounded wait no longer crashes them either (it did on main too).
- cron_drain_timeout is required: a 0.0 default meant "cron opted out",
the under-budget this fix exists to remove.
- Docstrings describe what the budget actually covers (PID exit, not
replacement startup).
- Tests assert the observer outlasts after-turn + the supervisor stop
envelope and that configured cron reaches the CLI wait, instead of
re-deriving the formula; the negative wording assertion on the pending
footer is dropped (change-detector).
The serve a remote Desktop spawns over SSH has no local spawner, so the
inventory read it as manual-serve: an update filed a manual-restart
reminder nobody on this host can discharge, reported the stale process
as unaccounted, and abort recovery could try an argv respawn without the
client's token file and owner nonce.
Classify it as desktop-ssh (using the canonical argv predicate, so rows
written before the ledger carried isolated are covered too) and treat it
like the local Desktop's own serve: skipped by the restart phase,
deferred to its client, never owed by abort recovery. A hand-started
serve --isolated stays manual-serve.
The fleet-restart-pending discharge parsed the marker inventory and
probed a gateway-less host inline (CC 36). _marker_owed_gateways now
turns the inventory into the owed set (raising ValueError, which the
caller's existing handler already maps to "keep the marker"), and
_discharge_gatewayless_marker owns the #118742 host probe. Every
verdict is unchanged; the caller is at CC 21.
- tools/browser_tool_install.py: keep pm-clean's frozen old-updater stub; main's
UTF-8 decode fix touched only the npx prefetch body it replaces.
- tests/hermes_cli/test_update_scoped_reconciliation.py: keep pm-clean's test
subset (catch-up rides the PM completion owner) and take main's gateway-less
host evidence (#120740): the updated seed that holds the host at a running
gateway, and the two gateway-less matrices for the source change that merged
cleanly into update_cmd_fleet.py.
* refactor(update): one predicate for serve rows outside the gateway matrix
The inventory branch of `_marker_only_restart_obsolete` inlined the rule for which
serve/dashboard rows the gateway matrix neither covers nor needs to (supervisor-owned,
or a manual serve handed to its own reminder). The inventory-less branch needs the same
rule for #118742, so it moves to `update_cmd_fleet_gatewayless.runtime_outside_gateway_evidence`
and both branches will read one definition. No behaviour change.
* fix(update): a Desktop-only host settles an inventory-less restart obligation
A host that runs no gateway (the Desktop app alone) can be left with an inventory-less
fleet-restart obligation: an updater that died before recording its inventory, or the
pre-inventory writer. With no owed set, the live gateway matrix is its only evidence, and
on that host the matrix is empty forever, so `_marker_only_restart_obsolete` never settled
and every later `hermes update` exited 1 with "gateways are still off the checkout code"
(#118742).
An empty fleet alone cannot tell that host from one whose gateway the dying update stopped,
so the inventory-less branch now asks the live host, never a historical receipt:
`host_owes_no_gateway_restart` settles only when no profile's `gateway_state.json` claims a
state other than stopped/startup_failed (a gateway that went away without a clean stop keeps
the obligation) and every live runtime sits outside the gateway matrix
(`runtime_outside_gateway_evidence`, shared with the inventory branch). HEAD must still
contain the pulled SHA (`checkout_contains`, same rule as #119367). Probe failures keep it.
Tests: the scoped-reconciliation matrix now holds its host at "the update stopped a gateway"
so it keeps pinning receipt independence; a new host-evidence matrix covers Desktop-only,
clean stop, carried commit, stopped gateway in a named profile, unclassified and
unidentified serves, a gateway row without fleet identity, and a diverged checkout. Two
manual-serve tests that assumed an empty fleet always stays pending now stub the live host
and assert the manual reminder survives the gateway obligation settling.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
---------
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
`_marker_only_restart_obsolete` bailed out with "a newer pull moved HEAD" whenever
`checkout_sha != expected_sha`, and otherwise held the fleet to `expected_sha` exactly.
A cherry-picked hotfix on top of the pulled SHA moves HEAD without any pull arming a
fresh obligation, so the recorded one could never discharge: every CLI start warned
and every no-op `hermes update` exited 1 through the armed veto in
`_pending_fleet_restart_needed`, while the gateway verifiably served HEAD.
The bail-out now fires only when HEAD no longer CONTAINS `expected_sha`
(`checkout_contains`, `git merge-base --is-ancestor`, fail-closed on any probe
failure), and the live fleet is held to the checkout SHA — the code it actually runs.
A gateway on genuinely stale code still keeps the obligation armed.
Closes#119367
`hermes update --yes` from a cron job inside the gateway takes the #100179
ancestor branch (`_drain_or_signal_gateway_for_update` -> self-restart request,
fire-and-forget) — the gateway can only restart after the updater exits. The
post-restart fleet matrix then saw that same gateway on the pre-update code_sha,
printed STALE + "Update not complete" and exited 1 on every nightly run (#119597).
The restart phase now records the pids that accepted a self-restart
(`_GatewayRestartOutcome.self_restart_pending_pids`, threaded from the systemd,
manual and launchd paths) and `collect_fleet_versions(self_restart_pending=)`
turns exactly those rows into `restart_pending` — rendered as "restart pending
(deferred until this process exits)" and excluded from the stale_or_down
verdict. Every other stale/down gateway keeps its verdict; the same pid without
the recorded acceptance is still STALE. The receipt keeps the row's old sha, so
the next `hermes update` / CLI startup hint still verifies the restart landed
via the live fleet (`_receipt_reports_stale_runtime` -> `_live_fleet_covers_receipt`).
Closes#119597
hermes-gateway*/hermes-serve* units, ai.hermes.gateway* LaunchAgents and
`gateway run` processes are account-wide namespaces shared by every Hermes
install on the box. The restart phase enumerated them by name, so a scratch
home's `hermes update` drained and restarted the account's real
hermes-gateway.service and SIGTERMed sibling installs' gateways (live: #93349
comment, 2026-09-23), then warned "Fleet version check returned no rows"
because none of those runtimes belonged to the updating home.
Ownership is now judged from what a runtime actually runs on, never from the
label: the live process environment (`_hermes_home_for_pid`), the unit's
declared Environment=HERMES_HOME, or the plist's pinned HERMES_HOME, compared
against the homes the update's plan inventories (the updating root and its
profiles/<name>). Foreign or unreadable ownership is named in the output and
left alone; it is not a failed restart. Applies to the systemd fleet loop, the
catch-up best-effort restart, the launchd derived-label loop, the manual
gateway sweep, and the pre/post-restart PID snapshots that drive the
fail-closed verdict.
Update stale test assumptions for PM install hints and terminal heartbeat,
keep Unix socket fixtures below AF_UNIX path limits, and remove timing and
host-process races from gateway tests. Read systemd units with BOM-tolerant
UTF-8 so the Windows footgun scan remains clean.
A pre-2026.7 `Restart=on-failure` system unit has no
RestartPreventExitStatus=78, so a permanent refusal restarted it ~180
times while the regenerated user units parked (#118282). The gateway
refreshes its USER unit at boot (`gateway run` -> refresh_systemd_unit_if_needed
(system=False)) but nothing ever rewrote an installed SYSTEM unit unless
the operator ran `sudo hermes gateway install|start|restart --system`.
The update-time fleet restart is the only routine contact with that unit:
as root it now regenerates the stale unit before the drain; without root
it names the exact repair command. A unit that already parks on 78 is
left alone. HERMES_HOME is restored after the refresh so the rest of the
update keeps running for the invoking profile.
Closes#118282
Every update route now finishes through update_completion._complete_selected,
which restarted the whole fleet unconditionally -- including the "Already up to
date" route that main sent through the pending-restart catch-up. Net effect:
each cron tick and each profile's `hermes update` drained and re-killed the one
multiplexed gateway.
Port the catch-up path's two live guards into the completion tail:
- host_restart_already_completed(checkout sha): a sibling profile attaches to
the restart this host already stamped (#95294); the restart phase now stamps
it via mark_host_restart_completed.
- every planned runtime AND every live fleet row current at the checkout sha
(#117051, d6b0d37ece). Both are required: the live matrix lists gateways
only, so a planned serve still on pre-update code keeps the restart.
The already-current route also arms the host obligation with the checkout sha;
an SHA-less arm replaced the standing record and wiped the restarted proof.
Host-scoped update-restart obligation (061195fac1 / 953b6f6f08 / 3be255eca6)
lands on the PM model: the obligation record, its readers and the legacy
per-home marker compat come in as-is. The catch-up restart path
(`_apply_pending_fleet_restart_catchup` / `_run_pending_fleet_restart`) is
retired here (the fleet restart rides the completion owner), so main's
per-host restart-once guard on that path is not carried; its unit→live
MainPID collapse IS ported into the live post-update systemd pass
(`_restart_systemd_gateway_units`), with the two collapse tests rewritten
against that function (red on the pre-port tree: `_unit_main_pid` absent).
Tests that exercised only the retired catch-up path are dropped.
Desktop: main's shared log-rotation planner replaces the inline constants
in main.ts; the merge keeps our machine-profile import beside its import.
utf-8 → utf-8-sig on the three new BOM-intolerant reads (footguns lint).
Review findings on the host-scoped update→restart obligation.
- update_cmd_fleet: an unwritable host state dir (read-only HERMES_GATEWAY_LOCK_DIR,
container UID that does not own $HOME) made write_host_obligation return False and the
caller ignored it, so an interrupted update left ZERO obligation — stale code, no
warning, no catch-up restart. The return is now propagated: the legacy per-home marker
(still read by every reader) carries it, and a host that can write neither says so.
- update_cmd_fleet::_obligation_fields: a PRESENT but unparseable/foreign-versioned host
record no longer falls through to the legacy marker; terms nobody can read cannot be
discharged by another record's terms.
- update_cmd_fleet::_restart_identity_sha: zip/pip/Docker installs resolve no checkout
SHA, so the restart-once stamp was "" and could never match — every profile's update
re-killed the one shared multiplexer. Falls back to the record's expected_sha, then the
receipt's post-update identity.
- update_host_obligation: any main_pid probe error is unproven identity (keep its own
restart), never an aborted restart pass.
- run_notifications: the online notice dedupes per home CHAT, so two served profiles
sharing one chat get one message (accounting stays per profile); transport resolution is
isolated per profile, so one broken adapter no longer starves the rest of the fan-out.
- run_adapters / run_profile_reconcile: _profile_configs is pruned with the served set, so
a failed or removed profile no longer owes a notice nothing can deliver and
.restart_pending.json is unlinked.
Tests cover the new format's own hazards: unwritable record dir, foreign-version record
with a legacy marker present, non-git install, shared home chat, broken adapter, pruned
config, plus a parity test for the duplicated host-state-dir resolver.
One host runs one multiplexing gateway, but the update pipeline still treated
the pull->restart obligation, enumerated units, recovery payloads and the
planned-restart notice as per-profile. Two profiles updating meant two outages
of the same process, and a served profile's channels were never told.
- hermes_cli/update_host_obligation.py: new host-scoped obligation record in
gateway.host_rendezvous.host_state_dir() (host-update-restart.json), plus the
unit->live-MainPID collapse rule. The legacy per-home marker stays readable
and clearable so an in-flight obligation is still discharged.
- update_cmd_fleet: arm/clear/read the host record; the catch-up restart is
idempotent per host (a completed restart onto the checkout SHA is never
repeated); leftover per-profile units resolving to one MainPID restart once.
- update_restart_recovery: payload profiles served by one host process are one
restart target, reported under "covered".
- gateway notices: owed targets and the online notice span every served
profile's home channels; the marker survives until each was reached.
After `hermes update` fast-forwards the checkout under a running multiplex gateway,
its cron ticker yields every tick ("stale code: booted on A, disk is at B") for as
long as the process lives. Two things made that a silent total dispatch outage:
* `hermes cron status` weighed only the liveness heartbeat (kept fresh by the yielding
loop) and the success marker; with no success marker on disk it printed
"✓ Gateway is running — cron jobs will fire automatically", and with a stale one
it pointed at "Check the gateway log" instead of naming the cause. The persisted
`CronTickYielded` error is now recognised (`cron.scheduler.stale_code_yield_labels`)
and reported as "Gateway is running STALE code — fires NOTHING", with both
revisions and the restart command. A fresh heartbeat with a recorded error and no
success marker is no longer green either.
* The post-update fleet version matrix flagged a `stale` gateway and exited 1, but
left it running. `_verify_fleet_after_update` now hands every proven-stale survivor
to the existing drain-first `request_restart` path (SIGUSR1) via
`hermes_cli/update_cmd_stale_survivors.py`: a supervised gateway respawns on the new
code, a bare `gateway run` is stopped and listed under "Restart manually" — the same
contract the restart phase already uses for unmapped manual gateways. The drain
budget computation is shared (`_gateway_drain_budget`).
The yield itself is unchanged: a stale-code process still never dispatches while a
fresher lock holder exists (design of 9a7732b45f).
Fixes#117275
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
`hermes update` on macOS printed "✓ Restarted ai.hermes.gateway" for the
profile running the update whenever launchd reported *any* pid for the label —
including the pre-update process the restart was supposed to replace.
f29ee96dd3 closed this for SIBLING labels only
(`_wait_for_launchd_service_pid(label, old_pid=old_pid, ...)`); the invoking
profile kept `wait_for_launchd_gateway_supervision` →
`_launchctl_label_supervising_process`, a predicate that is true for "launchctl
list exits 0 and some positive pid" and never compares against the pid seen
before the restart.
- `_launchctl_supervised_pid()` exposes the pid `_launchctl_label_supervising_process`
already parsed; the boolean is now a thin wrapper over it.
- `wait_for_launchd_gateway_supervision(..., old_pid=...)` accepts a fresh pid
only, mirroring the sibling loop's contract. `old_pid=None` keeps the old
"any supervised pid" meaning for callers with no pre-restart observation.
- `_restart_launchd_gateway_after_update()` snapshots the pid before
`launchd_restart()` and passes it in; the failure line now says launchd is
not supervising a NEW process. The snapshot is verification-only, so the
no-`launchctl list`-gating invariant of #74973 is untouched.
Also from round-2 review of this PR:
- the supervised-serve discharge test parametrized 5 supervisors, 3 of which
the inventory writer can never put on a serve/dashboard row; cut to the two
it can emit and documented the parity-only members.
- `_marker_only_restart_obsolete` now names who still owns a discharged
launchd-supervised serve (update-time `report_unaccounted_runtimes`, exit 1).
A `fleet_restart_pending` marker records the pre-update plan's runtimes as its inventory,
including `serve`/`dashboard` rows. `_marker_only_restart_obsolete` rejected every non-gateway
inventory row outright, so on any host that runs a dashboard or the Desktop `serve` backend the
marker could never discharge: `hermes version` kept printing "a previous `hermes update` ... did
not restart running gateways" after every gateway was already current on the pulled SHA.
Receipts already draw this boundary (`_receipt_owed_gateways`, #115090) and so does the restart
phase for the Desktop backend (#111494); the marker path was the one place left counting a
supervisor-owned serve row as evidence against the gateways it does not cover. A manual-serve row
still needs its durable handoff and an unclassified backend still fails closed.
Also corrects `_owed_stale_serve_rows`'s docstring, which justified the Desktop exclusion by the
"armed forever" consequence this change removes.
The catch-up rewrote the receipt's fleet matrix before checking whether that
actually cleared the debt, so an exit-1 run (marker still owes a gateway,
blocked serve_restart_pending storage) left latest.json mutated —
tests/hermes_cli/test_update_scoped_reconciliation.py pins the receipt as
byte-identical on the failure path. Decide on the in-memory settled copy and
write only when the pending restart is discharged (#117051).
A failed update receipt whose plan rows cannot be matched to a live gateway
(unknown profile identity, pre-pull SHAs, empty fleet matrix) could never be
discharged: every up-to-date `hermes update` ran the pending restart, printed
"Pending fleet restart completed." followed by "Fleet restart incomplete",
exited 1 and wrote another failed receipt — and every CLI start kept warning
about mixed sys.modules, even after a manual `hermes gateway restart` had put
the fleet on the checkout code (#117051, atoms 3/4).
In _apply_pending_fleet_restart_catchup, when the restart succeeded but the
receipt still owes gateways, probe the live fleet; if every row is current at
the checkout SHA, persist that matrix as latest.json's post-restart `fleet`
(update_receipt.settle_latest_receipt_fleet) so _receipt_reports_stale_runtime
and the startup warning see the recovery. The "incomplete" line now only
prints when the restart itself failed; the residual case names what is still
off the checkout code instead of contradicting the completion line.
Fixes#117051
The catch-up restart stopped every gateway before re-checking whether any was actually on
pre-update code, so the gateway the same `hermes update` had just cold-started was killed
and (on Windows) the stop/start pair printed "No gateway was running" followed by a second
spawn. _run_pending_fleet_restart now skips when the fleet probe shows every live gateway
current on the checkout SHA (identity known; any stale/down/unknown row still restarts).
The Windows direct-spawn report no longer prints the PID twice ("(PID n) (PID: n)").
Part of #117051
Conflicts resolved toward the PM model: main's lazy_deps/update_cmd_deps/npm
stamp machinery stays deleted (PM + scripts/build/node-deps.mjs own it), the
systemd ExecStop stop-mark rides the installation launcher, legacy
linux_only/macos_only/windows_only markers are rewritten to platforms(), and
finalize_update_receipt carries pending manual-serve obligations forward
again (lost when the ContextVar receipt rewrite crossed c0aa3ce354).
Test harness: the real-home I/O guard exempts /proc/<pid>/fd metadata reads
(deleted-WAL holder scans) and run_tests.sh drops ~/.hermes PATH entries so
shutil.which() cannot trip the tripwire.
A KeepAlive LaunchAgent backend's recorded ledger spawner (the bootstrap
shell) is long dead, so _collect_ledger_runtimes read the row as
manual-serve: the plan proposed a respawn-argv restart that fights the
job's own KeepAlive, and a stale survivor was reported as "manual" with
no kickstart hint.
Classify via the existing loaded-job matcher
(_loaded_launchd_backend_jobs / _launchd_job_owning_backend) before the
spawner probe, record the job's domain/label in the row detail, keep the
recovery-partition skip reason and the receipt's gateway-coverage matrix
consistent with the new supervisor value, and name the launchctl
kickstart command in the stale-survivor warning on macOS.
Units labelled before the profile-name suffix scheme (ai.hermes.gateway-<hash>
from the historical _profile_suffix) are invisible to the label derivation, so
the macOS update restart pass left them on pre-update code with no warning.
legacy_launchd_labels_for_install() credits such a plist only when attribution
to THIS install is provable — its ProgramArguments run this install's venv
python, or its pinned HERMES_HOME is one of this install's homes — keeping the
#41403 boundary (a sandboxed HERMES_HOME never touches another install's
fleet). Unreadable or unattributable plists are skipped fail-closed.
Fixes#115254
Two further dead ends in the fleet_restart_pending class that the salvaged
commits left open:
* An inventory-less marker whose expected_sha is no longer HEAD. Every pull
that does not come from `hermes update` (deploy scripts, `git pull` by hand,
a patch-preservation wrapper) skips both the arm and the clear, so a marker
left by an old update survives for weeks with an expected_sha the checkout
will never be at again; `checkout_sha != expected_sha` kept it pending
forever (#115638). Such a marker records no owed set, so the fleet being
current on the checkout is the whole of the evidence its warning can be
about: settle it against checkout HEAD instead. Inventoried markers keep the
moved-HEAD gate — their owed set belongs to the SHA they recorded.
* `_receipt_owed_gateways` returned None for any serve/dashboard row that was
not a manual-serve row already handed to its reminder — so a host running a
Desktop-hosted, systemd, or launchd `hermes dashboard`/`serve` carried a veto
in every receipt and the gateway warning could never be discharged by
evidence (#115090). Those backends are their supervisor's to restart and say
nothing about gateway coverage; only an unclassified backend or a failed
manual transfer still makes coverage unverified.
Tests: the empty-inventory shape moves out of the stays-pending list
(#115311), the identity test's "marker" case gets the inventory its comment
describes (an owed identity at a SHA the fleet does not serve), and the
plan-order test's unsupported row becomes a genuinely unclassified serve
row rather than a Desktop-hosted one.
A pre-update plan that records zero runtimes (Desktop-hosted `serve` with no
gateway services, or a fleet of only manually-deferred serve/dashboard
processes) armed `fleet_restart_pending` with an empty inventory. Nothing
navigates the marker's fail-closed discharge path for an empty `owed` set —
`_marker_only_restart_obsolete` returned False before ever probing, so every
later `hermes update` on the host reported "Fleet restart incomplete" and
exited 1 with no gateway to restart.
- `_write_fleet_restart_pending_marker`: never arm a marker for an explicit
empty inventory (`runtimes=[]`); `None` still arms the conservative
"unknown obligation" breadcrumb.
- `_marker_only_restart_obsolete`: build the owed set before the SHA/probe
gates; an inventory owing nothing is discharged and cleared immediately,
without a fleet probe. Inventories that DO owe restarts keep the existing
fail-closed probe-until-proven-current path; legacy/malformed inventories
stay fail-closed.
Regression tests: marker not armed for `runtimes=[]`; pre-fix empty-inventory
marker discharged without touching the fleet probe; an already-up-to-date
`hermes update` carrying the legacy marker exits 0 with no restart run.
Fixes#115311
_marker_only_restart_obsolete() stayed fail-closed forever when the marker
carried no inventory= line (legacy markers, or a pull that raced a missing
pre-update plan), so every later CLI call printed the interrupted-update
warning even after the fleet provably served expected_sha (#115638).
An inventory-less marker records no owed set, so it now settles on the same
live-fleet bar as an inventoried one minus the owed-coverage check: at least
one row, every row a current gateway under a known profile at expected_sha,
checkout not moved past the marker. Malformed or unsupported inventories stay
fail-closed, as do empty probes and stale/down rows.
Tests: discharge + stale-keep for inventory-less markers; legacy-marker arms
of the scoped-reconciliation matrix now pin live-evidence discharge with the
receipt left intact; explicit inventory=null settles the same way.
`_update_owes_fleet_restart` held a receipt whose restart phase completed to exactly
`post_update.sha`. A gateway restarted afterwards onto a checkout moved by hand runs code
NEWER than the update pulled, so the remedy the warning itself names (`hermes gateway
restart`) could not clear it until the next `hermes update` wrote a fresh receipt
(#113350 steps 3-4). A completed update is discharged when every owed gateway serves the
code it pulled OR today's checkout; the catch-up predicate `hermes update` runs is
unchanged.
`_live_fleet_covers_receipt` and `_marker_only_restart_obsolete` each looped the fleet
twice to fold `served_profiles` into the covered identities and re-validated a shape
`_fleet_row` already enforces. One `_fleet_covered_gateways(fleet)` answers the
`(kind, profile)` set (or None for an unidentified row) for both.
Keep the pending marker until a fresh fleet observation covers its owned gateway inventory. Transfer identified manual serve obligations to durable reminders so an already-verified gateway restart does not keep asking for another restart. Failed transfers retain the marker.
Pin the non-deferred alpha/beta gap and verified-restart-survival cases, including failed probes, storage recovery, and current-successor controls. Both regressions fail without the production change.
Reported-by: ehz0ah
Reported-by: cadamec
Manual serve/dashboard runtimes with no restart mechanism kept re-arming fleet_restart_pending: every CLI start warned about a completed update. This keeps the false alarm out while never hiding a real obligation: reminders are keyed by (pid, create_time), survive receipt rotation and gateway recovery, and clear only when the incarnation is provably gone or explicitly handed off. The pending marker owns its runtime inventory from the moment it is written; settlement derives coverage only from that inventory, so an older receipt can never discharge a newer update's obligation. Markers without an inventory stay pending. Startup keeps the completed-restart evidence path: a receipt whose restart phase finished vouches for the fleet at its post-update SHA.
Reported-by: adamkrawczyk
Diagnosis credit: KoNit-K (#107237)
Mechanism findings: g3org3yo, kokhlo, fmercurio
Reported-by: andrexibiza (P1 generation ownership)
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
`hermes` printed "A previous `hermes update` pulled new code but did not restart
running gateways" on every launch after an update whose receipt was `failed` for a
post-restart crash: `fleet` was empty (the matrix never ran), so the check fell back to
the pre-pull `plan.runtimes[].code_sha` against today's checkout and the live gateway,
correctly restarted onto the pulled commit, still read as "not restarted" once a manual
`git pull` had moved the checkout again. Reproduced on the maintainer's box against the
#112604 receipt: gateway on the update's `post_update.sha`, checkout two pulls ahead.
The startup hint now holds a receipt whose restart phase completed (`gateway_restart`
present, `incomplete` false, no `phase_error`) to the code it actually pulled: every owed
gateway serving `post_update.sha` discharges the update. Later drift is
`gateway/code_skew.py`'s job, not a warning that misreports the update. The catch-up
predicate `hermes update` itself runs keeps its checkout semantics (the fleet must reach
HEAD), so the same box still gets its gateway restarted on the next update.
`_live_fleet_covers_receipt` compares stamped `code_sha` against the expected sha for
`current` AND `stale` rows (both labels are relative to the checkout, not to what the
update owed); `unknown`/`down` rows still never cover.
