fix(update): a Desktop-only host settles an inventory-less restart obligation (#120740)
* refactor(update): one predicate for serve rows outside the gateway matrix The inventory branch of `_marker_only_restart_obsolete` inlined the rule for which serve/dashboard rows the gateway matrix neither covers nor needs to (supervisor-owned, or a manual serve handed to its own reminder). The inventory-less branch needs the same rule for #118742, so it moves to `update_cmd_fleet_gatewayless.runtime_outside_gateway_evidence` and both branches will read one definition. No behaviour change. * fix(update): a Desktop-only host settles an inventory-less restart obligation A host that runs no gateway (the Desktop app alone) can be left with an inventory-less fleet-restart obligation: an updater that died before recording its inventory, or the pre-inventory writer. With no owed set, the live gateway matrix is its only evidence, and on that host the matrix is empty forever, so `_marker_only_restart_obsolete` never settled and every later `hermes update` exited 1 with "gateways are still off the checkout code" (#118742). An empty fleet alone cannot tell that host from one whose gateway the dying update stopped, so the inventory-less branch now asks the live host, never a historical receipt: `host_owes_no_gateway_restart` settles only when no profile's `gateway_state.json` claims a state other than stopped/startup_failed (a gateway that went away without a clean stop keeps the obligation) and every live runtime sits outside the gateway matrix (`runtime_outside_gateway_evidence`, shared with the inventory branch). HEAD must still contain the pulled SHA (`checkout_contains`, same rule as #119367). Probe failures keep it. Tests: the scoped-reconciliation matrix now holds its host at "the update stopped a gateway" so it keeps pinning receipt independence; a new host-evidence matrix covers Desktop-only, clean stop, carried commit, stopped gateway in a named profile, unclassified and unidentified serves, a gateway row without fleet identity, and a diverged checkout. Two manual-serve tests that assumed an empty fleet always stays pending now stub the live host and assert the manual reminder survives the gateway obligation settling. Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com> --------- Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
This commit is contained in:
Austin Pickett
GitHub
@@ -334,7 +334,10 @@ def _marker_only_restart_obsolete() -> bool:
|
||||
that died before its inventory was recorded, #115638) clears once every live gateway is
|
||||
current on the checkout — there is no recorded owed set, so the fleet running the code on disk
|
||||
is the whole of the evidence the marker's warning can be about, even after HEAD moved past
|
||||
``expected_sha`` by an out-of-band pull.
|
||||
``expected_sha`` by an out-of-band pull. With no live gateway at all, the inventory-less marker
|
||||
asks the host instead (``update_cmd_fleet_gatewayless``): it clears when no profile left a
|
||||
gateway that should be running and every live runtime is supervisor-owned or handed off, so a
|
||||
Desktop-only install stops failing every later update (#118742).
|
||||
|
||||
A serve/dashboard row whose supervisor owns the restart (Desktop backend, systemd/launchd
|
||||
unit, Windows service) is outside the gateway matrix's evidence, not evidence against it —
|
||||
@@ -347,7 +350,7 @@ def _marker_only_restart_obsolete() -> bool:
|
||||
phase never touched it — this marker only stops re-warning about it on every later startup.
|
||||
"""
|
||||
from hermes_cli.update_cmd_fleet_checkout import checkout_contains
|
||||
from hermes_cli.update_serve_obligations import defer_manual_serve
|
||||
from hermes_cli.update_cmd_fleet_gatewayless import host_owes_no_gateway_restart, runtime_outside_gateway_evidence
|
||||
|
||||
try:
|
||||
fields = _obligation_fields()
|
||||
@@ -366,10 +369,7 @@ def _marker_only_restart_obsolete() -> bool:
|
||||
for runtime in runtimes:
|
||||
if not isinstance(runtime, dict):
|
||||
return False
|
||||
if runtime.get("kind") in ("serve", "dashboard") and (
|
||||
defer_manual_serve(runtime)
|
||||
or runtime.get("supervisor") in _SUPERVISOR_OWNED_SERVE_BACKENDS
|
||||
):
|
||||
if runtime_outside_gateway_evidence(runtime):
|
||||
continue
|
||||
if runtime.get("kind") != "gateway":
|
||||
return False
|
||||
@@ -403,7 +403,20 @@ def _marker_only_restart_obsolete() -> bool:
|
||||
logger.debug("Fleet probe failed; keeping fleet-restart-pending marker: %s", exc)
|
||||
return False
|
||||
if not fleet:
|
||||
return False # Absence cannot prove recovery of the recorded inventory.
|
||||
if owed is not None:
|
||||
return False # Absence cannot prove recovery of the recorded inventory.
|
||||
# No recorded owed set and no live gateway: settle only when the host itself shows nothing
|
||||
# the update could still owe a restart to, and HEAD still holds the code it pulled (#118742).
|
||||
try:
|
||||
gatewayless = (checkout_sha == expected_sha or checkout_contains(expected_sha)) and host_owes_no_gateway_restart()
|
||||
except Exception as exc:
|
||||
logger.debug("Gateway-less host probe failed; keeping fleet-restart-pending marker: %s", exc)
|
||||
return False
|
||||
if not gatewayless:
|
||||
return False
|
||||
_clear_fleet_restart_pending_marker()
|
||||
logger.debug("Fleet-restart-pending marker discharged: host runs no gateway at %s", checkout_sha[:10])
|
||||
return True
|
||||
covered = _fleet_covered_gateways(fleet)
|
||||
if covered is None:
|
||||
return False # unidentified runtime: the matrix cannot vouch for it
|
||||
|
||||
49
hermes_cli/update_cmd_fleet_gatewayless.py
Normal file
49
hermes_cli/update_cmd_fleet_gatewayless.py
Normal file
@@ -0,0 +1,49 @@
|
||||
"""Gateway-less host evidence for the update-restart obligation (``update_cmd_fleet`` sibling).
|
||||
|
||||
An inventory-less obligation (the pre-inventory writer, or a tail that died before recording one)
|
||||
has no owed set, so the live gateway matrix is its whole evidence. On a host that runs no gateway
|
||||
(the Desktop app alone, #118742) that matrix is empty forever and the obligation could never
|
||||
settle. An empty probe cannot tell that host from one whose gateway the dying update stopped, so
|
||||
these readers ask the live host directly. A historical receipt is never consulted: it can belong
|
||||
to an older update and says nothing about what runs now.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import asdict
|
||||
|
||||
|
||||
def runtime_outside_gateway_evidence(runtime: dict) -> bool:
|
||||
"""A serve/dashboard row the gateway matrix neither covers nor needs to.
|
||||
|
||||
Its supervisor owns the restart (Desktop backend, launchd/systemd unit, Windows service), or it
|
||||
is a manual serve whose restart ``defer_manual_serve`` has handed to its own durable reminder.
|
||||
Unclassified backends and failed transfers stay evidence against settlement (#115090, #111494).
|
||||
"""
|
||||
from hermes_cli.update_cmd_fleet import _SUPERVISOR_OWNED_SERVE_BACKENDS
|
||||
from hermes_cli.update_serve_obligations import defer_manual_serve
|
||||
|
||||
return runtime.get("kind") in ("serve", "dashboard") and (
|
||||
defer_manual_serve(runtime) or runtime.get("supervisor") in _SUPERVISOR_OWNED_SERVE_BACKENDS
|
||||
)
|
||||
|
||||
|
||||
def host_owes_no_gateway_restart() -> bool:
|
||||
"""True when no profile expects a gateway to be running and every live runtime is outside the matrix.
|
||||
|
||||
A ``gateway_state.json`` that does not say ``stopped``/``startup_failed`` belongs to a gateway
|
||||
that went away without a clean stop, which is what an update that died mid-restart leaves
|
||||
behind; it keeps the obligation. Profiles that never ran a gateway have no record at all.
|
||||
"""
|
||||
from gateway.status import read_runtime_status
|
||||
from hermes_cli.update_inventory import collect_runtime_inventory
|
||||
from hermes_cli.update_receipt import _NOT_EXPECTED_STATES, _profile_homes
|
||||
|
||||
for _profile, home in _profile_homes():
|
||||
record = read_runtime_status(home / "gateway_state.json")
|
||||
if record is None:
|
||||
continue
|
||||
state = record.get("gateway_state") if isinstance(record, dict) else None
|
||||
if not (isinstance(state, str) and state in _NOT_EXPECTED_STATES):
|
||||
return False
|
||||
return all(runtime_outside_gateway_evidence(asdict(runtime)) for runtime in collect_runtime_inventory().runtimes)
|
||||
@@ -84,16 +84,17 @@ def test_historical_manual_obligation_does_not_block_healthy_gateway(monkeypatch
|
||||
monkeypatch.setattr("hermes_cli.update_cmd._current_checkout_sha", lambda: "new")
|
||||
monkeypatch.setattr(process_identity, "_pid_alive_matches", lambda *a: alive)
|
||||
monkeypatch.setattr(update_receipt, "collect_fleet_versions", lambda **k: [{"profile": "default", "state": "current", "code_sha": "new"}] if gateway_present else [])
|
||||
monkeypatch.setattr("hermes_cli.update_inventory.collect_runtime_inventory", lambda: UpdatePlan(runtimes=[runtime] if alive is not False else []))
|
||||
if marker:
|
||||
fleet._write_fleet_restart_pending_marker(expected_sha="new")
|
||||
# An inventory-less marker never inherits inventory from a historical receipt, but it
|
||||
# discharges when the live fleet provably serves its expected SHA (#115638).
|
||||
pending = marker and not gateway_present
|
||||
assert fleet._pending_fleet_restart_needed() is pending
|
||||
# An inventory-less marker never inherits inventory from a historical receipt. It discharges
|
||||
# when the live fleet provably serves its expected SHA (#115638), or when the host runs no
|
||||
# gateway and the manual serve has its own reminder (#118742).
|
||||
assert not fleet._pending_fleet_restart_needed()
|
||||
fleet._warn_pending_fleet_restart_on_startup()
|
||||
warning = capsys.readouterr().err
|
||||
assert ("serve [work] pid 900" in warning) is (alive is not False)
|
||||
assert ("hermes gateway restart" in warning) is pending
|
||||
assert "hermes gateway restart" not in warning
|
||||
assert json.loads((root / "latest.json").read_text()) == receipt
|
||||
|
||||
|
||||
@@ -108,13 +109,16 @@ def test_stamped_manual_only_history_has_no_gateway_obligation(monkeypatch, caps
|
||||
monkeypatch.setattr("hermes_cli.update_cmd._current_checkout_sha", lambda: "new")
|
||||
monkeypatch.setattr(fleet, "_current_checkout_sha", lambda: "new")
|
||||
monkeypatch.setattr(update_receipt, "collect_fleet_versions", lambda **k: [])
|
||||
monkeypatch.setattr("hermes_cli.update_inventory.collect_runtime_inventory", lambda: UpdatePlan(runtimes=[RuntimeRecord(**runtime)]))
|
||||
if marker:
|
||||
fleet._write_fleet_restart_pending_marker(expected_sha="new")
|
||||
assert fleet._pending_fleet_restart_needed() is marker
|
||||
# With no gateway on the host, the live manual serve carries its own reminder and an
|
||||
# inventory-less marker has nothing left to hold (#118742).
|
||||
assert not fleet._pending_fleet_restart_needed()
|
||||
fleet._warn_pending_fleet_restart_on_startup()
|
||||
warning = capsys.readouterr().err
|
||||
assert "serve [work] pid 900" in warning
|
||||
assert ("hermes gateway restart" in warning) is marker
|
||||
assert "hermes gateway restart" not in warning
|
||||
|
||||
|
||||
@pytest.mark.parametrize("manual_first", [True, False])
|
||||
|
||||
@@ -4,13 +4,24 @@ import json
|
||||
|
||||
import pytest
|
||||
|
||||
from hermes_cli import process_identity, update_cmd_fleet as fleet, update_receipt
|
||||
from hermes_cli import process_identity, update_cmd_fleet as fleet, update_inventory, update_receipt
|
||||
from hermes_cli.update_inventory import RuntimeRecord, UpdatePlan
|
||||
from hermes_constants import get_hermes_home
|
||||
import hermes_cli.update_host_obligation as host_obligation
|
||||
|
||||
MANUAL = {"kind": "serve", "profile": "work", "pid": 900, "supervisor": "manual-serve", "restart_via": "respawn-argv", "code_sha": "old", "detail": {"create_time": 1000.0}}
|
||||
CURRENT = {"profile": "alpha", "state": "current", "code_sha": "new"}
|
||||
GATEWAY = {"kind": "gateway", "profile": "alpha", "code_sha": "old"}
|
||||
DEAD_PID = 2**22 - 7
|
||||
|
||||
|
||||
def write_gateway_state(home, state):
|
||||
"""``gateway_state.json`` for a gateway whose pid is gone; ``state=None`` omits the field."""
|
||||
record = {"pid": DEAD_PID, "code_sha": "old"}
|
||||
if state is not None:
|
||||
record["gateway_state"] = state
|
||||
home.mkdir(parents=True, exist_ok=True)
|
||||
(home / "gateway_state.json").write_text(json.dumps(record))
|
||||
|
||||
CASES = [
|
||||
("receipt-successor", {"outcome": "failed", "plan": {"runtimes": [GATEWAY]}}, None, [CURRENT], False),
|
||||
@@ -37,6 +48,11 @@ def seed(monkeypatch, old, marker, live, alive=True):
|
||||
monkeypatch.setattr(fleet, "_current_checkout_sha", lambda: "new")
|
||||
monkeypatch.setattr("hermes_cli.update_cmd._current_checkout_sha", lambda: "new")
|
||||
monkeypatch.setattr(update_receipt, "collect_fleet_versions", lambda **k: live)
|
||||
# Hold the host constant at one that still owes a gateway (the update stopped it and nothing
|
||||
# replaced it), so an empty live fleet stays unproven and only the receipt varies. Hosts that run
|
||||
# no gateway settle on host evidence: test_gatewayless_host_settles_on_host_evidence.
|
||||
write_gateway_state(get_hermes_home(), "running")
|
||||
monkeypatch.setattr(update_inventory, "collect_runtime_inventory", UpdatePlan)
|
||||
if marker is not None:
|
||||
fleet._write_fleet_restart_pending_marker(expected_sha=marker)
|
||||
return target
|
||||
@@ -79,6 +95,62 @@ def test_empty_marker_never_inherits_receipt_ownership(monkeypatch, capsys, aliv
|
||||
assert target.read_bytes() == before
|
||||
|
||||
|
||||
DESKTOP = RuntimeRecord(kind="serve", profile="default", pid=901, supervisor="desktop", restart_via="desktop-respawn")
|
||||
DESKTOP_RECEIPT = {"outcome": "failed", "plan": {"runtimes": [{"kind": "serve", "profile": "default", "supervisor": "desktop"}]}}
|
||||
|
||||
# (name, receipt, live runtimes, gateway_state per profile, checkout, pending)
|
||||
GATEWAYLESS_CASES = [
|
||||
# #118742: Desktop app only, no gateway was ever installed.
|
||||
("desktop-only", {}, [DESKTOP], {}, "new", False),
|
||||
("nothing-running", {}, [], {}, "new", False),
|
||||
("gateway-stopped-cleanly", {}, [DESKTOP], {"default": "stopped"}, "new", False),
|
||||
("gateway-startup-failed", {}, [], {"default": "startup_failed"}, "new", False),
|
||||
# HEAD carries a local commit on top of the pulled SHA (#119367).
|
||||
("carried-local-commit", {}, [DESKTOP], {}, "hotfix", False),
|
||||
# Receipts neither discharge nor block: the old manual row is history, not the live host.
|
||||
("manual-receipt-gatewayless-host", {"outcome": "success", "plan": {"runtimes": [MANUAL]}}, [], {}, "new", False),
|
||||
("desktop-receipt-stopped-gateway", DESKTOP_RECEIPT, [DESKTOP], {"default": "running"}, "new", True),
|
||||
("named-profile-gateway-gone", {}, [DESKTOP], {"work": "running"}, "new", True),
|
||||
("state-record-without-state", {}, [], {"default": None}, "new", True),
|
||||
("unclassified-serve", {}, [RuntimeRecord(kind="serve", profile="default", pid=902, supervisor="manual")], {}, "new", True),
|
||||
("manual-serve-without-identity", {}, [RuntimeRecord(kind="serve", profile="work", pid=903, supervisor="manual-serve", restart_via="respawn-argv")], {}, "new", True),
|
||||
("gateway-runtime-without-fleet-row", {}, [RuntimeRecord(kind="gateway", profile="default", pid=904, supervisor="manual")], {}, "new", True),
|
||||
("checkout-diverged", {}, [DESKTOP], {}, "elsewhere", True),
|
||||
]
|
||||
|
||||
|
||||
@pytest.mark.parametrize("name,receipt,runtimes,states,checkout,pending", GATEWAYLESS_CASES, ids=[case[0] for case in GATEWAYLESS_CASES])
|
||||
def test_gatewayless_host_settles_on_host_evidence(monkeypatch, capsys, name, receipt, runtimes, states, checkout, pending):
|
||||
"""An inventory-less marker with no live gateway settles on what the host runs now (#118742)."""
|
||||
from hermes_cli.profiles import _get_default_hermes_home, _get_profiles_root
|
||||
|
||||
seed(monkeypatch, receipt, "new", [])
|
||||
(get_hermes_home() / "gateway_state.json").unlink()
|
||||
for profile, state in states.items():
|
||||
write_gateway_state(_get_default_hermes_home() if profile == "default" else _get_profiles_root() / profile, state)
|
||||
monkeypatch.setattr(update_inventory, "collect_runtime_inventory", lambda: UpdatePlan(runtimes=list(runtimes)))
|
||||
monkeypatch.setattr(fleet, "_current_checkout_sha", lambda: checkout)
|
||||
monkeypatch.setattr("hermes_cli.update_cmd._current_checkout_sha", lambda: checkout)
|
||||
monkeypatch.setattr("hermes_cli.update_cmd_fleet_checkout.checkout_contains", lambda sha: checkout == "hotfix")
|
||||
|
||||
assert fleet._pending_fleet_restart_needed() is pending
|
||||
assert host_obligation.host_obligation_path().exists() is pending
|
||||
fleet._warn_pending_fleet_restart_on_startup()
|
||||
assert ("hermes gateway restart" in capsys.readouterr().err) is pending
|
||||
|
||||
|
||||
def test_gatewayless_probe_failure_keeps_marker(monkeypatch):
|
||||
seed(monkeypatch, {}, "new", [])
|
||||
(get_hermes_home() / "gateway_state.json").unlink()
|
||||
|
||||
def unavailable():
|
||||
raise OSError("process table unreadable")
|
||||
|
||||
monkeypatch.setattr(update_inventory, "collect_runtime_inventory", unavailable)
|
||||
assert fleet._pending_fleet_restart_needed()
|
||||
assert host_obligation.host_obligation_path().exists()
|
||||
|
||||
|
||||
@pytest.mark.parametrize("consumer", ["predicate", "startup"])
|
||||
def test_reconciliation_uses_one_receipt_snapshot(monkeypatch, capsys, consumer):
|
||||
old = {"outcome": "partial", "plan": {"runtimes": [MANUAL]}, "fleet": []}
|
||||
|
||||
@@ -132,7 +132,7 @@ The same inventory is embedded in every real update's receipt (`~/.hermes/logs/u
|
||||
|
||||
Every `hermes update` run writes a machine-readable receipt to `~/.hermes/logs/update_receipts/` (last 20 kept, `latest.json` always points at the most recent): the pre-update fleet plan, each step taken, anything skipped and why, the gateway restart outcome, and the final fleet version matrix. The SQLite runtime repair is one of those steps (`sqlite_runtime_repair`): a failed repair records the actual reason (for example the `uv sync` error) and the SQLite version pair, a deferred or not-applicable repair lands in the skips with its reason. After the restart phase the updater compares each live gateway's running code against the freshly updated checkout and prints a per-profile matrix — a gateway still serving pre-update code is reported loudly with the exact restart command, and the update exits non-zero so automation never treats a mixed-version fleet as healthy. Both `--plan` and the fleet check ask each running gateway directly over its local control socket (`gateway.sock` in the profile's data directory, a named pipe on Windows) when available, so version and supervisor information comes from the gateway itself; gateways from older versions are still discovered through their state files as before.
|
||||
|
||||
A multiplexed default gateway is one process serving several profiles, so it appears once in the matrix and vouches for every profile in its `served_profiles` record. The same coverage clears the "A previous `hermes update` pulled new code but did not restart running gateways" hint: once that gateway (or, after a manual `git pull`, every gateway an update restarted) runs the current code, `hermes gateway restart` is enough — the hint no longer waits for the next `hermes update` to write a fresh receipt. The same is true of the restart obligation left by an update that died before recording which gateways it owed (or by an older updater that never recorded them): once every live gateway runs the current checkout, the obligation is retired and the hint stops. That obligation is recorded once per HOST, in the cross-profile rendezvous directory (`$HERMES_GATEWAY_LOCK_DIR`, else `$XDG_STATE_HOME/hermes/gateway-locks`) as `host-update-restart.json`, so every profile's CLI sees the same one: `hermes -p coder update` and `hermes -p writer update` restart the shared multiplexed gateway once between them, not once each. An obligation left behind by an older per-profile updater (`fleet_restart_pending` in one profile's Hermes home) is still read and cleared. An update whose pre-update plan found no gateway at all owes nothing and leaves no breadcrumb. A backend supervised by Desktop, systemd or launchd is restarted by its supervisor and never blocks this settlement; only a manual backend whose reminder could not be saved keeps the obligation open.
|
||||
A multiplexed default gateway is one process serving several profiles, so it appears once in the matrix and vouches for every profile in its `served_profiles` record. The same coverage clears the "A previous `hermes update` pulled new code but did not restart running gateways" hint: once that gateway (or, after a manual `git pull`, every gateway an update restarted) runs the current code, `hermes gateway restart` is enough — the hint no longer waits for the next `hermes update` to write a fresh receipt. The same is true of the restart obligation left by an update that died before recording which gateways it owed (or by an older updater that never recorded them): once every live gateway runs the current checkout, the obligation is retired and the hint stops. On a host that runs no gateway at all (the Desktop app alone), it is retired once no profile has a gateway that went away without a clean stop and every running backend is restarted by its own supervisor or has its own reminder. That obligation is recorded once per HOST, in the cross-profile rendezvous directory (`$HERMES_GATEWAY_LOCK_DIR`, else `$XDG_STATE_HOME/hermes/gateway-locks`) as `host-update-restart.json`, so every profile's CLI sees the same one: `hermes -p coder update` and `hermes -p writer update` restart the shared multiplexed gateway once between them, not once each. An obligation left behind by an older per-profile updater (`fleet_restart_pending` in one profile's Hermes home) is still read and cleared. An update whose pre-update plan found no gateway at all owes nothing and leaves no breadcrumb. A backend supervised by Desktop, systemd or launchd is restarted by its supervisor and never blocks this settlement; only a manual backend whose reminder could not be saved keeps the obligation open.
|
||||
|
||||
### Manual backend restart reminders
|
||||
|
||||
|
||||
Reference in New Issue
Block a user