hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

fix(update): restart all macOS launchd gateways on hermes update

Browse Source 
The macOS branch of the update's fleet-restart step only restarted the
invoking profile's LaunchAgent. Sibling ai.hermes.gateway-<profile>
services kept pre-update modules cached in sys.modules and died on their
next agent turn (ImportError on new lazy imports, or TypeError/
AttributeError with garbled tracebacks on wider version gaps). The
systemd branch already iterates every hermes-gateway* unit; this brings
launchd to parity:

- _restart_macos_launchd_gateways(): the invoking profile keeps the
  existing launchd_restart() path; every other gateway of this install
  is drained via SIGUSR1 (same as systemd siblings), then hard-
  kickstarted unless KeepAlive already respawned it, then verified on a
  fresh PID. TimeoutExpired is isolated per label (#68523 parity) and
  counts toward failed_or_stale_units — including timeouts during
  liveness discovery, which must not read as "unloaded".
- Install-scoped fleet enumeration: launchd_gateway_labels_for_install()
  derives labels from THIS install's profiles (get_default_hermes_root),
  not by globbing the shared per-user ~/Library/LaunchAgents — a
  sandboxed HERMES_HOME (tests, capture sandboxes, side-by-side
  installs) must never enumerate, let alone restart, another install's
  fleet. This also keeps the hermetic test suite blind to a dev
  machine's real gateways.
- Domain-explicit sibling handling via _locate_launchd_gateway_service():
  liveness, kickstart, and fresh-PID verification all use the domain the
  service was actually located in (gui/<uid> vs user/<uid> probed per
  label via `launchctl print`). This addresses the #41403 review defect:
  the process-wide _launchd_domain() cache resolves the current profile's
  domain and must never be reused for a sibling. _launchd_domain() itself
  becomes a thin caching wrapper; behavior unchanged.
- _get_service_pids(all_profiles=...): the update path's manual-process
  sweep excludes every gateway service PID (mirror of the systemd
  hermes-gateway* pattern) so it cannot mistake a freshly respawned
  sibling service for a stale manual gateway. Default-scope callers
  (gateway status, cron checks, stop_profile_gateway's orphan reaper —
  which kills what it is fed) keep the current-profile-only contract.
- _warn_incomplete_gateway_fleet_restart() prints launchctl recovery
  hints for launchd labels alongside the systemctl ones.

Supersedes and completes #41403, addressing its review feedback
(per-label domain resolution + mocked regression tests).

Co-authored-by: David Neyra <vyr.agent@vyrgs.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
PT
2026-07-28 11:24:15 -07:00
committed by Teknium
parent ef04d846e9
commit f29ee96dd3
3 changed files with 983 additions and 99 deletions
Download Patch File Download Diff File Expand all files Collapse all files

292
hermes_cli/gateway.py
View File

@@ -107,11 +107,14 @@ def _get_service_pids(all_profiles: bool = False) -> set:
returns (true for both systemd and launchd in practice).
``all_profiles`` widens the launchd branch to every installed
``ai.hermes.gateway*`` agent — the update path needs the whole fleet
excluded from its sweep so sibling-profile launchd gateways found by the
ps scan aren't misclassified as manual processes (#73626). Default-scope
callers (``gateway status``, cron checks) keep seeing only the current
profile's service.
``ai.hermes.gateway*`` LaunchAgent — the update path needs the whole
fleet excluded from its sweep (#41403, #73626): sibling-profile launchd
gateways found by the (BSD-fixed) ps scan must not be misclassified as
manual processes and killed. Default-scope callers (``gateway status``,
cron checks) keep seeing only the current profile's service; the orphan
reaper passes all_profiles=True for the same friendly-fire reason. The
systemd branch has always been fleet-wide (``hermes-gateway*``) and is
unaffected.
"""
pids: set = set()
@@ -155,13 +158,29 @@ def _get_service_pids(all_profiles: bool = False) -> set:
# --- launchd (macOS) ---
if is_macos():
try:
if all_profiles:
# Enumerate every ai.hermes.gateway* agent across profiles
# so the update sweep's exclude set is complete (#73626).
# Without this, sibling-profile launchd gateways found by the
# (now-working) ps scan would be misclassified as manual and
# killed, racing with KeepAlive → duplicate gateways.
labels = {get_launchd_label()}
if all_profiles:
# Every gateway LaunchAgent, not just the invoking profile's —
# mirrors the systemd branch's ``hermes-gateway*`` pattern above.
# The update path restarts the whole fleet, and its stale-process
# sweep must not mistake a sibling service's fresh PID for a
# manual gateway it should kill (#41403).
labels.update(launchd_gateway_labels_for_install())
for label in sorted(labels):
try:
_domain, pid = _locate_launchd_gateway_service(label)
except subprocess.TimeoutExpired:
continue
if pid is not None and pid > 0:
pids.add(pid)
if all_profiles:
# Belt-and-suspenders for the EXCLUDE use case (#74075): a bare
# ``launchctl list`` prefix scan also catches ai.hermes.gateway*
# agents the label derivation can't map (renamed profiles, other
# installs sharing this user). Over-inclusion is safe here —
# these PIDs are only ever protected from the kill sweep, never
# targeted. Restart paths use the label-derived set only.
try:
result = subprocess.run(
["launchctl", "list"],
capture_output=True,
@@ -180,33 +199,8 @@ def _get_service_pids(all_profiles: bool = False) -> set:
pids.add(pid)
except ValueError:
pass
else:
label = get_launchd_label()
result = subprocess.run(
["launchctl", "list", label],
capture_output=True,
text=True, encoding='utf-8', errors='replace',
timeout=5,
)
if result.returncode == 0:
# Try plist format first (macOS 26+): "PID" = <N>;
pid = _parse_launchd_pid_from_list_output(result.stdout)
if pid is not None and pid > 0:
pids.add(pid)
else:
# Fall back to legacy tab-separated format:
# "PID\tStatus\tLabel"
for line in result.stdout.strip().splitlines():
parts = line.split()
if len(parts) >= 3 and parts[2] == label:
try:
pid = int(parts[0])
if pid > 0:
pids.add(pid)
except ValueError:
pass
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
return pids
@@ -1437,6 +1431,87 @@ def _parse_launchd_pid_from_list_output(output: str) -> int | None:
return None
def _parse_launchd_pid_from_print_output(output: str) -> int | None:
"""Extract the live PID from ``launchctl print`` output (``pid = <N>``).
A bootstrapped-but-not-running service prints no ``pid =`` line; the
first (service-level) occurrence wins over any nested endpoint state.
Returns ``None`` when no PID is found or the PID is non-positive.
"""
for line in output.splitlines():
stripped = line.strip()
if stripped.startswith("pid = "):
try:
pid = int(stripped[len("pid = "):].strip())
return pid if pid > 0 else None
except ValueError:
return None
return None
def _launchd_print_service_pid(domain: str, label: str) -> tuple[bool, int | None]:
"""Return ``(loaded, pid)`` for ``domain/label`` via ``launchctl print``.
Domain-explicit on purpose: legacy ``launchctl list`` infers its domain
from the caller's execution context, which is exactly the ambiguity that
sank the first fleet-restart attempt (#41403 review). ``TimeoutExpired``
propagates — fleet-restart callers own per-label failure accounting (a
wedged launchctl call must be reported, not read as "unloaded").
"""
try:
result = subprocess.run(
["launchctl", "print", f"{domain}/{label}"],
capture_output=True,
text=True, encoding='utf-8', errors='replace',
timeout=5,
)
except FileNotFoundError:
return (False, None)
if result.returncode != 0:
return (False, None)
return (True, _parse_launchd_pid_from_print_output(result.stdout))
def _launchd_service_registered(label: str) -> bool:
"""True when launchd knows ``label`` (``launchctl list <label>`` exit 0).
Registration is domain-agnostic and — unlike the ``launchctl print``
domain probes in ``_locate_launchd_gateway_service`` — stays true on
macOS 26+ hosts whose per-user domains reject service management, so
the update path can still hand the label to ``launchd_restart()``,
which owns that fallback. ``FileNotFoundError``/``TimeoutExpired``
propagate: the caller treats gate errors as a best-effort skip,
matching the pre-fleet inline behavior.
"""
result = subprocess.run(
["launchctl", "list", label],
capture_output=True,
text=True, encoding='utf-8', errors='replace',
timeout=5,
)
return result.returncode == 0
def _locate_launchd_gateway_service(label: str) -> tuple[str | None, int | None]:
"""Return ``(domain, pid)`` for ``label``, probing both per-user domains.
Probes ``gui/<uid>`` first (Aqua sessions), then ``user/<uid>``
(Background/SSH sessions). ``domain`` is None when the label is not
bootstrapped in either; ``pid`` is None when the service has no live
process. Sibling profile services resolve independently — a fleet can
legitimately mix domains (a profile installed over SSH lands in
``user/<uid>`` while the rest live in ``gui/<uid>``), so the current
profile's cached domain (``_launchd_domain()``) is never consulted.
``TimeoutExpired`` propagates (see ``_launchd_print_service_pid``).
"""
uid = os.getuid() # windows-footgun: ok — POSIX launchd (macOS) helper, never invoked on Windows
for domain in (f"gui/{uid}", f"user/{uid}"):
loaded, pid = _launchd_print_service_pid(domain, label)
if loaded:
return (domain, pid)
return (None, None)
def _probe_launchd_service_running() -> bool:
"""Return True when launchd is actively supervising the gateway process.
@@ -3018,6 +3093,36 @@ def get_launchd_plist_path() -> Path:
return _launchd_user_home() / "Library" / "LaunchAgents" / f"{name}.plist"
def launchd_gateway_labels_for_install() -> list[str]:
"""Return the launchd gateway label for every profile of THIS install.
Derived from the install's profile layout (rooted at
``get_default_hermes_root()``), NOT by globbing ``~/Library/LaunchAgents``:
the LaunchAgents directory is shared per-user, so a sandboxed
``HERMES_HOME`` (tests, capture sandboxes, side-by-side installs) must
never enumerate — let alone restart — another install's fleet.
Root label first, then profile labels sorted by name. Profile names
that cannot map to a service suffix (see ``_profile_suffix``'s naming
rule) are skipped — ``gateway install`` could never have created a
predictable label for them. Profiles without an installed gateway are
harmless to include: their labels simply aren't bootstrapped and
callers skip them after a failed locate.
"""
import re as _re
from hermes_cli.profiles import list_profiles
root_label: list[str] = []
profile_labels: list[str] = []
for profile in list_profiles():
if profile.is_default:
root_label.append("ai.hermes.gateway")
elif _re.match(r"^[a-z0-9][a-z0-9_-]{0,63}$", profile.name):
profile_labels.append(f"ai.hermes.gateway-{profile.name}")
return root_label + sorted(profile_labels)
def _detect_venv_dir() -> Path | None:
"""Detect the active virtualenv directory.
@@ -4231,52 +4336,35 @@ def get_launchd_label() -> str:
_resolved_launchd_domain: str | None = None
def _launchd_domain() -> str:
"""Return the launchd domain that actually manages the gateway service.
def _probe_launchd_domain_for_label(label: str) -> str:
"""Resolve the launchd domain that manages ``label`` — uncached, per label.
Probes ``gui/<uid>`` first (Aqua sessions), then ``user/<uid>``
(Background/SSH sessions). When neither domain contains a loaded
service, falls back to ``launchctl managername`` as a heuristic.
The result is cached for the lifetime of the process so that repeated
calls (``start``, ``stop``, ``restart``) use a consistent domain.
See #40831, #23387.
Sibling profile services resolve independently: a fleet can legitimately
mix domains (a profile installed over SSH lands in ``user/<uid>`` while
the rest live in ``gui/<uid>``), so the current profile's cached domain
(``_launchd_domain()``) must never be reused for another label.
"""
global _resolved_launchd_domain
if _resolved_launchd_domain is not None:
return _resolved_launchd_domain
uid = os.getuid() # windows-footgun: ok — POSIX launchd (macOS) helper, never invoked on Windows
label = get_launchd_label()
gui_domain = f"gui/{uid}"
user_domain = f"user/{uid}"
# 1. Probe gui/<uid> first — in Aqua sessions the service is loaded here.
try:
subprocess.run(
["launchctl", "print", f"{gui_domain}/{label}"],
check=True,
timeout=5,
capture_output=True,
)
_resolved_launchd_domain = gui_domain
return gui_domain
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# 2. Probe user/<uid> — in Background/SSH sessions this is the working domain.
try:
subprocess.run(
["launchctl", "print", f"{user_domain}/{label}"],
check=True,
timeout=5,
capture_output=True,
)
_resolved_launchd_domain = user_domain
return user_domain
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# 2. Then user/<uid> — in Background/SSH sessions this is the working domain.
for domain in (gui_domain, user_domain):
try:
subprocess.run(
["launchctl", "print", f"{domain}/{label}"],
check=True,
timeout=5,
capture_output=True,
)
return domain
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# 3. Neither domain has the service loaded — use managername as heuristic.
# Aqua → gui/<uid>, anything else (Background, loginwindow) → user/<uid>.
@@ -4288,17 +4376,32 @@ def _launchd_domain() -> str:
timeout=5,
)
if "Aqua" in (result.stdout or ""):
_resolved_launchd_domain = gui_domain
return gui_domain
except (subprocess.CalledProcessError, subprocess.TimeoutExpired, FileNotFoundError):
pass
# 4. Default to user/<uid> (matches the pre-probing behavior for
# Background/SSH sessions and is the recommended domain on macOS 26+).
_resolved_launchd_domain = user_domain
return user_domain
def _launchd_domain() -> str:
"""Return the launchd domain that actually manages the gateway service.
Per-label probing lives in ``_probe_launchd_domain_for_label``; this
wrapper resolves the *current* profile's label and caches the result for
the lifetime of the process so that repeated calls (``start``, ``stop``,
``restart``) use a consistent domain.
See #40831, #23387.
"""
global _resolved_launchd_domain
if _resolved_launchd_domain is not None:
return _resolved_launchd_domain
_resolved_launchd_domain = _probe_launchd_domain_for_label(get_launchd_label())
return _resolved_launchd_domain
# On macOS, exit code 125 ("Domain does not support specified action") and
# 3/113 ("Could not find service") all mean the job isn't currently loaded in
# the target domain, so start/restart should re-bootstrap the plist and retry.
@@ -5163,6 +5266,43 @@ def _wait_for_gateway_exit(
return True
def _launchd_kickstart(label: str, domain: str) -> None:
"""Hard-restart ``domain/label`` via ``launchctl kickstart -k``.
Raises ``CalledProcessError``/``TimeoutExpired`` — callers own the
per-label failure accounting during fleet restarts.
"""
subprocess.run(
["launchctl", "kickstart", "-k", f"{domain}/{label}"],
check=True,
capture_output=True,
text=True, encoding='utf-8', errors='replace',
timeout=90,
)
def _wait_for_launchd_service_pid(
label: str, old_pid: int | None, timeout: float = 10.0, *, domain: str
) -> bool:
"""Poll ``domain/label`` until the service runs on a fresh PID.
launchd's exit → ``KeepAlive`` respawn transition is not instantaneous;
a one-shot check races that window and falsely reports the service as
down (same rationale as the systemd ``is-active`` poll in the update
path). Poll every 0.5s up to ``timeout`` seconds before giving up.
``TimeoutExpired`` from launchctl propagates — callers own per-label
failure accounting.
"""
deadline = time.monotonic() + max(timeout, 0.5)
while True:
_loaded, pid = _launchd_print_service_pid(domain, label)
if pid is not None and pid > 0 and pid != old_pid:
return True
if time.monotonic() >= deadline:
return False
time.sleep(0.5)
def launchd_restart():
label = get_launchd_label()
target = f"{_launchd_domain()}/{label}"

148
hermes_cli/update_cmd.py
View File

@@ -4497,8 +4497,123 @@ def _warn_incomplete_gateway_fleet_restart(failed_units: list) -> None:
print(" Skipped units may still be running pre-update code (mixed")
print(" sys.modules). Restart them manually, then verify:")
print(" hermes gateway status")
print(" systemctl --user restart <unit> # user-scope")
print(" sudo systemctl restart <unit> # system-scope")
if any(not name.startswith("ai.hermes.") for name in ordered):
print(" systemctl --user restart <unit> # user-scope")
print(" sudo systemctl restart <unit> # system-scope")
if any(name.startswith("ai.hermes.") for name in ordered):
print(" launchctl kickstart -k gui/$UID/<label> # macOS (or user/$UID)")
def _restart_macos_launchd_gateways(
restarted_services: list,
failed_or_stale_units: list,
drain_budget: float,
) -> None:
"""Restart every launchd-managed gateway after an update (macOS).
The code update (git pull) is shared across all profiles, so every
``ai.hermes.gateway*`` LaunchAgent must reload it — restarting only the
invoking profile's service leaves siblings on pre-update ``sys.modules``
until their next agent turn imports a symbol the old module generation
doesn't have (#41403). Parity with the systemd fleet path.
The invoking profile keeps the existing ``launchd_restart()`` treatment
(self-restart request → graceful drain → kickstart). Siblings get the
same drain-first sequence, with their launchd domain resolved per label:
a sibling bootstrapped in the other supported domain (``gui/<uid>`` vs
``user/<uid>``) must not be kickstarted in the current profile's domain.
``subprocess.TimeoutExpired`` is isolated per label so one wedged
launchctl call cannot leave the rest of the fleet on old code (#68523).
"""
from hermes_cli.gateway import (
get_launchd_label,
get_launchd_plist_path,
launchd_restart,
launchd_gateway_labels_for_install,
_graceful_restart_via_sigusr1,
_launchd_kickstart,
_launchd_service_registered,
_locate_launchd_gateway_service,
_wait_for_launchd_service_pid,
)
# --- Current profile: unchanged single-service path ---------------------
# Gate order and predicate mirror the pre-fleet inline block exactly:
# plist first (no plist → zero launchctl calls), then the domain-agnostic
# `launchctl list` registration check — NOT a domain locate, which fails
# on macOS-26 hosts whose per-user domains reject service management
# even though launchd_restart() owns that fallback. Gate errors skip
# silently (best-effort, as before); only launchd_restart() itself
# failing counts toward the incomplete-update warning.
current_label = get_launchd_label()
try:
if get_launchd_plist_path().exists() and _launchd_service_registered(
current_label
):
try:
launchd_restart()
restarted_services.append(current_label)
except subprocess.CalledProcessError as e:
stderr = (getattr(e, "stderr", "") or "").strip()
print(f" ⚠ Gateway restart failed: {stderr}")
failed_or_stale_units.append(current_label)
except (FileNotFoundError, subprocess.TimeoutExpired):
pass
# --- Sibling profiles ---------------------------------------------------
for label in launchd_gateway_labels_for_install():
if label == current_label:
continue
try:
# Locate = liveness + domain in one domain-explicit probe; the
# kickstart and fresh-PID verification below reuse the located
# domain, so a sibling in the other gui/user domain can never be
# probed in one domain and restarted in another.
domain, old_pid = _locate_launchd_gateway_service(label)
if domain is None:
# Installed but not bootstrapped (stopped/uninstalled
# mid-way) — nothing is running old code here.
continue
graceful_ok = False
if old_pid is not None and old_pid > 0:
print(f" → {label}: draining (up to {int(drain_budget)}s)...")
graceful_ok = _graceful_restart_via_sigusr1(
old_pid, drain_timeout=drain_budget
)
if graceful_ok and _wait_for_launchd_service_pid(
label, old_pid=old_pid, timeout=10.0, domain=domain
):
# Unconditional KeepAlive already respawned it on the new
# code — a hard kickstart now would kill the fresh process.
restarted_services.append(label)
continue
try:
_launchd_kickstart(label, domain)
except subprocess.CalledProcessError as e:
stderr = (getattr(e, "stderr", "") or "").strip()
failed_or_stale_units.append(label)
print(
f" ⚠ Failed to restart {label}: {stderr}\n"
f" Recover manually: launchctl kickstart -k {domain}/{label}"
)
continue
if _wait_for_launchd_service_pid(
label, old_pid=old_pid, timeout=15.0, domain=domain
):
restarted_services.append(label)
else:
failed_or_stale_units.append(label)
print(
f" ✗ {label} failed to come back after restart.\n"
f" Check logs, then: launchctl kickstart -k {domain}/{label}"
)
except subprocess.TimeoutExpired:
failed_or_stale_units.append(label)
print(
f" ⚠ launchctl timed out restarting {label}; "
"continuing with remaining gateways"
)
def _surviving_gateway_pids_after_failed_restart():
"""Best-effort PIDs of gateways still running after the restart phase died.
@@ -6966,30 +7081,17 @@ def _cmd_update_impl(args, gateway_mode: bool):
)
# --- Launchd services (macOS) ---
# Restart EVERY ai.hermes.gateway* LaunchAgent, not only the
# invoking profile's — parity with the systemd branch above
# (#41403). Per-label TimeoutExpired isolation happens inside.
if is_macos():
try:
from hermes_cli.gateway import (
launchd_restart,
get_launchd_label,
get_launchd_plist_path,
_restart_macos_launchd_gateways(
restarted_services,
failed_or_stale_units,
_drain_budget,
)
plist_path = get_launchd_plist_path()
if plist_path.exists():
check = subprocess.run(
["launchctl", "list", get_launchd_label()],
capture_output=True,
text=True, encoding="utf-8", errors="replace",
timeout=5,
)
if check.returncode == 0:
try:
launchd_restart()
restarted_services.append(get_launchd_label())
except subprocess.CalledProcessError as e:
stderr = (getattr(e, "stderr", "") or "").strip()
print(f" ⚠ Gateway restart failed: {stderr}")
except (FileNotFoundError, subprocess.TimeoutExpired, ImportError):
except (FileNotFoundError, ImportError):
pass
# --- Manual (non-service) gateways ---

642
tests/hermes_cli/test_update_launchd_fleet_restart.py Normal file
View File

@@ -0,0 +1,642 @@
"""Regression for #41403 — ``hermes update`` must restart ALL macOS launchd gateways.
The macOS branch of the update's fleet-restart step only restarted the
invoking profile's LaunchAgent (``get_launchd_label()`` is profile-scoped).
Sibling ``ai.hermes.gateway-<profile>`` services kept running pre-update
modules cached in ``sys.modules`` and died on their next agent turn once the
new code lazily imported a symbol the old module generation didn't have
(``ImportError: cannot import name ...`` — or, with a wider version gap,
``TypeError``/``AttributeError`` on changed call signatures with garbled
tracebacks, because the source files on disk no longer match the loaded
code objects).
Also covers the launchd-domain review feedback on PR #41403: every sibling
interaction (liveness discovery, kickstart, fresh-PID verification) must be
domain-explicit — ``_launchd_domain()`` caches the *current* profile's
domain, and a sibling bootstrapped in the other supported domain
(``gui/<uid>`` vs ``user/<uid>``) would otherwise be probed or kickstarted
in a domain it does not live in.
"""
from __future__ import annotations
import subprocess
import sys
import pytest
import hermes_cli.gateway as gw
import hermes_cli.profiles
from hermes_cli.gateway import (
_locate_launchd_gateway_service,
_parse_launchd_pid_from_print_output,
_probe_launchd_domain_for_label,
launchd_gateway_labels_for_install,
)
from hermes_cli.update_cmd import (
_restart_macos_launchd_gateways,
_warn_incomplete_gateway_fleet_restart,
)
pytestmark = pytest.mark.skipif(
sys.platform == "win32",
reason="launchd fleet restart is macOS-only; helpers use POSIX os.getuid",
)
UID = 501
PRINT_RUNNING = (
"system/com.example = {\n"
"\tactive count = 1\n"
"\tstate = running\n"
"\tpid = 4242\n"
"\tprogram = /usr/bin/true\n"
"}\n"
)
PRINT_LOADED_NOT_RUNNING = (
"system/com.example = {\n"
"\tactive count = 0\n"
"\tstate = not running\n"
"\tprogram = /usr/bin/true\n"
"}\n"
)
@pytest.fixture(autouse=True)
def _fixed_uid(monkeypatch):
monkeypatch.setattr(gw.os, "getuid", lambda: UID)
def _completed(returncode: int = 0, stdout: str = "") -> subprocess.CompletedProcess:
return subprocess.CompletedProcess(
args=[], returncode=returncode, stdout=stdout, stderr=""
)
class _Profile:
def __init__(self, name, is_default=False):
self.name = name
self.is_default = is_default
class TestLaunchdGatewayLabelsForInstall:
def test_labels_derive_from_this_installs_profiles(self, monkeypatch):
"""The fleet is THIS install's profiles, root first — never a glob of
the shared per-user LaunchAgents dir. A sandboxed HERMES_HOME (tests,
side-by-side installs) must not enumerate — and restart — another
install's services, and the hermetic test suite must not see the dev
machine's real fleet."""
monkeypatch.setattr(
hermes_cli.profiles,
"list_profiles",
lambda: [
_Profile("tfl-wiki"),
_Profile("default", is_default=True),
_Profile("merit-ops"),
_Profile("Bad Name!"), # cannot map to a service suffix — skipped
],
)
assert launchd_gateway_labels_for_install() == [
"ai.hermes.gateway",
"ai.hermes.gateway-merit-ops",
"ai.hermes.gateway-tfl-wiki",
]
def test_no_profiles_means_no_fleet(self, monkeypatch):
monkeypatch.setattr(hermes_cli.profiles, "list_profiles", lambda: [])
assert launchd_gateway_labels_for_install() == []
class TestParseLaunchdPidFromPrintOutput:
def test_running_service_pid(self):
assert _parse_launchd_pid_from_print_output(PRINT_RUNNING) == 4242
def test_loaded_but_not_running_has_no_pid(self):
assert _parse_launchd_pid_from_print_output(PRINT_LOADED_NOT_RUNNING) is None
class TestLocateLaunchdGatewayService:
def test_domains_resolve_per_label_not_from_cache(self, monkeypatch):
"""The #41403 review defect: sibling domains are independent."""
gui_loaded = {"ai.hermes.gateway-a"}
def fake_run(cmd, **kwargs):
assert cmd[:2] == ["launchctl", "print"]
domain, _, label = cmd[2].rpartition("/")
in_gui = domain == f"gui/{UID}" and label in gui_loaded
in_user = domain == f"user/{UID}" and label not in gui_loaded
if in_gui or in_user:
return _completed(0, PRINT_RUNNING)
return _completed(113)
monkeypatch.setattr(gw.subprocess, "run", fake_run)
# Simulate a prior current-profile resolution having populated the
# process-wide cache — per-label lookups must not consult it.
monkeypatch.setattr(gw, "_resolved_launchd_domain", f"gui/{UID}")
assert _locate_launchd_gateway_service("ai.hermes.gateway-a") == (
f"gui/{UID}",
4242,
)
assert _locate_launchd_gateway_service("ai.hermes.gateway-b") == (
f"user/{UID}",
4242,
)
def test_loaded_without_live_process(self, monkeypatch):
monkeypatch.setattr(
gw.subprocess,
"run",
lambda *a, **k: _completed(0, PRINT_LOADED_NOT_RUNNING),
)
assert _locate_launchd_gateway_service("ai.hermes.gateway-x") == (
f"gui/{UID}",
None,
)
def test_not_loaded_in_either_domain(self, monkeypatch):
monkeypatch.setattr(gw.subprocess, "run", lambda *a, **k: _completed(113))
assert _locate_launchd_gateway_service("ai.hermes.gateway-x") == (None, None)
def test_timeout_propagates_to_caller(self, monkeypatch):
"""A wedged launchctl must surface as a failure, not read as
'unloaded' — the update path owns per-label failure accounting."""
def fake_run(cmd, **kwargs):
raise subprocess.TimeoutExpired(cmd=cmd, timeout=5)
monkeypatch.setattr(gw.subprocess, "run", fake_run)
with pytest.raises(subprocess.TimeoutExpired):
_locate_launchd_gateway_service("ai.hermes.gateway-x")
class TestProbeLaunchdDomainForLabel:
def test_unloaded_label_falls_back_to_managername(self, monkeypatch):
def fake_run(cmd, **kwargs):
if cmd[:2] == ["launchctl", "print"]:
raise subprocess.CalledProcessError(113, cmd)
if cmd == ["launchctl", "managername"]:
return _completed(0, "Aqua\n")
raise AssertionError(f"unexpected command {cmd}")
monkeypatch.setattr(gw.subprocess, "run", fake_run)
assert _probe_launchd_domain_for_label("ai.hermes.gateway-x") == f"gui/{UID}"
def test_unloaded_label_defaults_to_user_domain(self, monkeypatch):
def fake_run(cmd, **kwargs):
if cmd[:2] == ["launchctl", "print"]:
raise subprocess.CalledProcessError(113, cmd)
if cmd == ["launchctl", "managername"]:
return _completed(0, "Background\n")
raise AssertionError(f"unexpected command {cmd}")
monkeypatch.setattr(gw.subprocess, "run", fake_run)
assert _probe_launchd_domain_for_label("ai.hermes.gateway-x") == f"user/{UID}"
class TestGetServicePidsScoping:
def _wire(self, monkeypatch):
monkeypatch.setattr(gw, "is_macos", lambda: True)
monkeypatch.setattr(gw, "supports_systemd_services", lambda: False)
monkeypatch.setattr(gw, "get_launchd_label", lambda: "ai.hermes.gateway")
monkeypatch.setattr(
gw,
"launchd_gateway_labels_for_install",
lambda: ["ai.hermes.gateway", "ai.hermes.gateway-a", "ai.hermes.gateway-b"],
)
located = {
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-a": (f"gui/{UID}", 200),
"ai.hermes.gateway-b": (None, None), # not bootstrapped
}
monkeypatch.setattr(
gw, "_locate_launchd_gateway_service", lambda label: located[label]
)
def test_all_profiles_returns_every_gateway_service_pid(self, monkeypatch):
"""The update sweep's exclude-set must protect ALL freshly-restarted
services, not only the invoking profile's (else the sweep SIGTERMs
gateways launchd just respawned)."""
self._wire(monkeypatch)
assert gw._get_service_pids(all_profiles=True) == {100, 200}
def test_default_stays_scoped_to_current_profile(self, monkeypatch):
"""Regression guard: default-scope callers (gateway status, cron,
stop_profile_gateway's orphan reaper) must NOT start seeing sibling
service PIDs — the reaper SIGTERM/SIGKILLs what they feed it."""
self._wire(monkeypatch)
assert gw._get_service_pids() == {100}
def test_find_gateway_pids_passes_profile_scope_through(self, monkeypatch):
calls: list[bool] = []
monkeypatch.setattr(
gw,
"_get_service_pids",
lambda all_profiles=False: (calls.append(all_profiles), set())[1],
)
monkeypatch.setattr(gw, "_scan_gateway_pids", lambda *a, **k: [])
monkeypatch.setattr(gw, "supports_systemd_services", lambda: True)
gw.find_gateway_pids(all_profiles=False)
gw.find_gateway_pids(all_profiles=True)
assert calls == [False, True]
def _fleet(monkeypatch, tmp_path, *, current, labels, located,
registered=None, plist_exists=True,
drain_results=None, kick_errors=None, wait_results=None):
"""Wire a fake launchd fleet through hermes_cli.gateway seams.
``located`` maps label -> (domain, pid) as ``_locate_launchd_gateway_service``
would return it (values may also be exceptions to raise). ``registered``
maps label -> bool for the current-profile ``launchctl list`` gate and
defaults to "located in some domain". Returns a SimpleNamespace of
recorder lists: rec.kickstarts, rec.drains, rec.current_restarts, rec.waits, locates,
registered_checks.
"""
from types import SimpleNamespace
rec = SimpleNamespace(
kickstarts=[], drains=[], current_restarts=[], waits=[],
locates=[], registered_checks=[],
)
plist = tmp_path / f"{current}.plist"
if plist_exists:
plist.write_text("<plist/>")
def fake_locate(label):
rec.locates.append(label)
value = located[label]
if isinstance(value, Exception):
raise value
return value
def fake_registered(label):
rec.registered_checks.append(label)
if registered is not None:
return registered[label]
value = located.get(label)
return (
value is not None
and not isinstance(value, Exception)
and value[0] is not None
)
monkeypatch.setattr(gw, "get_launchd_label", lambda: current)
monkeypatch.setattr(gw, "get_launchd_plist_path", lambda: plist)
monkeypatch.setattr(gw, "launchd_gateway_labels_for_install", lambda: list(labels))
monkeypatch.setattr(gw, "_locate_launchd_gateway_service", fake_locate)
monkeypatch.setattr(gw, "_launchd_service_registered", fake_registered)
monkeypatch.setattr(
gw,
"_graceful_restart_via_sigusr1",
lambda pid, drain_timeout: (rec.drains.append(pid), (drain_results or {}).get(pid, False))[1],
)
def fake_kickstart(label, domain):
err = (kick_errors or {}).get(label)
if err is not None:
raise err
rec.kickstarts.append(f"{domain}/{label}")
monkeypatch.setattr(gw, "_launchd_kickstart", fake_kickstart)
def fake_wait(label, old_pid, timeout, domain):
rec.waits.append(f"{domain}/{label}")
return (wait_results or {}).get(label, True)
monkeypatch.setattr(gw, "_wait_for_launchd_service_pid", fake_wait)
monkeypatch.setattr(
gw, "launchd_restart", lambda: rec.current_restarts.append(current)
)
return rec
class TestRestartMacosLaunchdGateways:
def test_current_delegates_and_siblings_kickstart_in_own_domains(
self, monkeypatch, tmp_path
):
"""Current profile keeps launchd_restart(); every sibling (including
the root gateway when a named profile invokes the update) is
kickstarted — and verified — in the domain IT was located in."""
current = "ai.hermes.gateway-merit-ops"
rec = _fleet(
monkeypatch,
tmp_path,
current=current,
labels=["ai.hermes.gateway", current, "ai.hermes.gateway-user-scoped"],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
current: (f"gui/{UID}", 200),
"ai.hermes.gateway-user-scoped": (f"user/{UID}", 300),
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.current_restarts == [current]
assert rec.kickstarts == [
f"gui/{UID}/ai.hermes.gateway",
f"user/{UID}/ai.hermes.gateway-user-scoped",
]
assert rec.waits == [
f"gui/{UID}/ai.hermes.gateway",
f"user/{UID}/ai.hermes.gateway-user-scoped",
]
assert restarted == [
current,
"ai.hermes.gateway",
"ai.hermes.gateway-user-scoped",
]
assert failed == []
# Siblings were drained before the hard kickstart.
assert set(rec.drains) == {100, 300}
def test_current_profile_without_plist_makes_no_launchctl_calls(
self, monkeypatch, tmp_path
):
"""Upstream gate order preserved: no plist → the current profile is
skipped without ANY launchctl interaction (no registered probe, no
locate) — and definitely without inventing a failure. Siblings are
still processed."""
current = "ai.hermes.gateway"
rec = _fleet(
monkeypatch,
tmp_path,
current=current,
labels=[current, "ai.hermes.gateway-a"],
located={"ai.hermes.gateway-a": (f"gui/{UID}", 200)},
plist_exists=False,
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.current_restarts == []
assert current not in rec.registered_checks
assert current not in rec.locates
assert restarted == ["ai.hermes.gateway-a"]
assert failed == []
def test_current_profile_registered_but_unlocatable_still_restarts(
self, monkeypatch, tmp_path
):
"""macOS-26 quirk: a label can be `launchctl list`-registered while
both explicit gui/user `launchctl print` probes fail (domain doesn't
support service management). The gate must use the registered
predicate and hand off to launchd_restart(), which owns the
domain-unsupported fallback — locate is for siblings only."""
current = "ai.hermes.gateway"
rec = _fleet(
monkeypatch,
tmp_path,
current=current,
labels=[current],
located={current: (None, None)},
registered={current: True},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.current_restarts == [current]
assert current not in rec.locates
assert restarted == [current]
assert failed == []
def test_unbootstrapped_sibling_is_skipped_not_failed(
self, monkeypatch, tmp_path
):
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=["ai.hermes.gateway", "ai.hermes.gateway-idle"],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-idle": (None, None),
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.kickstarts == []
assert restarted == ["ai.hermes.gateway"]
assert failed == []
def test_loaded_but_not_running_sibling_is_kickstarted(
self, monkeypatch, tmp_path
):
"""A bootstrapped service with no live process still holds the old
code path for its next launch trigger — kickstart it (no drain)."""
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=["ai.hermes.gateway", "ai.hermes.gateway-dormant"],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-dormant": (f"gui/{UID}", None),
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.drains == []
assert rec.kickstarts == [f"gui/{UID}/ai.hermes.gateway-dormant"]
assert restarted == ["ai.hermes.gateway", "ai.hermes.gateway-dormant"]
assert failed == []
def test_graceful_drain_with_keepalive_respawn_skips_kickstart(
self, monkeypatch, tmp_path
):
"""When SIGUSR1 rec.drains the sibling and KeepAlive already respawned it
on a fresh PID, a second hard kickstart would kill the new process."""
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=["ai.hermes.gateway", "ai.hermes.gateway-a"],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-a": (f"gui/{UID}", 200),
},
drain_results={200: True},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert rec.drains == [200]
assert rec.kickstarts == []
assert rec.waits == [f"gui/{UID}/ai.hermes.gateway-a"]
assert restarted == ["ai.hermes.gateway", "ai.hermes.gateway-a"]
assert failed == []
def test_kickstart_failure_is_recorded_and_rest_continue(
self, monkeypatch, tmp_path
):
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=[
"ai.hermes.gateway",
"ai.hermes.gateway-bad",
"ai.hermes.gateway-good",
],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-bad": (f"gui/{UID}", 200),
"ai.hermes.gateway-good": (f"gui/{UID}", 300),
},
kick_errors={
"ai.hermes.gateway-bad": subprocess.CalledProcessError(
5, ["launchctl", "kickstart"]
)
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert failed == ["ai.hermes.gateway-bad"]
assert rec.kickstarts == [f"gui/{UID}/ai.hermes.gateway-good"]
assert restarted == ["ai.hermes.gateway", "ai.hermes.gateway-good"]
def test_timeout_during_discovery_is_failed_and_rest_continue(
self, monkeypatch, tmp_path
):
"""A wedged launchctl during liveness discovery must be accounted as
a failure (the sibling may still be on old code), not silently
skipped — and must not abort the remaining fleet (#68523 parity)."""
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=[
"ai.hermes.gateway",
"ai.hermes.gateway-wedged",
"ai.hermes.gateway-after",
],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-wedged": subprocess.TimeoutExpired(
cmd=["launchctl", "print"], timeout=5
),
"ai.hermes.gateway-after": (f"gui/{UID}", 300),
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert failed == ["ai.hermes.gateway-wedged"]
assert rec.kickstarts == [f"gui/{UID}/ai.hermes.gateway-after"]
assert restarted == ["ai.hermes.gateway", "ai.hermes.gateway-after"]
def test_timeout_during_kickstart_is_failed_and_rest_continue(
self, monkeypatch, tmp_path
):
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=[
"ai.hermes.gateway",
"ai.hermes.gateway-wedged",
"ai.hermes.gateway-after",
],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-wedged": (f"gui/{UID}", 200),
"ai.hermes.gateway-after": (f"gui/{UID}", 300),
},
kick_errors={
"ai.hermes.gateway-wedged": subprocess.TimeoutExpired(
cmd=["launchctl", "kickstart"], timeout=90
)
},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert failed == ["ai.hermes.gateway-wedged"]
assert rec.kickstarts == [f"gui/{UID}/ai.hermes.gateway-after"]
assert restarted == ["ai.hermes.gateway", "ai.hermes.gateway-after"]
def test_sibling_that_never_comes_back_is_failed(self, monkeypatch, tmp_path):
rec = _fleet(
monkeypatch,
tmp_path,
current="ai.hermes.gateway",
labels=["ai.hermes.gateway", "ai.hermes.gateway-zombie"],
located={
"ai.hermes.gateway": (f"gui/{UID}", 100),
"ai.hermes.gateway-zombie": (f"gui/{UID}", 200),
},
wait_results={"ai.hermes.gateway-zombie": False},
)
restarted: list[str] = []
failed: list[str] = []
_restart_macos_launchd_gateways(restarted, failed, drain_budget=0.0)
assert restarted == ["ai.hermes.gateway"]
assert failed == ["ai.hermes.gateway-zombie"]
class TestWaitForLaunchdServicePid:
def test_returns_true_once_pid_changes(self, monkeypatch):
pids = iter([200, 200, 4242])
monkeypatch.setattr(
gw,
"_launchd_print_service_pid",
lambda domain, label: (True, next(pids)),
)
monkeypatch.setattr(gw.time, "sleep", lambda _s: None)
assert gw._wait_for_launchd_service_pid(
"ai.hermes.gateway-x", old_pid=200, timeout=5.0, domain=f"gui/{UID}"
)
def test_returns_false_when_pid_never_changes(self, monkeypatch):
clock = iter(float(i) for i in range(100))
monkeypatch.setattr(gw.time, "monotonic", lambda: next(clock))
monkeypatch.setattr(gw.time, "sleep", lambda _s: None)
monkeypatch.setattr(
gw,
"_launchd_print_service_pid",
lambda domain, label: (True, 200),
)
assert not gw._wait_for_launchd_service_pid(
"ai.hermes.gateway-x", old_pid=200, timeout=3.0, domain=f"gui/{UID}"
)
class TestIncompleteWarningMentionsLaunchctl:
def test_launchd_labels_get_launchctl_hint(self, capsys):
_warn_incomplete_gateway_fleet_restart(["ai.hermes.gateway-merit-ops"])
out = capsys.readouterr().out
assert "Update incomplete" in out
assert "launchctl kickstart -k" in out
def test_systemd_units_keep_systemctl_hint(self, capsys):
_warn_incomplete_gateway_fleet_restart(["hermes-gateway-coder"])
out = capsys.readouterr().out
assert "systemctl" in out
assert "launchctl" not in out