docs(agents-md): update pipeline architecture, process-identity pitfall, gateway lifecycle contract, wine2e lane

Captures the durable invariants from the fleet-update campaign (#91277)
so contributors and the sweeper review against them:

- Update Pipeline section: the transactional shape now on main
  (plan → snapshot → apply → restart-per-kind → verify → report), the
  per-stage invariants (no partial snapshot tiers, ZIP only on real git
  failure + dirty-tree refusal + release-dir graft, fleet-wide drain-first
  restarts, code-sha verify, exactly-once receipts), deployment kinds as
  first-class, and the #92091 socket direction.
- Gateway lifecycle vs Desktop app: serve dies with the app by design,
  the detached gateway survives it; the Windows shim-unlock tree-kill is
  the known breach (#85265) and its replacement is pause-for-update —
  with the two anti-fix warnings.
- Known Pitfall: process identity is never inferred from argv substrings
  (canonical matchers, parser-derived flag sets, ancestor carve-out,
  full-cmdline rule, socket-first for new heuristics).
- Testing: the on-demand wine2e live Windows lane and its
  reproduce-first workflow.
This commit is contained in:
Teknium
2026-08-22 00:18:58 -07:00
parent ff88f27403
commit 4ba038b51d

110
AGENTS.md
View File

@@ -1201,6 +1201,79 @@ Full user-facing docs: `website/docs/user-guide/features/kanban.md`.
---
## Update Pipeline (`hermes update`)
The updater is transactional in shape (fleet-update campaign, #91277 —
Aug 2026). Every stage exists because its absence was a real field
failure; PRs that weaken a stage need to answer for the failure class it
guards:
```
plan → snapshot → apply → restart-per-kind → verify → report
```
- **Plan** (`hermes_cli/update_inventory.py`, `hermes update --plan`):
read-only inventory — install kind, all profiles, every live gateway
with supervisor + running code version. Deployment kinds are
first-class: `git` updates in place; `docker`/`nix`/`apt` are NOT
in-place-updatable and the updater reports the correct external
command instead of fighting the deployment model.
- **Snapshot** (`hermes_cli/backup.py`): pre-update quick snapshot for
EVERY profile (the code swap + fleet restart touch all of them), each
into its own `state-snapshots/`, identical file set + 1 GiB per-file
cap + keep=1. **Never add a partial/tiered snapshot set** — mixed
coverage creates torn-restore states across schema generations. Quick
snapshots are FILE-LOSS RECOVERY (the per-profile cron-jobs safety
net restores from them), NOT code-rollback insurance; `--backup` full
mode owns rollback.
- **Apply**: git pull, or the Windows ZIP fallback — which fires ONLY
when git itself failed (`_should_zip_fallback_on_update_error`,
argv-classified; a dependency-install failure must never trigger a
tree-clobbering re-download), REFUSES a dirty working tree
(`-uall`, plus a pre-swap TOCTOU re-check), and grafts the live
`apps/desktop/release/` into the staged swap (the GitHub source ZIP
has no built desktop app; without the graft the swap deletes it).
- **Restart-per-kind**: systemd and launchd restarts are FLEET-WIDE
(every `hermes-gateway*` unit / `ai.hermes.gateway*` LaunchAgent),
drain-first (SIGUSR1) with per-unit/per-label failure isolation.
Restarting only the invoking profile's service leaves siblings on
stale `sys.modules` until they crash — the largest dupe-PR cluster in
the repo's history came from that bug.
- **Verify**: gateways stamp their running `code_sha`/`code_version`
into `gateway_state.json` on every runtime-status write
(`gateway/status.py`); after the restart phase the updater compares
each live gateway against the fresh checkout and prints a fleet
version matrix. A provably-stale gateway fails the update (exit 1) —
automation must never treat a mixed-version fleet as healthy.
- **Report**: every run writes a machine-readable receipt to
`~/.hermes/logs/update_receipts/` (`latest.json` pointer; steps,
skips WITH reasons, restart outcome, plan, fleet snapshot).
Finalization is owned by the `cmd_update` command boundary — early
`sys.exit` paths (preflight refusals, fetch failures) still persist
a receipt with the real exit code. A begun-but-unwritten receipt is
a bug: the refused/failed runs are the ones receipts exist for.
Architecture direction: process-scan-based coordination between the
updater, serve/dashboard, and the gateway is being replaced by a
gateway-owned control socket (#92091). Do not add new scan heuristics
without checking that design; scans are the fallback layer.
### Gateway lifecycle vs. the Desktop app
`hermes serve` (control plane, desktop-spawned child) dies with the app
— by design. The messaging gateway (`gateway run`) SURVIVES the app: the
serve backend's `/api/gateway/*` endpoints spawn it detached
(`_spawn_hermes_action` — `start_new_session` / `DETACHED_PROCESS`), so
`before-quit`'s backend SIGTERM never reaches it. Bots keep running
when the user closes the app. The known breach of this contract is the
Windows shim-unlock teardown (`taskkill /T /F` on venv-shim holders,
#85265) — it exists to let updates proceed, and its replacement is
#92091's `pause-for-update`. Do not "fix" gateway-dies-with-app reports
by re-parenting the gateway under the backend, and do not "fix" update
locks by widening the tree-kill.
---
## Important Policies
### Prompt Caching Must Not Break
@@ -1314,6 +1387,27 @@ automatically scope to the active profile.
## Known Pitfalls
### DO NOT infer process identity from argv substrings
The bug class behind ~10 fleet-update issues (#90778, #87594, #78089,
#76129, #91964, ...): classifying a process by `"serve" in cmdline` or
similar. `kanban --preserve-cache` contains "serve"; a flag VALUE can
equal a subcommand (`-m dashboard serve`); truncated cmdlines hide the
real subcommand. Rules:
- Use the canonical matchers: `gateway.status.looks_like_gateway_command_line`
(gateway run), `hermes_cli.update_cmd._hermes_holder_subcommand`
(top-level subcommand of any Hermes argv). Never hand-roll token scans.
- Flag sets must be DERIVED from the parser
(`_holder_value_flags()` introspects `build_top_level_parser()`), never
hand-written lists — they drift.
- Never blanket-exclude ancestors from process scans: when `/update` runs
as the gateway's child, a gateway ancestor must stay visible to the
pause machinery (#87594). Exclude interactive ancestry, carve out
gateway-shaped ancestors.
- Match on FULL cmdlines; truncate only at display time (#78089).
- Before adding any new scan heuristic, read #92091 — the gateway control
socket replaces scans as the primary coordination mechanism; scans are
the fallback layer for old/crashed processes.
### DO NOT hardcode `~/.hermes` paths
Use `get_hermes_home()` from `hermes_constants` for code paths. Use `display_hermes_home()`
for user-facing print/log messages. Hardcoding `~/.hermes` breaks profiles — each profile
@@ -1491,6 +1585,22 @@ in order to pass, it belongs on that OS.**
When one test body walks several platforms in sequence, split it.
Keep the host-native arm on the Linux lane and move the other arm into its own marked test.
**Live Windows process-topology E2E: the `wine2e` lane.** For claims about
real Windows process behavior that mocks cannot reproduce (venv-holder
scans, process-tree parentage, launcher/worker chains, detach semantics),
there is an on-demand workflow `windows-venv-e2e.yml` that runs
`tests/hermes_cli/test_venv_holder_windows_live.py` on a real
`windows-latest` runner — spawning actual processes and driving the real
detection code, no mocked psutil. It fires ONLY on pushes to `wine2e/**`
branches (inert on PRs and main; costs nothing on normal work). The proven
workflow: write probes that pin CORRECT behavior, push to a `wine2e/`
branch to reproduce the bugs live on unfixed code, build the fix, iterate
until the lane is green, then open the PR — the live receipt on the exact
head is the Windows proof reviewers ask for. Extend the live suite when
touching that subsystem; assert against the gateway ANCESTOR found by
argv, not the direct parent (the venv shim makes every spawn a
launcher/worker chain).
**Use the marker, never a bare `skipif`.** `scripts/ci/list_os_marked_tests.py`
decides which files the macOS/Windows lanes import by grepping for the marker
*name*, and the lane then filters with `-m <marker>`. A test gated with