Review on #136509 (Greptile): assert_nothing_changed accepts any non-zero
exit, so an unrelated early failure could pass. The cell now asserts the
proxy recorded the EOF on github.com and the output names the failed fetch.
rc.10-v0.21.7 failed 13 release jobs because the release-only E2E still
assumed the old update behavior:
- Install/update E2E (Linux, macOS, Windows; hermes-update and desktop-app
routes). A fresh install of the official repo now defaults to the stable
channel. Installed at HEAD it is newer than v0.21.6 and correctly waits
("Already up to date"), so the HEAD -> NEXT legs never moved ("checkout
landed on NEXT", "no successful update completion receipt"). The drivers
now record `hermes update --set-channel main` before the user-state
snapshot, the same way the pytest sandboxes already do (3a4b4b6f83).
Releases older than --set-channel follow main already and are skipped by a
help probe.
- Network E2E (test_offline_and_flaky, test_proxy_only_egress). They asserted
that `main` reads and retries releases/channels/main.json through the
proxy. #136429 removed that read on purpose (regional WAF 403s, #128295).
Rewritten to the new contract: `main` reads no release record, survives a
release-host outage, and a cut tunnel to github.com still fails fast and
changes nothing.
The desktop unit test with the same assumption was already fixed on main
(0aa764fedd).
#135847 made an official-origin source checkout default to the stable channel,
which resolves only from GitHub's latest published release. The upgrade-network,
Windows install+update and Desktop update harnesses keep the official URL as
origin, publish commits on main and never a release, so every update in them
stopped at the stable default:
- e2e-upgrade (network): "Could not resolve the stable source channel: No
published stable release could be verified" (9 failed)
- Windows install + update E2E: "Release: stable v0.21.6; this checkout is
newer and waits for the next release", checkout never moved (12 failed)
- Desktop update E2E (app-update, update-concurrency, build-fail): the update
check answered {"channel":"stable","error":"release-unavailable"}
These suites test moving between commits, so each seeded install now records
`hermes update --set-channel main` right after its installer (new
_install_helpers.pin_main_channel; Windows Machine.install). Stable resolution
keeps its own coverage: test_release_channel_records sets stable explicitly on
the same seeded install, and tests/hermes_cli/test_source_release_channels.py.
Local: tests/e2e/core/upgrade/network on main plus this change, 21 passed
(4 failed in test_offline_and_flaky.py on main before it).
Publishing a vX.Y.Z GitHub release is the whole promotion: the source
updater's stable channel now reads only GitHub's latest published release
(non-draft, non-prerelease, strict vX.Y.Z) and verifies the tag on origin.
It no longer asks R2 for releases/channels/stable.json first, and it
ignores the R2 release-candidates pointer, which the stable pipeline writes
before the GitHub draft is published.
Agent Plugins v1 has no trust field, and Hermes's portable mcp.json reader
accepts only type/url/headers for a remote server, so a package server always
ran at the default trust: full. A package whose tools trade, spend or send
(senpi resolving trade approvals on a user's Hyperliquid account) could only
ask the model to check with the user; Hermes itself never prompted.
plugin.json's Hermes extension now accepts
extensions.com.nousresearch.hermes.servers.<name>.trust: untrusted
which is applied to that mcp.json server exactly like a config.yaml
'trust: untrusted': every write-capable call asks first and fails closed
when unattended. 'full' (the default) is accepted and changes nothing, any
other value refuses the package, so a package can narrow access but never
widen it, and a config.yaml server with the same name still replaces the
package's entry. 'hermes plugins validate' lists the gated servers.
The finally SIGCONT raised "No such process" when the gateway was already
gone, masking the exception that ended the try block. A stop-request wait
timeout now carries the pause record and the update log tail.
Review fixes for the pre-swap gateway pause (independent review of a029ec2):
- Every gateway is drained before the code swap: the control socket's
pause-for-update runs the request_restart drain (the post-update restart's
SIGUSR1 path) and the update waits within the fleet's canonical budget
(_gateway_drain_budget, 1905 s by default) with progress lines. systemd
units and launchd jobs are stopped through their supervisor only after
their drained gateway has exited, which also keeps Restart=/KeepAlive from
relaunching it mid-update. A gateway that does not take the drain gets the
supervisor stop / SIGTERM as before. The request is recorded (stop_sent)
before it is sent and carries no planned-stop marker: the marker watcher
takes the immediate signal-stop path and could fire first. Before: a
supervised gateway got systemctl stop / bootout at once
(restart_drain_timeout=0 cuts an in-flight turn), a bare one was SIGKILLed
after 70 s.
- The pause record stores canonical identities (pause_record.identity), so
recovery recognises a still-live gateway; the native birth stamp stays on
the entry for _alive.
- Supervised posix_units go through drop_never_stopped / split_draining /
backoff / hand-back like every other entry: a killed update no longer owes a
live unit it never stopped, nor relaunches one still draining.
- Unknown cgroup membership (either side unreadable) never authorises
stopping the unit; cgroup v1 name=systemd is read when v2 is absent.
- A record that cannot be written restores the token exactly (fresh or
adopted), so the checkout's tree gate is not armed by a pause that never
happened.
- A replayed bare gateway's environment follows the restart watcher it
replaces: the launch profile's own named home keeps the launcher env (an
exported bot token can be its only credential); every other home and the
multiplexing root get the served-profile factory.
- Resuming a paused unit runs the definition repair the fleet restart runs
first (_repair_unit_without_fatal_exit_park; launchd plist refresh).
- Shared POSIX tests use platforms("posix") instead of a module skipif, so
the macOS lane imports them.
e2e cell (e) now reads the recorded stop request (record stop_sent) instead of
the planned-stop marker, which the drain path no longer writes.
The post-commit cell assumed a manual `gateway run` that died at boot leaves the retry nothing to
relaunch. The pre-swap pause records its argv, so the retry relaunches it on HEAD; the cell now
asserts that, keeping the invariant that the obligation discharges only once a gateway serves HEAD
(observed: PID 6 -> 1210, code_sha == HEAD). The operator-restart leg is gone: it would start a twin.
Explicit encoding + DEVNULL stdin on every subprocess.run, SIGKILL via getattr, BOM-tolerant
reads in the pause e2e cells (writes stay plain utf-8), and the macOS-only launchd test waives
os.getuid inline. The footgun lint runs on every file regardless of platform guards.
#134320 registered ProviderProfiles for lmstudio and tencent-tokenhub without env_vars. That put
both into the provider-catalog E2E matrix for the first time with key_env=None. The shard fixture
then died on keys[""] (KeyError, every row of the shard errored), TokenHub turns could not find a
key, and LM Studio's /model switch could not reach the native /api/v1/models listing the fake
never served.
The profiles now name the same env vars their hermes_cli/auth.py registry rows use (that row stays
the lookup source of truth). The catalog fake also answers LM Studio's native listing.
tirith ran `tirith check` on every terminal command, default-on with fail-open, as a
pinned PM binary auto-downloaded at startup. Measured against the shipped 0.4.2 binary
it caught 14/16 attack commands but flagged 12/24 everyday agent commands
(`echo $((6*7))`, `"$VAR/x.sh"`, `gh api | python3`, LAN curls, `.dev` hosts,
`pip install` via analysis_incomplete), and in cron/-q deny mode each of those is a
hard block. It only ever saw terminal strings (execute_code, writes and the browser
bypass it), fail-open meant no protection wherever the binary could not install
(Windows, Termux, x64 musl), and it produced a steady issue/PR stream (fail_open
semantics four times, install loops filling disks, a SIGSEGV hang, session-approval
scoping, third-party promo text on approval cards).
Removed: tools/tirith_security.py, the tirith branches of the command guard (the
is_tirith tuple, session-only "always" downgrade, import-error/fail-open paths,
unattended tirith pass), the CLI and gateway startup installers, the PM package and
lock pins, the nix runtime dep, security.tirith_* defaults, the CLI warning string
and three tirith tips (gaps refilled from the tail so translations survive), plus
the gateway's "no automated assessor" warning that only existed to point at tirith.
Config v50 drops security.tirith_* and enables nothing in its place. Hermes ships
no tirith plugin; if tirith upstream publishes one, it goes through normal catalog
admission under their name.
test_crash_cells carried its own marker parser and identity rule, and it
decides the orphaned_update verdict. It disagreed with the product on three
inputs: a float started_at (product: malformed, copy: live), a create time
1.5 s off (product: live within its 2.0 s tolerance, copy: dead at 1.0 s) and
a run: line before delegate: (product: delegate live, copy: dead, it read
line 4 only). The self-check claimed parity but never called update_lock and
used the test's own pid, which the product judges by exact incarnation.
_marker_live now asks update_lock._parse_marker(...).owner_live() /
delegate_live(); the crash cells run in a pytest process on the same Windows
host as the marker's pids, so the product's process table reads them as the
updater does. The self-check marks a separate live process with an integer
started_at, compares the oracle to judge_marker, and adds the three rows.
The ORPHAN_RELEASE_GRACE poll is unchanged.
Round 5's Start-MarkerCustodian takes a killed hand-off script's claim over and releases it on its
next 1 s poll after the update ends. Strict run 37393368977 read the marker ~2 s before that
release (custodian removed it at 01:01:32) and called it a surviving marker. The cell now waits up
to 30 s for the release; a genuinely stuck marker lives until the heartbeat ceiling, minutes.
run_tests.sh runs one pytest per file and reports a file whose tests were
all deselected (pytest exit 5) or all skipped as passed. 8ce4ce35fc2 made
the -m selection and platforms('windows') marker a tests/ci check, but any
other zero-execution path (the HERMES_E2E_WINDOWS_INSTALL opt-in lost from
the step env, a new skip condition) still ran zero crash cells green.
The crash journey now writes crash-cells-ran.txt (each journey step and
ok/failed) into HERMES_E2E_ARTIFACTS once it has run, and a new required
step "Every crash cell ran" after the run step throws when the manifest
is missing. The manifest is uploaded with the other artifacts.
Test (tests/ci, every PR): the step exists after the run step, is neither
disabled nor advisory (workflow_steps.required), reads the same
HERMES_E2E_ARTIFACTS and names the module's manifest constant; and
_record_cells_ran writes exactly the steps that ran, nothing without the
env. Red on the start workflow (no step) and with the step made
continue-on-error.
Review 5411223855 (F52: collected > 0 / executed-cells assertion).
All six crash cells share one module-scoped machine. A cell whose
follow-up update failed (torn checkout, .hermes-update-in-progress or
.git/index.lock left, or a step that raised) handed that state to every
later cell, which then reported the first cell's breakage as its own.
_run_cells now runs the cells in order and, before each one, checks what
the previous cell left: the checkout exactly its target, no marker, no git
lock, follow-up rc 0. When it is not, the harness stops the machine's
processes, drops the lock and the marker and resets the checkout with the
harness's git to that cell's target (else to the HEAD it left), and notes
the restore and its reason in machine.timings, so the evidence of any
later failure names the culprit. The broken cell keeps its own verdict:
its result was captured before the restore. If the harness cannot reset,
the next cell fails "not run: <cell> left the shared machine unsound
(...)" instead of running on it. This is harness plumbing like the stale
index.lock clean-up _crash already did; every target only appends
statements to modules, so the venv still runs the restored tree.
Tests: a broken cell, a raising cell and a sound cell in sequence (each
later cell sees the reset checkout, no marker, no lock, and the timings
name the culprit), and an unrestorable machine failing the next cell
with the culprit's name. Both fail on the start sha (no _run_cells).
Review 5411223855 (crash-cell cascade).
The mid_git cell names an unmodelled git (the hold filter ran, but no
merge/reset/checkout with .git/index.lock appeared) after 30 s of wall
time. While the filter holds, git, its op and index.lock cannot move: the
state is static and the poll only has to observe it, but on a starved
runner a tree walk can take seconds, so 30 s could be a handful of looks
and a slow runner could false-fail the cell. The verdict now also needs
HOLD_SETTLE_LOOKS (60) complete looks; a normal runner passes 60 looks
well inside 30 s, so its behaviour is unchanged, and the wait is still
bounded by _kill_when's UPDATE_TIMEOUT.
Test drives _GitHold.point with a 10 s-per-look clock: the start code
raised on the 4th look (40 s), now it raises only after 60 looks.
Review 5411223855 (crash-cell timing).
What: _kill_when judges the kill by `_kill_delivered`: taskkill rc 0, or its stdout carrying the
update root's own SUCCESS line ("with PID <pid> (child process of ...").
Why: the update's git now runs inside the update's kill-on-close job. Killing the update closes the
job, a member dies while `taskkill /T /F` is still walking the tree, and taskkill exits 128 even
though it terminated the update (rc 1 is the code /F forces). The mid-fetch and tree-moved cells
then reported "the update was not running when killed" on the strict composite run (artifact
receipt: outcome interrupted, transcript stops at the fetch). The mid-git cells passed only because
their git is frozen by a hold. "Not found" for the root stays a refusal.
Red: test_kill_point_counts_only_an_update_killed_while_running[None-128-...SUCCESS...-True-True]
fails on the old rc-only check. Green: tests/ci + tests/e2e/core/windows_update 546 passed.
ORPHAN_MARKER_GAP excused the orphaned_update cell's marker assertion with no
expiry. PR CI never runs that cell (Windows, opt-in) nor strict acceptance,
so after #132354 (hand-off script names its update child as the line-4
delegate) and #132365 (the judge honours a live delegate) land, a regression
of exactly that contract would still pass as an expected failure.
Now the fix's footprint is read from the tree (ORPHAN_MARKER_FIX):
- while any half is absent, _orphan_gap_excuse applies known_failure as before;
- once every half is present the excuse is off, so the cell is strict on any
runner whatever the wrapper says;
- tests/ci/test_windows_update_crash_oracles.py (runs on every PR) fails with
"delete ORPHAN_MARKER_GAP ..." once the fix is in the tree, the same expiry
shape as _LANDS_IN_SIBLING_PR, and fails if the Python half already on this
branch (update_lock.delegate_live) loses its footprint, so a rename cannot
keep the excuse alive.
Checked against the scripts branch: this tree + its marker.ps1/windows.ps1
reads no half missing.
Review CI1.
_git_op and _update_child skip any child whose name/argv read raises. Both
catches were broad with no (or a one-word) reason. The skip is fail-closed:
an unreadable child is never counted as the kill point, and _kill_when's
bounded wait turns a point never seen into a harness verdict, never a pass.
Kept broad on purpose (psutil on Windows raises more than psutil.Error for a
racing child) and now say so with a BLE001 waiver naming the boundary.
Review CI4.
What: _orphan baselines <HERMES_HOME>/logs/update_receipts before the hand-off starts; the cell passes on rc 0, or on rc 120 only when the newest receipt written since says outcome success.
Why: CPython replaces ANY exit status with 120 when the final flush fails, so a failure exit after the success line read as a completed update.
Red: pre-fix cell -> [120-failed] and [120-None] accepted; receipt helper missing (9 passed, 3 failed).
Green: tests/ci/test_windows_update_crash_oracles.py 12 passed, 0 failed.
What: the test's _identity_live takes the marker age and, for an identity without a creation time, is live only within hermes_cli.update_lock.UPDATE_MARKER_MAX_AGE_SECONDS.
Why: the oracle read a ct-less (v1) marker for a live pid as LIVE past the product's 20-minute ceiling, so it could disagree with the verdict update_lock gives every other updater.
Red: pre-fix test_crash_cells.py -> [1260-False-False] FAILED (6 passed, 1 failed).
Green: 7 passed, 0 failed.
What: _kill_when requires proc.poll() is None after the point is seen and taskkill_tree's rc 0 (it now returns the CompletedProcess); otherwise a harness AssertionError naming the point.
Why: tree_moved's point (HEAD at target) persists after a clean exit, so the cell could pass on an update that was never interrupted.
Red: pre-fix _kill_when/_machine -> exited-0 + truthy point and taskkill rc 128 cases accepted (2 passed, 2 failed).
Green: tests/ci/test_windows_update_crash_oracles.py 4 passed, 0 failed.
The launch repair's own entry module, unlinked by a merge held at git's smudge filter and
taskkilled with index.lock held. The next launch must restore the pre-update tree and run a
turn; the follow-up update must complete. Runs after mid_git: a launch that cannot repair this
tree leaves no runnable install for later cells.
(cherry picked from commit 1999a3868dc573dde1ec97745988823b9be1c30e)
(cherry picked from commit 232936de515c094292a56a9e2f6631457bc9d53d)
(cherry picked from commit 8c22ecbd465f80771eeb6d87ddea2be21e06ad55)
(cherry picked from commit a5ff463ce85b2b372661856525aff27d230f9226)
(cherry picked from commit 9f7a4d177929ca32cf1d25afd32dbb5005338db0)
The known_failure on the mid-merge launcher gap is gone: #132361's launchers import
hermes_bootstrap (the interrupted-pull repair) before any other checkout module, so the
merge-time kill must leave a runnable install with no gate.
(cherry picked from commit 96fd6724a86aa9b48ac7f88b56230e94f3447d7e)
(cherry picked from commit f975552309b97b7b2e4ac97b63ad278d185e2369)
Native run 37190464303 (first merge-time kill): git merge was held writing
hermes_constants.py with index.lock held and killed there. Every next
launch then died with ModuleNotFoundError: the launcher
(hermes_cli/_launchers.py::_launcher_script) imports hermes_constants from
the checkout before import hermes_bootstrap, whose
restore_interrupted_pull is the repair, so the repair never runs and even
hermes update fails. The bricked install then failed every later cell.
mid_git now runs last, and its recovery assertions sit under a
known_failure keyed on exactly that traceback (owner upd-txn: strict
acceptance fails on it). The hold precondition (index.lock left by the
kill) is asserted outside the gate.
(cherry picked from commit 7aca86f2341f6135b4da2666d58e24e3e14b7945)
(cherry picked from commit 95595e12f6a0e6ddc022a43e871c8e654149fbbe)
Every green native run killed the mid_git cell during git fetch (fetch was
in GIT_OPS and is the long step), so the merge-time index.lock schedule the
cell is named for never ran. A smudge filter the harness names in the
install's .git/config + .git/info/attributes (git's own filter hook; no
product change, no env var) now holds git merge while it writes
hermes_constants.py: index.lock held, the cell's new file and
hermes_cli/main.py already at the target, hermes_constants.py unlinked.
The kill point requires the hold, a merge/reset/checkout git child by argv
and index.lock; the cell asserts the lock was left. The fetch-time kill
keeps its own cell (mid_fetch).
(cherry picked from commit 592d57c3a9f78e04d617be659342e5338e080381)
(cherry picked from commit ba54f871bd580bb3fe9de7f840a7a21baf564dd2)
Moving the marker known_failure after the other assertions exposed a
harness race that the early XFAIL had hidden. PR run 37180721655 failed
the orphan cell with "did not finish (rc=None, 'Update complete!'
logged=True, tree at target, clean)". psutil.Process.wait returns None
for a non-child that has already exited. The orphaned update exits
between two 0.25 s samples, so its rc was lost.
_ExitCode opens PROCESS_QUERY_LIMITED_INFORMATION|SYNCHRONIZE on the
child before the kill and waits/reads GetExitCodeProcess through that
handle. The exit code stays readable after the process is gone, and a
reused pid cannot answer for it.
(cherry picked from commit 85488842f43b2154340231e2dd1a5e7900622487)
(cherry picked from commit 05c47385379ecda49e717e517425f5a9f559d76e)
- _assert_recovered compared HEAD SHAs only. A tree with HEAD at the target
and half the files from the old commit passed. Each target now appends a
statement to two modules every launch imports (hermes_constants.py,
hermes_cli/main.py), and right after the first launch the checkout must be
exactly the pre-update commit or the target: HEAD, `git status
--porcelain --untracked-files=no` clean, `git diff --quiet HEAD`, and the
target-only file present iff HEAD is the target. Same after the follow-up
update (target only) and after the orphaned update.
- The orphan cell's two known_failure blocks came first, so pytest.xfail
ended the test before its finish / first-launch / follow-up assertions.
Every unrelated assertion now runs first; the marker contract is one
block at the end (strict acceptance fails it: upd-txn owns the gap).
- _orphan took its completion-banner baseline after taskkill + wait, so a
completion logged during those waits counted as "not fresh". Both
baselines are taken before the script starts.
Linux probe of the oracle (real git, minted target, torn tree):
.../r5-ci/tree_probe.out — torn tree (HEAD=target, old hermes_constants.py)
matches neither commit; the old SHA-only check accepted it.
(cherry picked from commit 72308a85965635d9b746e52eccf35d91fcca75cf)
(cherry picked from commit 11df688a5831f2ed2e4a88b6a1fed5e7c9234a02)
known_failure XFAILs a gap while its fix is an open PR, which is right
on a PR and wrong on the acceptance run of the integrated batch: there
it accepts exactly the gap the batch claims to close (the final
composite's Windows run passed with `test_crash_cells.py (3✓ 1xf)`).
HERMES_E2E_STRICT_ACCEPTANCE=<owner>[,<owner>] (or 1 = every gap) makes
the matching known_failure / known_gate blocks re-raise the gap's own
assertion. It is set only from the `strict_acceptance` dispatch input of
ci.yaml (threaded through tests.yml / tests-os.yml to every E2E step)
and of windows-install-update-e2e.yml; pull requests never set it.
run_tests.sh forwards it through its env -i allowlist.
Red on 8cc78279a36: test_pending_fixes_strict.py (a real pytest process
per mode) — the batch-owned gap XFAILs under strict mode.
(cherry picked from commit ca1e7e749a61263181e89aa687dec366d95fda18)
(cherry picked from commit 7768bf3ef8e15fb7f228ebad619d6a9a85ad4100)
Run 37149266852 (delegate writers merged): the marker stayed LIVE via the delegate,
the update logged "Update complete!", its receipt says success and the checkout is at
the target, but the process exits 120: CPython's status when the final stdout flush
fails, and its stdout reader is the killed script. Require the checkout at the target
plus a fresh "Update complete!" in update.log, and rc 0 or 120.
(cherry picked from commit d9d965fc1e6f1ab3f5bcdfafe044e1439cf7d3da)
(cherry picked from commit e0ef639ebcc797bd4ff8f28ec1d6d8fd9d3cddcf)
Run 37148409574 (with the delegate writers merged) killed windows.ps1 0.4 s after its
`hermes update` child appeared, before that child reached UpdateLock.acquire(), so it
measured python start-up instead of the delegate contract. Wait for the child's
"Updating Hermes Agent..." banner in update.log (printed only after acquire) too.
(cherry picked from commit 274861c4ac40802afa03389a668d08e9c568d92c)
(cherry picked from commit 1f4d1ecd724285d00f898603f85fbf491b00900f)
