`_kill_tree` SIGKILLs the per-file pytest process group, which misses any
child that left the group with `start_new_session=True` (an auto-started
`gateway run`, a setsid server). Those are reparented to init and run
forever: one host accumulated 215 orphan gateways (~52 GB RSS, up to 21 h
old), and every `hermes update` listed each as "another Hermes home — left
alone".
After every attempt the runner now SIGKILLs any process whose environment
still points into that attempt's unique temp root (PYTEST_DEBUG_TEMPROOT /
TMPDIR, or a HOME / HERMES_HOME a fixture created under it). The root is
unique per attempt, so the match names only that attempt's processes.
Linux-only (/proc); the process-group kill stays the fallback elsewhere.
With --no-checkout the file download moved out of the retried clone into the single unretried pin checkout, which ran after the empty clone was already published: a throttled pin left INSTALL_DIR with no index, and the next rerun stashed a 'delete every file' autostash. The pin now runs in the staged tree inside the deferred-checkout retry loop (which also stops materializing the tip first), and nothing is published until it succeeds.
Mirrors the install.sh change from #135092 on Windows. With -Commit (what
the desktop bootstrap passes), the fresh clone runs with --no-checkout and
the pin is checked out with retries inside the staging dir, so the branch
tip is never materialized first and a refused or failed pin never publishes
a checkout. A rerun over an existing checkout keeps pinning after the fetch.
Review findings on the local build scripts:
- Cleanup ran before desktop.py took the checkout lock, so a second run deleted
the outputs of a build that was still going. desktop.py gains --clean, which
removes this checkout's outputs after build_lock is held. Both scripts pass it
instead of deleting files themselves.
- build-bundle.ps1 left RELEASE_CLAIM_* and HERMES_PAYLOAD_* in the caller's
session. It now saves those variables and restores them in the finally block.
- The scripts said "unsigned" but inherited signing credentials. The ps1 removes
AZURE_* for the build. The sh removes CSC_* and APPLE_* and sets
CSC_IDENTITY_AUTO_DISCOVERY=false, the switch the desktop rebuild command uses.
The wording now says what is ignored.
- The Store bundle step needs the claim timestamp, and the script deletes the
claim. The ps1 prints it as HERMES_RELEASE_EPOCH, and BUILDING.md documents it.
* fix(install-e2e): tolerate the update-result consume race and held lock files
Two HEAD -> NEXT legs of the stable-release matrix went red for harness reasons,
not product ones.
macOS (app-driven update): source-update-observer listed
.hermes-update-result.json and then read it. The relaunched app consumes
(unlinks) that file on boot, so the read could land after the unlink and throw
ENOENT, which launch-from-spec reported as a failed update although the update
receipt said success. A vanished file is now "not there", the same as before it
was listed.
Windows (user-state verify): verify-user-state hashed every judged file. The
gateway's cron ticker holds an msvcrt byte-range lock on cron/.tick.lock, so
open() raised PermissionError and the upgrade was reported as changing user
state. Proven on a real Windows host: base raises Errno 13, the fix records the
lock as a sentinel. A lock file is held by a live process and is never user
content, so it is recorded without being read. Any other unreadable entry now
surfaces as a ScanError that names the path instead of a bare errno.
* test(install-e2e): an unreadable user file still fails the scan with its path
Skipping lock files must not let a user file the verifier cannot read pass
silently. The test makes a real mode-000 file under memories/ and asserts
snapshot_home raises ScanError naming the path, with the PermissionError kept
as the cause. It goes red when the OSError -> ScanError wrapper is removed.
abandon refused with 'no outstanding attempt' when a version had two, the
same branch as zero, despite its docstring promising to clear one. It now
clears all of them, oldest first.
release.py changelog prints the notes the next draft would carry without
claiming an attempt. When the body is over GitHub's limit, the refusal now
says to cut with --no-changelog, dump the changelog, edit it down by hand,
and paste it into the draft.
A local-only rc.* tag (a hand-made ref or a claim that never pushed) counted
as an outstanding attempt, so release refused and abandon reported 'no
outstanding attempt'. The remote is the only authority on claims now.
The pipeline seeded its version line at 0.21.4, but the legacy CalVer line
already published "Hermes Agent v0.21.5" as v2026.9.24. A first `--bump patch`
cut therefore derived 0.21.5 again: installs already reporting 0.21.5 would
compare equal and never see the new build as an update, and the release page
would carry two v0.21.5 entries.
Seed at 0.21.5 so the first pipeline patch is 0.21.6 and the first minor stays
0.22.0. Entrypoint fixtures now pin their own published head instead of
tracking the production seed.
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
test_script_killed_before_publishing_the_delegate_runs_no_update failed on
the Windows arm64 lane with PermissionError reading
logs/desktop-update-handoff.log: windows.ps1 appends each line with
Add-Content, and a poll that lands while that write holds the file is a
Windows sharing violation, not a missing line. The poll helper now
retries the read (5 s budget) instead of failing the test on it.
Same test is green on main's latest arm64 run; nothing in this PR
touches the desktop-update scripts.
Nothing tested mac_bundle_recover(), the start-of-run repair that finishes or
rolls back a Hermes.app swap a killed hand-off left half done.
posix.sh --self-test-swap-interrupt (body in swap-selftest.sh) runs the REAL
mac_swap on a fixture bundle tree in its own process group. mac_swap now
names its step boundaries (swap_checkpoint start/staged/aside/installed); the
self-test makes the swapper SIGSTOP itself at one of them and SIGKILLs the
whole group there, or resumes it and SIGKILLs it mid-ditto / mid-cleanup rm.
It checks the wreck has the expected shape, runs the REAL mac_bundle_recover,
and asserts the target is the complete old or complete new bundle (whole-tree
checksum, version marker included), no .old/.new/.hermes-update-* is left,
and a second recovery changes and logs nothing. Seven cells: before-stage,
mid-stage, staged, aside, installed, mid-cleanup, complete (control).
The seams are inert in a real run: SWAP_PAUSE_AT is set only inside the
self-test, and MAC_DITTO stays /usr/bin/ditto unless the self-test runs where
ditto is absent (Linux: cp -R).
test_desktop_update_mac_swap_interrupt.py drives the arm under
platforms("posix"): the Linux suite and the macOS lanes of tests-os.yml
(real /usr/bin/ditto, macOS /bin/bash 3.2).
The seeded-state test (#130557) was skipped on macOS as a fixture problem,
but the failure was in its own new assertions, on every OS:
- enabled-features.json is {"schema": 1, "extras": [...]}, so the fact's
extras compare against its "extras" list, not the whole file.
- venv_is_current read the user store the fixture points HERMES_RUNTIME_DIR
at, so it never saw the fact native.py writes. The desktop app starts the
bundled backend with HERMES_RUNTIME_DIR set to the payload's tools dir
(electron/main.ts resolveHermesBackend); the check now does the same.
The skip is gone. Without the native.py line the test fails on
"venv fact is None".
command_flags sorts bundle names, so the example key must sort after
HERMES_HOME for the unset-then-env order this test pins. HERMES_SKIP_INTRO
did; HERMES_DATA_DIR_SUFFIX does not. Use HERMES_SHARED_AUTH_DIR.
Select the expected archive and embedded app version from the admitted
channel request, or the release tag for tagged builds. Channel builds
have no tag, so indexing it crashed before macOS packages could stage.
Keep filename, identity and provenance validation intact.
Regression coverage reproduces the crash on both architectures and checks
that wrong filenames, embedded versions and commits remain rejected.
Within moments of its claim the daemon hands marker line 1 to the custodian it
forked (fdab580431e). The launcher's start check (contract C2) still waited for
the daemon's own pid, so on a host where the handover beat its next 0.1 s poll
it never matched, and once the daemon finished it reported a running update as
a failed launch (exit 70). `marker_names_handoff` now accepts line 1 naming the
daemon or a process the daemon forked; a grandchild or an unrelated pid still
does not count.
Sabotage A/B (daemon held alive 2 s after the custodian takeover, real
hand-off via the shim-progress harness): old acceptance exits 70, new exits 0
and publishes the stage. Test: the predicate over custodian/daemon/grandchild/
unrelated, red on the previous marker.sh.
Same gap as the POSIX hand-off: the -MarkerOp custody watcher took line 1
over only after windows.ps1 exited (WaitForExit, then the marker lock), so
an old packaged Desktop reading in between (lines 1 and 2 only) deleted the
marker while the resumed `hermes update` or a completion survivor still
held the checkout.
Start-MarkerCustodian now names the watcher on line 1 under the marker lock
right after starting it, before any update work, and the hand-off judges
its claim as that watcher from then on ($script:MarkerOwner in
New-MarkerContext): delegate publication, heartbeat and release keep
working. Invoke-MarkerRelease stops the watcher only after the release, so
line 1 stays live until then. Invoke-MarkerCustody keeps a claim that
already names it (judged by the creation time the hand-off recorded) and
falls back to the old takeover only when the handover failed.
Tests (Windows CI lane): the kill cell holds the marker lock across
taskkill /F and runs the shipped legacy reader at once; the pre-publication
kill cell and the A7 race expect the custodian on line 1.
The previous fix let the line-2 refresher take the marker over only AFTER
the hand-off died: it polls once a second and then waits for the A7 lock,
and until it rewrote line 1 the marker named the dead hand-off. An old
packaged Desktop (judges lines 1 and 2 alone, ignores A7 and the delegate)
reading in that gap deleted the marker and booted its backend while the
`hermes update` the hand-off started, or a completion survivor, still held
the checkout lock -- and the custodian then found no marker to keep.
Now the custodian is named on line 1 BEFORE any update work starts:
marker_refresher_start forks the refresher, then, under A7, rewrites the
claim's line 1 to the refresher's exact pid + creation time and adopts that
identity (MY_PID/MY_CT) for every later marker mutation (delegate, refresh,
release). posix.sh checks its own liveness with $$. The refresher keeps
line 2 young; if the hand-off dies it keeps the marker until the delegate
and the checkout lock are gone, then releases it. The hand-off releases the
marker before it stops the custodian, so line 1 stays live until the
release. If the handover fails (A7 busy), the old takeover-on-death path
still applies.
Test: the orchestrator-killed cell now holds the A7 lock across the SIGKILL
and runs the shipped legacy reader at once (no takeover is possible), then
keeps the post-takeover phases. Red on 5f855a2392c: {'live': None,
'kept': False}. Tests that asserted line 1 == the hand-off pid now expect
the custodian (that assertion encoded the vulnerable state).
marker.sh::checkout_lock_held and marker.ps1::Test-CheckoutLockHeld answered
"not held" when the lock file exists but cannot be opened (root-owned after
`sudo hermes update`, or held open without sharing on Windows), while the
Desktop's own probes (checkout_held / WINDOWS_CHECKOUT_LOCK_PS) answer "held".
The script helper's reclaim then deleted a dead marker the Desktop's probe
keeps. Both scripts now fail closed on every open/probe error except a
missing file: reclaim needs a provably free checkout. Parity tests: the
POSIX probe with perl, python3 and flock(1), plus an end-to-end reclaim; the
Windows reclaim against a lock file held with no sharing (Windows CI lane).
Same shape as the POSIX hand-off: windows.ps1 heartbeats line 2 from its
own process, so `taskkill /F` (no finally runs) left line 1 naming a dead
pid while the resumed `hermes update` (the line-4 delegate) or a completion
survivor still held the checkout, and an old packaged Desktop - which judges
lines 1 and 2 alone - deleted the marker and booted its backend.
After its claim the hand-off starts a hidden watcher (-MarkerOp custody
-CustodyOf <pid>, Start-MarkerCustodian). Once the hand-off exits with its
claim still published and the delegate running or the checkout lock held,
the watcher names itself on line 1 under the marker lock, heartbeats line 2
and releases the marker when both end (bounded like the R6 wait).
Invoke-MarkerRelease stops the watcher once the hand-off's own release is
decided. The Windows kill-cell test now also runs the reader be3fd671d7
shipped; that reader runner moves to a helper shared with the POSIX test.
An older packaged Desktop judges the update marker by line 1 (pid alive) and
line 2 (under 20 minutes) alone and unlinks it otherwise, never consulting
the protocol-2 delegate line or the marker helper. The line-2 refresher
exited with the hand-off script, so a SIGKILLed hand-off left line 1 naming
a dead pid while the `hermes update` it started, or a completion survivor,
still held the checkout lock: the old Desktop deleted the marker and booted
its backend into a checkout still being mutated.
The refresher now outlives the hand-off as its custodian (marker_custody):
once the hand-off is gone, if its claim is still published and the delegate
runs or the checkout lock is held, it names itself on line 1 (exact pid +
creation time, under the A7 lock), keeps line 2 young, and releases the
marker only after both are gone (bounded like the R6 wait). The test runs
the reader be3fd671d7 shipped (vendored verbatim under tests/fixtures).
The hand-off scripts gave the Desktop a fixed 30 s (windows.ps1) or 60 s
(posix.sh) to exit, but the Desktop's quit can first join a managed SSH update
it is running, which legitimately takes longer. The hand-off then recorded a
failed update although the Desktop exited safely moments later (review H2).
Both twins now share one 150 s ceiling, measured on the clock rather than as a
loop count, and still refuse with "Nothing was changed" if the Desktop is alive
at the ceiling, so nothing is ever replaced under a live Desktop. The ceiling is
overridable through HERMES_UPDATE_DESKTOP_EXIT_SECONDS only so the self-tests
need not sit it out, like the existing pipe-drain and step-idle overrides.
Tests: a Desktop still alive at the ceiling refuses with exit 4 and never runs
the update (red on the start sha, which waited out its hardcoded 60 s); a
Desktop that exits inside the ceiling lets the update run (control).
The script helper's `reclaim` answered `absent` the moment the marker file
was missing, before the checkout lock was looked at, and the Desktop gate
(update-marker-gate.ts liveMarkerProbe) returned "nothing runs" for a
missing marker without asking the helper at all. An update that holds the
checkout lock before it publishes its marker, or after it retired it (the
CLI paths of the stacked PRs do both), therefore let the Desktop boot a
backend into a checkout still being written.
- marker.sh marker_op_reclaim_locked / marker-claim.ps1 Invoke-MarkerOp:
no marker -> `held` when the checkout lock (owner byte + lease bytes on
Windows) is held, else `absent`, under the same marker lock.
- update-marker-gate.ts (gate-lane file, additive): a missing marker is also
routed through the helper; only its positive `held` keeps the gate shut
(blocked screen, Retry, Start anyway as for a dead marker). Any other
answer, `error` included, proceeds as before, so a broken script never
blocks a markerless boot.
The gate test "a live owner never reaches the helper; an absent marker
neither" pinned the missing check for the absent half; that half is now the
new "no marker: only held keeps the gate shut" case.
Review 5411223284 (open thread: update-marker-gate.ts ~193 / helper absent).
The S3 cell's root skip referenced os.geteuid directly, which
check-windows-footguns flags (absent on Windows). Same skip, resolved through
getattr.
review S3
Both hand-off scripts treated ANY nonzero `hermes update` exit as ok:false
"Update failed". After the commit point `hermes update` can still exit
nonzero by design -- 130 (Ctrl-C after the code moved: receipt outcome
`interrupted`) and 1 (a parked autostash: `partial`) -- or crash after a
committed run (`success`). The Desktop then said the update failed while the
new code was installed (contract C3 / invariant 3).
The scripts now hand the update a correlation id (HERMES_UPDATE_CORRELATION_ID,
already honoured by update_receipt._launcher_correlation_id; a caller's own id
is kept) and, on a nonzero exit other than 2 (refused), read the root's
logs/update_receipts/latest.json. Only THIS run's finalized receipt counts
(matching correlation_id, finished_at set, outcome success|partial|
interrupted); a record reconciled as interrupted keeps finished_at null and
never matches. Then the result is ok:true, manual, with an `update` warning
naming the exit and outcome, and the post-commit steps (Windows: verify,
gateway start --all) still run.
posix.sh also routes its own signal handler through the same judgement: a
HUP/INT/QUIT/TERM that lands after the update committed (exit 0, or a
committed receipt) is an owed `handoff` follow-up, exit 0, not
"hand-off was interrupted".
Tests: posix live cells (RED at start) for interrupted/130, partial/1,
success/1 and an INT delivered while the committed child finishes, plus
negative controls (no receipt, failed, another run's receipt, a reconciled
unfinished record); the Windows twin cells.
review S5
posix.sh::finish and windows.ps1's finally write the result (finished_at =
now) BEFORE the R6 release wait, which lasts up to 7200 s while a survivor
holds the checkout lock, and before the relaunch. The relaunched Desktop has
no expected run id and handoff-result.ts drops a non-manual result older than
30 min, so a failure or warnings result was silently discarded after a long
wait.
The early write stays (a kill during the wait still leaves a result); when the
wait actually ran, the result is published again after it with the real
finish time, unless something already consumed it. The release records that
it waited (RELEASE_WAITED / $script:MarkerReleaseWaited).
Tests: posix live cell (RED at start: finished_at 3 s before the survivor
let go) and the Windows twin with a timed checkout-lock holder.
review S2
Write-Result writes <result>.<pid>.tmp and then Moves/Replaces it over the
result. When both fail (result path locked, a directory, ACLs) the catch only
logged, leaving one tmp file per hand-off beside the result. The tmp is now
removed in a finally (it is gone already after a successful publish), the
same as posix.sh's write_result.
Test (Windows live cell): with the result path occupied by a directory the
hand-off still exits 0 and leaves no .hermes-update-result.json.*.tmp.
review S4
json_escape escaped only \ " \n \r \t. Every result/status string it builds
(FINAL_MSG, branch, channel, warnings) can carry text from git/ps/hermes
output; a raw ESC, BEL or any other char below 0x20 made the whole
.hermes-update-result.json invalid, and the Desktop dropped the outcome.
The remaining control chars are now written as \u00XX. Bash 3.2 compatible:
printf -v, no forks; the loop runs only when the string holds a control char.
Test (RED at start): json_escape over chars 1..127 plus UTF-8 and an ANSI
colour sequence round-trips through json.loads, in the C and C.UTF-8 locales.
review S7
Two parity gaps between marker.sh and marker-claim.ps1:
- adopt-run (--handoff-run), withdraw's `withdrawn` and `taken` accepted
J_OWNER_STATE==2, which also covers "alive, creation time unreadable, line 2
younger than 20 min". marker-claim.ps1 requires Test-ProcessIdentityExact
(both creation times known, within 2 s). An unreadable ct is no proof the
pid is still that process; these three now require ident_state 0, and
`taken` skips our own pid like the ps1.
- reclaim / withdraw printed `reclaimed` / `withdrawn` after an `rm -f` that
exits 0 even when it cannot unlink; the Desktop then proceeded as if the
marker were gone. marker_remove checks the marker is gone; else `busy`, the
ps1's answer when Remove-MarkerLocked fails.
Tests (RED at start): the three identity cases with every foreign ct
unreadable (and the readable control), and a reclaim inside a read-only
home that must answer `busy` and keep the marker.
review S3
Get-LiveProcessCt caches a pid's creation time for the script's lifetime and
forgets it only when a probe SEES the pid dead. A delegate (the `hermes update`
child) that exits and whose pid is reused between two probes keeps the old
incarnation's time, so the release judged the unrelated new process live and
handed the marker to it (A7 rule 5), parking every later update behind it.
Every judgement made through New-MarkerContext (claim, delegate, heartbeat,
release) and the delegate line Add-MarkerDelegate writes now pass -Fresh,
which re-reads the time. Pollers that only ask "still alive?" (Wait-DesktopExit)
keep the cache.
Test (Windows live cell): a release with the delegate's pid cached at the
recorded ct but really owned by another incarnation deletes the marker instead
of handing it over. marker.sh reads /proc or ps on every call: no cache.
review S6
marker.ps1::Test-CheckoutLockHeld probed only the owner byte at 1 MiB. Since R5b
every joiner the kill-on-close job refused also locks one of the 16 lease bytes
after it, and hermes_cli/update_lock.py::_try_lock treats any held lease as a
held lock. With the owner dead and a leased completion child still running, the
hand-off read the checkout as free, so the R6 release wait ended and the
Desktop's reclaim deleted the marker next to a live mutator.
Lock/Unlock now cover 1+16 bytes in one LockFile (it fails if any byte is held).
marker.sh needs no change: POSIX takes the checkout lock with flock on the whole
file and the lease bytes exist only on win32 (_take_lease).
Tests: a live Windows cell holds only a lease byte (1 and 16) and expects
`held`; a static cell, runnable on Linux, pins the probed range (RED at start:
(1048576, 1)).
review S1
What: run_bounded's no-perl branch starts the probe under bash job control (set -m), so it gets its own process group and the timeout's kill -KILL -- -pid reaches its children.
Why: without perl the probe stayed in the caller's group, the group kill failed, and a hung hermes --version / update --help left its python child running.
Red: pre-fix posix.sh fails test_timed_out_probe_is_killed_with_its_whole_process_tree[absent] (grandchild alive after rc=124). Green: both [present] and [absent] pass.
Review R8 m7: after the merge with main's 732b63d61a the probe decodes launcher
stdout as UTF-8 through ProcessStartInfo.StandardOutputEncoding and never touches
[Console]::OutputEncoding, so the test docstrings no longer claim the call sites
scope and restore it. runtime.ps1 keeps a one-line #124526 pointer at the
encoding line. Behaviour and assertions are unchanged.
87c67de3778 made the Windows claim carry its stable run:<id> line, so the
self-test marker is pid / started / ct / run. Pin the four-line body and the
run token grammar instead of the pre-run three-line count.
(cherry picked from commit f2bc6d679a17b17d996a27797116fe4dd7304776)
The module imported fcntl at top level, so collecting it on Windows failed
with ModuleNotFoundError before the platforms("linux") mark could skip it
(seen in every wine2e desktop_update run). pytest.importorskip skips the
module there instead.
(cherry picked from commit 06ff00e654db11083d044d4a87d94877a04a895e)
