The hand-off pid test only checked which pid the first update child was
told, and it was gated to Linux, so the macOS runner (where the refusal
was reported) never ran it.
It now drives the real posix.sh hand-off, has the update start a second
generation detached from it (the relaunch / takeover shape), and runs the
real UpdateLock there: on main that generation refuses with the custodian
as holder; with the fix it adopts. Gated "posix" so the macOS lane runs it
against marker.sh's whole-second creation times.
The salvaged test grepped posix.sh for the export line (tests must not read source). This
runs the real posix.sh hand-off against a launcher-backed stub install and records, inside the
update child, the HERMES_UPDATE_HANDOFF_PID it was given and marker line 1 at that moment:
they must be the same live custodian. Red on main: the child is told the script pid while
line 1 names the custodian.
The posix hand-off hands marker line 1 to a custodian subshell
(marker_refresher_start) and then exported HERMES_UPDATE_HANDOFF_PID="$$" —
the hand-off script's own pid.
`hermes update` adopts a live claim only when the owner is its own pid, the
HERMES_UPDATE_HANDOFF_PID value, or an ancestor
(update_lock.UpdateLock._is_partner). The custodian is a *sibling* of the
hand-off script, so it is none of the three and the ancestry walk can never
reach it: every desktop-initiated update refused the claim its own script had
just made and exited 2 ("Another Hermes update is already running ...
process <custodian>"), leaving the Desktop to relaunch and retry forever.
Export $MY_PID instead: the custodian once the handover succeeded, and the
script's own pid when it did not.
Fixes#134268
The builder shipped ~1.9k skills.sh rows per run with no resolved_github_id.
Most were rows whose repo 404s or whose fully-read tree has no matching
SKILL.md dir (renamed/removed upstream); every client install of one spent
~40 GitHub calls and failed as stale_index.
- fuzzy match ignores both '-' and '_' (pdb-database slug vs pdb_database dir)
- a single-skill repo (root or generic skills/ SKILL.md) resolves like the client does
- repo 404 or complete tree with no match: the row is dropped, counts logged
- truncated tree: bounded contents/<candidate>/SKILL.md probes, row kept if none hit
- transient errors keep the row unresolved, as before
A release that skipped bundles never writes the stable baseline pointer,
so its absence is normal at any point, not only before the first bundle
release. Drop the final-tag scan. A 404 on the default pointer now always
plans no upgrade arm. A supplied baseline-manifest that does not exist
still blocks, because that is a bad input.
No stable release has shipped signed packages yet. Every final tag so far
skipped bundles or predates the claim pipeline. So
releases/stable/release-candidates.json does not exist, and the three
"Pin actual OLD and NEW signed packages" jobs refused with "No published
stable package baseline".
transitions now treats a 404 as a first release only when no final stable
tag records a candidate manifest and no baseline-manifest input was given.
It still validates the receipt, emits empty matrices and baseline=none.
The packaged upgrade arms skip on that output. The gate accepts their skip
only when their own planner succeeded with baseline=none, so a missing
baseline after a real bundle release still blocks.
`_kill_tree` SIGKILLs the per-file pytest process group, which misses any
child that left the group with `start_new_session=True` (an auto-started
`gateway run`, a setsid server). Those are reparented to init and run
forever: one host accumulated 215 orphan gateways (~52 GB RSS, up to 21 h
old), and every `hermes update` listed each as "another Hermes home — left
alone".
After every attempt the runner now SIGKILLs any process whose environment
still points into that attempt's unique temp root (PYTEST_DEBUG_TEMPROOT /
TMPDIR, or a HOME / HERMES_HOME a fixture created under it). The root is
unique per attempt, so the match names only that attempt's processes.
Linux-only (/proc); the process-group kill stays the fallback elsewhere.
With --no-checkout the file download moved out of the retried clone into the single unretried pin checkout, which ran after the empty clone was already published: a throttled pin left INSTALL_DIR with no index, and the next rerun stashed a 'delete every file' autostash. The pin now runs in the staged tree inside the deferred-checkout retry loop (which also stops materializing the tip first), and nothing is published until it succeeds.
Mirrors the install.sh change from #135092 on Windows. With -Commit (what
the desktop bootstrap passes), the fresh clone runs with --no-checkout and
the pin is checked out with retries inside the staging dir, so the branch
tip is never materialized first and a refused or failed pin never publishes
a checkout. A rerun over an existing checkout keeps pinning after the fetch.
Review findings on the local build scripts:
- Cleanup ran before desktop.py took the checkout lock, so a second run deleted
the outputs of a build that was still going. desktop.py gains --clean, which
removes this checkout's outputs after build_lock is held. Both scripts pass it
instead of deleting files themselves.
- build-bundle.ps1 left RELEASE_CLAIM_* and HERMES_PAYLOAD_* in the caller's
session. It now saves those variables and restores them in the finally block.
- The scripts said "unsigned" but inherited signing credentials. The ps1 removes
AZURE_* for the build. The sh removes CSC_* and APPLE_* and sets
CSC_IDENTITY_AUTO_DISCOVERY=false, the switch the desktop rebuild command uses.
The wording now says what is ignored.
- The Store bundle step needs the claim timestamp, and the script deletes the
claim. The ps1 prints it as HERMES_RELEASE_EPOCH, and BUILDING.md documents it.
* fix(install-e2e): tolerate the update-result consume race and held lock files
Two HEAD -> NEXT legs of the stable-release matrix went red for harness reasons,
not product ones.
macOS (app-driven update): source-update-observer listed
.hermes-update-result.json and then read it. The relaunched app consumes
(unlinks) that file on boot, so the read could land after the unlink and throw
ENOENT, which launch-from-spec reported as a failed update although the update
receipt said success. A vanished file is now "not there", the same as before it
was listed.
Windows (user-state verify): verify-user-state hashed every judged file. The
gateway's cron ticker holds an msvcrt byte-range lock on cron/.tick.lock, so
open() raised PermissionError and the upgrade was reported as changing user
state. Proven on a real Windows host: base raises Errno 13, the fix records the
lock as a sentinel. A lock file is held by a live process and is never user
content, so it is recorded without being read. Any other unreadable entry now
surfaces as a ScanError that names the path instead of a bare errno.
* test(install-e2e): an unreadable user file still fails the scan with its path
Skipping lock files must not let a user file the verifier cannot read pass
silently. The test makes a real mode-000 file under memories/ and asserts
snapshot_home raises ScanError naming the path, with the PermissionError kept
as the cause. It goes red when the OSError -> ScanError wrapper is removed.
abandon refused with 'no outstanding attempt' when a version had two, the
same branch as zero, despite its docstring promising to clear one. It now
clears all of them, oldest first.
release.py changelog prints the notes the next draft would carry without
claiming an attempt. When the body is over GitHub's limit, the refusal now
says to cut with --no-changelog, dump the changelog, edit it down by hand,
and paste it into the draft.
A local-only rc.* tag (a hand-made ref or a claim that never pushed) counted
as an outstanding attempt, so release refused and abandon reported 'no
outstanding attempt'. The remote is the only authority on claims now.
The pipeline seeded its version line at 0.21.4, but the legacy CalVer line
already published "Hermes Agent v0.21.5" as v2026.9.24. A first `--bump patch`
cut therefore derived 0.21.5 again: installs already reporting 0.21.5 would
compare equal and never see the new build as an update, and the release page
would carry two v0.21.5 entries.
Seed at 0.21.5 so the first pipeline patch is 0.21.6 and the first minor stays
0.22.0. Entrypoint fixtures now pin their own published head instead of
tracking the production seed.
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
test_script_killed_before_publishing_the_delegate_runs_no_update failed on
the Windows arm64 lane with PermissionError reading
logs/desktop-update-handoff.log: windows.ps1 appends each line with
Add-Content, and a poll that lands while that write holds the file is a
Windows sharing violation, not a missing line. The poll helper now
retries the read (5 s budget) instead of failing the test on it.
Same test is green on main's latest arm64 run; nothing in this PR
touches the desktop-update scripts.
Nothing tested mac_bundle_recover(), the start-of-run repair that finishes or
rolls back a Hermes.app swap a killed hand-off left half done.
posix.sh --self-test-swap-interrupt (body in swap-selftest.sh) runs the REAL
mac_swap on a fixture bundle tree in its own process group. mac_swap now
names its step boundaries (swap_checkpoint start/staged/aside/installed); the
self-test makes the swapper SIGSTOP itself at one of them and SIGKILLs the
whole group there, or resumes it and SIGKILLs it mid-ditto / mid-cleanup rm.
It checks the wreck has the expected shape, runs the REAL mac_bundle_recover,
and asserts the target is the complete old or complete new bundle (whole-tree
checksum, version marker included), no .old/.new/.hermes-update-* is left,
and a second recovery changes and logs nothing. Seven cells: before-stage,
mid-stage, staged, aside, installed, mid-cleanup, complete (control).
The seams are inert in a real run: SWAP_PAUSE_AT is set only inside the
self-test, and MAC_DITTO stays /usr/bin/ditto unless the self-test runs where
ditto is absent (Linux: cp -R).
test_desktop_update_mac_swap_interrupt.py drives the arm under
platforms("posix"): the Linux suite and the macOS lanes of tests-os.yml
(real /usr/bin/ditto, macOS /bin/bash 3.2).
The seeded-state test (#130557) was skipped on macOS as a fixture problem,
but the failure was in its own new assertions, on every OS:
- enabled-features.json is {"schema": 1, "extras": [...]}, so the fact's
extras compare against its "extras" list, not the whole file.
- venv_is_current read the user store the fixture points HERMES_RUNTIME_DIR
at, so it never saw the fact native.py writes. The desktop app starts the
bundled backend with HERMES_RUNTIME_DIR set to the payload's tools dir
(electron/main.ts resolveHermesBackend); the check now does the same.
The skip is gone. Without the native.py line the test fails on
"venv fact is None".
command_flags sorts bundle names, so the example key must sort after
HERMES_HOME for the unset-then-env order this test pins. HERMES_SKIP_INTRO
did; HERMES_DATA_DIR_SUFFIX does not. Use HERMES_SHARED_AUTH_DIR.
Select the expected archive and embedded app version from the admitted
channel request, or the release tag for tagged builds. Channel builds
have no tag, so indexing it crashed before macOS packages could stage.
Keep filename, identity and provenance validation intact.
Regression coverage reproduces the crash on both architectures and checks
that wrong filenames, embedded versions and commits remain rejected.
Within moments of its claim the daemon hands marker line 1 to the custodian it
forked (fdab580431e). The launcher's start check (contract C2) still waited for
the daemon's own pid, so on a host where the handover beat its next 0.1 s poll
it never matched, and once the daemon finished it reported a running update as
a failed launch (exit 70). `marker_names_handoff` now accepts line 1 naming the
daemon or a process the daemon forked; a grandchild or an unrelated pid still
does not count.
Sabotage A/B (daemon held alive 2 s after the custodian takeover, real
hand-off via the shim-progress harness): old acceptance exits 70, new exits 0
and publishes the stage. Test: the predicate over custodian/daemon/grandchild/
unrelated, red on the previous marker.sh.
Same gap as the POSIX hand-off: the -MarkerOp custody watcher took line 1
over only after windows.ps1 exited (WaitForExit, then the marker lock), so
an old packaged Desktop reading in between (lines 1 and 2 only) deleted the
marker while the resumed `hermes update` or a completion survivor still
held the checkout.
Start-MarkerCustodian now names the watcher on line 1 under the marker lock
right after starting it, before any update work, and the hand-off judges
its claim as that watcher from then on ($script:MarkerOwner in
New-MarkerContext): delegate publication, heartbeat and release keep
working. Invoke-MarkerRelease stops the watcher only after the release, so
line 1 stays live until then. Invoke-MarkerCustody keeps a claim that
already names it (judged by the creation time the hand-off recorded) and
falls back to the old takeover only when the handover failed.
Tests (Windows CI lane): the kill cell holds the marker lock across
taskkill /F and runs the shipped legacy reader at once; the pre-publication
kill cell and the A7 race expect the custodian on line 1.
The previous fix let the line-2 refresher take the marker over only AFTER
the hand-off died: it polls once a second and then waits for the A7 lock,
and until it rewrote line 1 the marker named the dead hand-off. An old
packaged Desktop (judges lines 1 and 2 alone, ignores A7 and the delegate)
reading in that gap deleted the marker and booted its backend while the
`hermes update` the hand-off started, or a completion survivor, still held
the checkout lock -- and the custodian then found no marker to keep.
Now the custodian is named on line 1 BEFORE any update work starts:
marker_refresher_start forks the refresher, then, under A7, rewrites the
claim's line 1 to the refresher's exact pid + creation time and adopts that
identity (MY_PID/MY_CT) for every later marker mutation (delegate, refresh,
release). posix.sh checks its own liveness with $$. The refresher keeps
line 2 young; if the hand-off dies it keeps the marker until the delegate
and the checkout lock are gone, then releases it. The hand-off releases the
marker before it stops the custodian, so line 1 stays live until the
release. If the handover fails (A7 busy), the old takeover-on-death path
still applies.
Test: the orchestrator-killed cell now holds the A7 lock across the SIGKILL
and runs the shipped legacy reader at once (no takeover is possible), then
keeps the post-takeover phases. Red on 5f855a2392c: {'live': None,
'kept': False}. Tests that asserted line 1 == the hand-off pid now expect
the custodian (that assertion encoded the vulnerable state).
marker.sh::checkout_lock_held and marker.ps1::Test-CheckoutLockHeld answered
"not held" when the lock file exists but cannot be opened (root-owned after
`sudo hermes update`, or held open without sharing on Windows), while the
Desktop's own probes (checkout_held / WINDOWS_CHECKOUT_LOCK_PS) answer "held".
The script helper's reclaim then deleted a dead marker the Desktop's probe
keeps. Both scripts now fail closed on every open/probe error except a
missing file: reclaim needs a provably free checkout. Parity tests: the
POSIX probe with perl, python3 and flock(1), plus an end-to-end reclaim; the
Windows reclaim against a lock file held with no sharing (Windows CI lane).
Same shape as the POSIX hand-off: windows.ps1 heartbeats line 2 from its
own process, so `taskkill /F` (no finally runs) left line 1 naming a dead
pid while the resumed `hermes update` (the line-4 delegate) or a completion
survivor still held the checkout, and an old packaged Desktop - which judges
lines 1 and 2 alone - deleted the marker and booted its backend.
After its claim the hand-off starts a hidden watcher (-MarkerOp custody
-CustodyOf <pid>, Start-MarkerCustodian). Once the hand-off exits with its
claim still published and the delegate running or the checkout lock held,
the watcher names itself on line 1 under the marker lock, heartbeats line 2
and releases the marker when both end (bounded like the R6 wait).
Invoke-MarkerRelease stops the watcher once the hand-off's own release is
decided. The Windows kill-cell test now also runs the reader be3fd671d7
shipped; that reader runner moves to a helper shared with the POSIX test.
An older packaged Desktop judges the update marker by line 1 (pid alive) and
line 2 (under 20 minutes) alone and unlinks it otherwise, never consulting
the protocol-2 delegate line or the marker helper. The line-2 refresher
exited with the hand-off script, so a SIGKILLed hand-off left line 1 naming
a dead pid while the `hermes update` it started, or a completion survivor,
still held the checkout lock: the old Desktop deleted the marker and booted
its backend into a checkout still being mutated.
The refresher now outlives the hand-off as its custodian (marker_custody):
once the hand-off is gone, if its claim is still published and the delegate
runs or the checkout lock is held, it names itself on line 1 (exact pid +
creation time, under the A7 lock), keeps line 2 young, and releases the
marker only after both are gone (bounded like the R6 wait). The test runs
the reader be3fd671d7 shipped (vendored verbatim under tests/fixtures).
The hand-off scripts gave the Desktop a fixed 30 s (windows.ps1) or 60 s
(posix.sh) to exit, but the Desktop's quit can first join a managed SSH update
it is running, which legitimately takes longer. The hand-off then recorded a
failed update although the Desktop exited safely moments later (review H2).
Both twins now share one 150 s ceiling, measured on the clock rather than as a
loop count, and still refuse with "Nothing was changed" if the Desktop is alive
at the ceiling, so nothing is ever replaced under a live Desktop. The ceiling is
overridable through HERMES_UPDATE_DESKTOP_EXIT_SECONDS only so the self-tests
need not sit it out, like the existing pipe-drain and step-idle overrides.
Tests: a Desktop still alive at the ceiling refuses with exit 4 and never runs
the update (red on the start sha, which waited out its hardcoded 60 s); a
Desktop that exits inside the ceiling lets the update run (control).
