Nothing tested mac_bundle_recover(), the start-of-run repair that finishes or
rolls back a Hermes.app swap a killed hand-off left half done.
posix.sh --self-test-swap-interrupt (body in swap-selftest.sh) runs the REAL
mac_swap on a fixture bundle tree in its own process group. mac_swap now
names its step boundaries (swap_checkpoint start/staged/aside/installed); the
self-test makes the swapper SIGSTOP itself at one of them and SIGKILLs the
whole group there, or resumes it and SIGKILLs it mid-ditto / mid-cleanup rm.
It checks the wreck has the expected shape, runs the REAL mac_bundle_recover,
and asserts the target is the complete old or complete new bundle (whole-tree
checksum, version marker included), no .old/.new/.hermes-update-* is left,
and a second recovery changes and logs nothing. Seven cells: before-stage,
mid-stage, staged, aside, installed, mid-cleanup, complete (control).
The seams are inert in a real run: SWAP_PAUSE_AT is set only inside the
self-test, and MAC_DITTO stays /usr/bin/ditto unless the self-test runs where
ditto is absent (Linux: cp -R).
test_desktop_update_mac_swap_interrupt.py drives the arm under
platforms("posix"): the Linux suite and the macOS lanes of tests-os.yml
(real /usr/bin/ditto, macOS /bin/bash 3.2).
Get-DefaultBrowserExe read only the legacy UserChoice ProgId, which Windows
11 25H2 Settings no longer updates, so the update hand-off fell back to the
WinForms card (or a stale browser) on machines whose real default is
Chrome/Edge. Read UserChoiceLatest\ProgId first, legacy UserChoice second.
Fixes#108051. Follows #108062 / #108064.
Co-authored-by: KoNit-K <konit.block@protonmail.com>
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
Importing a/b/c.py runs a/__init__.py and a/b/__init__.py first. The
update-lane routing listed hermes_cli/local_runtime/processes.py (the
v2026.9.24 updater imports it after the pull) but not the package init
that import runs. #134116 changed what that init loads, its PR never ran
e2e-upgrade, and main went red until #134242.
classify_changes now routes the package inits above every module the
e2e_upgrade and e2e_desktop_update lanes own, and above each shared hub,
because a hub's inits run whenever the hub is imported. An init that is
itself a hub stays out. The routing guard's import walker counts the
package inits a module's imports run, so the next unrouted one fails it.
This routes eight inits that were missing: local_runtime, gateway, cron,
tools, tools/environments, plugins, hermes_platform,
hermes_cli/observability.
HERMES_PREVIEW_FULL_CONNECTORS=true|false makes the free-tier mint send
{"preview_full_connectors": <bool>} in the /api/anonymous/create body, so
the account service can record the cohort on the account and gate
toolkits from it. Any other value (unset included) keeps the body {}.
The variable joins the desktop bundle-env allowlist (release script,
banner writer, channel request decoder, BUILDING.md) so a bundle can
bake it in. It is self-reported and visible in the packaged JS: the
service must treat it as a preference, not proof.
tirith ran `tirith check` on every terminal command, default-on with fail-open, as a
pinned PM binary auto-downloaded at startup. Measured against the shipped 0.4.2 binary
it caught 14/16 attack commands but flagged 12/24 everyday agent commands
(`echo $((6*7))`, `"$VAR/x.sh"`, `gh api | python3`, LAN curls, `.dev` hosts,
`pip install` via analysis_incomplete), and in cron/-q deny mode each of those is a
hard block. It only ever saw terminal strings (execute_code, writes and the browser
bypass it), fail-open meant no protection wherever the binary could not install
(Windows, Termux, x64 musl), and it produced a steady issue/PR stream (fail_open
semantics four times, install loops filling disks, a SIGSEGV hang, session-approval
scoping, third-party promo text on approval cards).
Removed: tools/tirith_security.py, the tirith branches of the command guard (the
is_tirith tuple, session-only "always" downgrade, import-error/fail-open paths,
unattended tirith pass), the CLI and gateway startup installers, the PM package and
lock pins, the nix runtime dep, security.tirith_* defaults, the CLI warning string
and three tirith tips (gaps refilled from the tail so translations survive), plus
the gateway's "no automated assessor" warning that only existed to point at tirith.
Config v50 drops security.tirith_* and enables nothing in its place. Hermes ships
no tirith plugin; if tirith upstream publishes one, it goes through normal catalog
admission under their name.
Two ratchets this suite arms for its own siblings fired once they landed:
- the Windows pause (#132338) now imports gateway/status_inline_source.py
to recognise inline-bootstrapped gateways, so a change there starts the
e2e_upgrade suite like gateway/status.py (only two importers: not a hub)
- the marker-corpus consumers #132345 and #132354 ship are on main, so
_LANDS_IN_SIBLING_PR empties as its comment asks
`hermes_cli/_early_recovery_zip.py` (split out of `_early_recovery.py` on the
commit-point PR) is imported by the update path, so a change to it must start
the update pipeline and install E2E suites. Both lists and the install-e2e path
filter now match the `_early_recovery` prefix, which also routes any later
sibling. `remote-lifecycle-v2-marker.test.ts` reads the update-marker corpus
and is now a listed consumer; `remote-update-marker-programs.test.ts` exists in
this tree, so its sibling-PR allowance is gone.
Replaying the last 131 non-merge main commits (4787e4d56f) through the
classifier: main selects e2e_upgrade 5 / e2e_desktop_update 3, the CI branch
12 / 10. Four of the seven new hits per lane touched only plugins_cmd.py or
plugins_cmd_install.py. migrate_all_homes reaches them only when a home's
configured memory provider left core (the catalog install); no upgrade or
Desktop update journey configures one, so those suites cannot observe the
change, and the unit lane already runs on every .py edit. Measured on a
fresh home: memory_provider_migration, pm/plugins_state, agent/memory_provider
and plugins/memory run; plugins_cmd_install executes only its module body,
plugins_cmd is never imported.
Kept, because a fresh-home update runs them: memory_provider_migration.py
(migrate_all_homes on every source build), plugins/memory/__init__.py (the
post-update Honcho profile sync), pm/install.py (sealed() /
lazy_installs_allowed() gate every update's default-tool install), and
scripts/build/ (frontend-common.mjs is imported by freshness/web/tui/desktop).
After: 8 / 6. A guard test fails if the journey-home migration starts running
plugin-install code, the signal to route those modules again.
update_cmd_commit._admitted_python now resolves python3.N through hermes_platform.resolver
(cd471c9c55a), so the update path imports the resolver package and a change to it must start the
upgrade E2E, like hermes_platform.host already does.
What: scripts/ci/classify_changes.py names hermes_cli/git_pack_tidy.py in the update pipeline, so a
change to it runs e2e_upgrade.
Why: the derived routing guard (test_every_module_an_update_entry_point_imports_starts_its_suite)
found it imported by the update path and unrouted: main's partial-clone tidy (a0f47657b9) runs in
every update's pre-fetch via update_cmd_check.
- hermes_cli/web_routers/status.py (review CI2): /api/health `commit` is what
host-backend-attach.ts compares to the checkout before attaching after an
update, yet an edit ran only the python lane. Now also e2e_desktop_update,
the lane host-backend-attach already starts.
- Tauri updater (review CI3): marker.rs started e2e_upgrade and
e2e_desktop_update, neither of which builds or runs the bootstrap installer
(no workflow of either lane touches apps/bootstrap-installer or cargo),
while its callers update.rs/paths.rs started none. All three now route as
one unit to the lane that compiles and tests them: rust (cargo test, via
.rs) and nothing slow. _TAURI_UPDATE_MARKER is gone.
- remote-update-marker-programs.ts (+ test) (review CI3): the SSH remote's
marker judge/gate ran only frontend; it is the marker.sh/marker.ps1
contract run remotely, and its importers (remote-lifecycle.ts,
managed-ssh-update.ts) are desktop_updater files, so it is one too.
- gateway/status (review CI3): the prefix also matched status_phrases.py
(turn-status wording); the named status_inline_source sibling does not
exist. Tightened to gateway/status.py.
Changed assertion: _R6_CASES pinned marker.rs -> (rust, e2e_upgrade,
e2e_desktop_update); the finding is that the two e2e lanes are the bug (they
never build marker.rs), so it now pins (rust,), and a new test pins the
three Tauri updater files off every slow lane.
The stack's health refactors carved main_desktop_tcc.py out of main_desktop.py
and status_inline_source.py out of gateway/status.py; both are imported on the
update path, so an edit to either must start e2e_upgrade (and, for the desktop
module, e2e_desktop_update). Route each facade by prefix so its <stem>_*
siblings are covered without naming files this branch does not have yet.
The desktop gate's SSH-v2 marker fix added remote-update-marker-programs.test.ts,
which reads the shared marker corpus: list it as a consumer, allowed absent until
#132345 lands, like the other gate-PR consumer.
Route the exercised subprocess and provider-migration dependencies to both
real-update journeys. Keep backup routing under an explicit dispatch test.
Use the aggregate's existing detect outputs to reject skipped or missing
update consumers while retaining legitimate unselected PR skips.
Verified with the real classifier and aggregate subprocesses against
b9dec76a042, composite ff5c8f1a64b, and fixed code. Canonical runner:
571 passed, 0 failed, 1 native-Windows skip across CI and marker corpora.
(cherry picked from commit a082ca3c62ed49382f173a07882d4280d0ba7c9c)
(cherry picked from commit ed9f2028256b8b1d6af81ef97cd472f30d2cb042)
(cherry picked from commit 6d5cc740b6a75d476802222c2f4754855a5712b0)
tests/fixtures/update_marker_corpus.json is read by pytest (update_lock),
cargo (marker_tests.rs), vitest (update-marker-corpus.test.ts) and the
marker.sh / marker.ps1 corpus tests. By path it selected only python, so a
corpus edit never ran the Rust, Electron or Windows PowerShell readers.
_SHARED_FIXTURE_CONSUMERS routes a fixture with its consumers' test lanes;
the routing test derives the consumers from the tree and replays the
classifier through the workflow files to each lane's job (review D14).
(cherry picked from commit 4893ef31833b5fcbdfc527beb70724ea63a3eb8f)
(cherry picked from commit 20c04a27b2a04ebaabb9a9df3693339243d9380e)
main added hermes_cli/left_core_migration.py, imported by source_build.py; the
R15 import-graph test caught it unrouted on the composite of all seven PRs.
(cherry picked from commit 338aabcdda316a92b8bfcae10f52730fd6fcb344)
(cherry picked from commit 5a839bb475d08f82293f07bcdb60ded859814476)
known_failure XFAILs a gap while its fix is an open PR, which is right
on a PR and wrong on the acceptance run of the integrated batch: there
it accepts exactly the gap the batch claims to close (the final
composite's Windows run passed with `test_crash_cells.py (3✓ 1xf)`).
HERMES_E2E_STRICT_ACCEPTANCE=<owner>[,<owner>] (or 1 = every gap) makes
the matching known_failure / known_gate blocks re-raise the gap's own
assertion. It is set only from the `strict_acceptance` dispatch input of
ci.yaml (threaded through tests.yml / tests-os.yml to every E2E step)
and of windows-install-update-e2e.yml; pull requests never set it.
run_tests.sh forwards it through its env -i allowlist.
Red on 8cc78279a36: test_pending_fixes_strict.py (a real pytest process
per mode) — the batch-owned gap XFAILs under strict mode.
(cherry picked from commit ca1e7e749a61263181e89aa687dec366d95fda18)
(cherry picked from commit 7768bf3ef8e15fb7f228ebad619d6a9a85ad4100)
The classifier missed modules the update entry points import
(_subprocess_compat, desktop_build_lock, memory_provider_migration, the
scripts/build compilers) and never sent source_build/source_completion to
the Desktop update suite. ci.yaml gated tests-os on `python` alone, so a
TypeScript-only change to the Electron marker/result readers set
desktop_updater and still never reached the Windows hand-off tests.
- classify_changes.py: _UPDATE_ENTRY_POINTS / _DESKTOP_BUILD_ENTRY_POINTS,
an owner list of what they import, and an explicit _SHARED_HUBS set.
- tests/ci/test_update_ci_routing.py reads the entry points' real imports
(AST incl. lazy imports; ESM import/step() closure of the build scripts)
and fails until each is routed or is a real hub (>= 25 importers).
- It also replays the REAL classifier's output through the composite
action, ci.yaml detect outputs, job if:/with:, the called workflows' job
if: and the rendered Windows step, for every combination of the gated
lanes (tests/ci/_gha_expr.py models the expression subset and raises on
anything else).
- ci.yaml: tests runs for python|e2e|e2e_upgrade; tests-os for
python|desktop_updater|e2e|e2e_upgrade.
Red on 8cc78279a36: 30 of the new routing tests fail (R15 table cells,
16 lane combinations, ownership/closure tests).
(cherry picked from commit 3b17b5d2ad8081534bbeefcf5d8334cd79a7e6fd)
(cherry picked from commit 594f8b0c4ba32324da922d5289b0932299ffe7a9)
The e2e_upgrade lane (Linux e2e-upgrade + the Windows install + update
journey) only matched the update_* module family, so a PR touching
cmd_update in hermes_cli/main.py, _early_recovery.py, venv_sync.py,
relaunch.py, the gateway restart/pause surface, hermes_bootstrap.py or
the Desktop hand-off scripts merged without a real `hermes update`. The
e2e_desktop_update / desktop_updater lanes likewise missed main.ts, the
hand-off result reader, the install-kind / attach modules,
desktop_update_verify.py and main_desktop.py.
The added paths are the steps of the update pipeline (updater audit
code maps, step tables) plus the lock / marker / recovery state the
update_* modules import. Two invariant tests pin every such file to its
lane and fail when one is renamed.
(cherry picked from commit b02e2cc124e74033ccb4c6e1c7c939590ef65fac)
(cherry picked from commit 16068789ea98ce55e92eeb148ec9345d6778e781)
Offset the FILE_LINES growth the moa preset fixes add to hermes_cli/config.py
(4283 > its 4274 cap): the version-stamp trio (raw read of _config_version
plus check_config_version) moves verbatim into topical sibling
hermes_cli/config_version_stamp.py; config.py re-exports the two public
names (callers and tests import them from hermes_cli.config), the internal
_read_config_version_stamp callers now use read_config_version_stamp, and
scripts/docker_config_migrate.py imports from the new module directly.
- A "stdin" string used as a value (for example **{"cwd": "stdin"}) no longer marks a call safe; only a stdin keyword, dict key, subscript store or setdefault counts.
- An inline **dict(stdin=...) splat is read directly instead of being looked up as a helper named dict.
- age-keygen writes to <identity>.partial and moves it into place, so a timed-out run leaves nothing the next exists() check would accept.
Greptile 4198572811: the guard now finds subprocess/os/asyncio calls as ast.Call nodes, so multi-line calls are checked, and the 27 calls it newly flagged (pixel-art ffmpeg and 26 others) pass stdin=DEVNULL.
Optional-skill scripts can run inside the backend process, where a child
that inherits fd 0 can end the TUI/desktop JSON-RPC session with stdin EOF
(#14036). Add optional-skills/ to the guard's context dirs.
SKIP_DIRS used to match any part of the absolute path, so a context dir
named optional-skills (and any subdir named scripts, skills, gateway, cron
or hermes_cli) was never scanned. Match only "tests", relative to each
context dir.
The wider scan flags three calls; they now pass stdin=subprocess.DEVNULL:
comfyui auto_fix_deps.run_cmd, unbroker crypto.ensure_identity and
ast-grep cmd_install (its installers do not read stdin). Touching those
lines makes the code-health ratchet treat their missing timeout as new:
age-keygen gets timeout=30; the comfyui node/model installs and the
ast-grep installer download for as long as the network needs, so they
carry a reviewed HX006 allow comment instead.
The ladder's INT/TERM traps wrote the receipt and exited as soon as the
foreground stage returned, even when the stage's child caught Ctrl-C and
succeeded (setup's provider picker prints "Provider setup skipped." and
carries on). gateway and complete were skipped, the bootstrap marker was
never written and rc went 0 -> 130: a telemetry write changed installer
control flow.
install.sh:
- The ladder traps (INT, TERM and now HUP, a closed terminal) only note the
signal while a stage runs. rc==0 clears a Ctrl-C and the ladder carries on,
as bash's cooperative exit did on main; rc!=0 with a signal noted writes
`interrupted` and re-raises the signal. Outside a stage (platform check,
banner, between stages) the signal stops the run at once with a receipt.
The traps are installed with the EXIT trap, before check_platform.
- Inside a ladder stage, a signal marks STAGE_INTERRUPTED; stage_result
writes `interrupted` over the stage's own class, so pm.cli /
source_completion / `hermes setup` catching KeyboardInterrupt and exiting 1
no longer reads deps_install_failed / setup_failed. A child the signal
killed still ends the stage at once (status 128+N).
- A TERM/HUP sent to the installer alone takes effect when the running stage
returns (documented; the stage stays in the foreground because a
background stage's children ignore Ctrl-C under bash 3.2), and the receipt
names the stage the run stopped before, not a finished one.
- A stage killed outright (SIGKILL/OOM) now leaves a `failed/<stage>/other`
receipt; a failed write (full disk) removes its .tmp; the fallback id uses
$(( RANDOM*32768+RANDOM )) (no octal parse); no clock -> no receipt.
install.ps1:
- A native child that ended on Ctrl-C (130 or STATUS_CONTROL_C_EXIT) reads
`interrupted` over the stage's class; the receipt write removes its .tmp
on failure; the dead `other` class on the pre-ladder Initialize-ResolvedPaths
Fail is dropped (it runs before the ladder's try, so it never wrote one).
Tests (real install.sh main block with stub stage bodies, real SIGINT to the
process group): a handled Ctrl-C finishes the install with a success
receipt; a child that turns Ctrl-C into exit 1 reads interrupted. Both red
at 6246dd0f69d.
install.sh: a run id per ladder, first writer wins (a parent-only SIGTERM no longer
leaves two receipts); Ctrl-C / SIGTERM write failure_class=interrupted.
install.ps1: same once-only guard, finally{} records an interrupted run.
Reporter: latch a receipt id only after its row is saved, so a short-lived start
whose daemon reporter is cut off at exit re-reports on the next start.
install.sh / install.ps1 leave one closed-token receipt per full ladder run
(installer, outcome, failed_stage, failure_class, start/finish epoch) under
$HERMES_HOME/telemetry/shared_metrics/pending_installs. Every fail()/Fail call
site passes a closed failure class; the installer never sends anything.
The next Hermes start records each receipt once (latched by id, deleted after
the row is saved) only while collection is on; a start with collection off
purges the receipts unreported. Out-of-contract receipts are dropped.
Select the expected archive and embedded app version from the admitted
channel request, or the release tag for tagged builds. Channel builds
have no tag, so indexing it crashed before macOS packages could stage.
Keep filename, identity and provenance validation intact.
Regression coverage reproduces the crash on both architectures and checks
that wrong filenames, embedded versions and commits remain rejected.
plugin-catalog/mem0.yaml (#131267) points at mem0ai/mem0's own
integrations/hermes-plugin-mem0 at a reviewed pin. The entry stays dormant
while a bundled copy exists (bundled providers win in find_provider_dir), so
deleting the in-tree copy is what lets hermes_cli/memory_provider_migration.py
install it for every home whose memory.provider is "mem0". The vendor copy
keeps the provider name, mem0.json, the MEM0_* keys and the setup wizard.
Also dropped: the hermes-agent[mem0] extra, its win-arm64 platform row and
exclude-newer row, the pm/extras.py anchor (uv lock removes only mem0ai,
posthog, qdrant-client and sqlalchemy), the mem0-specific doctor check (the
generic check now names `hermes plugins install mem0`), and the tests that
imported plugins.memory.mem0. Prose that lists the bundled set now names mem0
as a catalog provider. MEM0_* env hints and redaction stay: the catalog copy
still reads them.
While this branch was in review, #132386 landed on main: a committed
`hermes update` that still owes work prints one whole "Desktop app build
owed:" line and one "Update follow-up '<step>' did not finish:" line per
owed step, and both Desktop hand-offs turn those into a manual outcome
naming the step (never a plain "Update complete."). This branch rewrites
both hand-off scripts, so the rebuild took its versions whole and this
commit carries main's behaviour into them, through this branch's own
post-commit design:
- posix.sh: owed_followup_steps() (unchanged from main: the parser contract
test runs it), the build-owed and owed-steps checks on a zero exit, each
setting DONE_NOTE (finish() makes it manual) and an add_warning entry so
the step also reaches the result's warnings; the Linux relaunch gate now
appends to DONE_NOTE instead of replacing an owed-step note.
- windows.ps1: the same two checks as Add-Followup -Manual calls before the
runtime verify, with main's exact [regex]::Matches expression.
Already equivalent here: the mac swap is skipped when the rebuild failed,
and finish() relaunches the kept app for a manual note unless the Linux
gate says skew/manual.
Before: 6 of main's contract tests fail on the rebuilt branch
(test_every_owed_followup_reaches_the_desktop_result x4, the truncated
build-owed case, the parser contract). After: tests/scripts/desktop_update/
plus test_update_post_commit_followups.py pass, 113 passed, 0 failed.
Within moments of its claim the daemon hands marker line 1 to the custodian it
forked (fdab580431e). The launcher's start check (contract C2) still waited for
the daemon's own pid, so on a host where the handover beat its next 0.1 s poll
it never matched, and once the daemon finished it reported a running update as
a failed launch (exit 70). `marker_names_handoff` now accepts line 1 naming the
daemon or a process the daemon forked; a grandchild or an unrelated pid still
does not count.
Sabotage A/B (daemon held alive 2 s after the custodian takeover, real
hand-off via the shim-progress harness): old acceptance exits 70, new exits 0
and publishes the stage. Test: the predicate over custodian/daemon/grandchild/
unrelated, red on the previous marker.sh.
Same gap as the POSIX hand-off: the -MarkerOp custody watcher took line 1
over only after windows.ps1 exited (WaitForExit, then the marker lock), so
an old packaged Desktop reading in between (lines 1 and 2 only) deleted the
marker while the resumed `hermes update` or a completion survivor still
held the checkout.
Start-MarkerCustodian now names the watcher on line 1 under the marker lock
right after starting it, before any update work, and the hand-off judges
its claim as that watcher from then on ($script:MarkerOwner in
New-MarkerContext): delegate publication, heartbeat and release keep
working. Invoke-MarkerRelease stops the watcher only after the release, so
line 1 stays live until then. Invoke-MarkerCustody keeps a claim that
already names it (judged by the creation time the hand-off recorded) and
falls back to the old takeover only when the handover failed.
Tests (Windows CI lane): the kill cell holds the marker lock across
taskkill /F and runs the shipped legacy reader at once; the pre-publication
kill cell and the A7 race expect the custodian on line 1.
The previous fix let the line-2 refresher take the marker over only AFTER
the hand-off died: it polls once a second and then waits for the A7 lock,
and until it rewrote line 1 the marker named the dead hand-off. An old
packaged Desktop (judges lines 1 and 2 alone, ignores A7 and the delegate)
reading in that gap deleted the marker and booted its backend while the
`hermes update` the hand-off started, or a completion survivor, still held
the checkout lock -- and the custodian then found no marker to keep.
Now the custodian is named on line 1 BEFORE any update work starts:
marker_refresher_start forks the refresher, then, under A7, rewrites the
claim's line 1 to the refresher's exact pid + creation time and adopts that
identity (MY_PID/MY_CT) for every later marker mutation (delegate, refresh,
release). posix.sh checks its own liveness with $$. The refresher keeps
line 2 young; if the hand-off dies it keeps the marker until the delegate
and the checkout lock are gone, then releases it. The hand-off releases the
marker before it stops the custodian, so line 1 stays live until the
release. If the handover fails (A7 busy), the old takeover-on-death path
still applies.
Test: the orchestrator-killed cell now holds the A7 lock across the SIGKILL
and runs the shipped legacy reader at once (no takeover is possible), then
keeps the post-takeover phases. Red on 5f855a2392c: {'live': None,
'kept': False}. Tests that asserted line 1 == the hand-off pid now expect
the custodian (that assertion encoded the vulnerable state).
marker.sh::checkout_lock_held and marker.ps1::Test-CheckoutLockHeld answered
"not held" when the lock file exists but cannot be opened (root-owned after
`sudo hermes update`, or held open without sharing on Windows), while the
Desktop's own probes (checkout_held / WINDOWS_CHECKOUT_LOCK_PS) answer "held".
The script helper's reclaim then deleted a dead marker the Desktop's probe
keeps. Both scripts now fail closed on every open/probe error except a
missing file: reclaim needs a provably free checkout. Parity tests: the
POSIX probe with perl, python3 and flock(1), plus an end-to-end reclaim; the
Windows reclaim against a lock file held with no sharing (Windows CI lane).
Same shape as the POSIX hand-off: windows.ps1 heartbeats line 2 from its
own process, so `taskkill /F` (no finally runs) left line 1 naming a dead
pid while the resumed `hermes update` (the line-4 delegate) or a completion
survivor still held the checkout, and an old packaged Desktop - which judges
lines 1 and 2 alone - deleted the marker and booted its backend.
After its claim the hand-off starts a hidden watcher (-MarkerOp custody
-CustodyOf <pid>, Start-MarkerCustodian). Once the hand-off exits with its
claim still published and the delegate running or the checkout lock held,
the watcher names itself on line 1 under the marker lock, heartbeats line 2
and releases the marker when both end (bounded like the R6 wait).
Invoke-MarkerRelease stops the watcher once the hand-off's own release is
decided. The Windows kill-cell test now also runs the reader be3fd671d7
shipped; that reader runner moves to a helper shared with the POSIX test.
An older packaged Desktop judges the update marker by line 1 (pid alive) and
line 2 (under 20 minutes) alone and unlinks it otherwise, never consulting
the protocol-2 delegate line or the marker helper. The line-2 refresher
exited with the hand-off script, so a SIGKILLed hand-off left line 1 naming
a dead pid while the `hermes update` it started, or a completion survivor,
still held the checkout lock: the old Desktop deleted the marker and booted
its backend into a checkout still being mutated.
The refresher now outlives the hand-off as its custodian (marker_custody):
once the hand-off is gone, if its claim is still published and the delegate
runs or the checkout lock is held, it names itself on line 1 (exact pid +
creation time, under the A7 lock), keeps line 2 young, and releases the
marker only after both are gone (bounded like the R6 wait). The test runs
the reader be3fd671d7 shipped (vendored verbatim under tests/fixtures).
The hand-off scripts gave the Desktop a fixed 30 s (windows.ps1) or 60 s
(posix.sh) to exit, but the Desktop's quit can first join a managed SSH update
it is running, which legitimately takes longer. The hand-off then recorded a
failed update although the Desktop exited safely moments later (review H2).
Both twins now share one 150 s ceiling, measured on the clock rather than as a
loop count, and still refuse with "Nothing was changed" if the Desktop is alive
at the ceiling, so nothing is ever replaced under a live Desktop. The ceiling is
overridable through HERMES_UPDATE_DESKTOP_EXIT_SECONDS only so the self-tests
need not sit it out, like the existing pipe-drain and step-idle overrides.
Tests: a Desktop still alive at the ceiling refuses with exit 4 and never runs
the update (red on the start sha, which waited out its hardcoded 60 s); a
Desktop that exits inside the ceiling lets the update run (control).
update-job.ps1 was one top-level `if (-not type) { Add-Type ... }` block.
It is now a named function, Initialize-HermesUpdateJob, that returns early
when the type already exists and is called once at the end of the file, so
the dot-source in windows.ps1 behaves exactly as before. The C# body is
unchanged byte for byte.
Checked with PowerShell 7.4 on Linux: dot-sourcing the file twice compiles
the type once and HermesUpdateJob::ProcessIds(0) answers; windows.ps1 still
parses with zero errors. Windows PowerShell 5.1 itself was not run here.
Review 5411223284 (shape: update-job.ps1 top-level code).
The script helper's `reclaim` answered `absent` the moment the marker file
was missing, before the checkout lock was looked at, and the Desktop gate
(update-marker-gate.ts liveMarkerProbe) returned "nothing runs" for a
missing marker without asking the helper at all. An update that holds the
checkout lock before it publishes its marker, or after it retired it (the
CLI paths of the stacked PRs do both), therefore let the Desktop boot a
backend into a checkout still being written.
- marker.sh marker_op_reclaim_locked / marker-claim.ps1 Invoke-MarkerOp:
no marker -> `held` when the checkout lock (owner byte + lease bytes on
Windows) is held, else `absent`, under the same marker lock.
- update-marker-gate.ts (gate-lane file, additive): a missing marker is also
routed through the helper; only its positive `held` keeps the gate shut
(blocked screen, Retry, Start anyway as for a dead marker). Any other
answer, `error` included, proceeds as before, so a broken script never
blocks a markerless boot.
The gate test "a live owner never reaches the helper; an absent marker
neither" pinned the missing check for the absent half; that half is now the
new "no marker: only held keeps the gate shut" case.
Review 5411223284 (open thread: update-marker-gate.ts ~193 / helper absent).
Both hand-off scripts treated ANY nonzero `hermes update` exit as ok:false
"Update failed". After the commit point `hermes update` can still exit
nonzero by design -- 130 (Ctrl-C after the code moved: receipt outcome
`interrupted`) and 1 (a parked autostash: `partial`) -- or crash after a
committed run (`success`). The Desktop then said the update failed while the
new code was installed (contract C3 / invariant 3).
The scripts now hand the update a correlation id (HERMES_UPDATE_CORRELATION_ID,
already honoured by update_receipt._launcher_correlation_id; a caller's own id
is kept) and, on a nonzero exit other than 2 (refused), read the root's
logs/update_receipts/latest.json. Only THIS run's finalized receipt counts
(matching correlation_id, finished_at set, outcome success|partial|
interrupted); a record reconciled as interrupted keeps finished_at null and
never matches. Then the result is ok:true, manual, with an `update` warning
naming the exit and outcome, and the post-commit steps (Windows: verify,
gateway start --all) still run.
posix.sh also routes its own signal handler through the same judgement: a
HUP/INT/QUIT/TERM that lands after the update committed (exit 0, or a
committed receipt) is an owed `handoff` follow-up, exit 0, not
"hand-off was interrupted".
Tests: posix live cells (RED at start) for interrupted/130, partial/1,
success/1 and an INT delivered while the committed child finishes, plus
negative controls (no receipt, failed, another run's receipt, a reconciled
unfinished record); the Windows twin cells.
review S5
posix.sh::finish and windows.ps1's finally write the result (finished_at =
now) BEFORE the R6 release wait, which lasts up to 7200 s while a survivor
holds the checkout lock, and before the relaunch. The relaunched Desktop has
no expected run id and handoff-result.ts drops a non-manual result older than
30 min, so a failure or warnings result was silently discarded after a long
wait.
The early write stays (a kill during the wait still leaves a result); when the
wait actually ran, the result is published again after it with the real
finish time, unless something already consumed it. The release records that
it waited (RELEASE_WAITED / $script:MarkerReleaseWaited).
Tests: posix live cell (RED at start: finished_at 3 s before the survivor
let go) and the Windows twin with a timed checkout-lock holder.
review S2
Write-Result writes <result>.<pid>.tmp and then Moves/Replaces it over the
result. When both fail (result path locked, a directory, ACLs) the catch only
logged, leaving one tmp file per hand-off beside the result. The tmp is now
removed in a finally (it is gone already after a successful publish), the
same as posix.sh's write_result.
Test (Windows live cell): with the result path occupied by a directory the
hand-off still exits 0 and leaves no .hermes-update-result.json.*.tmp.
review S4
json_escape escaped only \ " \n \r \t. Every result/status string it builds
(FINAL_MSG, branch, channel, warnings) can carry text from git/ps/hermes
output; a raw ESC, BEL or any other char below 0x20 made the whole
.hermes-update-result.json invalid, and the Desktop dropped the outcome.
The remaining control chars are now written as \u00XX. Bash 3.2 compatible:
printf -v, no forks; the loop runs only when the string holds a control char.
Test (RED at start): json_escape over chars 1..127 plus UTF-8 and an ANSI
colour sequence round-trips through json.loads, in the C and C.UTF-8 locales.
review S7
Two parity gaps between marker.sh and marker-claim.ps1:
- adopt-run (--handoff-run), withdraw's `withdrawn` and `taken` accepted
J_OWNER_STATE==2, which also covers "alive, creation time unreadable, line 2
younger than 20 min". marker-claim.ps1 requires Test-ProcessIdentityExact
(both creation times known, within 2 s). An unreadable ct is no proof the
pid is still that process; these three now require ident_state 0, and
`taken` skips our own pid like the ps1.
- reclaim / withdraw printed `reclaimed` / `withdrawn` after an `rm -f` that
exits 0 even when it cannot unlink; the Desktop then proceeded as if the
marker were gone. marker_remove checks the marker is gone; else `busy`, the
ps1's answer when Remove-MarkerLocked fails.
Tests (RED at start): the three identity cases with every foreign ct
unreadable (and the readable control), and a reclaim inside a read-only
home that must answer `busy` and keep the marker.
review S3
Get-LiveProcessCt caches a pid's creation time for the script's lifetime and
forgets it only when a probe SEES the pid dead. A delegate (the `hermes update`
child) that exits and whose pid is reused between two probes keeps the old
incarnation's time, so the release judged the unrelated new process live and
handed the marker to it (A7 rule 5), parking every later update behind it.
Every judgement made through New-MarkerContext (claim, delegate, heartbeat,
release) and the delegate line Add-MarkerDelegate writes now pass -Fresh,
which re-reads the time. Pollers that only ask "still alive?" (Wait-DesktopExit)
keep the cache.
Test (Windows live cell): a release with the delegate's pid cached at the
recorded ct but really owned by another incarnation deletes the marker instead
of handing it over. marker.sh reads /proc or ps on every call: no cache.
review S6
marker.ps1::Test-CheckoutLockHeld probed only the owner byte at 1 MiB. Since R5b
every joiner the kill-on-close job refused also locks one of the 16 lease bytes
after it, and hermes_cli/update_lock.py::_try_lock treats any held lease as a
held lock. With the owner dead and a leased completion child still running, the
hand-off read the checkout as free, so the R6 release wait ended and the
Desktop's reclaim deleted the marker next to a live mutator.
Lock/Unlock now cover 1+16 bytes in one LockFile (it fails if any byte is held).
marker.sh needs no change: POSIX takes the checkout lock with flock on the whole
file and the lease bytes exist only on win32 (_take_lease).
Tests: a live Windows cell holds only a lease byte (1 and 16) and expects
`held`; a static cell, runnable on Linux, pins the probed range (RED at start:
(1048576, 1)).
review S1
What: run_bounded's no-perl branch starts the probe under bash job control (set -m), so it gets its own process group and the timeout's kill -KILL -- -pid reaches its children.
Why: without perl the probe stayed in the caller's group, the group kill failed, and a hung hermes --version / update --help left its python child running.
Red: pre-fix posix.sh fails test_timed_out_probe_is_killed_with_its_whole_process_tree[absent] (grandchild alive after rc=124). Green: both [present] and [absent] pass.
Review R8 m7: after the merge with main's 732b63d61a the probe decodes launcher
stdout as UTF-8 through ProcessStartInfo.StandardOutputEncoding and never touches
[Console]::OutputEncoding, so the test docstrings no longer claim the call sites
scope and restore it. runtime.ps1 keeps a one-line #124526 pointer at the
encoding line. Behaviour and assertions are unchanged.
