What: one shared remote marker judge (remote-update-marker-programs.ts: system-python REMOTE_MARKER_JUDGE_PY/GATE, PowerShell WINDOWS_MARKER_JUDGE_PS) ports the corpus parse (ct:, delegate:, run:) and liveness rule (pid alive at |ct-actual|<=2s, else v1 1200s ceiling); the POSIX relaunch probe, POSIX spawn wrapper, managed-SSH observer, Windows relaunch probe and Windows spawn all use it. Both SSH spawn sidecars move from <marker>.mutex to <marker>.lock (POSIX flock, Windows FileShare.None, bounded 10s) and a dead claim is unlinked only inside that hold.
Why: the remote readers full-matched 'pid\nstarted', so every v2 marker was UNCERTAIN and a dead v2 claim refused SSH startup forever (Windows spawn refused on any marker), and no updater locks .mutex, so remote spawns were never serialized with the updaters' claims.
Red/green: pre-fix remote-lifecycle.ts fails the dead-v2 probe test ('unreadable or malformed') and the <marker>.lock race test ('Missing expected rejection'); pre-fix managed-ssh-update.ts reports a live v2 delegate as 'malformed'; all green after, plus the 50 corpus judge cases through the embedded Python.
Review R8 M6 (D3 "bounded, recoverable"). Only the primary boot wait published
its hold, and the hold IPC refuses when none is shown, so a profile backend that
met a hold waited forever with a single log line: a remote primary's
forced-local profile, or a profile switch after a local primary had booted
(e.g. a CLI `hermes update` that leaked a git child).
- Every blocked wait now registers its hold on one board (update-marker-gate
UpdateHoldBoard): the primary boot wait's hold wins, otherwise the first
blocked pool wait's is shown. A pool wait publishes it past the same 5 s grace
and only touches `updateHold`, never the window's boot phase; the primary's
own clear no longer drops a pool hold.
- Check again / Quit / Start anyway act on the shown hold id as before
(primary window only, confirmed, logged); Start anyway releases that pool wait.
- The renderer applies `updateHold` from boot progress after boot has completed
(it ignored every post-boot progress payload), so the screen also appears
over a running app.
Review R8 M5 + m7 (D3: the gate never opens on its own while an update holds
the install).
- M5: a marker body this process already saw block (held/busy/error/live) no
longer clears when the helper answers `unsupported`. That answer only means
the checkout's script is missing or unreadable for a moment (git unlinks and
rewrites a changed posix.sh; Windows sharing violations); it now counts as
`error`, keeps the gate closed and is re-asked. A script that exists but
cannot be read (anything but ENOENT) is `error`, never an older checkout.
main's reclaim is now exactly runMarkerHelper('reclaim'), which owns that rule.
- m7: the hold id mixes in the marker file's inode and mtime, so two empty (or
byte-identical) markers from different updates are different holds and a
Start anyway over one never passes the next.
- m7: helper verdicts are validated per op: `foreign`/`withdrawn`/`taken` from
reclaim (or reclaim's words from withdraw) are `error`, not clearance.
- m7: a wait that ends because the user chose Start anyway is reported as an
override (onOverride) and skips the bundle-swap relaunch, instead of being
logged as "update finished".
Review R8 D3. A verified `held` verdict (the update exited, a process it
started still holds the checkout lock) used to stop blocking after 7200 s and
let the local backend start while that process could still be writing the
install. The gate now never opens on its own while an update hold lasts.
- update-marker-gate.ts: no held ceiling. Each blocking state carries a stable
hold id (sha256 of the marker body) and `blocking` (held/busy/error, not a
live pid). `allowStartOverHold(id)` is the only way through while the hold
lasts, scoped to that exact marker body and process-wide (pool waits honour
it; a new body blocks again). `requestHoldRecheck()` makes every probe
re-ask the script helper on its next poll. The marker is never touched.
- main.ts: past a 5 s grace the boot shows the blocked screen (boot progress
`updateHold`): what holds the install, since when, last check. IPC from the
primary window only: recheck, quit, and start-anyway, which needs
`confirmed: true` and the hold id on screen, logs a USER OVERRIDE line and
flushes desktop.log. The non-blocking "started anyway" dialog is gone. Pool
waits log the hold once and wait for it to end or the primary's override.
- Renderer: UpdateHoldOverlay with Check again, Quit Hermes, Open logs and a
two-step Start anyway (confirmation copy says what can break). en/de/fr/es.
Tests: the ceiling cells are replaced by two invariants (held stays closed 10x
past the old ceiling and Retry re-asks at once; a confirmed Start anyway opens
only its hold, for every wait, and a new body blocks again). Both RED on
8855353024e, GREEN here. A real-process probe (SIGKILLed UpdateLock updater,
custody child holding the real checkout flock, real posix.sh) with the clock
jumped 3 h: the old gate opens, this one stays closed until Start anyway.
(cherry picked from commit 324a489259ab90e1f7158dcc2fa7d1c8a2da20f8)
A dead marker whose checkout lock a leftover process still holds (`held` /
`busy` from the script helper) parked the Desktop with no deadline (review R6
m7): a lock leaked into a long-lived process held start-up hostage while the
scripts themselves give up after RELEASE_WAIT_S = 7200 s.
The gate's marker probe now counts a held wait from the marker's line 2 (the
last time an owner wrote it), or from the first sighting in this process for
an unreadable line 2, and stops blocking after HELD_CEILING_MS = 7200 s: it
logs it once, reports it through onHeld, and leaves the marker untouched
(A7: the Desktop never mutates it). While held, boot progress names the
update process that exited and the time left; past the ceiling a
non-blocking dialog tells the user what still holds the install and how to
move on. A helper `live <pid>` keeps no ceiling (C1 rule 3). The ceiling is a
test seam option, not an env var.
(cherry picked from commit ac513e33ce0f5c99338d51607f894cf0dc3b4a8c)
The corpus now pins started_at to u64 (any digit count): Rust already
rejected 18446744073709551616, the TS judge took it (as an imprecise
Number). parseUpdateMarker compares line 2 as a BigInt, which parses any
digit string exactly and cannot throw behind the digits-only check.
(cherry picked from commit 9ef0529f1b2f531238b02977347f4dc8b8038531)
The boot/pool gate no longer deletes a dead marker. A dead or malformed
marker is routed through the checkout script's `reclaim` helper when the
script speaks hand-off protocol 2: held/busy/live keep the gate closed
(a completion can still hold the checkout lock after the script's child
died), reclaimed/absent open it, and an older checkout without the helper
proceeds as before minus the deletion. The helper is asked once per
distinct dead body per wait; held is re-asked every 5 s. The cached
create-time probe per wait is kept.
The recovery hand-off keeps its read-only conflict check and makes the
staged-updater pre-write exclusive-create only (logged when skipped).
(cherry picked from commit 60ddb61fddfda6af46954f42a19ab133520faf68)
A7 rules 3/4/6, R3/R4/R5 (Desktop side).
- update-marker-judge.ts: parse + judge exactly per the shared corpus
(tests/fixtures/update_marker_corpus.json): u32 pids, pid 0 dead, BOM/CR/
space-tab trimming, delegate/run on any line >= 4 (first wins), own pid
ours only at the exact incarnation (5 ms; no ct = previous incarnation),
2 s for other pids, v1/unreadable ct live only inside 1200 s.
- update-marker.ts never deletes or rewrites the marker: compareAndDelete
is gone; readLiveUpdateMarker/updateHandoffConflict are read-only; the
bridge and the staged-updater pre-write are a single exclusive create
that reports an existing dead marker / stale bridge of ours instead of
reclaiming it. Our own bridge is `ours`, not someone else's update.
- waitForHandoffClaim acknowledges only a LIVE line-1 owner that is not the
Desktop: protocol 2 requires a ct-verified ('match') identity and the
run id the Desktop passed; legacy scripts require liveness and the
started_at they echo. A claimant that died before the ack is not taken
(the reviewer's dead-daemon probe flips taken:true -> taken:false).
- checkout.ts protocol 2 (SPEC 4a): run id desk-<pid>-<b36 ms>-<hex4>,
bridge carries run:<id>; EEXIST -> live foreign refuses, dead/malformed
-> helper reclaim, stale bridge of ours -> helper withdraw with its run,
then one retry; --handoff-run / -HandoffRun passed; timeout -> helper
withdraw (`taken <pid>` = late success) then killHandoffTree. Legacy
scripts (SPEC 4b): no bridge, no run arg.
- update-marker-gate.ts: gate probe for R6 (wired into main.ts next).
Tests: corpus consumer (all 43 judge cases) + every export leaves every
corpus marker byte-identical; A7 rule 3 read-only cells and R5/legacy ack
cells with real child processes (fail on the previous head: ENOENT /
taken:true); checkout protocol-2 flows against a real fake helper script.
(cherry picked from commit 102ba6f4a27985ceccca06a47d528b6e1cca4cda)
A7 rule 3 makes the checkout script the only marker mutator Electron may
use. Add the runner for `posix.sh --marker-op <op>` / `windows.ps1
-MarkerOp <op>` (reclaim|withdraw, optional desktop pid + hand-off run),
HERMES_HOME set, hidden, hard 20 s bound (process group killed on POSIX),
and a strict one-line verdict parser (absent|reclaimed|held|busy|
live <pid>|withdrawn|taken <pid>|foreign; anything else, a nonzero exit or
a timeout is `unsupported`). Protocol negotiation reads the script text for
`^# hermes-handoff-protocol: (\d+)\s*$`; < 2 is a legacy script.
Tests run a real fake script process (argv, env, exit 64, garbage, two
lines, timeout, missing script) and pin the Windows command line through an
injected spawn.
(cherry picked from commit f1ef8fed94a510445b8417d9083618a6ccde15a7)
Contract C2: a script hand-off is confirmed only when the script takes the
Desktop's bridge marker in its own name, not when the launcher exits 0. The
channel hand-off test mocked spawnUpdaterProcess with a script that never
claimed, so apply() waited out the 20 s claim window and correctly reported
updater-spawn-failed. The fake now takes the marker as a live foreign pid,
and the test releases it after each hand-off (the update finished) so the
next apply does not see a live foreign owner. Claim timeout lowered to 2 s.
(cherry picked from commit fb43b97f0f501a2693130b4d8828014a380ae211)
plain Node has no process.getCreationTime, so on Windows the bridge claim
from tooling/tests was written without a ct line (v1). Found by the wine2e
Windows run of update-marker.test.ts (V4 cell).
(cherry picked from commit 5b530c7579b41dc413448c616125de16ee6ce5ae)
runs only on win32 (the wine2e lane): csrss.exe is access-denied to Get-Process
.StartTime; a marker whose pid a SYSTEM process reused must read DEAD.
(cherry picked from commit d10a47b71402140c4e474d3b1b60d2b6ed5d7ec6)
Round-2 review findings on the Electron side of contract C1/C2:
- A1/MAJOR-1: read a foreign pid's creation time on Windows through CIM
(Win32_Process.CreationDate, limited query rights) instead of
Get-Process .StartTime, which is access-denied for SYSTEM/elevated/
other-user processes. The update gate memoises one probe per pid per
wait (no powershell spawn per 1 s poll). A live pid whose creation time
cannot be read is live only inside the 20-minute ceiling, so a reused
pid can no longer park Desktop boot forever. macOS ps lstart is printed
and parsed as UTC.
- r1 M1: the staged Tauri updater pre-write names the spawned updater's
pid AND creation time (v2), so the updater adopts a v2 claim, no age
ceiling applies, and its hermes update child can add the delegate line.
- A3/m1: claims are published by writing a tmp sibling and hard-linking it
into place (EEXIST never replaces); O_EXCL+write remains only as the
no-hardlink fallback, and readers treat a 0-byte marker younger than 5 s
as live. Dead writers' tmp siblings are reclaimed.
- A2/MINOR-4: positional parse identical to the contract (BOM, CRLF,
integer started_at else malformed, bad ct => v1, bad delegate ignored).
- MINOR-3: on hand-off timeout, a missed compare-and-delete that finds the
script already took the marker counts as a started hand-off; otherwise
the Desktop kills the launcher tree it spawned (POSIX process group;
Windows: the wrapper's children carrying our -DesktopPid).
(cherry picked from commit 0615bd7c0565166afec2dff1f7b1ef5e835af56c)
The Electron side of the update-marker contract (C1/C2):
- update-marker: parse v2 (`ct:` creation time, `delegate:` line); an owner
is live iff its pid is alive, not a zombie, and its creation time matches
within 2 s (unknown = live). A live v2 owner is never aged out; the
20-minute ceiling now applies only to v1 markers. The marker is live when
the owner OR the delegate is live. Every delete is compare-and-delete.
- The bridge marker names Electron's own pid + creation time, claimed by
exclusive create BEFORE the spawn (it named the cmd.exe wrapper, dead
within ms). A live foreign owner refuses the hand-off.
- The hand-off counts as started only when the script takes the marker
within 20 s; a wrapper's exit 0 no longer quits the app. Otherwise the
Desktop stays up, restarts its backend, shows the error and
compare-deletes its bridge marker.
- update-gate: a failed receipt no longer outranks a live marker, and a
live marker owner keeps the boot parked past the old 20-minute deadline.
- handoff-result: parse before unlink, keep malformed files as .corrupt,
discard a result whose started_at belongs to another run, surface
ok:true warnings as a non-blocking notice.
- desktop-installation: the repair lock names its holder and a dead
holder's lock is reclaimed instead of throwing on every launch.
(cherry picked from commit 5d7f36bfaec2213e5c959ad1e0d21d514b9f94b7)
A CLI/launchd `hermes serve` that outlived an update kept serving the old
code; the new Desktop attached on token alone, got 503 "Restart required",
and Restart re-attached the same stale process. `/api/health` now publishes
the boot commit and attach falls to the next ladder rung (next record, then
spawn) unless it equals the checkout HEAD. Unknown Desktop identity
(packaged/non-git) keeps the token-only attach.
(cherry picked from commit e59c7e64bdf417253d3ed9f85710f70adbba3e11)
* fix(bedrock): route GPT-6 family and prefixed OpenAI inference profiles, size their 1M window
An enterprise deployment on provider bedrock with model us.openai.gpt-6.1-sol
compressed at 96,000 tokens with a 400K compression cap configured. Three gaps,
all on origin/main c53536929e:
1. agent/bedrock_adapter.py:77-79 BEDROCK_OPENAI_RESPONSES_MODEL_IDS held the
GPT-5.5 and GPT-5.6 IDs only (unchanged since 16476fad10). AWS now publishes
cards for GPT-6 Astra, GPT-6 Sol, GPT-6 Luna and GPT-6.1 Sol.
2. agent/bedrock_adapter.py:165-167 is_openai_bedrock_model was an exact set
match, so us.openai.gpt-5.6-sol returned False while openai.gpt-5.6-sol
returned True. Every cross-Region inference profile of a Bedrock OpenAI model
was misclassified.
3. agent/bedrock_adapter.py:1169-1195 BEDROCK_CONTEXT_LENGTHS had no GPT-6
rows, so get_bedrock_context_length fell to the 128,000 default silently
(the fallback WARNING fired only for application-inference-profile ARNs).
agent/context_compressor.py:1181 applies the 75% small-window floor, which
is exactly the observed 96,000.
Per the AWS model cards (model-card-openai-gpt-6-1-sol.html and siblings):
the bare openai.* ID is served in-Region by bedrock-mantle (Responses, Chat
Completions, no geo or global IDs), and the us./global. profile IDs are served
by bedrock-runtime where Converse is listed as supported. GPT-6.1 Sol states a
1M context window with 131,072 max output tokens, the other cards state
1,050,000. 272K on these cards is the long-context pricing tier boundary, not
the window.
Changes
- parse_bedrock_openai_model_id(model_id) -> (bare_id, profile_prefix) | None.
is_openai_bedrock_model is True for every endpoint form of an allowlisted ID,
GPT-OSS stays False. New bedrock_openai_uses_mantle is True only for the bare
in-Region ID. Both callers (hermes_cli/runtime_provider_backends.py primary,
agent/auxiliary_client.py auxiliary) route Mantle on that predicate and keep
prefixed IDs on Converse via bedrock-runtime, the path AWS documents for them.
The Claude profile-prefix regex now derives from the same prefix tuple.
- GPT-6 generation added to the allowlist, the static picker catalog and the
context table. OpenAI rows sized from the cards (6.1 Sol 1,000,000, others
1,050,000). The table/metadata pairing test covers the new rows.
- get_bedrock_context_length logs one WARNING per model ID per process when
the static default is used for any ID, naming the model, the value, that it is
a fallback, and the model.context_length override.
- website/docs/guides/aws-bedrock.md: route table, model table and the
cross-Region note describe both endpoint forms.
The empty compression summary the deployment also reported is not addressed
here. Converse is a valid path for the prefixed ID, so routing does not explain
it, and no request trace exists yet.
* chore(bedrock): keep auxiliary_client.py within its code-health line cap
Rewrap the _build_bedrock_client docstring back to two lines so the file does not grow past its value on main (code health ratchet, FILE_LINES).
---------
Co-authored-by: unsupportedpastels <theoldwizard123@pm.me>
On main (#132386) a --prepared child that exits without a result file is
a lost completion: _prepare settles it as owed and re-arms the tail for
the next launch. The L3 test's fake child returned 0 and wrote nothing,
so after the rebase the recorded events gained a trailing
arm_completion from that settle path. The fake now behaves like a
successful child and writes its result; the ordering assertion the test
exists for (lease first, then everything that writes the checkout) is
unchanged.
main.py may only shrink vs main (code-health FILE_LINES); this branch grew it
by 9 lines. _probe_container and _exec_in_container (the NixOS container-mode
re-exec, 96 lines) move verbatim to the topical sibling
hermes_cli/main_container.py; the entry point is now public
(exec_in_container) and main() late-imports it at its only call site. The
test import and the managed-runtime resolution allowlist entry follow the
move. No re-export shim. Behaviour is unchanged.
Two mocked tests drive _legacy_kill_process_tree down its shared-group
branch with a MagicMock proc whose pid is 999. That branch snapshots
psutil.Process(proc.pid).children(recursive=True) and SIGKILLs each
descendant, and psutil was never patched: the tests killed the real
descendants of whatever the machine runs as PID 999. On macOS runners
PIDs are allocated system-wide, and when 999 was an ancestor of the
test worker the file died mid-run (CRASHED, killed by SIGKILL), which
is how it surfaced on a PR whose diff never touches the browser tool.
Both tests now run against a fake process table, and the shared-group
test asserts the snapshotted descendant is killed individually (the
contract it was written for).
Live repro: inside an unprivileged user+PID namespace with PID 999 an
ancestor of the test runner, main's file reports CRASHED (killed by
SIGKILL); with this change it passes 9/9.
checkout_lock_held answered "free" when the lock file existed but could not be
opened. The hand-off scripts (marker.sh / marker.ps1) and the Desktop probes
(checkout_held, WINDOWS_CHECKOUT_LOCK_PS) answer "held" for the same file, so a
Python reader could reclaim a marker or admit a launch the others would park.
Only a missing lock file is free now; an existing file that cannot be opened
answers held. A lock call that fails outright (ENOLCK on NFS without lockd,
EOPNOTSUPP on some SMB shares) still answers free: nothing can hold such a
lock, and the interrupted-pull repair relies on that to run unguarded there.
Both callers that act on "held" are bounded (_reclaim_dead keeps the dead
marker; the acquire path waits its contention budget).
Both SIGSTOP/SIGCONT reader regressions resume their own stopped reader child with os.kill.
The live-system guard proves a pid is the test's descendant through psutil; in a test
interpreter without psutil it cannot, and it blocked the SIGCONT before the assertion (the
reviewer's F2 re-run hit exactly this). Reproduced by hiding psutil on PYTHONPATH: both tests
raised the guard's RuntimeError on os.kill(<reader>, 18); with the marker both pass. They signal
only fixture-owned children, so they carry live_system_guard_bypass like the custody tests.
`update_custody.run` (a timed-out Windows update child) re-implemented the tree kill and the
bounded pipe drain of `_subprocess_compat.bounded_probe_run`: its own `taskkill /T /F` copy of
`kill_process_tree`, then `communicate()` bounded, abandoning pipes a missed descendant still
holds. Both now call `kill_and_drain`. It stays correct for the job-bound child: the tree kill
never touches the update's job (which also holds the update's other children), the drained
output still rides on the re-raised TimeoutExpired, and abandoned pipes are detached so the
`with popen()` exit never closes a pipe a reader thread is blocked on. The probe keeps closing
its own private job first.
Tests: test_a_timed_out_child_never_waits_on_a_grandchild_holding_its_pipes and
tests/hermes_cli/test_bounded_probe_run.py (unchanged) pass. (kshitijk4poor review)
`incarnation_live` (holder records, the pause record) and `_identity_live` (the marker
reader, matching marker.rs) restated the same (pid, creation time) rule and disagreed when our
own creation time is unreadable: the marker reader treated a no-ct claim naming our pid as ours
(a degraded updater writes no-ct claims) and a ct one as dead, while `incarnation_live`
answered "unprovable" for both, so `_lock_holder` named a previous incarnation as the holder.
Both now go through `_incarnation`; `_identity_live` only adds the v1 age ceiling for an
unprovable identity. `_compare_and_delete` had no callers and is removed.
Test: test_one_incarnation_rule_when_our_own_creation_time_is_unreadable fails on the previous
head and passes here; the shared marker corpus is unchanged and green. (kshitijk4poor review)
`run_contained` starts the POSIX build launcher with `subprocess.run`, which SIGKILLs its own
child when KeyboardInterrupt escapes the wait. That child was the custodian holding the
checkout lock fd: moving it out of the caller's group (E) survives a group kill, not a direct
kill. A command still cleaning up from the same Ctrl-C then exited, dropped the last lock fd,
and a writer in a session of its own went on writing under the next lock owner (review R4).
The process the runner sees is now a stand-in: it forks the custodian into a group of its own,
relays SIGINT/SIGTERM/SIGHUP to it and returns its exit status. A killed stand-in leaves the
custodian to settle the tree: node gets the same 10 s it gets after a forwarded signal, then
every descendant is killed and reaped, and only then is the fd dropped.
Test: test_a_ctrl_c_keeps_custody_until_a_detached_writer_is_gone (verbose + captured runner,
slow and instant SIGINT cleanup; real run_in_custody, real SIGINT to the caller's group, real
flock contender) fails on bc9311b904c and passes here.
The POSIX relaunch gate (REMOTE_UPDATE_MARKER_PROBE), the managed-update
observer (marker_state) and the spawn-time recheck (marker_clear) each carried
their own marker parser, narrower than update_lock._parse_marker: a BOM, a
space-padded or zero-padded pid, an out-of-range first delegate line before a
valid one, or started_at past 2^53 read as malformed (UNCERTAIN), and liveness
was PID (+argv text) only, not the (pid, creation time) identity Python checks.
The spawn recheck read line 1 only, so a dead owner with a live delegate let a
serve start beside the delegate.
All three now embed one program, REMOTE_MARKER_JUDGE_PY
(remote-update-marker-programs.ts, byte-identical to the judge #132345 ships),
and vitest replays every tests/fixtures/update_marker_corpus.json judge case
through it, so the corpus covers the Electron remote readers as the parser
docstring claims. Fixtures that wrote a live pid with started_at 0/1 now write
the current time: under the shared v1 rule such a claim is past the 20-minute
ceiling and dead, exactly as Python judges it.
A delegate is live only when its pid AND creation time match (the updater's identity rule,
which #132345's shared judge enforces), so a made-up `ct:1700000001.500` on a live process is
the reused-pid case and correctly reads dead. The fixtures now record the spawn time of a real
delegate (inside the 2 s tolerance), so they hold under both the two-line-era readers and the
judge that replaces them.
The F1 fix runs every updater git mutation with core.hooksPath=<null device>, so the
pre-commit hook this test used to stall `_git_run`'s git never ran and the owner exited
before it could be killed mid-write. R2's invariant is about git and whatever it runs, not
hooks: a clean filter on `git add` (a local mutator) stalls git the same way and inherits the
same lock fd. Still red when `add` does not receive the fd ("a contender owns the checkout
while git still writes").
The old updater printed "→ Handing off ..." before spawning the takeover
child. On Windows, a piped stdout encodes in the ANSI code page, so the
arrow raised UnicodeEncodeError (a ValueError) before the child started:
stop_for_relaunch reported "Update takeover failed" and exited 1 for a
committed update. The banner and the Windows job-refusal notice are now
ASCII.
The takeover status test now also runs with strict cp1252 stdio (review S1).
The POSIX build launcher (_REAP_TREE) holds the checkout-lock fd for node's tree because node
marks the donated fd close-on-exec. It stayed in the caller's process group, so the
completion child's interruption path (`os.killpg(group, SIGKILL)`) killed it along with node,
while a lifecycle descendant that started its own session survived without custody: a
contender acquired the checkout while that writer kept writing (review E).
The launcher now leaves the caller's group and starts node back in it (`process_group=`):
every group kill still reaches node and what it starts in that group, but the custodian
outlives it, kills and reaps the rest of the tree (subreaper / recorded tree, as on a normal
return) and only then exits and releases the fd. A launcher that already leads its group
(a caller gave it a session) cannot leave it and keeps the old topology.
Local mutators inherit the checkout-lock fd (POSIX) so an interrupted writer keeps custody,
but git hands that fd on to repository hooks. A post-merge hook that backgrounds a process
kept the flock after the merge returned and the owner released, so every later update and
recovery was refused until the daemon exited (review F1, helix4u's report).
The custody config for local mutators now carries `-c core.hooksPath=<os.devnull>` next to
the fsmonitor/credential suppression: internal updater mutations never run repository hooks.
Per command only; the repository's own hook configuration is untouched. Genuine unfinished
mutators still hold the fd (test_lock_fd_reaches_local_mutators_only stays green).
The updater now writes a creation-time line 3 (and the hand-off scripts
append delegate:/run: lines), but the client-supplied remote probes still
accepted exactly "<pid>\n<started_at>\n". The SSH relaunch gate answered
UNCERTAIN for an exited writer's v2 claim, kept it, and refused every
reconnect/spawn until some other reader cleared it; the managed-SSH observer
reported the same claim as malformed.
Both POSIX programs now parse the A2 shape: line 1 pid, line 2 started_at,
line 3 ignored (ct or blank, as the Python reader treats a bad line 3), and
the first well-formed delegate:<pid> ct:<ct> as a second holder. The claim is
dead only when the owner AND the delegate are dead, so dead-holder detection
is never weaker than v1. The relaunch gate's delete now runs under the
updaters' <marker>.lock mutex (non-blocking; held -> UNCERTAIN) and only while
the bytes still equal the ones judged dead, so a claim published after the
read is never removed.
Review F3 (PR #132365 thread 3).
read_live_update() read a dead marker, then rechecked it under the marker
mutex in _reclaim_dead(). When a marker-only claimant (the launch hand-off,
no checkout lease) published in between, the recheck correctly kept the
replacement and returned "live" -- but the reader only handled "held" and
fell through to None. update_in_progress() then answered False for a live
update, and the skew watchdog counted that poll toward retirement; the
checkout-lock OR cannot correct it because a marker-only claim holds no
checkout lease.
A "live" recheck verdict now re-judges the marker once and returns its holder.
Review F2 (PR #132365 thread 2).
Follow-up to the R3 watchdog (review R3, kshitijk4poor 5411135165): after killing a timed-out
fetch, its output is drained for at most _DRAIN_SECONDS, as update_custody.run does on Windows,
so a git-remote-https helper still holding the pipes never hangs the update (health HX006). The
watchdog's SIGKILL line is marked POSIX-only for the Windows footgun scan: run_git never takes
that path on win32.
Review P2 (kshitijk4poor 5411135165): an unwritable marker directory/marker or checkout lock
now refuses the update where main ran it unlocked, and the refusal said only "not writable".
Decision: the refusal stays. Before the commit point it changes nothing (the update has not
started), and the marker is what the Desktop gate, the SSH remote probes, the Tauri updater and
the hand-off scripts read; running without it re-opens the race the lock exists to close.
nix/docker installs are stopped earlier by _update_preflight_handled. The checkout lock lives in
the git directory, which the update's own git commands must write anyway, so refusing there up
front only replaces a later git failure.
What changed is the message: each reason names what the path is for (the marker: the Desktop
app, gateways, installers and other updaters wait on it; the checkout lock: the git directory
git must write too), and describe_holder's remedy covers a read-only filesystem (remount
read-write, or fix owner/permissions) besides running as the owning user.
Test: an unwritable marker location's message names the path, its readers and the read-only
filesystem remedy; RED on the start sha.
The updater's fetch runs without the checkout lock fd (m3: its network half and any
credential daemon must never hold the checkout). On Linux PR_SET_PDEATHSIG ends it when the
owner dies; macOS has no parent-death signal, so a killed `hermes update` left the fetch
rewriting refs (and holding live `*.lock` ref files) under the next lock owner. The docstring
pointed at a NOT_COVERED note that does not exist.
Where no parent-death signal can be armed, run_git now runs an fd-less ref writer (`fetch`)
under a stdlib watchdog: its stdin is a pipe only the owner writes; a byte means the fetch is
done, EOF without one means the owner died, and it SIGKILLs the fetch, as the death signal
would. It holds no lock fd and runs in its own session. A watchdog that cannot start is logged
and the fetch runs as before. Readers and ls-remote write nothing and stay as they were;
promisor lazy fetches add content-addressed objects only and are not watched.
Test: the existing killed-owner fetch cell, run with the parent-death signal disabled (the
macOS condition); RED on the start sha ("git fetch outlived its killed update owner").
Review: kshitijk4poor 5411135165, open thread R3 (partly fixed).
What: git_pack_tidy's `pack-objects --stdin-packs` merge (main a0f47657b9) now goes through
update_custody.run_git, and `pack-objects` is a LOCAL_MUTATOR: it holds the checkout lock fd, and
inside an update on Windows it is job-bound. A timeout or spawn failure still reads as out of time;
a custody refusal propagates like every other updater git's. bounded_probe_run loses the
`popen_kwargs` parameter this branch added for the gc fold, which #132927 removed; nothing passes it.
Why: the tidy replaced the gc fold this branch had put in custody, and arrived as a bare git
spawned during `hermes update` under the update lock. It writes a pack the update then moves into
the object store; outside custody a killed update could leave it writing into
hermes-tidy-staging after the lock is free, and on Windows it was outside the update's job.
The tidy only runs under the lock (main.py acquires UpdateLock before _cmd_update_impl), so on
Windows it always takes custody's bounded drain, not the bare subprocess.run timeout cleanup
bounded_probe_run exists to avoid (#68609).
Red: test_merges_down_to_the_target_into_one_pack_whatever_the_size_limit saw custody spawn only
[('config', False), ('config', False)]. Green: pack tidy, custody, bounded probe, build stamp and
no-window-flag suites, 45 passed.
What: _old_updater._run_in_custody's POSIX `subprocess.run(argv, ...)` gets the HX006 waiver and
the reason the Windows `proc.wait()` already had: the takeover is the rest of a committed update
(sync + build), and it holds the checkout lock fd, so the checkout stays locked while it runs.
Why: both branches wait unbounded by design (a timeout would abandon a committed update mid-write),
but only the Windows one said so; the POSIX twin read as an unreviewed missing timeout (review L7).
Comment-only: no behaviour change, so no RED test (old-updater suites pass: 699).
What: the BLE001 waiver and docstring of read_live_update now say what makes the never-raises
probe safe: an unjudgeable marker fails open on the MARKER only; every caller that acts on None
still consults the checkout kernel lock (update_in_progress ORs checkout_lock_held,
UpdateLock.acquire/acquire_checkout take it). Behaviour unchanged.
Why: the old waiver ("unjudgeable = no live update") read as a fail-open where main caught only
OSError/ValueError (review L5). Narrowing the catch was rejected: the judge also runs the
incarnation probes (psutil/ctypes/proc readers), so a narrow catch would let an unexpected error
escape into update_in_progress and the launch path, which rely on never-raises; the guard that
actually prevents a race is the kernel lock, now named and pinned.
Test: an unjudgeable marker (the parse raising an unexpected exception) over a checkout lock held
by another process -> read_live_update None, update_in_progress True, a contender's acquire
refused. Comment-only fix, so this pins the guard (GREEN on bbfa23b137d too), it is not a RED test.
What: update_completion._prepare on win32 joins the checkout lock first (update_lock._acquire_checkout,
the joiner path that takes an R5b lease byte, exactly what the --prepared child does through
complete_source_checkout's UpdateLock). It holds that lease for its lifetime (through the --prepared
child); a refused join raises before arm_completion, so main() fails the preparation boundary.
Why: a bootstrap child whose job bind was refused runs outside the kill-on-close job and outlives a
killed owner, yet it ran arm_completion, pm.sync_venv (uv children) and
collect_superseded_generations holding no lease; the lease was only taken later in the --prepared
grandchild, so the checkout could be handed to a contender while the sync still wrote (review L3).
Tests (new file, RED on bbfa23b137d, GREEN here; Windows faked for update_completion only): the
lease is the first event, before arm_completion/sync/collector/--prepared; a refused join writes
nothing.
What: the POSIX build launcher (_OWN_GROUP -> _REAP_TREE) no longer starts node in a session of
its own. Node stays in the caller's process group; the launcher holds the checkout lock fd and,
once node exits, kills every remaining descendant: on Linux it is a child subreaper
(PR_SET_CHILD_SUBREAPER) and SIGKILLs + reaps its own children until none is left; elsewhere
(macOS) it records node's descendant tree from ps while node runs and kills the recorded
processes whose start time still matches. SIGINT/SIGTERM/SIGHUP are forwarded to node (killed
10 s later if still running), then the descendants are killed the same way.
Why: start_new_session=True moved node out of the caller's group, so every existing group kill
(update_completion's killpg on Ctrl-C, Desktop's kill(-pid), killHandoffTree) reached only the
launcher; node was orphaned and its npm/esbuild descendants kept writing after the checkout lock
was free (review L1, probe review/lock/custody/p1: grandchild wrote after abort, lock free).
The old killpg after p.wait() also signalled a reaped leader's group (pid-reuse window, review
L6): the launcher now signals only its own unreaped children (Linux) or start-time-verified
pids (ps walk), never a group.
Tests (RED on bbfa23b137d, GREEN here): a group kill of the caller-visible group stops node and
its child; a detached (setsid) grandchild outliving node is killed before the build returns, on
the subreaper and the ps-walk paths; SIGTERM to the launcher exits 143 and leaves no writer.
JOB_OBJECT_LIMIT_SILENT_BREAKAWAY_OK is only accepted through
JOBOBJECT_EXTENDED_LIMIT_INFORMATION (class 9); class 2 rejects it, so the Windows lane
failed building the control job before reaching the custody launcher. Same shape as
process_identity's job setup; the assertion now carries GetLastError.
Patching update_custody.sys.platform patches the sys module itself, so any lazy
import inside the timed run took its Windows branch: the suite's Popen guard imports
gateway.status there, which then imports msvcrt (red on macOS when that import came
first). A sys proxy scoped to update_custody drives the same Windows branch of run().
The launch-time completion tail now spawns through update_custody.run (checkout
custody), but this file's fixture still patched venv_sync.subprocess.call: the owner
test ran a real source_completion.py child against a scratch checkout (red on macOS)
and the borrowing test's 'no tail ran' assertion was vacuous. Patch the seam
production uses, as test_update_launch_completion.py already does.
The six findings are deliberate boundaries, so each gets an explicit
`# health: allow` with its reason instead of a narrowed catch:
- _old_updater._run_in_custody / update_completion._prepare (HX006): the
waited child is the rest of the committed update (sync + build); it is
supervised and dies with the updater, a timeout would abandon it mid-write.
- _old_updater._checkout_custody (BLE001): calls whatever older release's
update_lock is loaded; no fds means the takeover takes the lock itself.
- update_custody._past_commit (BLE001): fail-closed, unknown = past commit.
- update_custody._held (BLE001): a torn tree's half-written module can raise
anything at import.
- update_lock.process_create_time (BLE001): liveness probe; any psutil failure
(including partial/stub modules) degrades to the stdlib probe instead of
escaping to read_live_update.
- update_lock.read_live_update (BLE001): documented never-raises probe.
Existing `# noqa: BLE001` comments become the health waiver (ruff does not
select BLE here). Comments only; behaviour is unchanged.
The tests added for N06/F12/PID-reuse/readonly-lock used bare write_text/read_text('utf-8'); check-windows-footguns flags those. Bytes I/O, same assertions.
What: platform_default_home() appends HERMES_DATA_DIR_SUFFIX literally to the hermes/.hermes leaf via platform_default_home_with(suffix), which fixes hermes_home(), hermes_root() and the update marker together.
Why: Python (_get_platform_default_hermes_home) and Desktop (platformDefaultHermesHome) both append it, so a hermes-setup launched in a suffix-only channel environment read and wrote the unsuffixed home and marker.
Red: paths::tests::data_dir_suffix_names_the_default_home_and_its_marker_root on the 0ec53574011 resolution (byte-verbatim extract, suffix ignored): left '.hermes' right '.hermes-channel-build-x'.
Green: passes on the real fixed source extract; also cargo check clean for x86_64-pc-windows-msvc (Tauri crate itself cannot build on this host).
What: the Windows pid_is_alive opens the process with SYNCHRONIZE and returns WaitForSingleObject(h, 0) == WAIT_TIMEOUT; access denied still reads alive via liveness_from_open_error.
Why: GetExitCodeProcess == STILL_ACTIVE (259) is also what a process that exited with code 259 reports while any handle keeps its object, and its creation time still matches the v2 marker, so Busy had no age bound. Python's _early_recovery already uses the wait probe.
Red: Windows-only; marker_tests::pid_is_alive_false_for_an_exited_process_whose_exit_code_is_259 (cfg(windows)) needs a native runner.
Green: cargo check --tests --target x86_64-pc-windows-msvc clean on the real marker.rs (std-only harness); Linux marker tests 49 passed.
What: the Python and Rust tmp sweeps reclaim a tmp sibling named for our own pid once its mtime is older than 60 s (OWN_PID_TMP_STALE[_SECONDS]); other pids keep the liveness check, and an ASCII pid past u32 is dead in both (Python had kept it since N06; Rust always swept it).
Why: with deterministic container pids a previous holder of our pid number left a tmp that every later run skipped as 'our own in-flight write', so it was never cleaned. Every write of ours finishes far inside 60 s, so a fresh same-pid tmp is still kept.
Red: test_stale_tmp_under_our_own_pid_number_is_reclaimed FAILED and marker_tests::stale_tmp_under_our_own_pid_number_is_swept FAILED on 07c014355b5 (real marker.rs via a std-only harness; the Tauri crate cannot build here without webkit2gtk/dbus).
Green: both pass; test_update_lock.py 59 passed/1 skipped; marker tests 49 passed/2 ignored.
