A CLI/launchd `hermes serve` that outlived an update kept serving the old
code; the new Desktop attached on token alone, got 503 "Restart required",
and Restart re-attached the same stale process. `/api/health` now publishes
the boot commit and attach falls to the next ladder rung (next record, then
spawn) unless it equals the checkout HEAD. Unknown Desktop identity
(packaged/non-git) keeps the token-only attach.
(cherry picked from commit e59c7e64bdf417253d3ed9f85710f70adbba3e11)
* fix(bedrock): route GPT-6 family and prefixed OpenAI inference profiles, size their 1M window
An enterprise deployment on provider bedrock with model us.openai.gpt-6.1-sol
compressed at 96,000 tokens with a 400K compression cap configured. Three gaps,
all on origin/main c53536929e:
1. agent/bedrock_adapter.py:77-79 BEDROCK_OPENAI_RESPONSES_MODEL_IDS held the
GPT-5.5 and GPT-5.6 IDs only (unchanged since 16476fad10). AWS now publishes
cards for GPT-6 Astra, GPT-6 Sol, GPT-6 Luna and GPT-6.1 Sol.
2. agent/bedrock_adapter.py:165-167 is_openai_bedrock_model was an exact set
match, so us.openai.gpt-5.6-sol returned False while openai.gpt-5.6-sol
returned True. Every cross-Region inference profile of a Bedrock OpenAI model
was misclassified.
3. agent/bedrock_adapter.py:1169-1195 BEDROCK_CONTEXT_LENGTHS had no GPT-6
rows, so get_bedrock_context_length fell to the 128,000 default silently
(the fallback WARNING fired only for application-inference-profile ARNs).
agent/context_compressor.py:1181 applies the 75% small-window floor, which
is exactly the observed 96,000.
Per the AWS model cards (model-card-openai-gpt-6-1-sol.html and siblings):
the bare openai.* ID is served in-Region by bedrock-mantle (Responses, Chat
Completions, no geo or global IDs), and the us./global. profile IDs are served
by bedrock-runtime where Converse is listed as supported. GPT-6.1 Sol states a
1M context window with 131,072 max output tokens, the other cards state
1,050,000. 272K on these cards is the long-context pricing tier boundary, not
the window.
Changes
- parse_bedrock_openai_model_id(model_id) -> (bare_id, profile_prefix) | None.
is_openai_bedrock_model is True for every endpoint form of an allowlisted ID,
GPT-OSS stays False. New bedrock_openai_uses_mantle is True only for the bare
in-Region ID. Both callers (hermes_cli/runtime_provider_backends.py primary,
agent/auxiliary_client.py auxiliary) route Mantle on that predicate and keep
prefixed IDs on Converse via bedrock-runtime, the path AWS documents for them.
The Claude profile-prefix regex now derives from the same prefix tuple.
- GPT-6 generation added to the allowlist, the static picker catalog and the
context table. OpenAI rows sized from the cards (6.1 Sol 1,000,000, others
1,050,000). The table/metadata pairing test covers the new rows.
- get_bedrock_context_length logs one WARNING per model ID per process when
the static default is used for any ID, naming the model, the value, that it is
a fallback, and the model.context_length override.
- website/docs/guides/aws-bedrock.md: route table, model table and the
cross-Region note describe both endpoint forms.
The empty compression summary the deployment also reported is not addressed
here. Converse is a valid path for the prefixed ID, so routing does not explain
it, and no request trace exists yet.
* chore(bedrock): keep auxiliary_client.py within its code-health line cap
Rewrap the _build_bedrock_client docstring back to two lines so the file does not grow past its value on main (code health ratchet, FILE_LINES).
---------
Co-authored-by: unsupportedpastels <theoldwizard123@pm.me>
main.py may only shrink vs main (code-health FILE_LINES); this branch grew it
by 9 lines. _probe_container and _exec_in_container (the NixOS container-mode
re-exec, 96 lines) move verbatim to the topical sibling
hermes_cli/main_container.py; the entry point is now public
(exec_in_container) and main() late-imports it at its only call site. The
test import and the managed-runtime resolution allowlist entry follow the
move. No re-export shim. Behaviour is unchanged.
checkout_lock_held answered "free" when the lock file existed but could not be
opened. The hand-off scripts (marker.sh / marker.ps1) and the Desktop probes
(checkout_held, WINDOWS_CHECKOUT_LOCK_PS) answer "held" for the same file, so a
Python reader could reclaim a marker or admit a launch the others would park.
Only a missing lock file is free now; an existing file that cannot be opened
answers held. A lock call that fails outright (ENOLCK on NFS without lockd,
EOPNOTSUPP on some SMB shares) still answers free: nothing can hold such a
lock, and the interrupted-pull repair relies on that to run unguarded there.
Both callers that act on "held" are bounded (_reclaim_dead keeps the dead
marker; the acquire path waits its contention budget).
`update_custody.run` (a timed-out Windows update child) re-implemented the tree kill and the
bounded pipe drain of `_subprocess_compat.bounded_probe_run`: its own `taskkill /T /F` copy of
`kill_process_tree`, then `communicate()` bounded, abandoning pipes a missed descendant still
holds. Both now call `kill_and_drain`. It stays correct for the job-bound child: the tree kill
never touches the update's job (which also holds the update's other children), the drained
output still rides on the re-raised TimeoutExpired, and abandoned pipes are detached so the
`with popen()` exit never closes a pipe a reader thread is blocked on. The probe keeps closing
its own private job first.
Tests: test_a_timed_out_child_never_waits_on_a_grandchild_holding_its_pipes and
tests/hermes_cli/test_bounded_probe_run.py (unchanged) pass. (kshitijk4poor review)
`incarnation_live` (holder records, the pause record) and `_identity_live` (the marker
reader, matching marker.rs) restated the same (pid, creation time) rule and disagreed when our
own creation time is unreadable: the marker reader treated a no-ct claim naming our pid as ours
(a degraded updater writes no-ct claims) and a ct one as dead, while `incarnation_live`
answered "unprovable" for both, so `_lock_holder` named a previous incarnation as the holder.
Both now go through `_incarnation`; `_identity_live` only adds the v1 age ceiling for an
unprovable identity. `_compare_and_delete` had no callers and is removed.
Test: test_one_incarnation_rule_when_our_own_creation_time_is_unreadable fails on the previous
head and passes here; the shared marker corpus is unchanged and green. (kshitijk4poor review)
`run_contained` starts the POSIX build launcher with `subprocess.run`, which SIGKILLs its own
child when KeyboardInterrupt escapes the wait. That child was the custodian holding the
checkout lock fd: moving it out of the caller's group (E) survives a group kill, not a direct
kill. A command still cleaning up from the same Ctrl-C then exited, dropped the last lock fd,
and a writer in a session of its own went on writing under the next lock owner (review R4).
The process the runner sees is now a stand-in: it forks the custodian into a group of its own,
relays SIGINT/SIGTERM/SIGHUP to it and returns its exit status. A killed stand-in leaves the
custodian to settle the tree: node gets the same 10 s it gets after a forwarded signal, then
every descendant is killed and reaped, and only then is the fd dropped.
Test: test_a_ctrl_c_keeps_custody_until_a_detached_writer_is_gone (verbose + captured runner,
slow and instant SIGINT cleanup; real run_in_custody, real SIGINT to the caller's group, real
flock contender) fails on bc9311b904c and passes here.
The old updater printed "→ Handing off ..." before spawning the takeover
child. On Windows, a piped stdout encodes in the ANSI code page, so the
arrow raised UnicodeEncodeError (a ValueError) before the child started:
stop_for_relaunch reported "Update takeover failed" and exited 1 for a
committed update. The banner and the Windows job-refusal notice are now
ASCII.
The takeover status test now also runs with strict cp1252 stdio (review S1).
The POSIX build launcher (_REAP_TREE) holds the checkout-lock fd for node's tree because node
marks the donated fd close-on-exec. It stayed in the caller's process group, so the
completion child's interruption path (`os.killpg(group, SIGKILL)`) killed it along with node,
while a lifecycle descendant that started its own session survived without custody: a
contender acquired the checkout while that writer kept writing (review E).
The launcher now leaves the caller's group and starts node back in it (`process_group=`):
every group kill still reaches node and what it starts in that group, but the custodian
outlives it, kills and reaps the rest of the tree (subreaper / recorded tree, as on a normal
return) and only then exits and releases the fd. A launcher that already leads its group
(a caller gave it a session) cannot leave it and keeps the old topology.
Local mutators inherit the checkout-lock fd (POSIX) so an interrupted writer keeps custody,
but git hands that fd on to repository hooks. A post-merge hook that backgrounds a process
kept the flock after the merge returned and the owner released, so every later update and
recovery was refused until the daemon exited (review F1, helix4u's report).
The custody config for local mutators now carries `-c core.hooksPath=<os.devnull>` next to
the fsmonitor/credential suppression: internal updater mutations never run repository hooks.
Per command only; the repository's own hook configuration is untouched. Genuine unfinished
mutators still hold the fd (test_lock_fd_reaches_local_mutators_only stays green).
read_live_update() read a dead marker, then rechecked it under the marker
mutex in _reclaim_dead(). When a marker-only claimant (the launch hand-off,
no checkout lease) published in between, the recheck correctly kept the
replacement and returned "live" -- but the reader only handled "held" and
fell through to None. update_in_progress() then answered False for a live
update, and the skew watchdog counted that poll toward retirement; the
checkout-lock OR cannot correct it because a marker-only claim holds no
checkout lease.
A "live" recheck verdict now re-judges the marker once and returns its holder.
Review F2 (PR #132365 thread 2).
Follow-up to the R3 watchdog (review R3, kshitijk4poor 5411135165): after killing a timed-out
fetch, its output is drained for at most _DRAIN_SECONDS, as update_custody.run does on Windows,
so a git-remote-https helper still holding the pipes never hangs the update (health HX006). The
watchdog's SIGKILL line is marked POSIX-only for the Windows footgun scan: run_git never takes
that path on win32.
Review P2 (kshitijk4poor 5411135165): an unwritable marker directory/marker or checkout lock
now refuses the update where main ran it unlocked, and the refusal said only "not writable".
Decision: the refusal stays. Before the commit point it changes nothing (the update has not
started), and the marker is what the Desktop gate, the SSH remote probes, the Tauri updater and
the hand-off scripts read; running without it re-opens the race the lock exists to close.
nix/docker installs are stopped earlier by _update_preflight_handled. The checkout lock lives in
the git directory, which the update's own git commands must write anyway, so refusing there up
front only replaces a later git failure.
What changed is the message: each reason names what the path is for (the marker: the Desktop
app, gateways, installers and other updaters wait on it; the checkout lock: the git directory
git must write too), and describe_holder's remedy covers a read-only filesystem (remount
read-write, or fix owner/permissions) besides running as the owning user.
Test: an unwritable marker location's message names the path, its readers and the read-only
filesystem remedy; RED on the start sha.
The updater's fetch runs without the checkout lock fd (m3: its network half and any
credential daemon must never hold the checkout). On Linux PR_SET_PDEATHSIG ends it when the
owner dies; macOS has no parent-death signal, so a killed `hermes update` left the fetch
rewriting refs (and holding live `*.lock` ref files) under the next lock owner. The docstring
pointed at a NOT_COVERED note that does not exist.
Where no parent-death signal can be armed, run_git now runs an fd-less ref writer (`fetch`)
under a stdlib watchdog: its stdin is a pipe only the owner writes; a byte means the fetch is
done, EOF without one means the owner died, and it SIGKILLs the fetch, as the death signal
would. It holds no lock fd and runs in its own session. A watchdog that cannot start is logged
and the fetch runs as before. Readers and ls-remote write nothing and stay as they were;
promisor lazy fetches add content-addressed objects only and are not watched.
Test: the existing killed-owner fetch cell, run with the parent-death signal disabled (the
macOS condition); RED on the start sha ("git fetch outlived its killed update owner").
Review: kshitijk4poor 5411135165, open thread R3 (partly fixed).
What: git_pack_tidy's `pack-objects --stdin-packs` merge (main a0f47657b9) now goes through
update_custody.run_git, and `pack-objects` is a LOCAL_MUTATOR: it holds the checkout lock fd, and
inside an update on Windows it is job-bound. A timeout or spawn failure still reads as out of time;
a custody refusal propagates like every other updater git's. bounded_probe_run loses the
`popen_kwargs` parameter this branch added for the gc fold, which #132927 removed; nothing passes it.
Why: the tidy replaced the gc fold this branch had put in custody, and arrived as a bare git
spawned during `hermes update` under the update lock. It writes a pack the update then moves into
the object store; outside custody a killed update could leave it writing into
hermes-tidy-staging after the lock is free, and on Windows it was outside the update's job.
The tidy only runs under the lock (main.py acquires UpdateLock before _cmd_update_impl), so on
Windows it always takes custody's bounded drain, not the bare subprocess.run timeout cleanup
bounded_probe_run exists to avoid (#68609).
Red: test_merges_down_to_the_target_into_one_pack_whatever_the_size_limit saw custody spawn only
[('config', False), ('config', False)]. Green: pack tidy, custody, bounded probe, build stamp and
no-window-flag suites, 45 passed.
What: _old_updater._run_in_custody's POSIX `subprocess.run(argv, ...)` gets the HX006 waiver and
the reason the Windows `proc.wait()` already had: the takeover is the rest of a committed update
(sync + build), and it holds the checkout lock fd, so the checkout stays locked while it runs.
Why: both branches wait unbounded by design (a timeout would abandon a committed update mid-write),
but only the Windows one said so; the POSIX twin read as an unreviewed missing timeout (review L7).
Comment-only: no behaviour change, so no RED test (old-updater suites pass: 699).
What: the BLE001 waiver and docstring of read_live_update now say what makes the never-raises
probe safe: an unjudgeable marker fails open on the MARKER only; every caller that acts on None
still consults the checkout kernel lock (update_in_progress ORs checkout_lock_held,
UpdateLock.acquire/acquire_checkout take it). Behaviour unchanged.
Why: the old waiver ("unjudgeable = no live update") read as a fail-open where main caught only
OSError/ValueError (review L5). Narrowing the catch was rejected: the judge also runs the
incarnation probes (psutil/ctypes/proc readers), so a narrow catch would let an unexpected error
escape into update_in_progress and the launch path, which rely on never-raises; the guard that
actually prevents a race is the kernel lock, now named and pinned.
Test: an unjudgeable marker (the parse raising an unexpected exception) over a checkout lock held
by another process -> read_live_update None, update_in_progress True, a contender's acquire
refused. Comment-only fix, so this pins the guard (GREEN on bbfa23b137d too), it is not a RED test.
What: update_completion._prepare on win32 joins the checkout lock first (update_lock._acquire_checkout,
the joiner path that takes an R5b lease byte, exactly what the --prepared child does through
complete_source_checkout's UpdateLock). It holds that lease for its lifetime (through the --prepared
child); a refused join raises before arm_completion, so main() fails the preparation boundary.
Why: a bootstrap child whose job bind was refused runs outside the kill-on-close job and outlives a
killed owner, yet it ran arm_completion, pm.sync_venv (uv children) and
collect_superseded_generations holding no lease; the lease was only taken later in the --prepared
grandchild, so the checkout could be handed to a contender while the sync still wrote (review L3).
Tests (new file, RED on bbfa23b137d, GREEN here; Windows faked for update_completion only): the
lease is the first event, before arm_completion/sync/collector/--prepared; a refused join writes
nothing.
What: the POSIX build launcher (_OWN_GROUP -> _REAP_TREE) no longer starts node in a session of
its own. Node stays in the caller's process group; the launcher holds the checkout lock fd and,
once node exits, kills every remaining descendant: on Linux it is a child subreaper
(PR_SET_CHILD_SUBREAPER) and SIGKILLs + reaps its own children until none is left; elsewhere
(macOS) it records node's descendant tree from ps while node runs and kills the recorded
processes whose start time still matches. SIGINT/SIGTERM/SIGHUP are forwarded to node (killed
10 s later if still running), then the descendants are killed the same way.
Why: start_new_session=True moved node out of the caller's group, so every existing group kill
(update_completion's killpg on Ctrl-C, Desktop's kill(-pid), killHandoffTree) reached only the
launcher; node was orphaned and its npm/esbuild descendants kept writing after the checkout lock
was free (review L1, probe review/lock/custody/p1: grandchild wrote after abort, lock free).
The old killpg after p.wait() also signalled a reaped leader's group (pid-reuse window, review
L6): the launcher now signals only its own unreaped children (Linux) or start-time-verified
pids (ps walk), never a group.
Tests (RED on bbfa23b137d, GREEN here): a group kill of the caller-visible group stops node and
its child; a detached (setsid) grandchild outliving node is killed before the build returns, on
the subreaper and the ps-walk paths; SIGTERM to the launcher exits 143 and leaves no writer.
The six findings are deliberate boundaries, so each gets an explicit
`# health: allow` with its reason instead of a narrowed catch:
- _old_updater._run_in_custody / update_completion._prepare (HX006): the
waited child is the rest of the committed update (sync + build); it is
supervised and dies with the updater, a timeout would abandon it mid-write.
- _old_updater._checkout_custody (BLE001): calls whatever older release's
update_lock is loaded; no fds means the takeover takes the lock itself.
- update_custody._past_commit (BLE001): fail-closed, unknown = past commit.
- update_custody._held (BLE001): a torn tree's half-written module can raise
anything at import.
- update_lock.process_create_time (BLE001): liveness probe; any psutil failure
(including partial/stub modules) degrades to the stdlib probe instead of
escaping to read_live_update.
- update_lock.read_live_update (BLE001): documented never-raises probe.
Existing `# noqa: BLE001` comments become the health waiver (ruff does not
select BLE here). Comments only; behaviour is unchanged.
What: the Python and Rust tmp sweeps reclaim a tmp sibling named for our own pid once its mtime is older than 60 s (OWN_PID_TMP_STALE[_SECONDS]); other pids keep the liveness check, and an ASCII pid past u32 is dead in both (Python had kept it since N06; Rust always swept it).
Why: with deterministic container pids a previous holder of our pid number left a tmp that every later run skipped as 'our own in-flight write', so it was never cleaned. Every write of ours finishes far inside 60 s, so a fresh same-pid tmp is still kept.
Red: test_stale_tmp_under_our_own_pid_number_is_reclaimed FAILED and marker_tests::stale_tmp_under_our_own_pid_number_is_swept FAILED on 07c014355b5 (real marker.rs via a std-only harness; the Tauri crate cannot build here without webkit2gtk/dbus).
Green: both pass; test_update_lock.py 59 passed/1 skipped; marker tests 49 passed/2 ignored.
What: on win32, _acquire_checkout releases and refuses (reason: read-only; delete it or fix its permissions) when the lock file only opened read-only; the read-only fallback stays POSIX-only.
Why: Windows children find their owner via that record (_held_by_our_windows_ancestor); without it the parent admitted itself and its completion child was refused (RuntimeError mid-update).
Red: Linux stand-in probe (update_lock.sys.platform=win32, kernel-lock stubs) on 9afee3fc8a1: _acquire_checkout admitted=True with record b'0\\n0\\n'.
Green: same probe -> refused with the read-only reason; native test_read_only_lock_file_is_refused_on_windows (platforms windows) needs a Windows runner.
What: _open_lock_file opens with O_NOFOLLOW, and _acquire_checkout refuses (without unlinking) a lock file that is not a regular single-link file before locking or truncating it.
Why: the holder record is lseek/ftruncate/written into the lock file; a symlink or hard link at that name truncated a file outside the install.
Red: test_checkout_lock_never_writes_through_a_link_to_a_file_outside_the_install[symlink,link] both FAILED on d7dba9fb8d7 (sentinel overwritten).
Green: both pass; test_update_lock.py + _fd_provenance + _protocol 74/74 (incl. the read-only A5 lock-file test).
What: the no-hard-link fallback of _publish_exclusive treats a short write as ENOSPC and, on any write/fsync failure, closes the fd and unlinks the marker only if it is still the inode it created (compared under the marker mutex), then re-raises.
Why: the torn prefix named our live pid, so a process that never acquired blocked every updater for as long as it lived (disk full / EIO on FAT, exFAT, some network mounts).
Red: test_failed_exclusive_create_leaves_no_torn_claim FAILED on 1a545e69af9 (marker left behind).
Green: it passes; test_withdrawing_a_torn_claim_never_deletes_a_replacement (guard) passes before and after; test_update_lock.py 56/56.
What: _sweep_dead_tmp_siblings parses the pid field with _bounded_int (ASCII, u32, overflow-safe) and skips names it cannot attribute.
Why: str.isdigit() accepts Unicode digits like '²'/'①' that int() refuses, so one planted tmp name raised ValueError out of UpdateLock.acquire.
Red: test_tmp_litter_with_a_digit_lookalike_pid_never_breaks_admission[superscript-two,circled-one] both FAILED on 23b8a30cc4a.
Green: both pass; test_claim_publishes_whole_and_reclaims_dead_writers_tmp_files still passes.
What: _parse_marker binds the bounded delegate pid once and uses it, instead of re-parsing the raw token with int().
Why: a delegate line with >4300 leading zeros passed the _bounded_int filter but int() raised ValueError, hiding a live owner from read_live_update and crashing UpdateLock.acquire.
Red: corpus case delegate_pid_leading_zeros_past_int_limit -> ValueError at update_lock.py:594 on b0ced54db07.
Green: 62/62 test_update_marker_corpus.py; Rust corpus test runs the same case.
What: _update_takeover starts update_finish via update_custody.popen_post_commit and _old_updater
starts the takeover via the already-loaded update_lock helpers: created suspended, bound to the
lock owner's kill-on-close job, then resumed (POSIX keeps the lock fd). A refused bind still runs
(post-commit, D2) but is printed/receipted and the child joins the lock holding its own lease (R5b).
Why: both hops were plain unsuspended subprocess runs on Windows, so killing the takeover (or the
old updater) freed the checkout lock while the unbound child kept syncing and building.
Red/green: Windows-native cells added (refused bind -> receipt step / notice; pre-fix none); POSIX
takeover/finish/custody suites pass (144 passed).
What: every Windows process that joins its ancestor's checkout lock also locks one of 16 lease bytes
past the owner's byte, taken while the owner is seen alive; _try_lock/checkout_lock_held and the Tauri
marker probe treat any held lease as a held lock. The refused completion child still runs (D2: a
committed update never fails) but is no longer unfenced.
Why: a child outside the kill-on-close job outlived a killed owner, whose msvcrt lock died with it,
so a contender acquired while the child still wrote.
Red/green: Windows-native cells (Python owner-killed/unbound child; Rust held lease) need a native run;
POSIX paths unchanged (update_lock suites pass, same failure set as base).
What: venv_sync._finish_source_update starts the tail child through update_custody.run(inherit_lock=True):
POSIX keeps the lock fd; Windows creates it suspended and binds it to the lock owner's kill-on-close
job before it runs. A refused bind never runs the child; the launch fails as a failed tail does.
Why: a raw subprocess.call left the child outside any job on Windows, so killing the launching
backend freed the checkout lock while the tail kept building under a second updater.
Red/green: refused custody -> pre-fix 'DID NOT RAISE' (child ran); fixed -> passed. Native Windows cell added.
What: build_prepared_desktop runs npm run build / builder through source_build.run_in_custody.
Why: they used bare run_contained (no lock fd, no Windows job), so a killed owner freed the
checkout while the desktop build kept writing node_modules/release.
Red/green: fake npm recording the lock fd -> pre-fix 'no' (AssertionError); fixed -> 'yes', passed.
What: on POSIX inside an update, contained_command runs the build under a stdlib parent that holds
the lock fd, starts the command as its own process group, and SIGKILLs that group once it exits;
source_build.run_in_custody is the one seam for checkout-writing build commands.
Why: node marks the donated fd close-on-exec, so npm/esbuild under a dead node kept writing with the
checkout lock already free.
Red/green: fake node leaving a fd-less writer -> pre-fix AssertionError (writer ran on); fixed -> passed.
What: the job-joining launcher starts the real command CREATE_SUSPENDED, checks IsProcessInJob
against the update job, and resumes it only then; otherwise kills it and exits 87 (the refusal path).
Why: a redirecting interpreter (Store Python alias) could join while node landed outside the job,
so owner death freed the lock while node wrote. Verification replaces topology guessing.
Red/green: Windows-only cell (SILENT_BREAKAWAY_OK job control) added; needs a native run (pre-fix prints 'built').
What: update_custody.run's in-update (Windows) branch kills the child's tree (taskkill /T /F) on a
timeout, then drains its output for at most 5 s and detaches pipes a survivor still holds.
Why: it killed only git.exe and waited unbounded on git-remote-https, which inherits the pipes.
Red/green: pre-fix run(sh -c 'sleep 20 & sleep 20', timeout=0.5) -> AssertionError (waited); fixed -> passed.
What: delete update_custody.spawn_kwargs (no production caller since #132927 removed the gc fold) and
its helper-only test; a parametrized test drives run_git itself (gc holds the lock fd, a reader not).
Why: the old test exercised only the dead helper and returned early on win32, a false green.
Red/green: mutant 'run_git raises' -> new test 2 failed (old test passed under it); real code -> 8 passed.
#132927 deleted gitlock.consolidate_lazy_fetch_packs, the one updater git
that ran under its own bounded runner with spawn_kwargs custody. Every
updater git now goes through run_git, so the import and the docstring
exception go with it.
CustodyRefused is an OSError, and many readers swallow one (_git_stdout, _capture_head_sha),
so on a refusing host the first refusal could end the update as a misleading downstream
error. update_custody now keeps the run's first refusal (and whether any update child had
run in custody, or the update was past its apply stage), and the command's failure path
(cmd_update; the completion child's _finish) prints it in place of the generic error:
it names `hermes update`, says nothing was changed when no update step had run, and how to
retry. A refused bind and a job that cannot be handed to a build launcher are now receipted
like the refused launcher join; the resume moved to update_lock.resume_suspended_child.
Tests: Linux (the cmd_update failure path after a swallowed refusal, receipt step) and a
Windows live cell (the real readers swallow a real refused job assignment).
The Windows completion child (dependency sync, product builds) was started running and
bound afterwards, so a fast grandchild could escape the update's kill-on-close job, and
a refused bind was only a log line. It is now created suspended, bound, then resumed.
Post-commit a refused bind still runs it (ruling: refusing would fail a committed
update), but never silently: the refusal is printed and recorded as a failed
update_custody step in this run's receipt and in the receipt the child resumes (its
request is rewritten before it runs), so the terminal receipt carries it.
Windows live cell: the real AssignProcessToJobObject refuses an event handle; the
child must not have run at bind time and the step must reach the child's receipt.
The checkout's msvcrt byte lock belongs to the process that took it: an inherited
handle does not keep it, so a writer outside the update's kill-on-close job
outlived a killed owner and wrote the checkout behind a free lock. A refused
bind now kills the still-suspended child and raises CustodyRefused; a refused
launcher join (the Node build) exits before starting node and raises it too; a
job that cannot be created or handed over refuses the same way.
The merge used refs/remotes/upstream/main but the commit counts used the short
upstream/main, which a local branch of that name shadows (refs/heads before
refs/remotes): the sync printed '1 behind' and merged 2 commits. The sync now
resolves refs/remotes/upstream/main^{commit} once and uses that SHA for both
counts (against refs/remotes/origin/main) and the fast-forward.
m3: installer checkouts are partial clones, so a move (reset --hard, merge
--ff-only, checkout) lazily fetched the target's missing objects as a child
holding the checkout lock fd, with the user's credential helper; a
credential-cache daemon it started held the checkout for its lifetime.
Mutators now run with -c credential.helper=, and before a move run_git
fetches what it will read (git diff <HEAD or empty tree> <target>) without
the fd and with the helpers, so a private fork still authenticates.
m4: consolidate_lazy_fetch_packs ran gc --auto with neither the fd nor a
death signal; gc/repack/pack-objects outlived a killed owner and repacked
under the next one. gc (it packs refs) is a local mutator and its bounded
runner gets run_git's custody via update_custody.spawn_kwargs (POSIX: the
lock fd its children inherit). pull is dropped from LOCAL_MUTATORS (unused;
its network half must not hold the fd). The module doc names the exception
and the read-only source_releases/source_check readers.
run_contained merges the build's stderr into the output it drops on success,
so the launcher's 'update custody unavailable' notice was invisible in a
normal update. The launcher now also writes the notice to a report file;
contained_command turns it into a logger.warning (errors.log) and an
update_custody receipt step. The build outcome is unchanged.
venv_sync.prepare_launch and _prepare_borrowed_launch claimed the marker with
UpdateLock() (no install_root), and UpdateLock._claim_marker unlinked a dead
marker without asking whether the checkout lock was held. The claim then
failed acquire_checkout and released its own claim: the killed update's
marker was gone while its completion child still ran, and the Desktop gate
reopened over it.
- _claim_marker refuses as held (pid 0) and keeps the marker when the checkout
lock is held by another process tree (our own hold or an inherited lock fd
may still reclaim).
- UpdateLock(checkout_first=False): install_root names the guarding checkout
without taking its lock first; both venv_sync sites pass install_root=root.
- _lock_holder reports a dead recorded updater over a held lock as held, so
hermes update / launch repair / source completion refusals use the held
wording instead of naming the dead pid (m6).
A venv's Scripts\python.exe is a redirector that starts the base interpreter as its own child
inside a job of its own. That child cannot join the update job once the job holds the updater's
git children (AssignProcessToJobObject: ERROR_ACCESS_DENIED, 'could not join the update job: 5',
every install.ps1 TUI build on the Windows runners), and the redirector keeps the inherited job
handle open, so the job never closed on the owner's death (test_killed_owner_takes_its_node_build_down).
The stdlib launcher now runs on sys._base_executable, and a refused join runs the command outside
the job with a stderr notice instead of failing the build.
The stable-zip identity test injects its git faults at update_custody.run as well: on Windows
inside an update that seam is a suspended Popen bound to the job, not subprocess.run.
R2: hermes_cli/update_custody.run_git is THE updater git runner;
update_cmd._git_run, update_cmd_git._git_run, update_cmd_stash,
update_cmd_check._git, gitlock, update_cmd_zip and the fleet ancestry probe
all go through it.
- every updater git call carries -c gc.autoDetach=false
-c maintenance.auto=false: no detached gc/maintenance child ever forks
under (POSIX) or outlives (Windows) the checkout lock;
- POSIX: only local mutators (reset, checkout, merge, stash, rebase,
update-ref, clean, apply, commit, ...) inherit the lock fd, with
core.fsmonitor=false; fetch/ls-remote/credential and readers run without
it, so a credential-cache daemon never holds the checkout (reviewer: up to
2 h of installer waits). On Linux those fd-less children die with the
updater (PR_SET_PDEATHSIG) instead of rewriting refs under the next owner;
- Windows: inside an update every child is created suspended, assigned to
the owner's kill-on-close job and only then resumed; the Node build
(pm.progress.run_contained) joins it through a stdlib launcher.
The fork sync fast-forwards locally (merge --ff-only
refs/remotes/upstream/main after its fetch) instead of a networked pull
under the mutator fd.
R6: read_live_update/_reclaim_dead no longer delete a dead marker while the
checkout kernel lock is held (a killed updater's completion/build/git still
runs); the reader answers a 'held' UpdateHolder, matching marker.sh's held
verdict, and reclaims only once the tree is gone. update_in_progress and
read_live_update take install_root.
D17: the updater's checkout acquire waits up to 3 s on contention before
reporting busy: a marker.sh/marker.ps1/Python 'held?' probe holds the lock
for microseconds (reviewer: 257/3283 spurious hermes update failures).
D8: _parse_marker bounds pid (u32) / started_at (u64) / delegate pid by
significant digits before int(), so a 5000-digit field is malformed
(dead), never a ValueError out of read_live_update or acquire.
Windows: one kill-on-close job per lock owner (update_tree_job), shared by
every bound child, so a launcher can hand it to a child it cannot see.
Release now judges the marker by identity, and the identity check sent
os.kill(own_pid, 0) on every release. Our pid is alive by definition; only
the incarnation (5 ms ct) is in question. Same short-circuit in marker.rs.
R3 / A7 rule 1: every marker decision-and-mutation (reclaim of a dead claim,
adopt + delegate append, release, ownership hand-over) now runs while holding
the kernel lock on the never-deleted sidecar `<marker>.lock` (POSIX flock;
Windows: the file opened share-none, the same lock the PowerShell hand-off
takes), bounded to 10 s. Only the exclusive first publish stays lock-free
(rule 2). Python (update_lock.marker_mutex) and Rust (marker.rs lock_marker)
interoperate on the same lock. A stale reader paused before its unlink can no
longer delete a claim published after its read.
A7 rule 4 / R12: one incarnation rule. Our own pid is ours only within 5 ms of
our creation time; a no-ct claim naming our pid is a previous incarnation
(dead) — in Python (`update_lock.incarnation_live`, exported for the pause
record) and in marker.rs (no more 2 s self tolerance; Electron's v1 own-pid
pre-write is reclaimed and republished as v2; the exit-2 heal requires our
exact incarnation).
A7 rule 5: release is identity-based (`_release_decision` / `release_decision`).
The owner removes its claim regardless of delegate lines unless a LIVE
delegate other than itself exists, which then inherits the claim (rewritten
with the delegate as owner, started_at and run lines kept). A delegate drops
its line while the owner lives, else deletes. This is the native Windows
`orphaned_update` root cause: the hand-off script named `hermes update` as
delegate before it claimed, the child adopted without writing, and its exit
released nothing, leaving '<dead owner> ... delegate:<exited child>'.
Parser (A7 rule 7): lines 4+ are tagged — first well-formed delegate line,
every `run:<id>` line; pid must fit u32. Both consumers run
tests/fixtures/update_marker_corpus.json (judge + release cases).
R2: checkout mutators keep custody. `_git_run` passes the held checkout lock
fd to git (and its hooks); `run_source_script` passes it to the Node build;
`complete_source_checkout` ACQUIRES `UpdateLock(install_root=root)` instead of
claiming only its home's marker; venv_sync's interrupted-update finish acquires
the checkout lock before mutating instead of sampling it and hands the fd to
its completion child; the historical takeover chain passes it down. On Windows
a child of the update joins the lock when its holder record names a live
ancestor (it runs in that owner's kill-on-close job).
R11: a live pre-root-marker claim under <root>/profiles/*/ (the old per-profile
marker) is observed: a new update refuses while one is live and not a partner.
Real-process tests: tests/hermes_cli/test_update_lock_protocol.py (each fails on
6ff4055cd74), corpus consumers in Python and cargo.
A fresh pid namespace (bwrap, a container) hands a killed updater's pid to
the next launch about a second after the updater started: inside the 2 s
cross-writer create-time tolerance. The launch then read the dead claim as
its own live one, adopted it (acquired=False) and prepare_launch returned
early as "under the update's claim" -- silently skipping the source-update
tail the kill left owed (test_hostile_commit tail cells, red on the
composite CI only because their timing landed inside the window).
Compare our own pid against our own creation time exactly (3-decimal
marker rounding); other pids keep the 2 s skew window.
- A2: positional parse (BOM stripped, CRLF accepted, integer pid and
started_at or the marker is malformed = dead; a malformed ct line makes
it v1; only an exact line-4 `delegate:<pid> ct:<n>` counts).
- A1: a live pid whose creation time cannot be read is live only within
the v1 20-minute ceiling (it may be a reused pid).
- A3: claims are published whole (private tmp + exclusive hard link,
exclusive create where links are unsupported); a 0-byte marker younger
than 5 s is a claim in flight and is neither taken over nor deleted.
- A5: an existing lock file we cannot write (left root-owned by
`sudo hermes update`) is opened read-only for the kernel lock instead
of refusing every later update.
- m12: macOS `ps -o lstart` runs with TZ=UTC0 in a minimal env (no DST
skew, no raw environ copy) and is parsed as UTC.
- m10: tmp siblings left by a writer that died are reclaimed at claim.
- No delegate line is written when our own creation time is unknown.
