`git checkout <existing branch>` can exit 0 and move HEAD while leaving a file
unwritten ("unable to unlink old ..."): HEAD and the index name the new commit,
the file keeps the old bytes. `tree_is_whole` only judged baselines at an
unmoved HEAD, so that torn tree was admitted and the paused gateways started
on it.
`_checkout_move` now judges every move the moment git returns: the paths
between the commit it started from and the one HEAD reached that are changed
and were not already there byte for byte (`update_pause_record.torn_by_move`)
are recorded on the baseline as `landed[<head>]`. The gate holds while any of
those paths is still changed. Judging at return time, not at resume, keeps a
later dependency sync or stash restore that rewrites an upstream-changed file
(package-lock.json) from holding the set. A recorded move the updater died
before judging is judged at resume time against what the pause saw.
A step's own baseline is retired only when its move landed whole (it was
retired on any HEAD move before). The moved-HEAD gate reuses tracked_changes
and the dirty digests the record already carries.
recover() runs before unrelated commands (anything printing --json, the
serve/dashboard processes Desktop reads over a pipe). Its own notices already go
to stderr, but the Windows resume helpers it calls print with bare print(),
which landed in that command's stdout. The resume now runs with sys.stdout
pointed at stderr. The record tests' resume stand-ins report on the real stdout
(sys.__stdout__) accordingly; no assertion changed.
update_pause_record.recover() runs before every command but `update` and
`gateway run` (chat, doctor, config, the serve/dashboard processes Desktop
spawns). Its Windows resume waits out the relaunch readiness budget for each
profile/unmapped gateway; one that never became ready was handed back unowned
"so the next launch retries at once", so every following command stalled
~30 s again (measured: 31 s on each of three consecutive launches).
Profile and unmapped entries now get the stored backoff SCM services already
have (service_retry): an attempted entry the resume left owed gets
relaunch_retry {attempts, next_at} (60 s doubling, capped at 1 h); a launch
inside the window holds it owed beside the draining entries without relaunching
it. The debt itself is never dropped, a tree-gate deferral does not count as
an attempt, and `hermes update` (merge_into does not carry the backoff) still
retries at once.
The pause record's second evidence reader (_names_request) turned a refused or
malformed planned-stop marker / accepted receipt into False, i.e. "this update
never asked it to stop". Past the request's TTL recovery then dropped the live,
still-draining gateway (drop_never_stopped) and durably retired its debt.
Now only an absent file or a valid one naming something else is "not asked";
refused/malformed is unknown: the entry stays owed (held as draining while its
process lives), and unknown is never resolved into stop_sent, so the next pass
judges it again.
The consumer's receipt (mark_stop_accepted) swallowed its own write failure, so
an accepted stop could leave no durable trace at all. When the home refuses the
new receipt file it now stamps the kept request in place ("accepted", opened r+,
never created); a torn stamp reads malformed, hence unknown, hence owed. An
expired request nobody accepted still expires.
The unmoved-HEAD tree gate decided which paths the failed move could have
written from the CURRENT FETCH_HEAD and remote refs, so a later fetch whose
target no longer touched a half-written file turned the gate from hold to
resume on unchanged partial bytes. And a path dirty at pause was exempt by name
alone, so an autostashed edit let different half-written bytes through.
_pull_updates now records the resolved move target on the pause's own baseline
before git writes (mark_move; a failure aborts the move), and the gate diffs
HEAD against that recorded target. The pause also stores each dirty path's
digest, so only the same bytes count as "seen at pause". A pause with no
recorded move, or a record from before these fields, keeps the fetched-refs
and pathname behaviour.
When both the producer's mark_stop_sent and the consumer's checkpoint into the
pause record were refused (busy mutex, Windows replace refusal), the consumer
still accepted the stop and left only the 60-second request marker. An updater
that died meanwhile left recovery to drop the still-draining gateway as never
asked once that TTL passed, losing its restart.
The consumer's fallback now writes a receipt beside the marker (the gateway's
own home, not the record) naming the accepting incarnation; recovery treats it
as stop evidence without a TTL. Unconsumed requests still expire, and a user's
stop (another stopper) is still no evidence.
read() returned None for ENOENT, a refused read and malformed bytes alike, so
the carrier survey dropped an unreadable newer claim (its older alpha+beta copy
then executed again, replaying completed alpha) and write() replaced an
unreadable orphaned record (losing its restart obligation).
Only a missing file is absent now; a refused or malformed one raises
RecordUnknown (an OSError), so selection, claiming and the new pause's write
fail closed and leave every file for a later, readable retry.
Decision for review 5411136378's "same HEAD" item. At the pre-update HEAD the
gate counted every tracked change the pause had not seen as git's half-written
checkout. A no-op update (already at the target) still runs its dependency and
web builds; one that rewrites a tracked file (a lockfile, a generated asset)
made the end-of-update resume defer, and every later launch and every later
no-op `hermes update` (adoption keeps the old baseline) deferred again, so the
gateways stayed stopped until someone cleaned the file. A user editing a tracked
file had the same effect.
Rule chosen (a committed update's gateways never wait on a follow-up that cannot
succeed, and never start on a torn tree): a fast-forward, reset or merge only
writes paths that differ between HEAD and its target, and the update fetches
that target first. An unseen change now counts only on a path that differs
between HEAD and a commit in FETCH_HEAD or refs/remotes. When git cannot say or
nothing was fetched, every unseen change still counts (the previous behaviour),
so the torn-tree cells keep holding.
Decision for review 5411136378's "moved HEAD" item. tree_is_whole required
pm.venv_is_current at a moved HEAD and counted any exception as "not current".
The updater's early exits after the pull (wrong branch, channel sha mismatch)
reach the resume before the dependency sync, so they deferred; on a checkout
whose launches never sync dependencies (developer checkout, no install stamp,
lazy installs disabled, ZIP) or whose probe raises, the deferral never ended
and gateways stayed stopped where main restarted them.
Rule chosen (gateways never stay stopped after a committed update without an
owed follow-up that can succeed): defer only when the install is self-managed
(updateMechanism "self", the conditions venv_sync.prepare_launch syncs under),
because every later launch syncs dependencies before startup recovery runs, so
the owed record resumes on the next launch. Everywhere else, and when currency
cannot be established, resume now; the restarted gateway's own launch syncs or
prints the remedy.
Changed assertions: test_resume_waits_for_a_whole_tree pinned "dependencies"
deferral on a stamp-less checkout, which is the stranding this item reports; it
now expects a resume. test_an_adopted_baseline_never_replaces_this_runs_own
patches the renamed helper (_venv_is_current -> _deps_hold_resume).
update_pause_record._git ran a bare ["git", "-C", root] through subprocess.run.
On a Windows install whose only git is the copy install.ps1 staged in PM's store
(expose_pm_git puts it on PATH only inside the updater), every gate read died
with WinError 2: head_sha returned None and tree_is_whole refused on every
launch, so paused gateways stayed stopped. Reads now resolve git like the
updater does without installing anything (PATH, then PM's recorded entry, then
the lockfile's staged entry) and run through update_custody.run_git, so inside
an update they are job-bound on Windows and carry the custody config.
The resolver mirrors _early_recovery._git_executable from #132361 minus the
marker-recorded path; this lane does not carry that module's version.
Review 5411136378 (#132338) fix 3.
identity() took its creation time from process_identity._process_create_time
(psutil only), while identity_is_live judges with update_lock.incarnation_live,
whose producer update_lock.process_create_time falls back to the stdlib probe.
Without psutil every record was written with ct=None, which the reader can only
call unprovable (live), so a killed updater's pause was never orphaned on such a
host. identity() now calls update_lock.process_create_time, the same producer
the marker uses.
Review 5411136378 (#132338) fix 2.
Three hygiene gaps in the Windows paused-gateway record:
- _atomic_write left `<name>.<pid>.tmp` beside the record when os.replace
was refused (a Windows reader without FILE_SHARE_DELETE) or the write
failed: the temp is now unlinked and the error still propagates.
- _mutex counted re-entry process-wide, so a second thread of the same
process (gateway consumer, watcher) entered while the first held the
kernel lock: the depth is now threading.local and each thread's first
entry takes its own descriptor + flock/msvcrt lock, so threads exclude
each other like processes.
- A `.retired` list that reads but does not parse raised RetiredUnknown
from adopt_orphans -> every Windows update aborted, forever. With no
record or claim of this checkout left the list guards nothing: it is
renamed to `.corrupt` (warning on stderr) and the update goes on.
While a record/claim exists the list may be the only proof it was
completed, so it stays unknown (R9 fail-closed is unchanged); a list
that cannot be read at all (sharing violation) is unknown as before.
Review W3.
merge_into carried an adopted orphan's pre_sha/dirty_at_pause with
setdefault, so record_pause's later stamp_tree kept the ORPHAN's
baseline and dropped this run's own. When the killed run had moved
HEAD X->Y, this run's torn same-HEAD checkout at Y was judged against
X: HEAD "moved", so only the venv was checked and the gate passed
(probe: composite (True, '') vs own stamp (False, 'half-written')).
The token now carries `baselines`: one {pause_id, pre_sha,
dirty_at_pause} per pause (stamp_tree is idempotent per pause_id;
merge_into, finish_pause and abandon_pause carry the list). The gate
fails when ANY baseline taken at the current HEAD sees a tracked change
its own pause did not, so the oldest evidence still blocks a torn tree
the orphan left and this run's evidence blocks the tree it tore itself.
The record format is new on this stack (not on main): no compat read.
Review W1.
Code-health BLE001 in hermes_cli/update_pause_record.py. All three are
deliberate boundaries, now waived on the except line with the reason:
- _venv_is_current: fail closed -- any error reading venv state means "not
current", so paused gateways stay stopped.
- _resume_fenced: any resume error is reported and the claim handed back in
the finally path, never raised into the launch.
- recover: start-of-run recovery never bricks a launch.
Behaviour unchanged.
What: _resume_claimed takes the checkout lock (bounded contention wait) around the tree gate and
the verified gateway start; when an update already took the checkout after our claim, the claim
is handed back unowned and untouched, so that update or the next launch owns it.
Why: recovery judged the tree and launched gateways with no checkout custody, so a concurrent
`hermes update` could mutate the checkout under freshly started gateways (the updater skips a
live claim and may discover no running gateway to pause).
Red: pre-fix update_pause_record.py -> "gateways restarted while an update owned the checkout: resume ['default']".
Green: test_recovery_never_starts_gateways_under_an_update_that_took_the_checkout passes (set resumed by the next launch).
What: merge_into carries the adopted pause's pre_sha/dirty_at_pause (setdefault), so
record_pause's stamp_tree keeps the oldest baseline instead of stamping the current tree.
Why: re-stamping certified a tree the killed update left half-written as "dirty before the
pause", letting the adopting update's resume start gateways on a torn checkout.
Red: pre-fix update_pause_record.py -> test fails "adoption certified the torn tree as the pre-update baseline".
Green: test_an_update_adopting_an_orphan_never_certifies_the_tree_it_left_torn passes.
R8 re-review M2. `_retired()` read any OSError/ValueError as "nothing retired": while the list
was unreadable (a Windows sharing violation, an AV scanner) a completed obligation came back into
`orphans()`, `claim()` took it, and a retirement made in that window rewrote the list with only
its own ids, erasing the history for good.
- Only an absent list is empty. One that exists but cannot be read (or is malformed) raises
`RetiredUnknown`: `orphans()` reports it (recovery prints the skip), `claim()` takes nothing,
`_retire()` raises before rewriting the list or deleting a carrier.
- `orphans()` surveys under the pause mutex, so it no longer reads the list while a mutator
replaces it (Windows opens without FILE_SHARE_DELETE).
- `hermes update` aborts before its first stop with a clear message when it cannot read what an
earlier update paused, instead of adopting from an unknown list.
- `_mutex()` takes an optional wait (used by the planned-stop consumer in a later commit).
Test: test_an_unreadable_retired_list_claims_nothing_and_forgets_nothing (real file, mode 000 as
the stand-in for a Windows read refusal) — RED on 13017670871, GREEN here.
(cherry picked from commit 08ce58ae672457def25deb5e7f775f5166fa7eaf)
Completion (a recovering launch's finished claim, an update's discharged record)
now lists the obligation's ids (and every id it absorbed) on a durable retired
list before unlinking its files. A reader holding a file without
FILE_SHARE_DELETE makes the unlink fail; that leftover copy is now redundant by
id instead of an orphan whose stale set the next launch resumes again. Same-id
copies keep the furthest-progressed one (a claim's rev), so a partly resumed
claim is not shadowed by the stale source. The list is pruned once no readable
file carries an id.
Native windows tests hold the file open from a real second process.
(cherry picked from commit 7f5b98a0e10ce2f578ae576481b51dc8029f4e99)
(cherry picked from commit 4fba720f1e9879f93b9d17d4a5c6bb4980ffe2ae)
Checkpoint validated stop consumption before unlinking requests and reread
claimed tokens under custody. Reject stale and mismatched marker evidence,
and retain cold-start generations when orphan pauses are adopted.
Real-process regressions cover producer death, claim/consume ordering,
failed checkpoint writes, marker validation, and successive orphan adoption.
(cherry picked from commit 2f4fd37ebd61a2a6676763e62fa812acddb98a1b)
(cherry picked from commit 4e16ee29bbe0e681f6692006f2b44b95c4e08dc7)
mark_stop_requested recorded intent for every running gateway before the first
request; an updater killed in between left a never-asked, still-serving gateway
held as 'draining' forever, and restarted it after a later user stop.
Now the intent also names where each request goes (planned-stop marker) and the
stopper; mark_stop_sent records each request once issued (marker on disk, else a
positive socket ACK), before the socket call. A live entry is owed only when it
was recorded sent or this update's own marker still names it (a user's stop
marker is not evidence); the evidence is resolved once into stop_sent so a merge
or a hand-back carries it. Force-killed pids need no record (dead = owed).
(cherry picked from commit e199442e5690614065e3013f06ad54b18584811d)
(cherry picked from commit f2af22c6e8a45a7712046107b10c17fc9ad9d690)
- Every record/claim mutation decides read -> judge -> mutate under a kernel lock on a
never-deleted sidecar (.hermes-update-paused-gateways.lock, A7). A claim is published with the
claimer identity already inside (atomic replace), then its source is retired: no window where a
renamed claim still names the dead claimer.
- Stable obligation ids: a claim keeps its pause_id; an update folding orphaned sets lists their
ids under `absorbed`. A crash between publish and retire leaves a copy that readers skip and
the next mutator retires — a set is never resumed twice.
- Stop acknowledgment: the update records `stop_requested` right before its first stop request.
A live entry never asked is dropped (still serving); a live entry asked to stop is draining and
keeps its restart debt until it has exited (the claim goes back unowned meanwhile).
- One record per checkout (file name keyed by install root): another checkout sharing the home
never imports, relabels or certifies this checkout's paused set.
- Resume retires each runtime on its own readiness (profile home / unmapped argv), and a failed
SCM service or profile never keeps the independent ones stopped.
(cherry picked from commit 4301239b4ae702cf619cc90ddd3bb605159db94e)
(cherry picked from commit c54cb419ae943694883ab829f274b168772e13ca)
When the service pause fails and everything this run stopped is restarted
in-line, the record used to be discharged outright, dropping a set adopted
from an earlier killed update. Only this run's part is discharged now; the
adopted set goes back on disk unowned for the next launch.
(cherry picked from commit 8f82322877ed852b16739716c2712b4c7ec7530f)
(cherry picked from commit 41f91c428c79b11ef14f1a44694d0e861c07d656)
- A recovering launch killed mid-resume (taskkill, console close) left only its
.claim file, which nothing read: the paused gateways were never restarted.
Claims now carry the claimer's identity; a claim whose claimer is dead is
re-adopted like a dead owner's record. A launch that still owes part of the
set hands its claim back unowned so the next launch retries at once.
- Under a Desktop/Tauri-driven update the orchestrator holds the marker, so the
update saw its own partner as "another update", never adopted an orphaned
pause record and then overwrote it. The update tree holding the checkout lock
is no longer "elsewhere" (update_lock.update_in_progress for everyone else),
and write() merges another pause's orphaned set instead of overwriting it
(refuses when that pause's owner is live).
- Runtime-host detection (gateway run / serve / dashboard in the ancestry)
uses the canonical command-line matchers, so `--profile X gateway run` and
the hermes_cli/main.py path form are recognised; the rule sits in
_adopt_or_refuse's partner check.
- Whole-tree gate: at a moved HEAD only the interrupted-pull marker and current
dependencies count; a tracked file a build rewrote no longer keeps gateways
stopped forever. At the pre-update HEAD a new tracked change is still git's
torn checkout.
- _pause_windows_gateways_for_update split into record / cold-start / stop phases.
(cherry picked from commit a56eaa3a0f65c44f45600a9b2868859d98cc382e)
(cherry picked from commit a72664b05f5f789c0b238ee1fcd1e9e087851182)
The set of gateways/services `hermes update` stops on Windows lived only in the
updater's memory plus an atexit resume, so a killed updater (console closed,
taskkill, power loss) left them stopped with nothing on disk saying so, and the
atexit/foreground resume could restart them onto a half-written checkout or the
pre-sync dependencies.
- hermes_cli/update_pause_record.py: the paused set is written under the root
Hermes home BEFORE the first stop, owned by the updater (pid + creation time in
the update-marker ct format), mirrored after every resume attempt and removed
(compare-and-delete) only once the resume is done.
- Every in-process resume is gated on a whole tree: no interrupted-pull marker,
no tracked change beyond those present at pause time, and HEAD still at the
pre-update commit or dependencies current for the moved HEAD. Otherwise the
record stays for recovery.
- Start-of-run recovery (hermes_cli/main.py, beside the fleet-restart hint) claims
an orphaned record (dead owner, no live update) by rename and resumes it; a
later `hermes update` adopts an orphaned record into its own pause instead.
- update_lock: a claim is never adopted across a Hermes gateway/serve/dashboard
host in the process ancestry, so an update started from a gateway the first
update relaunched cannot run under that update's claim (cli §7 V9).
(cherry picked from commit fc0a9c3a53247a8779b5bd9f40b678a22a173521)
(cherry picked from commit 1c0292b5763ad82678309561da7a1eeec3fa0581)
