The early before_build resume ran for Windows sets too. The fleet restart skips only what the
POSIX pause restarted (already_restarted), so a Windows gateway resumed before the build was
then drained and restarted a second time by the fleet pass: one extra outage per update that
main never had. The early resume is now POSIX-only and goes through the main facade like every
other resume call; Windows resumes where it always did. Docs say so.
Arm the source-completion tail and the fleet obligation in the parent before
any tree move; cover the CP0 branch switch, the upstream fork ff and the syntax
rollback with the interrupted-pull marker; settle (never drop) the marker on a
non-kill git failure; refuse foreign-owned venvs and uncompilable targets before
HEAD moves; push the fork only after a validated update; ZIP fallback only from
an untouched checkout; journal the ZIP swap so start-of-run recovery finishes or
rolls it back and no staging/backup sibling wedges the next run.
(cherry picked from commit 584acad44f3d6c552b7c2767d12c612de0436498)
(cherry picked from commit c6567ac2d28c421f384d95e1a6fc28912cb30f86)
What: update_completion._prepare on win32 joins the checkout lock first (update_lock._acquire_checkout,
the joiner path that takes an R5b lease byte, exactly what the --prepared child does through
complete_source_checkout's UpdateLock). It holds that lease for its lifetime (through the --prepared
child); a refused join raises before arm_completion, so main() fails the preparation boundary.
Why: a bootstrap child whose job bind was refused runs outside the kill-on-close job and outlives a
killed owner, yet it ran arm_completion, pm.sync_venv (uv children) and
collect_superseded_generations holding no lease; the lease was only taken later in the --prepared
grandchild, so the checkout could be handed to a contender while the sync still wrote (review L3).
Tests (new file, RED on bbfa23b137d, GREEN here; Windows faked for update_completion only): the
lease is the first event, before arm_completion/sync/collector/--prepared; a refused join writes
nothing.
The six findings are deliberate boundaries, so each gets an explicit
`# health: allow` with its reason instead of a narrowed catch:
- _old_updater._run_in_custody / update_completion._prepare (HX006): the
waited child is the rest of the committed update (sync + build); it is
supervised and dies with the updater, a timeout would abandon it mid-write.
- _old_updater._checkout_custody (BLE001): calls whatever older release's
update_lock is loaded; no fds means the takeover takes the lock itself.
- update_custody._past_commit (BLE001): fail-closed, unknown = past commit.
- update_custody._held (BLE001): a torn tree's half-written module can raise
anything at import.
- update_lock.process_create_time (BLE001): liveness probe; any psutil failure
(including partial/stub modules) degrades to the stdlib probe instead of
escaping to read_live_update.
- update_lock.read_live_update (BLE001): documented never-raises probe.
Existing `# noqa: BLE001` comments become the health waiver (ruff does not
select BLE here). Comments only; behaviour is unchanged.
What: every Windows process that joins its ancestor's checkout lock also locks one of 16 lease bytes
past the owner's byte, taken while the owner is seen alive; _try_lock/checkout_lock_held and the Tauri
marker probe treat any held lease as a held lock. The refused completion child still runs (D2: a
committed update never fails) but is no longer unfenced.
Why: a child outside the kill-on-close job outlived a killed owner, whose msvcrt lock died with it,
so a contender acquired while the child still wrote.
Red/green: Windows-native cells (Python owner-killed/unbound child; Rust held lease) need a native run;
POSIX paths unchanged (update_lock suites pass, same failure set as base).
CustodyRefused is an OSError, and many readers swallow one (_git_stdout, _capture_head_sha),
so on a refusing host the first refusal could end the update as a misleading downstream
error. update_custody now keeps the run's first refusal (and whether any update child had
run in custody, or the update was past its apply stage), and the command's failure path
(cmd_update; the completion child's _finish) prints it in place of the generic error:
it names `hermes update`, says nothing was changed when no update step had run, and how to
retry. A refused bind and a job that cannot be handed to a build launcher are now receipted
like the refused launcher join; the resume moved to update_lock.resume_suspended_child.
Tests: Linux (the cmd_update failure path after a swallowed refusal, receipt step) and a
Windows live cell (the real readers swallow a real refused job assignment).
The Windows completion child (dependency sync, product builds) was started running and
bound afterwards, so a fast grandchild could escape the update's kill-on-close job, and
a refused bind was only a log line. It is now created suspended, bound, then resumed.
Post-commit a refused bind still runs it (ruling: refusing would fail a committed
update), but never silently: the refusal is printed and recorded as a failed
update_custody step in this run's receipt and in the receipt the child resumes (its
request is rewritten before it runs), so the terminal receipt carries it.
Windows live cell: the real AssignProcessToJobObject refuses an event handle; the
child must not have run at bind time and the step must reach the child's receipt.
The kill-on-close job that binds the completion child to `hermes update` allowed no breakaway,
so the gateways the completion restarts or resumes (gateway_windows._spawn_detached, which asks
for CREATE_BREAKAWAY_FROM_JOB) fell back into the job and were killed the moment `hermes update`
exited, on success too. Allow BREAKAWAY_OK (not SILENT: ordinary descendants stay bound).
A job that cannot be set up no longer raises out of run_completion after the commit point: it is
logged and the completion runs unbound; the bind sits inside the try whose cleanup kills the
child, so nothing can orphan it.
Windows cell: a real install, a real gateway, a real `hermes update`; the restarted gateway must
still be alive 30 s after the updater exited.
hermes_cli/update_lock.py implements update-marker contract C1 and the checkout lock (C1.7):
- the marker lives in the profile-tree ROOT home, never profiles/<p> (desktop V9c)
- v2 body pid / started_at / ct:<creation time>; an owner is live while its pid is
alive and its creation time matches (2.0 s) — no age ceiling for v2, the 20-min
ceiling only ages out v1 markers (cli V10/V11, desktop V3, V22)
- O_EXCL claim with one compare-and-delete reclaim of a dead owner; every delete
is compare-and-delete; never truncate-and-write over a marker (desktop V4)
- adopting a hand-off partner's claim writes a line-4 delegate and restores
lines 1-3 byte-identical (or deletes when the partner is dead) on exit
- `hermes update` holds <install root>/.hermes-update.lock (flock / msvcrt) so two
updates of one checkout from different homes exclude each other (cli V3); completion
children inherit the locked fd (pass_fds) on POSIX and run in a kill-on-close job on
Windows, so the lock is never free while the update tree runs
- an unwritable marker dir or install root refuses via exit 2 instead of running unlocked
- process_create_time: psutil when importable, a stdlib probe of the same clock otherwise
- update_in_progress(install_root) for readers: live marker or held checkout lock
Smallest hunks outside the lane: main.py passes install_root=PROJECT_ROOT;
update_completion.py passes the lock fd to both completion children.
settle_lost_completion (the completion child died without a result: OOM,
SIGKILL) recorded the owed follow-up and finalized from the parent's
pre-child context. That snapshot predates everything the child persisted
on the run's archive, so _persist_running wrote it over the child's
running record and the terminal receipt lost the stages that actually
ran (deps, build). finalize_interrupted_update_receipt and
_settle_after_commit already continue from the on-disk record.
update_receipt.resume_run_record is now that one step: it adopts the
run's archive when another process carried it further (and drops the
open receipt when the archive is already terminal). Both the interrupt
path and the lost-completion path use it.
The run's own archive was read five ways: _read_terminal_receipt,
_running_record and amend_terminal_followup (each globbing by update_id
with different error handling), plus the _persist_running stale-snapshot
guard and finalize_interrupted_update_receipt (each re-reading the
_run_file path inline). update_receipt.read_run_record is now the only
reader: unreadable, non-object and other-run files all mean "no record",
it never raises, and each caller keeps its outcome on top of that.
The divergence was a bug in one place: _read_terminal_receipt and
_running_record called .get() on whatever json.loads returned, so a run
archive holding valid JSON that is not an object (null, a list) raised
AttributeError out of settle_lost_completion -- a post-commit path that
must never fail the update. It now settles like a missing archive.
"record the follow-up, then amend the closed receipt" was hand-written at six
sites (three Windows-resume sites, the historical-takeover resume,
gateway_restart, channel_adoption). Two of them -- _complete_source_update's
Windows resume and channel_adoption -- amended without checking that the
run's receipt was already closed. On an open run record_followup has just
persisted the follow-up into the running record, so the amend appended a
second copy to that durable record and to latest.json (what an interrupted
run leaves behind).
update_receipt.owe_followup(update_id, step, reason) records the step and
amends the archive only once the run is closed; every site now calls it.
89e06e62dbe gave the bootstrap completion child -X utf8; the prepared
child (_prepare -> --prepared, which runs _complete_selected and prints the
✓/⚠ follow-up protocol into the same pipe) still started with -I and no
UTF-8 mode, so it relied on hermes_cli's import-time stdio repair alone.
Both children now carry -X utf8; a test captures both argvs.
Review: kshitijk4poor 5411222454 open thread "UTF-8 under -I".
After the run's receipt closed as `success`, three boundaries could still
write 1 to .update_exit_code (and the completion child could still answer
130), so the gateway watcher reported a failed update the receipt calls a
success:
- cmd_update's catch-alls (main.py, unchanged) call
_write_gateway_update_exit_code(False) for any escaping exception. The
writer now keeps 0 when this command's run is closed as a success
(update_receipt.committed_success: the receipt scope records the terminal
outcome it finalized, or adopted from the completion child).
- The parent adopts the child's terminal receipt and drops its stale
pre-child context as soon as the result is accepted, before the parent-
side Windows resume, so nothing after it can write that context.
- update_completion._finish: an interrupt that lands after verification
finalized the run answers 0 (the receipt's verdict); the 1 for a parked
stash (partial) or an interrupt before the run closed is unchanged, and
the status is written once the code is settled.
- update_finish (historical takeover): the same rule for its finally.
An interrupted run (receipt `interrupted`, exit 130) still writes 1, as
test_interrupt_after_child_success_demotes_gateway_marker_at_boundary pins.
Review: kshitijk4poor 5411222454 regression 3 / open thread F4.
After the tree moved, an OSError from run_completion's TemporaryDirectory,
_write_json or Popen (or anything the spawn path raises) reached
main.cmd_update as "Update failed": receipt failed, .update_exit_code 1,
exit 1. A completion child killed by OOM/SIGKILL, or one that exited
without a correlated terminal result, returned `code or 1` and exited
non-zero the same way. Both contradict invariant 3: the code is
committed, only the tail is owed.
_complete_source_update now settles both through
update_completion.settle_lost_completion (pre-swap module, everything it
uses is already imported in the parent): it arms the source-update tail
for the next launch (the bootstrap child may never have run to arm it),
names the owed Desktop build (P4), records a `completion` follow-up and
finalizes the run success-with-follow-ups, publishes the gateway
watcher's 0, and returns exit 0 -- so the parent also resumes paused
Windows gateways the lost child never resumed. Meanings kept: Ctrl-C
stays 130 (a 130 without a result finalizes `interrupted`, not
`failed`), parked local changes stay `partial` / exit 1 (#122557), and a
child that finalized then died keeps its own terminal record.
run_completion no longer prints "✗ ... did not finish"; it hands the
reason to the caller.
Test change: test_missing_child_result_fails_boundary_receipt_and_releases_lock
pinned exit `code or 1`, receipt `failed`, .update_exit_code 1 and
resume_needed still True -- exactly the outcome invariant 3 forbids. It is
now test_missing_child_result_is_an_owed_completion_and_releases_lock
(adds a SIGKILL case): exit 0, receipt success with a `completion`
follow-up, watcher 0, gateways resumed, lock released. New: spawn
OSError is owed; a lost completion with parked changes stays partial/1.
Review P2.
The only "Desktop app build owed:" line came from source_build when the
Desktop build itself failed. A committed run whose build never ran -- the
dependency sync failed (A6), the source-update tail was refused by the
update lock, the prepared completion child died -- printed no such line,
so posix.sh / windows.ps1 reported a plain "Update complete." while the
app was stale (F6 residual).
update_completion now owns the prefix (DESKTOP_BUILD_OWED, same text the
scripts match) and _report_desktop_build_owed prints the whole line
whenever the run requested the Desktop build and the run's record has no
`build` stage (the build never reached its own verdict, where the builder
prints the line itself): in _settle_after_commit (deps owed / prepared
child died), in _complete_selected when the tail raised, and in _finish
when the tail raised before the build stage.
Review P4.
_settle_after_commit wrote <home>/.update_exit_code unguarded. An OSError
there (a home that is a file, a read-only mount, ENOSPC) escaped before
_write_bootstrap_result, so the parent saw no result and reported
"Source update completion did not finish" + exit 1 for a run whose code
was committed and whose receipt already said success.
The write now goes through _publish_gateway_success, guarded exactly like
update_cmd_fleet._write_gateway_update_exit_code (which this stdlib-only
bootstrap cannot import): it warns and the result is still written, so
the run answers 0. The gateway watcher falls back to its own timeout.
Review P1 (invariant 3).
main's code-health ratchet (4156715960) ignores `# noqa: BLE001`, and ruff does not
select BLE here, so those markers waived nothing. Each catch-all is the C3 boundary:
after the commit point the failure is a receipt follow-up, never the update's status.
The stated reason moves into `# health: allow BLE001 -- <why>` unchanged.
The parent already decodes the child's piped output as UTF-8; -X utf8 makes the child encode it
that way on any console code page. The reviewed failure was invalid on re-verification; this is
the one-token hardening the triage allowed. test_update_completion_process.py failure set unchanged.
- A receipt store that refuses the terminal write after the commit point no
longer selects exit 1: update_receipt keeps the correlated terminal record
this process finalized (finalized_receipt) and the completion child answers
its parent from it; run_completion's correlation gate is unchanged.
- Restart failures (failed systemd unit, abort recovery, Windows service resume)
no longer write 1 to the gateway /update marker; the debt stays the
gateway_restart follow-up, the host obligation and the warning lines.
- TAIL_FOLLOWUPS includes profile_sync, so an owed profile sync keeps
source-completion-pending armed and withholds the install stamp.
Both completion paths marked the build stage failed whenever ANY tail follow-up
was owed, so a run whose only debt was config_migration (or maintenance /
dependencies) reported a failed build. record_build_stage marks it failed only
for launchers/build (what the build stage publishes) or completion (the tail
raised before proving them); other owed steps stay their own follow-ups.
Round 5 (R8, R14) on the post-commit completion:
- update_completion: the completion's runtime-safety verdict reaches fleet
verification, so an unsafe SQLite runtime still vetoes gateway multiplex
auto-migration; the committed update keeps exit 0 + the sqlite_runtime follow-up.
- update_cmd_config: a failed config-format write raises into the owed-step
guard: config_migration lands on the receipt and the tail stays pending.
- update_receipt: a handed-off receipt from an older interpreter (no
carried_manual_serves) snapshots the previous receipt's manual-serve debt
before the running record replaces latest.json.
- update_finish (historical takeover child): a post-commit build failure is a
build follow-up, maintenance and the fleet restart still run, exit 0, tail
stays pending until it finishes.
Contract C3 keeps post-commit step failures at exit 0, but the parked stash is not a
step anything retries: the receipt records a user_action, finalizes partial and
hermes update exits 1 (#122557). The Desktop handoff reports an owed app rebuild as
an action ("could not be rebuilt ... hermes desktop --force-build"), and the pm e2e
follows A6 (dependency sync after the tree moved is an owed follow-up).
The completion bootstrap's PM step (ensure_tools_for_sync / pm.sync_venv) runs
after the checkout moved. Its failure ended in exit 1 with no receipt, which the
Desktop reads as "still on the previous version" while the tree is new.
- update_completion._bootstrap: a failure there is now settled as a post-commit
follow-up: the tail obligation stays armed, the run's receipt is a success
naming a `dependencies` follow-up ("dependencies not installed yet — the next
launch retries"), exit 0 (.update_exit_code 0 in gateway mode). A prepared
child that dies without a result settles the same way (`completion`).
- The parent resumes paused Windows gateways itself when the bootstrap could
not (it has no dependencies to do so).
- Ctrl-C after the commit point (parent in run_completion, or the child) closes
the run as `interrupted` (never `failed`), exit 130, and prints that the new
code is in place and its remaining steps are owed.
- TAIL_FOLLOWUPS gains `dependencies`.
Restores the dropped zip-completion "preparation" case with the new expectation.
A Desktop built before the blobless fix keeps walking history during
`hermes update`. On a treeless (tree:0) checkout every walk downloads trees
again, and #129514 filled a 434 GB disk during the minutes of dependency work.
The one-time conversion (#133016) ran at the very end of the update, after that
window. Both update hand-offs (update_completion._prepare and an older
updater's _update_takeover.prepare) now run it first, in the bootstrap
interpreter, and the late call is gone.
Re-running install.sh / install.ps1 over a treeless checkout now converts it
too. The filter is recorded only after the refetch succeeds, so a failure is
retried by the next update instead of leaving a blob:none filter over a
history that has no trees.
The pack cleanup also sweeps tmp_* transfer files that killed fetches left
behind. They are more than an hour old, a storm leaves gigabytes of them,
and git itself only prunes them after two weeks. The docs gain the recovery
steps for a disk that the storm already filled.
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
Co-authored-by: Yuan Li <dskwelmcy@163.com>
The extra interpreter spawn ran on every platform before run_completion's
Popen. On Windows it went through subprocess.run first, which the taskkill
interrupt tests patch to assert only the completion child exists, so all
three failed. install_before_lock is a no-op off Linux anyway; skip the
spawn there.
run_completion runs in the OLD process after the checkout swap, and its
contract is never to import application helpers from the replacement tree.
Importing pm.libatomic there crashed the update when the new tree lacked
the module (downgrade or older pin) and on architectures where
current_target() raises. Run the pre-install as its own process from the
new tree, ignore its outcome, and let the in-lock repair report.
Also: gate on an openable /dev/tty first (cheapest check, and the detached
child exits immediately), treat an unsupported architecture as "skip",
bound the sudo -n probe, and drop the timeout on the interactive install
so a slow prompt or mirror is never killed mid-dpkg (it runs outside PM's
lock). Found in the final-head review of #132664.
hermes update's completion runs PM in a start_new_session child with no
controlling terminal, so the in-child sudo -v warm-up could never read a
password and the cold password-sudo update from #124926 still failed. A
cached ticket was also the wrong currency: under sudo's default per-tty
timestamps a ticket minted in one session is not visible to another.
Install the package itself, interactively, at the last point that still
owns the terminal: run_completion() before it detaches, plus the existing
ensure_tools_for_sync() entry for the other update paths. Gate on an
openable /dev/tty instead of isatty() (a detached child inherits a tty
stdin and a forwarded-isatty stdout). Inside PM's lock the repair is
unchanged: sudo -n, stdin closed. Reported by @JoaoMarcos44 on #132664.
The update pipeline already writes one machine-readable receipt per run, so the
run/stage counters are derived from it in the process that finalizes it instead
of instrumenting stages for metrics. The receipt gains minimal stage END marks
(plan, snapshot, apply[mode], deps, build, restart; verify is inferred from the
fleet matrix), an initiator fact (desktop when a live orchestrator claim
names another pid) and the pre-update commit date, which is all the derivation
needs for outcome, failed stage, duration and from-version-age buckets.
A pre-pull interpreter must never import pulled code: when it is the finalizer
(same pid, checkout sha moved) it parks the receipt (stdlib only) and the next
Hermes start records it. Collection off => nothing imported beyond a config read.
The completion-process test stub gains the new record_stage API the real
module now exposes (no assertion changed).
An update resolves the enabled plugin union against the NEW core. A plugin
admitted against the old core can stop fitting when core moves (managed
Python 3.13 -> 3.14 vs a member's requires-python <3.14, a requires_hermes
upper bound, a bumped pin), and the whole update then died after the
source swap with a non-resolver InstallError whose 'retry' hint failed the
same way every time.
Update syncs now pass evict_incompatible_plugins=True (update completion,
historical takeover, launch-time completion, venv_sync, post-update
drift). PM screens statically first (requires-python vs the target
interpreter, manifest/requires_hermes), then, if the rest still fails,
builds core alone to prove the plugins are the cause and re-adds members
in config order, disabling each one that breaks the build. Misfits land in
plugins.disabled (memory.provider cleared) in every home that enables
them, published through the existing journaled change hook (the journal
now carries several configs), and are reported on stderr + receipt
warnings. Admission and ordinary syncs still refuse; only a core that
cannot build on its own fails an update.
A normal `hermes update` only re-synced the venv. The sync pulls uv/python
in through its own dependency, but a ripgrep/ffmpeg/node/npm pin bump in
pm/lock.json was never installed, so PATH activation warned and skipped the
managed tool dirs on every CLI start and gateway boot. Only the takeover
route for historical releases ensured the tool roots.
Move the takeover's loop into pm.client.ensure_tools_for_sync() and call it
from both routes before the sync: update_completion._prepare (the CLI and
Desktop route, running from the new tree so the new lockfile applies) and
_update_takeover.prepare. It uses explicit=True like the takeover (an update
is an explicit user action) and a failed download fails the update.
post_update.step_provision_runtimes / MACHINE_STEPS stay: `python -m
hermes_cli.post_update --scope machine` and tests still reference them.
The ffmpeg docstring no longer claims that step re-ensures it.
Every update route now finishes through update_completion._complete_selected,
which restarted the whole fleet unconditionally -- including the "Already up to
date" route that main sent through the pending-restart catch-up. Net effect:
each cron tick and each profile's `hermes update` drained and re-killed the one
multiplexed gateway.
Port the catch-up path's two live guards into the completion tail:
- host_restart_already_completed(checkout sha): a sibling profile attaches to
the restart this host already stamped (#95294); the restart phase now stamps
it via mark_host_restart_completed.
- every planned runtime AND every live fleet row current at the checkout sha
(#117051, d6b0d37ece). Both are required: the live matrix lists gateways
only, so a planned serve still on pre-update code keeps the restart.
The already-current route also arms the host obligation with the checkout sha;
an SHA-less arm replaced the standing record and wiped the restarted proof.
hermes_cli.runtime_paths (venv generations, selection, activation) moves to
pm.environments, and gains venv_bin_dir / venv_python / project_python. Every
in-tree caller asks pm for an interpreter now; pm no longer reaches back into
hermes_cli for its own environment layout (pm.packages, pm.extras, pm.ensure,
pm.paths imported hermes_cli.runtime_paths). The three open-coded
"Scripts/python.exe or bin/python" ladders in pm collapse onto venv_python.
hermes_constants.venv_python_path / venv_bin_dir and hermes_cli.runtime_paths
stay as frozen-updater-surface shims only (tests/compat/old_updater_surface.json).
To keep the boot path light, pm/__init__ resolves its facade lazily (PEP 562)
and pm.registry loads the built-in package definitions on first read instead of
at import: `import hermes_bootstrap` now loads pm + pm.environments only (25ms,
was 37ms with the eager facade dragging in the downloader). The stripped-payload
fixtures that ship only pre-import files keep working for the same reason.
Also restores two frozen-surface re-exports the F401 sweep dropped
(banner._github_compare_behind, cua_backend.resolve_cua_driver_cmd).
The installer ladder stopped at node-deps/path/desktop with its own
semantics while an update ran launchers, product builds and post-build
maintenance, so a fresh install and a finished update ended in different
states: after re-running the installer at HEAD the products had no receipts
and the read-only source acceptance failed.
hermes_cli/source_completion.py now owns that tail -- publish launchers,
build the products, run the maintenance -- and update_completion's
_complete_selected calls it, so there is one implementation. install.sh and
install.ps1 keep the bootstrap stages (prerequisites, repository, venv,
python-deps, config) and hand off to it in a single `products` stage;
--include-desktop selects the desktop product inside that stage instead of
adding a second build stage, and `desktop` stays dispatchable via --stage for
external callers.
Windows keeps its installer-owned PATH publication (expose_cli answers
"windows-installer-owned" on Windows) plus the packaged-artifact probe, ACL
grant and shortcuts. The desktop stage no longer pre-syncs wake/voice: pm
lazy-installs them at first use, as the update path does.
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
