A time-based grace kept a just-abandoned lease 'held', so back-to-back `hermes pm repair` runs
never collected the generation the previous run execv'd out of (e2e test_generation_gc red).
Rename+replace is out (msvcrt byte locks refuse renames on Windows), so the writer now re-checks
its lease path after taking the flock and retakes when a peer's prune unlinked it in between; the
pruner only unlinks while holding the lock, so a visible path after the lock is always ours.
Review majors on #126179 (runtime_state._prune_unlocked_leases):
1. The boot-time prune opened every sibling lease with O_RDWR and caught
only FileNotFoundError, so one lease this user cannot open (a root-owned
0o600 file from `sudo hermes` on the same checkout, the #125525 class)
raised PermissionError inside activate_dependencies and hermes_bootstrap
exited 1 with "run hermes pm repair" on every launch. Any other OSError
now counts the lease as HELD: GC stays fail-closed, boot never aborts.
2. lease_directory creates (O_CREAT|O_EXCL) and flocks in two syscalls, and
pm/worker.py, pm/launch.py and pm/environments.py call it outside
runtime_lock, so a peer's prune in that window could take the
non-blocking lock and unlink the file; the first process then held a
lock on an unlinked inode and its pin was invisible for its lifetime
(collect_generations could rmtree a running generation). A lease younger
than LEASE_GRACE_SECONDS (60 s) is never a prune candidate and counts as
held. Chosen over a temp-name + os.replace scheme because it is three
lines, keeps a crash-between-create-and-flock leak collectable, and a
60 s margin dwarfs a microsecond window.
test_next_reader_removes_lease_left_by_hard_exit ages the hard-exited
lease past the grace window before asserting the next reader removes it;
its invariant is unchanged.
lease_directory releases its per-invocation lease file only through atexit,
which os.execv (hermes_bootstrap's re-exec into the managed environment) and
os._exit (gateway shutdown_watchdog._hard_exit, hermes_cli/main.py past
finalization) never run: a 4s supervisor restart loop left ~860 lease files an
hour in the SELECTED generation, which generation GC never visits.
The kernel lock dies with the process even when atexit does not, so the file
itself is the only leak: every reader (the next lease_directory and
leases_held) now unlinks lease files nobody holds a lock on. Both run under
runtime_lock at boot / in the collector, so a lease between O_CREAT|O_EXCL and
its lock is never pruned from under a live process.
An update resolves the enabled plugin union against the NEW core. A plugin
admitted against the old core can stop fitting when core moves (managed
Python 3.13 -> 3.14 vs a member's requires-python <3.14, a requires_hermes
upper bound, a bumped pin), and the whole update then died after the
source swap with a non-resolver InstallError whose 'retry' hint failed the
same way every time.
Update syncs now pass evict_incompatible_plugins=True (update completion,
historical takeover, launch-time completion, venv_sync, post-update
drift). PM screens statically first (requires-python vs the target
interpreter, manifest/requires_hermes), then, if the rest still fails,
builds core alone to prove the plugins are the cause and re-adds members
in config order, disabling each one that breaks the build. Misfits land in
plugins.disabled (memory.provider cleared) in every home that enables
them, published through the existing journaled change hook (the journal
now carries several configs), and are reported on stderr + receipt
warnings. Admission and ordinary syncs still refuse; only a core that
cannot build on its own fails an update.
pm imported runtime_state's private helpers (_lock, _atomic_bytes, _bytes,
_digest) at a dozen sites while runtime_state imports pm.environments at
module top. The primitives are pm's: move them into the stdlib-only
pm.filesystem as lock_fd, durable_write_bytes, read_bytes_or_none and
file_digest, and repoint every pm caller.
runtime_state keeps the private names only as import aliases: it still
calls them through its own globals, and pre-PM updaters load them by these
names mid-swap (tests/compat/old_updater_surface.json).
Boot-subset test fixtures now copy pm/filesystem.py, since runtime_state
imports it at process boot; worker-injection tests patch the name
pm.publication now reads.
pm-runtime/generations/<uuid> trees were never collected: a kill -9 during
staging orphaned a venv permanently and every input change left the old
runtime behind. `hermes pm gc` now sweeps them under .prepare.lock with the
same lease model as application generations: entry points (worker.py,
launch.py) pin their own runtime, prepare_runtime marks new generations
lease-managed, and the collector removes unpublished stages outright,
superseded generations only once no worker holds a lease, and never a
pre-lease generation.
When runtime_lock times out, activate_dependencies read the selection and
leased it without the lock. An installer committing a new generation in
between left the reader holding a lease on an unselected tree while
importing from it; that tree is exactly what `hermes pm gc` removes once
the marker is a day old. Re-read the selection after leasing and move the
lease when it changed.
lease_generation now returns a release callable that also unlinks its
lease file, so one zero-byte file per hermes invocation no longer
accumulates under .leases/.
hermes_cli.runtime_paths (venv generations, selection, activation) moves to
pm.environments, and gains venv_bin_dir / venv_python / project_python. Every
in-tree caller asks pm for an interpreter now; pm no longer reaches back into
hermes_cli for its own environment layout (pm.packages, pm.extras, pm.ensure,
pm.paths imported hermes_cli.runtime_paths). The three open-coded
"Scripts/python.exe or bin/python" ladders in pm collapse onto venv_python.
hermes_constants.venv_python_path / venv_bin_dir and hermes_cli.runtime_paths
stay as frozen-updater-surface shims only (tests/compat/old_updater_surface.json).
To keep the boot path light, pm/__init__ resolves its facade lazily (PEP 562)
and pm.registry loads the built-in package definitions on first read instead of
at import: `import hermes_bootstrap` now loads pm + pm.environments only (25ms,
was 37ms with the eager facade dragging in the downloader). The stripped-payload
fixtures that ship only pre-import files keep working for the same reason.
Also restores two frozen-surface re-exports the F401 sweep dropped
(banner._github_compare_behind, cua_backend.resolve_cua_driver_cmd).
Three ways a fresh install punished a second backend:
- `runtime_lock` waited forever, and its holder can be rebuilding the whole dependency
environment. `activate_dependencies` runs at every boot, so the second backend — the
onboarding profile — never bound. It now yields whether it holds the lock, and boot
proceeds without it: recovery is the holder's job, and the generation being leased is the
selected one, which the collector never removes. Explicit installs still pass
`timeout=None`; an opportunistic lazy install refuses instead of queueing behind a rebuild
it did not request. Same rule `boot_bootstrap._RecordLock` already states for home
maintenance.
- `stt-whisper` had no platform gate although its anchor `faster_whisper` cannot install on
win32/ARM64 (ctranslate2 ships no wheel) or darwin-x64, so the lazy install rebuilt the
whole environment and still failed the anchor on every retry. The extra is gated to match
its dependency markers; `voice` stays ungated because its sounddevice/numpy do install on
those targets.
- the first sync copies the payload's uv cache out to the machine cache. A copy that failed
still recorded `.seeded`, so the partial seed was permanent and every later offline sync
failed closed on the missing entries.
Validated: tests/pm/ and tests/hermes_cli/ for the touched modules. A/B on HEAD: the gate
test, the cache-seed test and both lock tests fail there, pass here.
Keep upstream's reviewed catalog as the only plugin name index.
Catalog pins and custom update sources share staged PM validation.
Publish code and dependencies with recovery after process death.
Reject a concurrent enablement change before publishing disabled code.
Use the manifest loader's supported version in the installer. Keep
probe cooldowns for timeouts, not TLS failures that a CA change fixes.
Preserve the backup, uninstall, browser and memory-provider repairs.
Verified with the canonical runner on native Windows ARM64, real Git
repositories, local TLS endpoints and UV dependency generations.
Desktop catalog tests and both TypeScript checks pass. The full suite
and native release builds were not run. No remote push.
Resolve dependency state and the plugin union from the active home without changing process environment. Use the existing home-root derivation for custom and named profiles. The journal validator checks the same root as the writer.
Verified 71 tests passed, 8 skipped across root resolution, union, recovery, and selection. The new context-only-home regression failed before the fix.
Dependency publication now recovers interrupted config/facts changes before
activation and leases live generations during collection. Receipts retain
update correlation and failed steps across nested command boundaries.
Doctor and desktop surfaces report those failures through shared owners.
Move checkout updates out of the desktop facade. Stage a detached Windows
relaunch waiter before shutdown, with bounded handshake and process-birth
checks. Keep packaged lifecycle tests isolated from the installed app.
Native verification exposed two production races: cron maintenance imported
the interactive CLI and rewrote TERMINAL_CWD, and install-ID reads collided
with first publication. Use the existing owners and locks. Plugin checks
now run at startup and each due-gated housekeeping tick, not after 60 ticks.
Share updater-test mutation boundaries and remove collection-root fixtures.
Separate cold MCP startup from command latency and give the real HTTP drip
test enough time to reach body handling.
Root npm check passed, including packaging. The fixed-tree Windows Python
run reported 44557 passed, one failed, and 1404 skipped, plus one retry-only
HTTP test. Those final failures now pass in a 35-test bounded batch. A real
isolated gateway wrote startup and periodic plugin-check receipts.
Full final-tree CI, bundled Sandbox deployment, and actual App Installer
relaunch remain unverified. docs/pm-audit-status.md records these limits.
