worker_bootstrap() logged a failed activate_dependencies() and let the worker
carry on through the inherited PYTHONPATH pin. That path is not leased: once
the gateway releases its generation the collector deletes it under the live
worker and the next dependency import dies mid-job (#122936 review, reproduced
with a committed generation that has no site-packages).
The failure now propagates, so the worker exits before its ownership ack and
_launch_external_cron_worker reports the dispatch failure with the worker's
stderr. PM's own no-op cases (venv, wheel, Nix, nothing committed under a venv)
still return normally.
Tests drive the real PM boot in a marked child: it imports a sentinel that
exists only in the committed generation, and when an update commits a newer
one mid-job the real collector keeps the worker's generation until the worker
exits. A damaged selection exits before cron.jobs is imported.
123456hyu made the same prelude move and fail-closed change on #122290
(dff16e12); this carries them without the argv gate and source-order test.
Co-authored-by: 123456hyu <123456hyu@users.noreply.github.com>
- Rename the boot marker HERMES_CRON_EXTERNAL_WORKER to _HERMES_CRON_WORKER_BOOT.
The old name differed by one leading underscore from
_HERMES_CRON_EXTERNAL_WORKER, the owning execution id read in the same worker.
- The subprocess probe now gets the marker name from WORKER_MARKER instead of
a literal, so a future rename cannot silently skip the consumed-marker check.
- Resolve the repo root inside worker_bootstrap(): only the marked worker uses it.
- Correct the docs to match the boot order: the boot re-selects and leases the
generation before any third-party import, and the PYTHONPATH pin is the
fallback for a failed boot. They are not two load-bearing halves, and the pin
does not resolve modules exactly as the gateway did.
`-m cron.scheduler` runs `cron/__init__.py` before the module itself. That
file's first import, `cron.jobs` -> `hermes_yaml` -> `ruamel`, is already a
third-party dependency. With the boot at the top of `cron/scheduler.py`, the
worker imported through the pinned PYTHONPATH alone:
- no lease on the generation was held yet;
- no `.pth` files had run;
- with the dependency pin removed, it still died with
"No module named 'ruamel'".
Run the marked-worker boot at the top of `cron/__init__.py`, under a private
alias, so the package attribute `cron.worker_bootstrap` stays the module. Drop
the now-redundant call in `scheduler.py`. Replace the in-process marker test
with a subprocess test: the boot runs before `cron.jobs` is imported, runs
exactly once, consumes the marker, and never fires for an unmarked importer.
Also add the missing trailing newlines on the two new files.
The restart-safe cron worker is spawned as `sys.executable -m cron.scheduler`, a Hermes
entry point that never goes through `hermes_bootstrap`. On a self-managed (PM) install its
interpreter is the store Python, which owns no dependencies and whose PYTHONPATH the
subprocess sanitizer has just stripped, so the worker died at its first dependency import
(`No module named 'ruamel'`, #122222) before it could publish its ownership
acknowledgement -- every scheduled job landed `last_status: failed`.
Salvaged from #122290 (thanks @123456hyu), whose two commits pinned the dependency
site-packages and leased the generation at the worker entry. Reworked onto PM's own
machinery:
* `pin_hermes_tree_on_pythonpath` asks `pm.environments.committed_venv` +
`site_packages()` for the generation, instead of deriving one from this process's
`sys.path` behind a `pyvenv.cfg` ancestor probe. The record is scoped to the active
home, so the child-spawn path no longer reads the real home -- that probe tripped
`tests/home_io_guard.py` when the suite runs from a self-managed install -- and the
uv-launcher, first-match-only and ordering edge cases of the old derivation go with it.
* `cron/worker_bootstrap.py` is now the PM call the worker actually needs: lease +
`site.addsitedir` + `sys.path`/`PYTHONPATH`/`PATH` are all
`pm.environments.activate_dependencies`, the same boot `hermes_bootstrap` runs, and it
holds the generation's lease for the process's life. It is gated on the
`HERMES_CRON_EXTERNAL_WORKER` marker the spawn site sets (consumed on read) rather than
sniffing `--external-worker-file` out of `sys.argv`, so the gateway keeps its launch
contract without argv identity inference.
Verified live on PM's store Python against a real generation (A/B, same child): tree-only
PYTHONPATH -> `ModuleNotFoundError: No module named 'ruamel'`, rc=1; pinned env -> imports,
`running_from_selected_environment` True, and the worker holds a real kernel lease while it
runs (released on exit). The real `-m cron.scheduler --external-worker-file ...` entry now
reaches its ownership handoff instead of dying at the import. `tests/cron/`: 1376 passed.
Tests: the two invariants that carry the contract -- the pin restores the committed
generation (and invents nothing without one), and the boot runs only in the marked worker,
once. The six construction tests that drove the old derivation are dropped with it.
Duplicate-cluster credit: #122238 (@webtecnica, earliest), #122382 (@Sakujakira), #122685
(@etrisnanto), #122719 (@aglowinthefield) and #122738 (@GabboAlarcon) diagnosed and fixed
this same worker-dependency failure in parallel while this branch was rebuilt. None of their
diffs is contained here -- the co-author trailers below record direction and duplicate-cluster
credit, not lifted code; their roles are listed in the PR body.
Co-authored-by: webtecnica <75556242+webtecnica@users.noreply.github.com>
Co-authored-by: Manuel S. <3703103+Sakujakira@users.noreply.github.com>
Co-authored-by: Eko Trisnanto <4469382+etrisnanto@users.noreply.github.com>
Co-authored-by: aglowinthefield <146008217+aglowinthefield@users.noreply.github.com>
Co-authored-by: GabboAlarcon <40816412+GabboAlarcon@users.noreply.github.com>
(cherry picked from commit 7c55715c0478280adf4e492b24edf331342470f7)
The pin in scheduler_worker_env.py restores the site-packages *directory* into the
child's PYTHONPATH, but a directory name is not a retention permit. The PM collector
removes an unselected, lease-managed generation once it is old enough and no process
holds its lease, so between the gateway's exit and the worker's next import the very
generation the worker needs can be collected away.
New cron/worker_bootstrap.py closes that gap at the worker entry, before the first
application/dependency import:
- lease_dependencies(): acquire the committed generation's kernel lease (the same
hermes_cli.runtime_state.lease_generation primitive activate_dependencies uses),
so the collector's 'no process holds the lease' safety condition keeps the
generation alive for the worker's whole lifetime. Unconditional and idempotent.
- activate_worker_dependencies(): delegate to pm.environments.activate_dependencies
(the same resolution + site.addsitedir + sys.path reordering the gateway's boot
uses) so the worker's own imports resolve without waiting for the gateway's
handoff. Gated on --external-worker-file in argv, so the in-process gateway path
keeps its exact launch contract and the runner-owns-its-own-deps case stays a
no-op.
Hooked into the top of cron/scheduler.py, after the stdlib-only sys.path pin and
before any Hermes package import, so hermes_yaml -> ruamel.yaml succeeds. Both
degrade to a logged no-op when no dependency generation is committed.
Tests: test_worker_bootstrap_lease_and_activation, _leases_but_skips_activation_in_gateway,
_no_committed_venv_is_noop, and a scheduler-import smoke test. The two home_io_guard
failures in this suite are pre-existing on a clean checkout and unrelated.
Fixes the P2 lifetime finding in @andrexibiza's review of #122290.
(cherry picked from commit 429f0dd191dbf719d05cbbbe3cda6546893b97d0)
