#135348 ran ruff's UP037 autofix with ruff.strict.toml's target-version = "py314". On 3.14
annotations are lazy (PEP 649), so stripping quotes is safe there; on 3.11-3.13 a function or
variable annotation is evaluated when the def or statement runs. Every unquoted annotation that
names the enclosing class (`def clone_for_agent(self) -> ContextEngine`), a class bound later in
the module, or a TYPE_CHECKING-only import now raises NameError on import: 300 modules on main
(gateway.config, run_agent, hermes_state, cron.jobs, the Telegram adapter, ...) fail to import
under 3.11, the update path crashes mid-`hermes update` on any install whose interpreter is
older than 3.14, and the e2e-upgrade suite's retry leg fails with
`NameError: name 'ContextEngine' is not defined`.
- Re-quote exactly the 79 annotations (36 files) that name something unbound when they are
evaluated (including tui_gateway/server.py's `_TurnScopes`, which only reaches that module's
globals through register() after import); no other text changes.
- agent/turn_scripted_prelude.py: `Generator[...]` gets its third argument back. d76cce4f11
applied UP043 (default type args, valid only from 3.13) to a module-level alias, so
`Generator[Y, S]` raises TypeError on 3.11/3.12 and takes gateway.run, agent.conversation_loop
and tui_gateway.{server,ws,entry} down with it.
- ruff.strict.toml target-version = "py311", the requires-python floor. With it UP037 leaves
these quotes alone and only strips the two that are genuinely redundant, which this commit
strips; any future version-gated fix stays valid on every supported Python.
- tests/ci/test_strict_ruff_target.py pins the target to the requires-python floor (red on main).
The #115692 reclaim released any running attempt whose claim was older than
max(3 x HERMES_CRON_TIMEOUT, script timeout, 2 h), even when the owner was
healthy and busy. Every cron job that legitimately ran longer than that was
marked `unknown` mid-run, its fire claim yanked from under it and the run
aborted with "lost its durable fire claim ownership" (two weekly jobs on one
install in a single day).
The run monitor now stamps `progress_at` on the execution row every 60 s
while the agent reports recent activity, and the sweep measures the stale
bound from that stamp (claim age for legacy rows). A wedged worker stops
stamping and is still reclaimed once the bound passes.
Shape: the stamper and the inactivity watchdog loop (which reads the same
idle signal) live in a new topical sibling cron/scheduler_liveness.py, so
cron/scheduler.py shrinks (4457 -> 4436 lines) instead of growing.
Two profiles whose cron/ resolves to one store share a store key; when one left, its departure dropped the degraded record the remaining profile still needed (gauge flipped to writable, a second outage notice, and the earlier outage start lost for one-shot cover). Departed stores now exclude the ones still ticked. The notices test pins it (red without the set difference).
forget_homes rebuilt each store path from a departed home KEY
(Path(key) / "cron"). cron/AGENTS.md forbids rebuilding a home from a key
(hermes_home_key normcases), and the realpath only follows a symlinked cron/
while the home still exists: a profile whose home was deleted kept its degraded
record, holding the host-wide hermes.cron.store.writable gauge at 0.
register_ticked_homes now records each home's resolved store (new public
store_health.store_key) while the home exists and hands the departed stores to
forget_homes. gateway/cron_store_notices.py uses the same store_key against
record.store (already resolved), so one normaliser decides "is this the served
home's store" everywhere.
Gate r1 (PM12, 2c + quality + 2ab toothless) Low finding.
clear_run_claim (the dispatch-failure path: interpreter shutdown, execution
creation failure, pool.submit failure) set run_claim to None. The claiming scan
had already ended the recovery window, and a restart drops the in-memory outage
records, so the next scan saw a past-grace one-shot with no claim and retired it
as missed: the job the outage skipped never fired.
Keep {"outage": True} instead. With no "at" it is a stale claim, so the job is
re-dispatched on the next tick, and _retire_expired_oneshot keeps it. At-most-once
is unchanged: the fire_claim CAS still fences every re-dispatch.
Gate r1 (PM12, 2c) Low finding.
Follow-up to #133283 (greptile review). Store records are keyed by the
resolved store path, but the notice sender rebuilt the owning profile from
that path's parent. A profile whose cron/ is a symlink to another disk has
a foreign parent, so neither its outage nor recovery notice reached any
home channel; forget_homes had the same mismatch and never dropped such a
departed profile's record.
Both now match by resolving each served home's own cron/ path.
Follow-up to #133283 (greptile review). When only cron/.tick.lock is
unwritable (e.g. root-owned after a sudo run, in a writable cron dir) the
tick degrades and returns 0, so the ticker records a success; probe_store
checked only jobs.json, so `hermes cron status` said jobs would fire and
`hermes doctor` reported a writable store while nothing ran.
probe_store now treats an existing tick lock it cannot write the same way
as a read-only jobs.json target: both status and doctor surface it with the
usual fix hint.
Follow-up to #133283 (greptile review). The due scan stamps a one-shot's
run_claim; when the later fire-claim save fails the run never starts. The
next saved scan skips it on that live claim and ends the recovery window,
so once the claim expires _retire_expired_oneshot saw a stale claim past
grace and skipped it forever: neither fired nor retired.
The scan now marks a run claim it stamps past grace (only an outage lets a
one-shot through there) with outage=true; such a claim without a fire claim
keeps the job covered, so it fires once the claim expires. A fire claim
still means the run may have started, so at-most-once is unchanged.
After the previous commit the worker's pre-ack recovery is the import of
hermes_bootstrap, not run_agent's copy; the module docstring and the
relaunch test still credited run_agent. Also restore the scheduler
comment's original wrap (whitespace-only churn from the first pick).
(Lead fold: quality + thermo + reuse lenses, Low.)
Importing hermes_bootstrap already runs _settle_interrupted_update
(restore_interrupted_pull + relaunch_after_restore) at module import, so the
explicit second call in finish_worker_boot was a duplicate. The worker never
imports hermes_bootstrap before __main__ (probed: not in sys.modules after
`import cron.scheduler`, marked or not), so the import in finish_worker_boot
is what runs the repair, before the payload is read and the ack published.
The interrupted_pull case of test_relaunch_replays_the_worker_before_its_ack
stays green.
run_agent's import also re-execs the process when it restores a tree a killed
`hermes update` half-wrote (_early_recovery.relaunch_after_restore). The
external worker never loads hermes_cli.main, so that check first ran inside
run_one_job, after the ack: the re-exec found its payload deleted, refused the
execution, and the adopted run was left `unknown`, the same class as the
source-update relaunch. finish_worker_boot() now runs the recovery before the
payload is read, so its re-exec replays the whole worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b28779fd3bbaa1b92272448d17ccbb47f2794e0f)
The external worker first imported hermes_bootstrap through run_agent inside
run_one_job, after it had read (and deleted) its payload and published its
ownership ack. When the dependency inputs were not current, prepare_launch
synced and relaunched the worker into an -I interpreter that ignores the
pinned PYTHONPATH, and worker_bootstrap had already popped the boot marker:
the new process died on `import ruamel` and the adopted run was left
`unknown` (#122290 follow-up).
The worker's __main__ now runs hermes_bootstrap (finish_worker_boot) before
it touches the payload, and the marker is dropped only after that, so a
relaunch replays the whole worker and boots its dependencies again. The
call cannot live in worker_bootstrap(): during `-m cron.scheduler`'s package
import sys.argv[0] is still "-m" and __main__ has no spec, so the relaunch
command could not name the module. Covers the POSIX execv and Windows
subprocess.call relaunch branches alike.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b537b59787acf3b3f5828c0fb6ce82482de40820)
Final-stack review (hermes-pr-review 2c) on #133283:
- The tick lock reported its outage under the unresolved profile home (_get_lock_paths), while
every save, probe and grace check uses the resolved store path. With a symlinked HERMES_HOME
(or macOS /var -> /private/var) the lock outage never cleared: writable stuck at 0, no
"recovered" notice, and a skipped one-shot retired as missed. All store_health records are now
keyed through one _key() (realpath). Pinned with a symlinked home; red without it.
- unwritable -> recovered (before delivery) -> unwritable posted nothing: the dropped first
notice left the "recovered" timer pending, so the re-entry cancelled it as if an outage notice
had gone out. A dropped "unwritable" now cancels that timer, so the re-entry announces. The
notice test runs on a manual loop clock (no wall-clock timing; it had flaked once under load)
and pins the sequence.
- cron status printed "0 due run(s) not fired" and an empty summary when the jobs could not be
read from an unwritable store; it now says the jobs could not be read.
/simplify-code quality pass on the final #133283 stack, no behaviour change:
- One null-safe is_recurring in cron/constants.py replaces scheduler_tick's new copy and
unreachable_retry's (whose .get("schedule", {}) raised on a null schedule).
- cron status / cron list share _probe_then_list_jobs (probe, print, guarded read) instead of
two near-identical blocks.
- StoreDegraded.notice_fields and probe_report build their fields through one _report_fields.
- The notice listener's pre-checks no longer repeat deliver()'s store re-check; deliver() is the
single staleness gate (still red when removed).
Greptile r2 on #133283:
- #1: probe_store built one byte string the size of jobs.json; a large store made cron status,
doctor and the idle recovery check allocate that much. It now writes the same total in 64 KiB
chunks.
- #2: the host-wide store gauges read every degraded record, so a profile that left this
gateway while its store was down kept it at writable=0 until a save that would never come.
register_ticked_homes already discards a departed home's worker pool; it now also drops that
home's store state (store_health.forget_homes). The export test pins it.
The read-only tick-lock fallback (85f06c4777) pushed _tick_admitted to CC 22 against the
code-health target of 20 (was 17 on main). Move the lock-or-degrade step and the
advance-or-drop-recurring step into _acquire_tick_lock_or_degrade and
_advance_or_drop_recurring, with a small _is_recurring predicate. Behaviour is unchanged:
the cron, monitoring and cron CLI suites pass as-is, and the read-only-lock test still
fails with the errno filter removed.
Greptile r1 on #133283:
- #3: probe_store created an empty file in cron/, while a save stages a full payload beside a
symlinked jobs.json's real target. A read-only or full target passed the probe, so status and
doctor reported a writable store, and a fixed 1 MiB free-space cutoff could flag a store
whose small save still fits. The probe now stages a jobs.json-sized payload with
mkstemp_beside (the save's own staging helper), fsyncs it, and checks the resolved target is
writable; the free-space cutoff is gone.
- #5: on a read-only cron/ without cron/output, list_jobs tries to create output and raised
before `cron status` / `cron list` printed the warning and fix. Both print the store warning
first, and a read failure on an unwritable store no longer hides it.
Greptile r1 on #133283:
- #1: on a read-only (or full / permission-denied) mount the tick lock cannot be opened for
writing, so _acquire_tick_lock raised before the degraded-store path ever ran and the tick
still failed. An unwritable-class errno (EROFS/EACCES/EPERM/ENOSPC/EDQUOT) from the lock now
degrades the store (site "lock") and returns 0 with the MCP orphan sweep; EMFILE/ENFILE and
other errors still fail the tick (#87644). Live: a real read-only HFS mount raised errno 30
on the previous head and returns 0 here.
- #2: end_recovery_window ran before the scan's save, so a store that failed again before that
save landed lost the first outage's window and a later scan retired the skipped one-shot as
missed. The window now closes only after the scan's save lands, and a re-entry before then
keeps the first outage's start.
Follow-up review of the three fixes above:
- recheck_idle cleared the outage on a passing probe alone. The probe only creates an empty
file, so EDQUOT or a read-only jobs.json in a writable dir passed it while every real save
failed, flapping the state (and its WARN pair and metric) every idle minute. A passing probe
now does a real jobs.json save; its note_writable ends the outage, a failure keeps it.
- first_report still double-counted a miss at outage start (counted before the failing save
created the record) and after recovery by another save. Replace it: the scan queues its
miss/catch-up metrics and emits them only after its save lands, which counts each persisted
fast-forward exactly once with no per-outage bookkeeping.
- The ended-outage window lived for the process lifetime, so a one-shot due during the outage
but first scanned much later still fired late. The first due scan after recovery now closes it.
Review cleanup, no behaviour change:
- warn_store_unwritable only resolved the active store before calling
store_health.note_unwritable; note_unwritable now defaults cron_dir to the active store and
every caller uses it directly (scheduler_tick no longer imports both).
- The advance-skip split no longer zips a parallel bool list twice, and reads
(schedule or {}) so a null schedule cannot raise inside the OSError handler.
- skipped_runs metric: a walrus instead of getattr around an Optional record.
- Notices: runner._served_profile_homes is declared on the reconcile mixin, so no getattr.
- cron status loads the active jobs once (skipped count + summary) instead of twice.
While the store is unwritable the due scan's fast-forward cannot persist, so every tick finds
the same miss again and called record_catch_up_occurrence (or record_cron_missed when
catch_up_missed is false) once per tick per job for the whole outage, inflating the
catch-up/missed metrics.
store_health.first_report counts each (job, due instant) once per outage; with a writable
store it always counts, as before.
The degraded record cleared only when a jobs.json save landed in the gateway. With nothing due
(the user freed space and then paused or removed the due jobs, or CLI saves from another
process), no save lands, so writable stayed 0, the "recovered" notice never went out and the
one-shot grace gate stayed open indefinitely.
An idle tick now re-probes a degraded store at most once a minute (the same throttle as the
dispatch gate) and clears it when the probe passes.
The one-shot grace gate only looked at the live degraded record, and the first jobs.json save
that lands clears it. On a real recovery that save is usually another job's heartbeat or a
create, landing before the due scan, so every one-shot the outage skipped was retired as
"missed" anyway (review probe: degraded 15 min, one-shot due 10 min ago, an unrelated
create_job, then the scan retired it).
note_writable now keeps the ended outage per store (recovered_at), and the gate fires a
one-shot due inside [since - grace, recovered_at]. A one-shot due after recovery and then
missed is still retired, as on main.
Gate finding C12 r3 #9 (quality): degraded_since only wrapped
degraded_record(...).since, and jobs.py late-imported store_health in three
places for no cycle (store_health imports cron.jobs only lazily, inside
_active_cron_dir). Remove degraded_since, call degraded_record at the
one-shot gate, and import store_health once at module level next to
_claim_owner_is_dead. A fresh interpreter imports cron.jobs,
cron.scheduler, cron.store_health (either order) and
agent.monitoring.cron_health cleanly. dispatch_blocked loses the walrus and
the split locking: probe outside the lock, then update last_probe, error
and skipped under it in a single block. Behaviour is unchanged (the existing
cron tests pass as-is).
Gate finding C12 r3 #7 (efficiency): once dispatch had been tried, each
degraded tick could make two write attempts — the due scan's failing save,
then dispatch_blocked's minute-throttled probe_store — even though the scan
had just shown the store refusing writes. note_unwritable now refreshes
last_probe on any failing site once the record has been probed, so the
re-probe only runs on ticks where nothing else tried to write. A record
entered by a scan/load failure still has last_probe None, so its first
dispatch attempt and the one-shot "failed" row are unchanged. The degrade
test now asserts zero extra probes across the 5 outage ticks (2 before).
Gate finding C12 r3 #6 (2c): disk_usage().free leaves out the blocks the
filesystem reserves for root. A gateway running as root (common in
containers) can still write into them, yet the probe reported ENOSPC and
kept the store degraded while every real save succeeded. When geteuid() is
0, free_bytes now reads statvfs f_bfree * f_frsize. Platforms without
geteuid (Windows) keep disk_usage().free. The doctor low-space row uses the
same helper and so agrees with the probe.
Probe-verified (no third test function): with a faked root, statvfs and
disk_usage, HEAD returns 0 and this commit returns f_bfree * f_frsize.
A one-shot that came due while its store was unwritable could not settle (the claim write
failed), so it stayed due. The first tick after the store recovered found it past
ONESHOT_GRACE_SECONDS (120 s) and retired it as "missed": the outage silently ate the run.
The one-shot grace gate now treats a run that was due no earlier than grace before the store
entered its degraded state as skipped, not missed, and lets it fire once. Recurring jobs
already behaved: the unpersisted fast-forward re-runs each tick and the first writable tick
fires the job once under the existing catch-up policy.
The new lifecycle test drives 5 ENOSPC ticks plus recovery through a real tick(): one
"unwritable" transition, 2 probes for 5 ticks, only the scan's save attempted after entry,
then the recurring job and the one-shot each fire once and the one-shot keeps one failed row.
warn_store_unwritable logged one WARNING per (store, errno, site) every 15 minutes, and every
60s tick still ran the advance and the one-shot claims that could only fail again. Nothing could
tell the user "this store is degraded since X, N runs skipped".
cron/store_health.py (new; jobs.py is at its size cap) keeps one in-memory record per store dir:
since, error, the sites that failed, the distinct (job, due instant) runs skipped, last probe.
warn_store_unwritable delegates to it and WARNs once on entry with the fix (free disk space /
remount read-write / fix permissions on <path>). _save_jobs_unlocked, the single jobs.json write
chokepoint, clears it on the first save that lands and logs the recovery with the skip count.
While a store is known unwritable, tick() skips advance/claim for the due jobs (recording them as
skipped) and re-probes at most once per 60 s with a temp-file create+unlink plus a free-space
floor. The heartbeat and the MCP orphan sweep keep running. The first dispatch after entry still
goes through, so a due one-shot settles its single failed row, as before.
The OSError branch of _process_due_job's fire claim is the full-disk path,
yet it closed the receipt with a bare finish_execution while the sibling
branch already used settle_unstarted_execution. On a full disk the
executions.db write fails too, so the skip raised sqlite3.OperationalError
out of the worker instead of returning False. Use the best-effort settle
there as well (fold finding 1); a parametrize row covers the ledger failing.
The except branch covers both create_execution and the steps after it
(note_cron_execution, context copy). When the receipt WAS created the log
still said "execution creation failed", pointing operators at the ledger
while the row had been written and settled. Name the step from whether
`execution` was set (fold finding 3, quality review).
_submit_with_guard persists a `claimed` receipt before the worker runs, and
run_one_job only owns the terminal writes once it starts. Two exits in
between left the row `claimed` forever (a permanently in-flight attempt
for a run that never happened): a fire claim raising anything other than
OSError in _process_due_job, and a failure in _submit_with_guard after
create_execution returned (note_cron_execution, context copy). Both now
close the receipt through a best-effort settle_unstarted_execution in
cron/executions.py (the ledger owner; scheduler.py is a capped facade); the
claim failure still re-raises so the real error surfaces.
Co-authored-by: PRELVAS94 <86016656+PRELVAS94@users.noreply.github.com>
The late import in _claim_is_live was only there because the helper moved
to scheduler_ownership; that module's top-level imports never reach
cron.jobs, so there is no cycle. A function-level cron.* import inside
jobs.py is a first-time import on the completion tail, which #133407's
generation-pinning guard (test_completion_tail_imports[jobs.py]) rejects
once both land.
load_jobs() drops non-object entries (and flattens id-keyed maps, normalizes
repeat.completed) in memory, then saves the repair. That save was the one
store write left unguarded: on a full disk / read-only mount a single junk
entry in jobs.json raised ENOSPC out of every load_jobs(), so every tick died
before reaching the guarded advance/claim paths. The repaired list is already
what callers use; log through the rate-limited warn_store_unwritable(site=
'load') and let the next load retry the persist.
The due scan's save runs first on every tick and fails whenever a recurring job is due, so a shared (store, errno) key let its 'repairs not persisted' warning silence the tick's 'skipped N recurring job(s)' for the whole outage. Key on (store, errno, site) with site in scan/advance/claim: at most three WARNINGs per store per window, and the skip is always logged. The tick test now asserts the skip warning and that the recurring job never reached its fire claim (both red with the shared key / a keep-all filter). Gate r2 Medium (all five reviewers).
Gate r1 quality (Low): the size-gate move put get/record_catch_up_occurrence
into occurrences.py, whose docstring only described scheduled identities.
Widen it to name the counter (the larger cron/markers.py consolidation is
out of scope for this fix).
Gate r1 quality + reuse (Low): drop the third blank line before tick()
(E303); rewrite the test module docstring, which still claimed a writable
store is pinned and that the dispatch is never lost, to describe what the
two kept tests assert; lift one module-level _enospc stub shared by the
full_disk fixture and the tick test instead of two copies. No behaviour
change.
Gate r1 efficiency + quality (Low): when advance_next_runs fails and every
due job is recurring, the tick returned 0 without _sweep_mcp_orphans(),
unlike the idle and dispatch paths, so stdio children of earlier crashed
jobs were never reaped for the whole store outage. The partition also ran
the same predicate twice and filtered by O(n*m) dict-equality membership.
Partition in one pass (skipped count = n - len(filtered)) and sweep before
the early return, like the idle branch. The tick test is parametrized over
a recurring-only store (the skip-all branch) and asserts one sweep: red
without the sweep, green here.
Gate r1 2c (Low): _last_store_warning was keyed by errno alone and process
global. In a multiplexed gateway every profile ticks in one process, so
profile A's ENOSPC warning silenced profile B's for 15 minutes, and the text
gave no store path, so an operator could not tell which profile was stuck.
Key the limiter by (cron_dir, errno) via _current_cron_store() (same module,
no lazy import) and put the store path in the warning. The scan test now
also warns from a second profile's store: red on the errno-only key (1
warning), green here (2, each naming its store).
Gate r1 quality (Medium): the due-scan save guard in _get_due_jobs_locked
called logger.warning directly, so while the store stayed unwritable it
logged every 60s tick (the repairs are never persisted, so needs_save stays
true) -- the spam _STORE_WARN_INTERVAL_SECONDS exists to stop. Its wording
("due jobs still dispatched") also contradicted the tick-level warning that
recurring dispatch was skipped in the same tick.
Move warn_store_unwritable and its state into cron/jobs.py next to
_write_marker (the lower layer; jobs.py never imports scheduler_tick), route
the due-scan guard through it with neutral wording ("due-scan repairs not
persisted"), and have scheduler_tick.py and scheduler.py import it from
cron.jobs (scheduler.py via its existing top-level block, not inside the
except). The helper now takes the whole consequence phrase so each caller
states what it did.
The scan test now runs two scans and asserts exactly one neutral WARNING:
red on the previous code (2 warnings), green here.
Code-health ratchet: cron/jobs.py is capped at 3495 lines (its value on
main). The next commit moves the store-unwritable warning helper down into
jobs.py (gate r1 quality finding), which would push it over. Offset that by
a pure move of the self-contained claim-owner liveness probe into the
existing ownership sibling; body is byte-identical, its only caller
(_claim_is_live) late-imports it, and nothing in tests/ patches it.
