The #115692 reclaim released any running attempt whose claim was older than
max(3 x HERMES_CRON_TIMEOUT, script timeout, 2 h), even when the owner was
healthy and busy. Every cron job that legitimately ran longer than that was
marked `unknown` mid-run, its fire claim yanked from under it and the run
aborted with "lost its durable fire claim ownership" (two weekly jobs on one
install in a single day).
The run monitor now stamps `progress_at` on the execution row every 60 s
while the agent reports recent activity, and the sweep measures the stale
bound from that stamp (claim age for legacy rows). A wedged worker stops
stamping and is still reclaimed once the bound passes.
Shape: the stamper and the inactivity watchdog loop (which reads the same
idle signal) live in a new topical sibling cron/scheduler_liveness.py, so
cron/scheduler.py shrinks (4457 -> 4436 lines) instead of growing.
Two profiles whose cron/ resolves to one store share a store key; when one left, its departure dropped the degraded record the remaining profile still needed (gauge flipped to writable, a second outage notice, and the earlier outage start lost for one-shot cover). Departed stores now exclude the ones still ticked. The notices test pins it (red without the set difference).
The tick-lock case only asserted probe_store() returned an error, which any other
probe failure would also satisfy. Assert the error names .tick.lock so the case
proves that branch fired.
Gate r1 (PM12, quality) Low finding.
forget_homes rebuilt each store path from a departed home KEY
(Path(key) / "cron"). cron/AGENTS.md forbids rebuilding a home from a key
(hermes_home_key normcases), and the realpath only follows a symlinked cron/
while the home still exists: a profile whose home was deleted kept its degraded
record, holding the host-wide hermes.cron.store.writable gauge at 0.
register_ticked_homes now records each home's resolved store (new public
store_health.store_key) while the home exists and hands the departed stores to
forget_homes. gateway/cron_store_notices.py uses the same store_key against
record.store (already resolved), so one normaliser decides "is this the served
home's store" everywhere.
Gate r1 (PM12, 2c + quality + 2ab toothless) Low finding.
clear_run_claim (the dispatch-failure path: interpreter shutdown, execution
creation failure, pool.submit failure) set run_claim to None. The claiming scan
had already ended the recovery window, and a restart drops the in-memory outage
records, so the next scan saw a past-grace one-shot with no claim and retired it
as missed: the job the outage skipped never fired.
Keep {"outage": True} instead. With no "at" it is a stale claim, so the job is
re-dispatched on the next tick, and _retire_expired_oneshot keeps it. At-most-once
is unchanged: the fire_claim CAS still fences every re-dispatch.
Gate r1 (PM12, 2c) Low finding.
Follow-up to #133283 (greptile review). Store records are keyed by the
resolved store path, but the notice sender rebuilt the owning profile from
that path's parent. A profile whose cron/ is a symlink to another disk has
a foreign parent, so neither its outage nor recovery notice reached any
home channel; forget_homes had the same mismatch and never dropped such a
departed profile's record.
Both now match by resolving each served home's own cron/ path.
Follow-up to #133283 (greptile review). When only cron/.tick.lock is
unwritable (e.g. root-owned after a sudo run, in a writable cron dir) the
tick degrades and returns 0, so the ticker records a success; probe_store
checked only jobs.json, so `hermes cron status` said jobs would fire and
`hermes doctor` reported a writable store while nothing ran.
probe_store now treats an existing tick lock it cannot write the same way
as a read-only jobs.json target: both status and doctor surface it with the
usual fix hint.
Follow-up to #133283 (greptile review). The due scan stamps a one-shot's
run_claim; when the later fire-claim save fails the run never starts. The
next saved scan skips it on that live claim and ends the recovery window,
so once the claim expires _retire_expired_oneshot saw a stale claim past
grace and skipped it forever: neither fired nor retired.
The scan now marks a run claim it stamps past grace (only an outage lets a
one-shot through there) with outage=true; such a claim without a fire claim
keeps the job covered, so it fires once the claim expires. A fire claim
still means the run may have started, so at-most-once is unchanged.
Lead fold from the independent review (quality + thermo + 2c + efficiency
lenses, all Low): the fire startup gate and
APIServerAdapter._gateway_is_draining now share runner_is_draining, so a
future drain flag cannot be added to one and missed by the other; the gate
returns early on no runner instead of looping on an unchanging guard. The
cold-adapter test closes its loop and joins its thread, the slow-verify
test keeps the same ordering at a fifth of the wall-clock, and the Chronos
contract doc lists the startup 503.
The fixture hand-assembled a _TargetDelivery (a Slack origin paired with a hard-wired Telegram
target, a duck-typed transport, a hand-set origin_user_id), so it asserted a shape the real
prologue never produces. The targets now come from _prepare_target_delivery with a live relay (or
native) adapter, so origin matching, transport resolution and origin_user_id are the production
ones.
Review (greptile): after the boot moved before the ack, the tests
checked that a relaunch keeps the marker but none checked that a boot
without a relaunch consumes it. Job children would otherwise inherit it
and repeat the dependency boot.
After the previous commit the worker's pre-ack recovery is the import of
hermes_bootstrap, not run_agent's copy; the module docstring and the
relaunch test still credited run_agent. Also restore the scheduler
comment's original wrap (whitespace-only churn from the first pick).
(Lead fold: quality + thermo + reuse lenses, Low.)
run_agent's import also re-execs the process when it restores a tree a killed
`hermes update` half-wrote (_early_recovery.relaunch_after_restore). The
external worker never loads hermes_cli.main, so that check first ran inside
run_one_job, after the ack: the re-exec found its payload deleted, refused the
execution, and the adopted run was left `unknown`, the same class as the
source-update relaunch. finish_worker_boot() now runs the recovery before the
payload is read, so its re-exec replays the whole worker.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b28779fd3bbaa1b92272448d17ccbb47f2794e0f)
The external worker first imported hermes_bootstrap through run_agent inside
run_one_job, after it had read (and deleted) its payload and published its
ownership ack. When the dependency inputs were not current, prepare_launch
synced and relaunched the worker into an -I interpreter that ignores the
pinned PYTHONPATH, and worker_bootstrap had already popped the boot marker:
the new process died on `import ruamel` and the adopted run was left
`unknown` (#122290 follow-up).
The worker's __main__ now runs hermes_bootstrap (finish_worker_boot) before
it touches the payload, and the marker is dropped only after that, so a
relaunch replays the whole worker and boots its dependencies again. The
call cannot live in worker_bootstrap(): during `-m cron.scheduler`'s package
import sys.argv[0] is still "-m" and __main__ has no spec, so the relaunch
command could not name the module. Covers the POSIX execv and Windows
subprocess.call relaunch branches alike.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
(cherry picked from commit b537b59787acf3b3f5828c0fb6ce82482de40820)
Final-stack review (hermes-pr-review 2c) on #133283:
- The tick lock reported its outage under the unresolved profile home (_get_lock_paths), while
every save, probe and grace check uses the resolved store path. With a symlinked HERMES_HOME
(or macOS /var -> /private/var) the lock outage never cleared: writable stuck at 0, no
"recovered" notice, and a skipped one-shot retired as missed. All store_health records are now
keyed through one _key() (realpath). Pinned with a symlinked home; red without it.
- unwritable -> recovered (before delivery) -> unwritable posted nothing: the dropped first
notice left the "recovered" timer pending, so the re-entry cancelled it as if an outage notice
had gone out. A dropped "unwritable" now cancels that timer, so the re-entry announces. The
notice test runs on a manual loop clock (no wall-clock timing; it had flaked once under load)
and pins the sequence.
- cron status printed "0 due run(s) not fired" and an empty summary when the jobs could not be
read from an unwritable store; it now says the jobs could not be read.
Greptile r1 on #133283:
- #7: the "recovered" timer started an un-awaited send; if the store failed again while that
send waited on profile loading or its transport, the outage notice could land first and the
stale "recovered" last. Notices for a store now go through one asyncio.Lock in transition
order and re-check the store at delivery, so a notice that no longer matches is dropped and
the outage notice stays the last word.
- #6: t() ran before entering the owning profile's runtime scope, so a secondary profile's home
channel got the launch profile's display.language. The message is now rendered inside that
scope.
- #8: the notice test shortened the window to 0.05 s and slept 0.2 s, against tests/AGENTS.md
(no quiet-runner timing). It now uses a zero window and drains the loop until no timer or send
remains, holds a delivery open with an asyncio.Event to drive the flip-during-send case, and
asserts the full notice sequence.
Greptile r1 on #133283:
- #3: probe_store created an empty file in cron/, while a save stages a full payload beside a
symlinked jobs.json's real target. A read-only or full target passed the probe, so status and
doctor reported a writable store, and a fixed 1 MiB free-space cutoff could flag a store
whose small save still fits. The probe now stages a jobs.json-sized payload with
mkstemp_beside (the save's own staging helper), fsyncs it, and checks the resolved target is
writable; the free-space cutoff is gone.
- #5: on a read-only cron/ without cron/output, list_jobs tries to create output and raised
before `cron status` / `cron list` printed the warning and fix. Both print the store warning
first, and a read failure on an unwritable store no longer hides it.
Greptile r1 on #133283:
- #1: on a read-only (or full / permission-denied) mount the tick lock cannot be opened for
writing, so _acquire_tick_lock raised before the degraded-store path ever ran and the tick
still failed. An unwritable-class errno (EROFS/EACCES/EPERM/ENOSPC/EDQUOT) from the lock now
degrades the store (site "lock") and returns 0 with the MCP orphan sweep; EMFILE/ENFILE and
other errors still fail the tick (#87644). Live: a real read-only HFS mount raised errno 30
on the previous head and returns 0 here.
- #2: end_recovery_window ran before the scan's save, so a store that failed again before that
save landed lost the first outage's window and a later scan retired the skipped one-shot as
missed. The window now closes only after the scan's save lands, and a re-entry before then
keeps the first outage's start.
Follow-up review of the three fixes above:
- recheck_idle cleared the outage on a passing probe alone. The probe only creates an empty
file, so EDQUOT or a read-only jobs.json in a writable dir passed it while every real save
failed, flapping the state (and its WARN pair and metric) every idle minute. A passing probe
now does a real jobs.json save; its note_writable ends the outage, a failure keeps it.
- first_report still double-counted a miss at outage start (counted before the failing save
created the record) and after recovery by another save. Replace it: the scan queues its
miss/catch-up metrics and emits them only after its save lands, which counts each persisted
fast-forward exactly once with no per-outage bookkeeping.
- The ended-outage window lived for the process lifetime, so a one-shot due during the outage
but first scanned much later still fired late. The first due scan after recovery now closes it.
Review cleanup, no behaviour change:
- warn_store_unwritable only resolved the active store before calling
store_health.note_unwritable; note_unwritable now defaults cron_dir to the active store and
every caller uses it directly (scheduler_tick no longer imports both).
- The advance-skip split no longer zips a parallel bool list twice, and reads
(schedule or {}) so a null schedule cannot raise inside the OSError handler.
- skipped_runs metric: a walrus instead of getattr around an Optional record.
- Notices: runner._served_profile_homes is declared on the reconcile mixin, so no getattr.
- cron status loads the active jobs once (skipped count + summary) instead of twice.
While the store is unwritable the due scan's fast-forward cannot persist, so every tick finds
the same miss again and called record_catch_up_occurrence (or record_cron_missed when
catch_up_missed is false) once per tick per job for the whole outage, inflating the
catch-up/missed metrics.
store_health.first_report counts each (job, due instant) once per outage; with a writable
store it always counts, as before.
The degraded record cleared only when a jobs.json save landed in the gateway. With nothing due
(the user freed space and then paused or removed the due jobs, or CLI saves from another
process), no save lands, so writable stayed 0, the "recovered" notice never went out and the
one-shot grace gate stayed open indefinitely.
An idle tick now re-probes a degraded store at most once a minute (the same throttle as the
dispatch gate) and clears it when the probe passes.
The one-shot grace gate only looked at the live degraded record, and the first jobs.json save
that lands clears it. On a real recovery that save is usually another job's heartbeat or a
create, landing before the due scan, so every one-shot the outage skipped was retired as
"missed" anyway (review probe: degraded 15 min, one-shot due 10 min ago, an unrelated
create_job, then the scan retired it).
note_writable now keeps the ended outage per store (recovered_at), and the gate fires a
one-shot due inside [since - grace, recovered_at]. A one-shot due after recovery and then
missed is still retired, as on main.
The boot replay of degraded stores runs after the listener is installed, so a store that
recovered in between got an "unwritable" notice it would never retract. Recheck the live
record before announcing. The existing notice test now also pins the re-entry cancel and the
no-lone-recovered guard, each red when removed.
Gate r4 2ab (TOOTHLESS): reverting fe427445ce left the kept tests green. The
stub runner had no _served_profile_homes and yielded profile None, so it never
took the served-home branch. It now serves profile "p" homed at the store's
parent while the launch home is elsewhere, so the notice only arrives when the
lookup goes through runner._served_profile_homes.
Gate r4 (2c + quality, both probed): the r3 hysteresis dropped an "unwritable"
transition that came within an hour of the last notice. note_unwritable reports
entry only once, so for unwritable -> recovered -> unwritable the home channel's
last notice said "writable again" while the store stayed down, possibly for days.
Rate-limit the recovery instead. "unwritable" is sent only for a store whose
outage was not announced yet; "recovered" waits NOTICE_REPEAT_SECONDS on the
gateway loop and is sent only if the store is still writable then; a re-entry
meanwhile cancels the pending recovery and sends nothing. The last notice always
matches the store's real state, and a flapping disk posts at most one pair per
window. All bookkeeping now runs on the loop (call_soon_threadsafe), so the
ticker thread no longer needs a lock, and the notice coroutine is created
there too: a transition after the loop closed raises before any coroutine
exists (logged by store_health._notify), instead of leaking an un-awaited one
(gate r4 reuse item, loop-side variant).
Gate finding C12 r3 #7 (efficiency): once dispatch had been tried, each
degraded tick could make two write attempts — the due scan's failing save,
then dispatch_blocked's minute-throttled probe_store — even though the scan
had just shown the store refusing writes. note_unwritable now refreshes
last_probe on any failing site once the record has been probed, so the
re-probe only runs on ticks where nothing else tried to write. A record
entered by a scan/load failure still has last_probe None, so its first
dispatch attempt and the one-shot "failed" row are unchanged. The degrade
test now asserts zero extra probes across the 5 outage ticks (2 before).
Gate finding C12 r3 #2 (quality): the notice listener dropped the future
from run_coroutine_threadsafe, so a send that raised (adapter error, a
profile scope that failed to bind) vanished without a trace. Keep the future
and log failures from a done-callback, the same way plugin message injection
does in gateway/run_plugin_injection.py.
Gate finding C12 r3 #1 (2c): a store that crosses the disk-full line every
few ticks entered and left the degraded state each time, and every
transition posted a home-channel notice — an unwritable/recovered pair per
flap. The notice listener now remembers when it last messaged each store
and drops an "unwritable" notice within an hour of that, plus the matching
"recovered" when its "unwritable" was never sent. The cron log still records
every transition.
The tick runs on a worker thread inside the gateway, so an outage was visible only in the log.
cron.store_health now reports one transition when a store enters the degraded state and one
when it clears. gateway/cron_store_notices.py (new; run_notifications.py is at its size cap)
installs the listener once the adapters are connected, after the state.db warning. Each
transition is posted onto the gateway loop and sent to the home channels of the profile that
owns the store, via _served_home_channel_transports, _send_home_channel_message and
present_notification. The owning profile's suppress_warning_notifications opt-out applies,
under that profile's runtime scope. A store that degraded during boot is announced when the
listener is installed.
Copy goes through t(): gateway.cron_store.unwritable names the store, the error, since-when and
the fix. gateway.cron_store.recovered reads "Cron store writable again; N skipped runs, catching
up once per job". The keys are added to every locale (English text, as test_i18n's key-parity
contract requires).
The test probes a real 0500 store through `cron status` and checks the headline and fix. It
drives three failing sites and two saves, then checks that exactly one entry notice and one
recovery notice were sent.
A one-shot that came due while its store was unwritable could not settle (the claim write
failed), so it stayed due. The first tick after the store recovered found it past
ONESHOT_GRACE_SECONDS (120 s) and retired it as "missed": the outage silently ate the run.
The one-shot grace gate now treats a run that was due no earlier than grace before the store
entered its degraded state as skipped, not missed, and lets it fire once. Recurring jobs
already behaved: the unpersisted fast-forward re-runs each tick and the first writable tick
fires the job once under the existing catch-up policy.
The new lifecycle test drives 5 ENOSPC ticks plus recovery through a real tick(): one
"unwritable" transition, 2 probes for 5 ticks, only the scan's save attempted after entry,
then the recurring job and the one-shot each fire once and the one-shot keeps one failed row.
warn_store_unwritable logged one WARNING per (store, errno, site) every 15 minutes, and every
60s tick still ran the advance and the one-shot claims that could only fail again. Nothing could
tell the user "this store is degraded since X, N runs skipped".
cron/store_health.py (new; jobs.py is at its size cap) keeps one in-memory record per store dir:
since, error, the sites that failed, the distinct (job, due instant) runs skipped, last probe.
warn_store_unwritable delegates to it and WARNs once on entry with the fix (free disk space /
remount read-write / fix permissions on <path>). _save_jobs_unlocked, the single jobs.json write
chokepoint, clears it on the first save that lands and logs the recovery with the skip count.
While a store is known unwritable, tick() skips advance/claim for the due jobs (recording them as
skipped) and re-probes at most once per 60 s with a temp-file create+unlink plus a free-space
floor. The heartbeat and the MCP orphan sweep keep running. The first dispatch after entry still
goes through, so a due one-shot settles its single failed row, as before.
The OSError branch of _process_due_job's fire claim is the full-disk path,
yet it closed the receipt with a bare finish_execution while the sibling
branch already used settle_unstarted_execution. On a full disk the
executions.db write fails too, so the skip raised sqlite3.OperationalError
out of the worker instead of returning False. Use the best-effort settle
there as well (fold finding 1); a parametrize row covers the ledger failing.
tick(sync=True) collects worker futures and logs "Cron job future failed"
instead of re-raising, so the try/except RuntimeError around it could never
fire and only hid a regression that made tick() start raising. Verified the
test passes without it (fold finding 2).
The except branch covers both create_execution and the steps after it
(note_cron_execution, context copy). When the receipt WAS created the log
still said "execution creation failed", pointing operators at the ledger
while the row had been written and settled. Name the step from whether
`execution` was set (fold finding 3, quality review).
_submit_with_guard persists a `claimed` receipt before the worker runs, and
run_one_job only owns the terminal writes once it starts. Two exits in
between left the row `claimed` forever (a permanently in-flight attempt
for a run that never happened): a fire claim raising anything other than
OSError in _process_due_job, and a failure in _submit_with_guard after
create_execution returned (note_cron_execution, context copy). Both now
close the receipt through a best-effort settle_unstarted_execution in
cron/executions.py (the ledger owner; scheduler.py is a capped facade); the
claim failure still re-raises so the real error surfaces.
Co-authored-by: PRELVAS94 <86016656+PRELVAS94@users.noreply.github.com>
load_jobs() drops non-object entries (and flattens id-keyed maps, normalizes
repeat.completed) in memory, then saves the repair. That save was the one
store write left unguarded: on a full disk / read-only mount a single junk
entry in jobs.json raised ENOSPC out of every load_jobs(), so every tick died
before reaching the guarded advance/claim paths. The repaired list is already
what callers use; log through the rate-limited warn_store_unwritable(site=
'load') and let the next load retry the persist.
The due scan's save runs first on every tick and fails whenever a recurring job is due, so a shared (store, errno) key let its 'repairs not persisted' warning silence the tick's 'skipped N recurring job(s)' for the whole outage. Key on (store, errno, site) with site in scan/advance/claim: at most three WARNINGs per store per window, and the skip is always logged. The tick test now asserts the skip warning and that the recurring job never reached its fire claim (both red with the shared key / a keep-all filter). Gate r2 Medium (all five reviewers).
