Long-running cron jobs (>60 s, i.e. past one heartbeat interval) intermittently
ended with last_status=error / "Interrupted by shutdown before terminal
completion." while their output was complete. The fire-claim heartbeat thread
took ONE sample that read the claim as not ours, latched lost_ownership, and
_FireOwnership.lost() then trusted that latch without asking the store again —
so a run whose claim still validated (the very condition under which
_record_fire_ownership_lost writes that message) was recorded as interrupted,
its output never saved. The reporter's own error string proves the miss was
transient: it is only ever written when the owner re-validates True.
Why this shape:
- _FireOwnership.lost(): an explicit transport cancel stays terminal; a latched
heartbeat miss is re-checked against the store, and a claim that still
validates keeps the run's real outcome. The owner-fenced mark_job_run and
fire_claim_fence remain the authority, so a genuinely re-owned claim still
yields (no delivery, no terminal write over the new owner, ledger discard).
An unreachable store after a latched miss stays fail-closed.
- _heartbeat_loop: a miss is re-sampled once after
_FIRE_CLAIM_MISS_CONFIRM_SECONDS before it latches. Latching cancels the live
agent run and ends the lease refresh, so a single sample must not do that; a
genuinely re-owned claim misses twice and latches ~1 s later.
The issue's other hypothesis (worker process identity under a systemd-run
--scope worker) is falsified by the code: the owner token is stored in
fire_claim.by at claim time and compared to the stored value, never recomputed
from _machine_id(), and the heartbeat thread runs in a copied context so it
reads the same store.
Slimmer redo of #113364 (same direction: revalidate before trusting the latch)
without its production-dead isinstance(_CombinedCancelEvent) branch.
Co-authored-by: KoNit-K <konit.block@protonmail.com>
eafed27cf0