The run's own archive was read five ways: _read_terminal_receipt,
_running_record and amend_terminal_followup (each globbing by update_id
with different error handling), plus the _persist_running stale-snapshot
guard and finalize_interrupted_update_receipt (each re-reading the
_run_file path inline). update_receipt.read_run_record is now the only
reader: unreadable, non-object and other-run files all mean "no record",
it never raises, and each caller keeps its outcome on top of that.
The divergence was a bug in one place: _read_terminal_receipt and
_running_record called .get() on whatever json.loads returned, so a run
archive holding valid JSON that is not an object (null, a list) raised
AttributeError out of settle_lost_completion -- a post-commit path that
must never fail the update. It now settles like a missing archive.
For a partial run (committed, exit 1 after record_user_action) the toast said
only "Action failed (exit 1)" and dropped the receipt's recovery instruction;
on exit 0 it named follow-ups from whatever receipt the status route attached,
which is the latest one when this action has none.
Keep the update POST's action_id and read the debt through the shared
updateDebt (committed success|partial, matching action_id only). The toast
composes it with every verdict: follow-ups keep the rerun remedy, a user
action shows the producer's own instruction verbatim (never "re-run hermes
update", which does not restore a parked stash).
The backend-apply store consulted the receipt's owed steps only on exit 0. A
partial run (record_user_action turns a committed success into partial / exit
1, e.g. a stash whose restore conflicted) fell through to "Backend update
failed." and a receipt-proven partial after the restart to "did not come back
online", dropping the exact recovery instruction; and the success branch read
followups from whatever receipt the status route attached, which is the
latest receipt when this action has none.
Add @hermes/shared updateDebt(receipt, actionId): the debt of a committed
(success|partial) run, only when the receipt's action_id is the id the start
request returned. The store composes it on success and failure: follow-ups
keep the rerun remedy, a user action shows the producer's instruction verbatim
(a rerun does not restore a parked stash). The existing owed-success spec's
receipt gains the matching action_id the real route emits.
The status route rebuilt <root>/logs/update.log by hand while
hermes_cli.logs.log_file_path("update") already owns that placement (root
home for the update tee). One resolver, so a future move cannot split the
writer and this reader.
"record the follow-up, then amend the closed receipt" was hand-written at six
sites (three Windows-resume sites, the historical-takeover resume,
gateway_restart, channel_adoption). Two of them -- _complete_source_update's
Windows resume and channel_adoption -- amended without checking that the
run's receipt was already closed. On an open run record_followup has just
persisted the follow-up into the running record, so the amend appended a
second copy to that durable record and to latest.json (what an interrupted
run leaves behind).
update_receipt.owe_followup(update_id, step, reason) records the step and
amends the archive only once the run is closed; every site now calls it.
managedUpdateMessage returned a generic update/restore message before reading
the receipt's debt whenever the combined outcome was not `updated`. Two
reachable outcomes lost necessary recovery data from the displayed result
and the Update-all row: a committed success whose profile restoration failed
(owed follow-ups dropped) and a `partial` run whose record_user_action
carries the exact parked-local-change instruction (dropped; only "the remote
update failed" shown).
Compose the debt with every verdict: owed steps are listed for any terminal
receipt, `owed` is returned, follow-ups get the rerun advice by name, and a
user action is shown verbatim (never as "re-run hermes update", which does
not restore a parked stash). updateOk/restoreOk stay independent.
After a dashboard restart the only record of which update THIS dashboard
started was the `=== hermes-update started ... <id> ===` header, read from a
2,000-line tail. A build that printed more than 2,000 lines (or a rotated
log) hid it: spawned_action_id became None, the matching root completion
marker and the matching archived success receipt were both refused, and the
route answered running=false/exit_code=null, which the web poller shows as
"Action failed (exit ?)".
_spawn_hermes_action now also writes the id to a `<name>.action_id` sidecar
(atomic) next to the log; the status route falls back to it when no header
is visible. A visible header still wins (a dashboard predating the sidecar
wrote only the header, and the newest header is the latest spawn). The
identity selects B's own archived receipt even when another profile's run is
latest; failed/unfinished actions still report exit null.
C3 moved update receipts to the ROOT store (get_default_hermes_root) so the
Desktop, the SSH observer and the hand-off scripts find a sticky profile's
run. pm/receipt.py (byte-identical by ruling) still writes get_hermes_home(),
so under HERMES_HOME=<root>/profiles/foo the two stores split: `hermes -p foo
pm status` lost update receipts, read_latest_receipt() consumers (fleet
checks, serve obligations, dashboard) lost the profile's pm receipts, and
receipts written to the profile store before the move were never read.
Keep the root writer and add the profile store as a second store when it
differs: latest.json is mirrored into it (pm status reads only that one),
read_latest_receipt() takes the newest pointer of both (root on a tie, i.e.
last writer wins as when they shared one folder) and read_receipt_for_action
searches both archives. Default homes resolve to one store: unchanged.
main's convert_treeless_checkout_first (1c97b57a93) runs in update_completion._prepare before
the completion children, and its git probe went through the Popen this test replaces to capture
the children's argv, so it was counted as a third spawn.
`actionFinishedOwed` was added to the English catalog and the shared type only, so `tsc -b`
(what the Docker image and nix builds run) failed on the 15 locales typed against the full
shape, and Arabic fell back to English. Adds the translated string to all 16 catalogs.
Under C3 a run whose code committed is a success (exit 0) even while a
post-commit step is owed, but the dashboard status summary and the managed SSH
observer dropped the receipt's followups/user_action, so every client outside
the local shell hand-off reported plain success. Both summary protocols now
carry followups and user_action; the Desktop backend apply and managed SSH
result keep their C3 success/restore status but name each owed step and its
remedy, and the web dashboard toast does the same. The Desktop receipt
fallback also requires the receipt's action_id to match the started action
(when the backend reports one), so another action's run cannot certify it.
The receipt store moved to the root home, so profile A's later success in
latest.json became profile B's exit 0 once B's dashboard lost its in-memory
result: the receipt fallback had no identity to compare. Each receipt now
records the dashboard action (HERMES_ACTION_ID) that spawned the run, and the
status route resolves the action this dashboard started (its log header, else
the in-memory id), reads that action's receipt from latest.json or the
retained archive, and maps an outcome to an exit code only on an exact match.
The attached summary carries action_id so clients can tell whose run it is;
the update_id/PM correlation identity is unchanged.
_record_owed_gateway_inventory returned for any existing inventory, so a
same-SHA retry that stopped an unmapped gateway while the mapped one stayed
down recorded a gateway_restart follow-up but never added the stopped PID to
the durable owed set; once the mapped gateway recovered, startup settled the
obligation with no successor for the unmapped one. A readable inventory now
only grows (earlier rows and record provenance kept); a malformed or
unsupported one is still left untouched.
The owed-migration raise escaped before the sibling config migration and the
pre-update snapshot safety nets, so a failed active write left sibling configs
on the old version and skipped the cron/model-settings restore that only this
call can do (the snapshot ids are not carried to the next-launch retry). Hold
the error, run the independent work, then raise so config_migration stays owed.
The C3 rewrite of the stale-gateway line (review 5411222454 'Rebase against
#132361': the AGENTS.md line) pushed AGENTS.md + hermes_cli/AGENTS.md to
30217 chars, past the 30000 cap. The rule stays (owed follow-ups, exit 0;
parked stash 1, Ctrl-C 130; root receipts); the table lives in
developer-guide/source-update-completion.md.
cmd_update first retargets to the owning install, which probes the PM test
environment under the test HOME and trips the home I/O guard in a mixed
file run. The retarget is not what these tests (review regression 3) pin.
89e06e62dbe gave the bootstrap completion child -X utf8; the prepared
child (_prepare -> --prepared, which runs _complete_selected and prints the
✓/⚠ follow-up protocol into the same pipe) still started with -I and no
UTF-8 mode, so it relied on hermes_cli's import-time stdio repair alone.
Both children now carry -X utf8; a test captures both argvs.
Review: kshitijk4poor 5411222454 open thread "UTF-8 under -I".
After the run's receipt closed as `success`, three boundaries could still
write 1 to .update_exit_code (and the completion child could still answer
130), so the gateway watcher reported a failed update the receipt calls a
success:
- cmd_update's catch-alls (main.py, unchanged) call
_write_gateway_update_exit_code(False) for any escaping exception. The
writer now keeps 0 when this command's run is closed as a success
(update_receipt.committed_success: the receipt scope records the terminal
outcome it finalized, or adopted from the completion child).
- The parent adopts the child's terminal receipt and drops its stale
pre-child context as soon as the result is accepted, before the parent-
side Windows resume, so nothing after it can write that context.
- update_completion._finish: an interrupt that lands after verification
finalized the run answers 0 (the receipt's verdict); the 1 for a parked
stash (partial) or an interrupt before the run closed is unchanged, and
the status is written once the code is settled.
- update_finish (historical takeover): the same rule for its finally.
An interrupted run (receipt `interrupted`, exit 130) still writes 1, as
test_interrupt_after_child_success_demotes_gateway_marker_at_boundary pins.
Review: kshitijk4poor 5411222454 regression 3 / open thread F4.
A committed update exits 0 and prints one whole "Update follow-up '<step>'
did not finish:" line per owed step (update_receipt.record_followup). The
hand-offs only reacted to the Desktop build line, so a gateway proven to
run old code (gateway_restart), a refused Windows resume, a lost
completion or a failed channel adoption ended as a plain
"Update complete.".
posix.sh collects the distinct owed steps from that line
(owed_followup_steps) and turns them into the manual result message
(DONE_NOTE, the only channel the main-based script has); a gateway_restart
adds the `hermes gateway restart` hint. windows.ps1 does the same through
its manual outcome. With #132354's add_warning protocol the same
owed_followup_steps output feeds one add_warning per step.
record_followup's docstring names the line as a protocol; a contract test
runs the real posix.sh parser and the windows.ps1 regex against it.
Review: kshitijk4poor 5411222454 regression 1 / open thread F6.
The full outcome list (stale gateway, owed completion, re-armed tail) lives in
developer-guide/source-update-completion.md; the AGENTS.md chain stays under its 30,000-char cap.
What: an inventoried marker that owes an unmapped gateway (one the update stopped without a
profile it could name) now clears only when a local gateway no mapped owed identity explains is
running the checkout. The old guard only refused an all-external fleet.
Why: with `default` and an unmapped gateway both owed, `default` coming back current cleared the
whole marker, so the unmapped gateway stayed down with no warning and no owed restart. The new
rule subsumes the all-external refusal (no local row can be the successor).
Red: test_unmapped_stop_debt_is_not_settled_by_the_mapped_gateways_restart -> owes False with only
`default` current (review probe f7_mixed: marker cleared, pid 101 still down).
Green: owed while only `default` runs; settled once a further local gateway runs the checkout;
14 fleet/obligation files, 124 passed.
The tests added for review P1-P4 wrote and read text without an
encoding (or read with BOM-intolerant utf-8) and used a bare
signal.SIGKILL in a child script, which check-windows-footguns flags
(invariant 5). Writes now pass encoding="utf-8", reads "utf-8-sig", and
the posix-only child kills itself with signal 9 by number.
Review P1-P4 (invariant 5 hygiene for the new tests).
Follow-up to the lost-completion settle (review P2):
- A retired channel is adopted only after a VERIFIED completion: a lost
one exits 0 with the owed `completion` follow-up but leaves the config
on the retired channel; the next `hermes update` resolves the
retirement again and adopts it (the same retry the adoption-failure
follow-up already names).
- The settle triggers on a non-zero answer without a receipt (what
run_completion returns for every answer that is not a correlated
terminal receipt) and runs at most once per run (a spawn failure that
could not finalize does not settle twice).
Test changes (test_source_channel_integration.py), each pinning the
failure invariant 3 forbids:
- test_retirement_adopts_destination_only_after_success[failed]: the
completion answered exit 1 without a receipt and the test expected
SystemExit; it is now an owed completion (exit 0). The channel
assertion is unchanged: the destination is still not adopted.
- test_retirement_waits_for_correlated_completion_process[uncorrelated|
missing]: same, an uncorrelated/missing answer was SystemExit and is
now owed; the channel still stays on the retired one. [failure] (the
child answered a correlated failed receipt) still raises.
Review P2.
hermes_cli/AGENTS.md and cli-internals.md stated "after the commit point
nothing fails" without the one documented exception (an autostash whose
restore conflicted stays parked: `partial`, exit 1, #122557) and without
the lost-completion outcome P2 introduces (a completion that could not
start or died without a result is an owed `completion` follow-up, tail
re-armed, exit 0). update_receipt's module docstring still said receipts
live under <HERMES_HOME>; they live under the ROOT home
(get_default_hermes_root()), never a sticky profile's.
Review P6.
The two BLE001 waivers around the memory-provider and left-core plugin
migrations claimed the failure "is reported as a follow-up", but neither
calls record_followup: the failure is printed as ⚠ only. Recording a
follow-up here would be wrong for the install path that shares this
builder (it would print "Update follow-up ..." during an install), and
the migrations are re-run by every later update/tail anyway. Fix the
wording to name the real behaviour instead.
Review P5.
update_finish.finish_update (the historical takeover's completion) ran
_restart_gateway_fleet_after_update, the Windows resume merge and
_verify_fleet_after_update unguarded. They run after the commit point,
yet a raise (a systemctl timeout, a service-manager error) reached
update_finish.main's catch-all: record_step failed, receipt `failed`,
.update_exit_code rewritten to 1, exit 1.
The three calls now sit behind the same post-commit boundary as the
current completion: the error is a `gateway_restart` follow-up (amended
onto the terminal receipt when verification already finalized it), the
fleet restart obligation stays armed for the next launch / `hermes
update`, and the run exits 0 with the watcher's 0 intact.
Review P3 (invariant 3).
After the tree moved, an OSError from run_completion's TemporaryDirectory,
_write_json or Popen (or anything the spawn path raises) reached
main.cmd_update as "Update failed": receipt failed, .update_exit_code 1,
exit 1. A completion child killed by OOM/SIGKILL, or one that exited
without a correlated terminal result, returned `code or 1` and exited
non-zero the same way. Both contradict invariant 3: the code is
committed, only the tail is owed.
_complete_source_update now settles both through
update_completion.settle_lost_completion (pre-swap module, everything it
uses is already imported in the parent): it arms the source-update tail
for the next launch (the bootstrap child may never have run to arm it),
names the owed Desktop build (P4), records a `completion` follow-up and
finalizes the run success-with-follow-ups, publishes the gateway
watcher's 0, and returns exit 0 -- so the parent also resumes paused
Windows gateways the lost child never resumed. Meanings kept: Ctrl-C
stays 130 (a 130 without a result finalizes `interrupted`, not
`failed`), parked local changes stay `partial` / exit 1 (#122557), and a
child that finalized then died keeps its own terminal record.
run_completion no longer prints "✗ ... did not finish"; it hands the
reason to the caller.
Test change: test_missing_child_result_fails_boundary_receipt_and_releases_lock
pinned exit `code or 1`, receipt `failed`, .update_exit_code 1 and
resume_needed still True -- exactly the outcome invariant 3 forbids. It is
now test_missing_child_result_is_an_owed_completion_and_releases_lock
(adds a SIGKILL case): exit 0, receipt success with a `completion`
follow-up, watcher 0, gateways resumed, lock released. New: spawn
OSError is owed; a lost completion with parked changes stays partial/1.
Review P2.
The only "Desktop app build owed:" line came from source_build when the
Desktop build itself failed. A committed run whose build never ran -- the
dependency sync failed (A6), the source-update tail was refused by the
update lock, the prepared completion child died -- printed no such line,
so posix.sh / windows.ps1 reported a plain "Update complete." while the
app was stale (F6 residual).
update_completion now owns the prefix (DESKTOP_BUILD_OWED, same text the
scripts match) and _report_desktop_build_owed prints the whole line
whenever the run requested the Desktop build and the run's record has no
`build` stage (the build never reached its own verdict, where the builder
prints the line itself): in _settle_after_commit (deps owed / prepared
child died), in _complete_selected when the tail raised, and in _finish
when the tail raised before the build stage.
Review P4.
_settle_after_commit wrote <home>/.update_exit_code unguarded. An OSError
there (a home that is a file, a read-only mount, ENOSPC) escaped before
_write_bootstrap_result, so the parent saw no result and reported
"Source update completion did not finish" + exit 1 for a run whose code
was committed and whose receipt already said success.
The write now goes through _publish_gateway_success, guarded exactly like
update_cmd_fleet._write_gateway_update_exit_code (which this stdlib-only
bootstrap cannot import): it warns and the result is still written, so
the run answers 0. The gateway watcher falls back to its own timeout.
Review P1 (invariant 3).
test_web_server.py is over the 2,000-line target and may only shrink; the two
root-home update.log status cells (one changed, one added by this PR) move
unchanged into test_web_server_update_status.py.
main's code-health ratchet (4156715960) ignores `# noqa: BLE001`, and ruff does not
select BLE here, so those markers waived nothing. Each catch-all is the C3 boundary:
after the commit point the failure is a receipt follow-up, never the update's status.
The stated reason moves into `# health: allow BLE001 -- <why>` unchanged.
main moved the update-pipeline long form from hermes_cli/AGENTS.md into
developer-guide/cli-internals.md (f3af936694). The C3 contract (nothing fails
after the commit point; root-home receipts that are on disk from the start)
now lives there, and the AGENTS.md rule bullet states it in the 30k chain budget.
The parent already decodes the child's piped output as UTF-8; -X utf8 makes the child encode it
that way on any console code page. The reviewed failure was invalid on re-verification; this is
the one-token hardening the triage allowed. test_update_completion_process.py failure set unchanged.
What: build_update_products acquires npm inside the "Node dependencies" attempt and, whenever the
desktop product was requested but not built, prints `Desktop app build owed: <step> failed`.
posix.sh matches that line instead of the truncated follow-up text; windows.ps1 now projects it as
a manual outcome (it had no projection of the current follow-up at all).
Why: the 500-char follow-up leads with whichever product failed first, and an npm acquisition
failure escaped un-named, so a committed update whose Desktop was not rebuilt said "Update complete."
Red: pre-fix source_build.py -> test_unbuilt_desktop_is_named_on_one_whole_line[both] fail; pre-fix posix.sh -> truncated-followup hand-off test reports manual=false.
Green: test_source_build.py 15 passed; test_desktop_update_gateway_flag.py 3 passed. windows.ps1 needs a native run.
The behaviour stays as designed (source-update-completion.md): a per-profile sync error is printed
and not owed; only a sync that escapes the step becomes the profile_sync follow-up. The docstring
claimed every profile-sync failure was owed, contradicting the code and the developer doc.
What: _spawn_hermes_action writes the action id into the action log's start header; get_action_status
accepts a root `hermes-update completed <id>` marker only when <id> is that header's id, else falls
back to the receipt / null.
Why: the root update.log is shared by every profile, so profile A's success certified profile B's
failed (or never-started) action as exit 0 after B's dashboard lost its in-memory result.
Fixture: test_update_status_recovers_completed_result_after_dashboard_restart's action-log header
now carries the action id it recovers (its old id-less header is exactly the uncorrelated case).
Red: pre-fix actions.py/web_server_gateway.py -> new test fails (foreign completion -> exit_code 0).
Green: new test + recovery test pass; matching id still yields exit 0 and action_id.
What: OBSERVATION_SCRIPT's receipt() scans <install root>/logs/update_receipts (install_root was
already derived for the marker); status/ready/intent stay on the launch home.
Why: the CLI writes receipts under get_default_hermes_root(), so a named-profile managed SSH update
never found its correlated receipt and the Desktop reported a committed update as failed.
Red: pre-fix managed-ssh-update.ts -> new vitest case fails (reads the stale profile-path receipt).
Green: managed-ssh-update.test.ts 36/36 passed.
What: when the restart phase stops a gateway it cannot map to a profile, the owed obligation now
records it as an unmapped inventory row; _marker_owed_gateways turns that row into a sentinel that
only a live, current gateway of this checkout settles.
Why: the row-less obligation looked gateway-less at the next startup (an unmapped process leaves no
gateway_state.json), so the gateway-less discharge erased the debt while the gateway stayed down.
Red: pre-fix fleet/fleet_verify -> test_unmapped_stop_debt_survives_startup_until_a_current_gateway_runs fails (_update_owes_fleet_restart False, marker cleared).
Green: same test passes; reviewer probe unmapped_debt2 (with _pytest_owns_live_checkout stub) exits 0, inventory carries the unmapped row.
- After the completion child answers (or the A6 branch resumes), the post-commit
resume attempt is settled: its failure is already the windows_resume follow-up,
so the parent's atexit net is unregistered and the command's finally does not
replay it. A replay waited again and, on the ZIP route, its RuntimeError reached
main.py's BaseException handler: exit 1 and "/update failed".
- The ZIP and CalledProcessError finally blocks resume only what is still owed and
report a failure as the windows_resume follow-up instead of raising.
- update_finish (historical takeover) records a refused resume as a follow-up and
keeps its exit status (was code = 1), and reports the attempt as handled so the
historical parent does not replay it.
Tests (real processes, refusing SCM resume is the only service seam):
test_zip_update_never_replays_a_refused_resume_after_the_commit_point,
test_refused_gateway_resume_after_commit_is_a_followup_not_exit_1.
- A receipt store that refuses the terminal write after the commit point no
longer selects exit 1: update_receipt keeps the correlated terminal record
this process finalized (finalized_receipt) and the completion child answers
its parent from it; run_completion's correlation gate is unchanged.
- Restart failures (failed systemd unit, abort recovery, Windows service resume)
no longer write 1 to the gateway /update marker; the debt stays the
gateway_restart follow-up, the host obligation and the warning lines.
- TAIL_FOLLOWUPS includes profile_sync, so an owed profile sync keeps
source-completion-pending armed and withholds the install stamp.
Both completion paths marked the build stage failed whenever ANY tail follow-up
was owed, so a run whose only debt was config_migration (or maintenance /
dependencies) reported a failed build. record_build_stage marks it failed only
for launchers/build (what the build stage publishes) or completion (the tail
raised before proving them); other owed steps stay their own follow-ups.
- unsafe SQLite (3.46.1 interpreter) keeps exit 0 + sqlite_runtime follow-up but never
enters multiplex auto-migration; healthy 3.53.1 control does.
- a late PermissionError on the profile config write is a config_migration follow-up,
keeps the tail pending and withholds the install stamp.
- an old-schema handoff receipt keeps the previous run's manual-serve debt (1 -> 1).
- historical takeover: a real TypeScript build failure is a build follow-up, exit 0,
config migration + fleet restart still run (rewritten from the old exit-1 pin).
Round 5 (R8, R14) on the post-commit completion:
- update_completion: the completion's runtime-safety verdict reaches fleet
verification, so an unsafe SQLite runtime still vetoes gateway multiplex
auto-migration; the committed update keeps exit 0 + the sqlite_runtime follow-up.
- update_cmd_config: a failed config-format write raises into the owed-step
guard: config_migration lands on the receipt and the tail stays pending.
- update_receipt: a handed-off receipt from an older interpreter (no
carried_manual_serves) snapshots the previous receipt's manual-serve debt
before the running record replaces latest.json.
- update_finish (historical takeover child): a post-commit build failure is a
build follow-up, maintenance and the fleet restart still run, exit 0, tail
stays pending until it finishes.
The pm/receipt.py hunk is reverted (pm is not the updater's), so a sticky-profile update leaves
pm's own sync receipt under the profile, where `hermes pm status` reads it.
Contract C3 keeps post-commit step failures at exit 0, but the parked stash is not a
step anything retries: the receipt records a user_action, finalizes partial and
hermes update exits 1 (#122557). The Desktop handoff reports an owed app rebuild as
an action ("could not be rebuilt ... hermes desktop --force-build"), and the pm e2e
follows A6 (dependency sync after the tree moved is an owed follow-up).
