plugin-catalog/mem0.yaml (#131267) points at mem0ai/mem0's own
integrations/hermes-plugin-mem0 at a reviewed pin. The entry stays dormant
while a bundled copy exists (bundled providers win in find_provider_dir), so
deleting the in-tree copy is what lets hermes_cli/memory_provider_migration.py
install it for every home whose memory.provider is "mem0". The vendor copy
keeps the provider name, mem0.json, the MEM0_* keys and the setup wizard.
Also dropped: the hermes-agent[mem0] extra, its win-arm64 platform row and
exclude-newer row, the pm/extras.py anchor (uv lock removes only mem0ai,
posthog, qdrant-client and sqlalchemy), the mem0-specific doctor check (the
generic check now names `hermes plugins install mem0`), and the tests that
imported plugins.memory.mem0. Prose that lists the bundled set now names mem0
as a catalog provider. MEM0_* env hints and redaction stay: the catalog copy
still reads them.
While this branch was in review, #132386 landed on main: a committed
`hermes update` that still owes work prints one whole "Desktop app build
owed:" line and one "Update follow-up '<step>' did not finish:" line per
owed step, and both Desktop hand-offs turn those into a manual outcome
naming the step (never a plain "Update complete."). This branch rewrites
both hand-off scripts, so the rebuild took its versions whole and this
commit carries main's behaviour into them, through this branch's own
post-commit design:
- posix.sh: owed_followup_steps() (unchanged from main: the parser contract
test runs it), the build-owed and owed-steps checks on a zero exit, each
setting DONE_NOTE (finish() makes it manual) and an add_warning entry so
the step also reaches the result's warnings; the Linux relaunch gate now
appends to DONE_NOTE instead of replacing an owed-step note.
- windows.ps1: the same two checks as Add-Followup -Manual calls before the
runtime verify, with main's exact [regex]::Matches expression.
Already equivalent here: the mac swap is skipped when the rebuild failed,
and finish() relaunches the kept app for a manual note unless the Linux
gate says skew/manual.
Before: 6 of main's contract tests fail on the rebuilt branch
(test_every_owed_followup_reaches_the_desktop_result x4, the truncated
build-owed case, the parser contract). After: tests/scripts/desktop_update/
plus test_update_post_commit_followups.py pass, 113 passed, 0 failed.
Within moments of its claim the daemon hands marker line 1 to the custodian it
forked (fdab580431e). The launcher's start check (contract C2) still waited for
the daemon's own pid, so on a host where the handover beat its next 0.1 s poll
it never matched, and once the daemon finished it reported a running update as
a failed launch (exit 70). `marker_names_handoff` now accepts line 1 naming the
daemon or a process the daemon forked; a grandchild or an unrelated pid still
does not count.
Sabotage A/B (daemon held alive 2 s after the custodian takeover, real
hand-off via the shim-progress harness): old acceptance exits 70, new exits 0
and publishes the stage. Test: the predicate over custodian/daemon/grandchild/
unrelated, red on the previous marker.sh.
Same gap as the POSIX hand-off: the -MarkerOp custody watcher took line 1
over only after windows.ps1 exited (WaitForExit, then the marker lock), so
an old packaged Desktop reading in between (lines 1 and 2 only) deleted the
marker while the resumed `hermes update` or a completion survivor still
held the checkout.
Start-MarkerCustodian now names the watcher on line 1 under the marker lock
right after starting it, before any update work, and the hand-off judges
its claim as that watcher from then on ($script:MarkerOwner in
New-MarkerContext): delegate publication, heartbeat and release keep
working. Invoke-MarkerRelease stops the watcher only after the release, so
line 1 stays live until then. Invoke-MarkerCustody keeps a claim that
already names it (judged by the creation time the hand-off recorded) and
falls back to the old takeover only when the handover failed.
Tests (Windows CI lane): the kill cell holds the marker lock across
taskkill /F and runs the shipped legacy reader at once; the pre-publication
kill cell and the A7 race expect the custodian on line 1.
The previous fix let the line-2 refresher take the marker over only AFTER
the hand-off died: it polls once a second and then waits for the A7 lock,
and until it rewrote line 1 the marker named the dead hand-off. An old
packaged Desktop (judges lines 1 and 2 alone, ignores A7 and the delegate)
reading in that gap deleted the marker and booted its backend while the
`hermes update` the hand-off started, or a completion survivor, still held
the checkout lock -- and the custodian then found no marker to keep.
Now the custodian is named on line 1 BEFORE any update work starts:
marker_refresher_start forks the refresher, then, under A7, rewrites the
claim's line 1 to the refresher's exact pid + creation time and adopts that
identity (MY_PID/MY_CT) for every later marker mutation (delegate, refresh,
release). posix.sh checks its own liveness with $$. The refresher keeps
line 2 young; if the hand-off dies it keeps the marker until the delegate
and the checkout lock are gone, then releases it. The hand-off releases the
marker before it stops the custodian, so line 1 stays live until the
release. If the handover fails (A7 busy), the old takeover-on-death path
still applies.
Test: the orchestrator-killed cell now holds the A7 lock across the SIGKILL
and runs the shipped legacy reader at once (no takeover is possible), then
keeps the post-takeover phases. Red on 5f855a2392c: {'live': None,
'kept': False}. Tests that asserted line 1 == the hand-off pid now expect
the custodian (that assertion encoded the vulnerable state).
marker.sh::checkout_lock_held and marker.ps1::Test-CheckoutLockHeld answered
"not held" when the lock file exists but cannot be opened (root-owned after
`sudo hermes update`, or held open without sharing on Windows), while the
Desktop's own probes (checkout_held / WINDOWS_CHECKOUT_LOCK_PS) answer "held".
The script helper's reclaim then deleted a dead marker the Desktop's probe
keeps. Both scripts now fail closed on every open/probe error except a
missing file: reclaim needs a provably free checkout. Parity tests: the
POSIX probe with perl, python3 and flock(1), plus an end-to-end reclaim; the
Windows reclaim against a lock file held with no sharing (Windows CI lane).
Same shape as the POSIX hand-off: windows.ps1 heartbeats line 2 from its
own process, so `taskkill /F` (no finally runs) left line 1 naming a dead
pid while the resumed `hermes update` (the line-4 delegate) or a completion
survivor still held the checkout, and an old packaged Desktop - which judges
lines 1 and 2 alone - deleted the marker and booted its backend.
After its claim the hand-off starts a hidden watcher (-MarkerOp custody
-CustodyOf <pid>, Start-MarkerCustodian). Once the hand-off exits with its
claim still published and the delegate running or the checkout lock held,
the watcher names itself on line 1 under the marker lock, heartbeats line 2
and releases the marker when both end (bounded like the R6 wait).
Invoke-MarkerRelease stops the watcher once the hand-off's own release is
decided. The Windows kill-cell test now also runs the reader be3fd671d7
shipped; that reader runner moves to a helper shared with the POSIX test.
An older packaged Desktop judges the update marker by line 1 (pid alive) and
line 2 (under 20 minutes) alone and unlinks it otherwise, never consulting
the protocol-2 delegate line or the marker helper. The line-2 refresher
exited with the hand-off script, so a SIGKILLed hand-off left line 1 naming
a dead pid while the `hermes update` it started, or a completion survivor,
still held the checkout lock: the old Desktop deleted the marker and booted
its backend into a checkout still being mutated.
The refresher now outlives the hand-off as its custodian (marker_custody):
once the hand-off is gone, if its claim is still published and the delegate
runs or the checkout lock is held, it names itself on line 1 (exact pid +
creation time, under the A7 lock), keeps line 2 young, and releases the
marker only after both are gone (bounded like the R6 wait). The test runs
the reader be3fd671d7 shipped (vendored verbatim under tests/fixtures).
The hand-off scripts gave the Desktop a fixed 30 s (windows.ps1) or 60 s
(posix.sh) to exit, but the Desktop's quit can first join a managed SSH update
it is running, which legitimately takes longer. The hand-off then recorded a
failed update although the Desktop exited safely moments later (review H2).
Both twins now share one 150 s ceiling, measured on the clock rather than as a
loop count, and still refuse with "Nothing was changed" if the Desktop is alive
at the ceiling, so nothing is ever replaced under a live Desktop. The ceiling is
overridable through HERMES_UPDATE_DESKTOP_EXIT_SECONDS only so the self-tests
need not sit it out, like the existing pipe-drain and step-idle overrides.
Tests: a Desktop still alive at the ceiling refuses with exit 4 and never runs
the update (red on the start sha, which waited out its hardcoded 60 s); a
Desktop that exits inside the ceiling lets the update run (control).
update-job.ps1 was one top-level `if (-not type) { Add-Type ... }` block.
It is now a named function, Initialize-HermesUpdateJob, that returns early
when the type already exists and is called once at the end of the file, so
the dot-source in windows.ps1 behaves exactly as before. The C# body is
unchanged byte for byte.
Checked with PowerShell 7.4 on Linux: dot-sourcing the file twice compiles
the type once and HermesUpdateJob::ProcessIds(0) answers; windows.ps1 still
parses with zero errors. Windows PowerShell 5.1 itself was not run here.
Review 5411223284 (shape: update-job.ps1 top-level code).
The script helper's `reclaim` answered `absent` the moment the marker file
was missing, before the checkout lock was looked at, and the Desktop gate
(update-marker-gate.ts liveMarkerProbe) returned "nothing runs" for a
missing marker without asking the helper at all. An update that holds the
checkout lock before it publishes its marker, or after it retired it (the
CLI paths of the stacked PRs do both), therefore let the Desktop boot a
backend into a checkout still being written.
- marker.sh marker_op_reclaim_locked / marker-claim.ps1 Invoke-MarkerOp:
no marker -> `held` when the checkout lock (owner byte + lease bytes on
Windows) is held, else `absent`, under the same marker lock.
- update-marker-gate.ts (gate-lane file, additive): a missing marker is also
routed through the helper; only its positive `held` keeps the gate shut
(blocked screen, Retry, Start anyway as for a dead marker). Any other
answer, `error` included, proceeds as before, so a broken script never
blocks a markerless boot.
The gate test "a live owner never reaches the helper; an absent marker
neither" pinned the missing check for the absent half; that half is now the
new "no marker: only held keeps the gate shut" case.
Review 5411223284 (open thread: update-marker-gate.ts ~193 / helper absent).
Both hand-off scripts treated ANY nonzero `hermes update` exit as ok:false
"Update failed". After the commit point `hermes update` can still exit
nonzero by design -- 130 (Ctrl-C after the code moved: receipt outcome
`interrupted`) and 1 (a parked autostash: `partial`) -- or crash after a
committed run (`success`). The Desktop then said the update failed while the
new code was installed (contract C3 / invariant 3).
The scripts now hand the update a correlation id (HERMES_UPDATE_CORRELATION_ID,
already honoured by update_receipt._launcher_correlation_id; a caller's own id
is kept) and, on a nonzero exit other than 2 (refused), read the root's
logs/update_receipts/latest.json. Only THIS run's finalized receipt counts
(matching correlation_id, finished_at set, outcome success|partial|
interrupted); a record reconciled as interrupted keeps finished_at null and
never matches. Then the result is ok:true, manual, with an `update` warning
naming the exit and outcome, and the post-commit steps (Windows: verify,
gateway start --all) still run.
posix.sh also routes its own signal handler through the same judgement: a
HUP/INT/QUIT/TERM that lands after the update committed (exit 0, or a
committed receipt) is an owed `handoff` follow-up, exit 0, not
"hand-off was interrupted".
Tests: posix live cells (RED at start) for interrupted/130, partial/1,
success/1 and an INT delivered while the committed child finishes, plus
negative controls (no receipt, failed, another run's receipt, a reconciled
unfinished record); the Windows twin cells.
review S5
posix.sh::finish and windows.ps1's finally write the result (finished_at =
now) BEFORE the R6 release wait, which lasts up to 7200 s while a survivor
holds the checkout lock, and before the relaunch. The relaunched Desktop has
no expected run id and handoff-result.ts drops a non-manual result older than
30 min, so a failure or warnings result was silently discarded after a long
wait.
The early write stays (a kill during the wait still leaves a result); when the
wait actually ran, the result is published again after it with the real
finish time, unless something already consumed it. The release records that
it waited (RELEASE_WAITED / $script:MarkerReleaseWaited).
Tests: posix live cell (RED at start: finished_at 3 s before the survivor
let go) and the Windows twin with a timed checkout-lock holder.
review S2
Write-Result writes <result>.<pid>.tmp and then Moves/Replaces it over the
result. When both fail (result path locked, a directory, ACLs) the catch only
logged, leaving one tmp file per hand-off beside the result. The tmp is now
removed in a finally (it is gone already after a successful publish), the
same as posix.sh's write_result.
Test (Windows live cell): with the result path occupied by a directory the
hand-off still exits 0 and leaves no .hermes-update-result.json.*.tmp.
review S4
json_escape escaped only \ " \n \r \t. Every result/status string it builds
(FINAL_MSG, branch, channel, warnings) can carry text from git/ps/hermes
output; a raw ESC, BEL or any other char below 0x20 made the whole
.hermes-update-result.json invalid, and the Desktop dropped the outcome.
The remaining control chars are now written as \u00XX. Bash 3.2 compatible:
printf -v, no forks; the loop runs only when the string holds a control char.
Test (RED at start): json_escape over chars 1..127 plus UTF-8 and an ANSI
colour sequence round-trips through json.loads, in the C and C.UTF-8 locales.
review S7
Two parity gaps between marker.sh and marker-claim.ps1:
- adopt-run (--handoff-run), withdraw's `withdrawn` and `taken` accepted
J_OWNER_STATE==2, which also covers "alive, creation time unreadable, line 2
younger than 20 min". marker-claim.ps1 requires Test-ProcessIdentityExact
(both creation times known, within 2 s). An unreadable ct is no proof the
pid is still that process; these three now require ident_state 0, and
`taken` skips our own pid like the ps1.
- reclaim / withdraw printed `reclaimed` / `withdrawn` after an `rm -f` that
exits 0 even when it cannot unlink; the Desktop then proceeded as if the
marker were gone. marker_remove checks the marker is gone; else `busy`, the
ps1's answer when Remove-MarkerLocked fails.
Tests (RED at start): the three identity cases with every foreign ct
unreadable (and the readable control), and a reclaim inside a read-only
home that must answer `busy` and keep the marker.
review S3
Get-LiveProcessCt caches a pid's creation time for the script's lifetime and
forgets it only when a probe SEES the pid dead. A delegate (the `hermes update`
child) that exits and whose pid is reused between two probes keeps the old
incarnation's time, so the release judged the unrelated new process live and
handed the marker to it (A7 rule 5), parking every later update behind it.
Every judgement made through New-MarkerContext (claim, delegate, heartbeat,
release) and the delegate line Add-MarkerDelegate writes now pass -Fresh,
which re-reads the time. Pollers that only ask "still alive?" (Wait-DesktopExit)
keep the cache.
Test (Windows live cell): a release with the delegate's pid cached at the
recorded ct but really owned by another incarnation deletes the marker instead
of handing it over. marker.sh reads /proc or ps on every call: no cache.
review S6
marker.ps1::Test-CheckoutLockHeld probed only the owner byte at 1 MiB. Since R5b
every joiner the kill-on-close job refused also locks one of the 16 lease bytes
after it, and hermes_cli/update_lock.py::_try_lock treats any held lease as a
held lock. With the owner dead and a leased completion child still running, the
hand-off read the checkout as free, so the R6 release wait ended and the
Desktop's reclaim deleted the marker next to a live mutator.
Lock/Unlock now cover 1+16 bytes in one LockFile (it fails if any byte is held).
marker.sh needs no change: POSIX takes the checkout lock with flock on the whole
file and the lease bytes exist only on win32 (_take_lease).
Tests: a live Windows cell holds only a lease byte (1 and 16) and expects
`held`; a static cell, runnable on Linux, pins the probed range (RED at start:
(1048576, 1)).
review S1
What: run_bounded's no-perl branch starts the probe under bash job control (set -m), so it gets its own process group and the timeout's kill -KILL -- -pid reaches its children.
Why: without perl the probe stayed in the caller's group, the group kill failed, and a hung hermes --version / update --help left its python child running.
Red: pre-fix posix.sh fails test_timed_out_probe_is_killed_with_its_whole_process_tree[absent] (grandchild alive after rc=124). Green: both [present] and [absent] pass.
Review R8 m7: after the merge with main's 732b63d61a the probe decodes launcher
stdout as UTF-8 through ProcessStartInfo.StandardOutputEncoding and never touches
[Console]::OutputEncoding, so the test docstrings no longer claim the call sites
scope and restore it. runtime.ps1 keeps a one-line #124526 pointer at the
encoding line. Behaviour and assertions are unchanged.
The corpus now pins line 2 to digits that fit u64 (any count) and a pid to
digits that fit u32 after leading zeros. bash took a line 2 past u64 as a
far-future start (live); PowerShell rejected any line 2 over 18 digits
(so u64::MAX was malformed) and any pid over 10 characters (so a
zero-padded pid was malformed), and Windows PowerShell's [double]::Parse
throws on a creation time past double's range.
bash compares a 20-digit line 2 as two 10-digit halves (never in signed
shell arithmetic); PowerShell strips leading zeros before the width checks,
compares 20 digits as a decimal, and reads an out-of-range ct as +Infinity,
which never matches a live process. Rewrites carry line 2's exact digits;
only the age arithmetic clamps.
(cherry picked from commit 7208aa4af3c9e355dadee970dddb730fcfeb0301)
The two scripts disagreed on an old packaged Desktop's v1 bridge (round 5
D11): bash adopted a live launcher that is our parent but no longer the
Desktop's child when line 2 equals HERMES_UPDATE_STARTED_AT (the Desktop
quit first and the launcher was re-parented); PowerShell refused it, also
accepted a v2/delegate claim naming its parent, required a gone launcher to
be its recorded parent, and Trim()/TryParse'd the env value.
One rule now, as a pure function in both (marker_launcher_rule /
Test-MarkerLauncherRule) over the same facts, plus one env comparison
(plain ASCII digits of line 2's value). tests/scripts/desktop_update/
lineage_rule_cases.py is the shared table; the posix and Windows suites run
it, and a real-process cell per platform covers the divergent row.
(cherry picked from commit aa4b8ce704bc474d539690235c191edee3b637f2)
checkout_lock_held took the real checkout lock through flock(1) on a shell
fd and released it only when the shell closed that fd: the hold spanned
flock(1)'s exit, the wait and the close, so a concurrent `hermes update`
acquiring non-blocking was refused 223 of 3157 times in the reviewer's
tight-loop probe (round 5 D17). Now one process (perl, else python3) opens
the file, tries LOCK_EX|LOCK_NB and drops it in the next syscall: 0 of
3387. flock(1) stays the last-resort fallback; no tool still means held.
The old-Electron line-2 refresher ran only while `hermes update` ran, so
an old packaged Desktop could age-delete the marker 20 minutes into the R6
release wait (round 5 D18). The refresher now lives from the claim until
the release is decided (posix.sh), and the Windows heartbeat ticks inside
Invoke-MarkerRelease's wait. Tests shorten the cadence with the test-only
--self-test-refresh-every argument.
(cherry picked from commit 8339b25e9cfbd13f9f3e8f606e703d994b8b5410)
Release waits (bounded) while any process still holds the checkout lock's
byte (hermes_cli/update_lock.py takes it with msvcrt.locking at 1 MiB), and
-MarkerOp reclaim answers `held` instead of freeing a dead marker in that
state, so a Desktop gate never opens over a surviving completion. posix.sh
withdraw now keeps a dead adopter's claim (`foreign`), like marker.ps1.
(cherry picked from commit 8a5525d26f036a3c4b5176f11ee4a590c746e2db)
Invoke-HermesProbe handed a .cmd launcher to Process.Start with
UseShellExecute=false, letting CreateProcess pick the interpreter and
re-parse the arguments. Run it through ComSpec explicitly like the update
step does, and refuse arguments cmd.exe cannot be made to take literally.
(cherry picked from commit 0b24aab96962535268d71711c0ef564e01ac6e11)
Every marker mutation (reclaim, adopt, delegate, release/hand-over,
withdraw, line-2 refresh) now happens inside one hold of
[IO.File]::Open("<marker>.lock", OpenOrCreate, ReadWrite, None), retried
for 10 s then BUSY (claim refused). Read->judge->mutate all inside the
hold; the sidecar is never deleted. Replaces the compare-then-replace /
compare-then-delete helpers.
- parser/judge per tests/fixtures/update_marker_corpus.json (trimming,
BOM/CR, u32 pids, pid 0 dead, delegate/run on any line >= 4, own-pid
exact incarnation, 2 s tolerance, v1 1200 s ceiling)
- release per corpus: owner deletes or hands the claim to a live
delegate; a delegate releaser drops its line
- -HandoffRun <id>: adopt only the Desktop bridge carrying run:<id>
- -DesktopPid alone (old packaged Desktop): accept its cmd.exe wrapper's
v1 overwrite by lineage, never by dropping the check
- -MarkerOp reclaim|withdraw: one verdict line, exit 0 (64 usage)
- `# hermes-handoff-protocol: 2`
- the update child is created suspended in a kill-on-close job, named as
the delegate under the lock, then resumed: no update instruction runs
before (or without) the delegate line
- line 2 refreshed every 5 min while the child runs (old readers)
- the job helper moves to update-job.ps1 (windows.ps1 < 2000 lines)
(cherry picked from commit 8bdf467c768da314ce7817f114a6b1b154e11154)
marker.sh (sourced by posix.sh) owns the update marker: corpus parser and
judge, claim, delegate, release and the line-2 refresh, each read -> judge ->
mutate inside one flock(1) / perl Fcntl hold of <marker>.lock (busy 10 s
fails closed). Two live claimants of one dead marker no longer both win (R3).
Hand-off protocol 2 (`# hermes-handoff-protocol: 2`): --handoff-run R adopts
only the Desktop's live bridge carrying run:R; an older Desktop's bridge (its
launcher's pid, v1) is adopted by lineage -- the named pid is our parent and
the Desktop's child, or carries HERMES_UPDATE_STARTED_AT -- never by dropping
the check (R4). --marker-op reclaim|withdraw gives the Desktop one verdict
line; it never mutates the marker itself.
Release waits while any survivor still holds the checkout lock and the
reclaim helper answers `held` for a dead marker in that state (R6). The
update child is gated on a go-file created only after the delegate line is
published (pre-publication crash window), run_bounded kills the probe's
whole process group, and line 2 is refreshed every 5 min while ours so an
older Desktop's 20-minute age ceiling never deletes a live claim.
(cherry picked from commit f16dcc5583df8294cea96e6941cfdf5d4a7c8078)
A daemon killed in the second between spawning `hermes update` and that
child taking the update lock left a marker whose only identity was the dead
script: every reader judged it DEAD and a second update could start over the
first. The child now runs asynchronously just long enough for its pid to be
written as line 4 (`delegate:<pid> ct:<ct>`, compare-and-swap on our own
lines 1-3), then is waited on; HUP/INT/QUIT that land meanwhile are handled
after it exits, exactly as with the former foreground child.
(cherry picked from commit a09d2fb7cac0d730b491560ea0f970e5d227421e)
- marker helpers move to scripts/desktop-update/marker.ps1, dot-sourced
first (windows.ps1 2113 -> 1930 lines); a hand-off that cannot load it
exits 3 having changed nothing. It ships in the checkout next to
windows.ps1 and runtime.ps1, which is where the Desktop runs them from.
- creation time comes from Win32_Process.CreationDate (limited query
rights), cached per pid while it lives: Get-Process .StartTime is access
denied for SYSTEM/elevated/other-user pids under PowerShell 5.1, so a
pid reused by a service after a reboot read as a live owner forever.
Alive with an unreadable time is live only inside the 20-minute ceiling.
- -DesktopPid runs adopt the Desktop's live bridge claim only; a refused
run exits 2 without a result file and only reopens a Desktop that is gone.
- positional parse identical to the other readers; claims publish the
complete body with an exclusive hard link (rename fallback); a fresh
empty marker is a claim in flight; the delegate line always carries ct.
- a follow-up step whose process tree cannot be quiesced after the commit
point is ok:true + a manual warning (no relaunch, marker kept live),
not "still on the previous version".
- the interrupted post-update tail is always a manual warning: re-running
hermes update is the only remedy, so the copy says exactly that.
- the launcher probe bound is a -ProbeTimeoutSeconds parameter (default
60), not the HERMES_UPDATE_PROBE_TIMEOUT_SECONDS environment variable.
(cherry picked from commit 65663dc6a64f942c6385b7de3ba13cdecd393387)
Contract amendments A1-A4, posix side:
- --desktop-pid runs only adopt that Desktop's live bridge claim; with no
bridge (the Desktop gave up after its 20 s wait) or anyone else's marker
the script exits 2 without claiming, running, or writing a result
- a refused hand-off (exit 2) never writes .hermes-update-result.json; the
other update owns the outcome, and a stale "still on the previous
version" dialog could otherwise follow a successful CLI update
- positional parse: BOM/CRLF accepted, non-integer line 2 = malformed
(dead), bad ct line = v1, line 4 only as "delegate:<pid> ct:<ct>"
- a live pid whose creation time cannot be read is live only inside the
20-minute ceiling (unknown never means forever)
- an empty marker younger than 5 s is a claim in flight
- macOS ps lstart is rendered and parsed with TZ=UTC0 (DST-safe)
(cherry picked from commit 9d8b58063f0c2d8de6bc37b615e6e49f1abb7549)
windows.ps1 implements update-marker contract C1 v2 and the C3 outcome
vocabulary on the Windows Desktop hand-off:
- The marker claim is the script's first action (before any Add-Type, UI
or probe): exclusive create (tmp + File.Move, never overwrites) with a
v2 body "<pid>\n<started_at>\nct:<creation time>\n". A live marker owned
by exactly -DesktopPid is adopted by compare-and-swap keeping its
started_at; any other live owner refuses the run (exit 2, marker
untouched, "Nothing was changed"); a dead owner (pid gone, creation time
mismatch = pid reuse, v1 older than 20 min) is compare-and-deleted and
the claim retried.
- Release is compare-and-delete on lines 1-3 and keeps the marker while a
line-4 delegate is alive. When a stalled tree cannot be quiesced (exit 7)
the marker gains a delegate line naming a surviving job member
(JobObjectBasicProcessIdList), so it stays live exactly as long as the
tree does instead of naming the dying script.
- After `hermes update` exits 0 (or the 124 banner remap) the result is
ok:true, exit_code 0: verify failure (was exit 8), legacy rebuild
failure (was exit 6), a gateway restart miss, an interrupted post-banner
phase and any unexpected throw become `warnings` + manual, with a
"Hermes was updated, but ..." message; gateway start --all still runs
after a failed verify/rebuild.
- The idle watchdog counts job CPU/IO accounting deltas as progress, so a
pipe-silent but busy step (download, extraction) is not killed.
- Launcher probes (--print-runtime-command, --version, update --help) run
with a 60 s timeout and kill their tree on expiry.
- Desktop exit wait checks pid + creation time pinned at start, and a
still-open Desktop is never relaunched over (exit 4).
- The result file is written atomically, carries started_at (marker line
2) and warnings, and is no longer deleted at hand-off start.
- Stop-UiServer only removes its own profile dir and dirs whose pid is not
running (TEMP is shared across HERMES_HOMEs).
(cherry picked from commit af178fd695d7f94bf584bfabe9748cbb752fea13)
posix.sh now implements the update-marker contract (C1/C2) as a lock:
- the daemon claims `.hermes-update-in-progress` as its first action with a v2
body (pid, started_at, `ct:` creation time) via an atomic no-replace create
(link(2)); a LIVE foreign owner refuses with exit 2 regardless of age, a
marker owned by --desktop-pid is adopted keeping its started_at, a dead one
is compare-and-deleted; liveness honours a line-4 `delegate:` (the running
`hermes update`), so an orphaned update is never reclaimed;
- the launcher waits for the daemon's claim and exits non-zero when it never
lands, so the Desktop does not quit into nothing;
- release is compare-and-delete and keeps a marker whose delegate is alive.
Outcomes follow C3: a legacy rebuild failure, a mac bundle stage/swap failure
or a relaunch gate after `hermes update` exited 0 is ok:true + warnings (no
more exit 6 / exit 7 "still on the previous version"). The result carries
started_at and warnings and is written via a per-pid tmp + rename.
Crash safety: start-of-run recovery restores an interrupted .app swap from its
aside copy (both swappers' suffixes) and a killed TCC heal backup, and sweeps
UI/probe litter of dead pids; the two swap renames are a signal-deferred
critical section. `hermes desktop`'s Python swappers recover the same state
instead of deleting the only copy first.
Also: probes are bounded (60 s), `hermes update` streams into the log, the
update child gets default TERM and the daemon default HUP (no SIG_IGN leak into
gateways or the relaunched app), the relaunched Linux app drops the update env,
the Desktop wait is zombie/pid-reuse aware and never relaunches over a live
app, serve-ui.py exits with its orchestrator, and the job section is parsed as
one brace group so `hermes update` rewriting posix.sh cannot shift the rest.
(cherry picked from commit c7dd9fab00b8af09321ec982d07a62f03351355d)
The protect-main ruleset requires a check by NAME, and a green check run on a
PR head never expires. After the code-health ratchet joined the required set,
32 PRs whose last CI run predated it merged on their old "All required checks
pass" green without the ratchet ever running; 20 of them would have been
blocked (62 findings, incl. 15 new blind `except Exception`).
Renaming the gate to "All required checks pass (v2)" and pointing the ruleset
at the new name voids every older green: each open PR needs one fresh CI run,
which now includes the ratchet, before it can merge, whoever merges it.
The ci.yaml comment documents the bump as the standard step whenever the
required set gains a job every PR must pass.
live_comment.py keeps both names in its infra list (old runs still render),
and a test pins that the gate's current display name never shows as a job.
A committed update exits 0 and prints one whole "Update follow-up '<step>'
did not finish:" line per owed step (update_receipt.record_followup). The
hand-offs only reacted to the Desktop build line, so a gateway proven to
run old code (gateway_restart), a refused Windows resume, a lost
completion or a failed channel adoption ended as a plain
"Update complete.".
posix.sh collects the distinct owed steps from that line
(owed_followup_steps) and turns them into the manual result message
(DONE_NOTE, the only channel the main-based script has); a gateway_restart
adds the `hermes gateway restart` hint. windows.ps1 does the same through
its manual outcome. With #132354's add_warning protocol the same
owed_followup_steps output feeds one add_warning per step.
record_followup's docstring names the line as a protocol; a contract test
runs the real posix.sh parser and the windows.ps1 regex against it.
Review: kshitijk4poor 5411222454 regression 1 / open thread F6.
What: build_update_products acquires npm inside the "Node dependencies" attempt and, whenever the
desktop product was requested but not built, prints `Desktop app build owed: <step> failed`.
posix.sh matches that line instead of the truncated follow-up text; windows.ps1 now projects it as
a manual outcome (it had no projection of the current follow-up at all).
Why: the 500-char follow-up leads with whichever product failed first, and an npm acquisition
failure escaped un-named, so a committed update whose Desktop was not rebuilt said "Update complete."
Red: pre-fix source_build.py -> test_unbuilt_desktop_is_named_on_one_whole_line[both] fail; pre-fix posix.sh -> truncated-followup hand-off test reports manual=false.
Green: test_source_build.py 15 passed; test_desktop_update_gateway_flag.py 3 passed. windows.ps1 needs a native run.
Contract C3 keeps post-commit step failures at exit 0, but the parked stash is not a
step anything retries: the receipt records a user_action, finalizes partial and
hermes update exits 1 (#122557). The Desktop handoff reports an owed app rebuild as
an action ("could not be rebuilt ... hermes desktop --force-build"), and the pm e2e
follows A6 (dependency sync after the tree moved is an owed follow-up).
Every core spec drives open, resume and switch, yet changes under
apps/desktop/src/app/session/ skipped the suite on the PR; #132017
changed resume that way and main went red on remote-secondary.
The reuse receipt check in node-deps.mjs accepted a tree where every locked
directory existed but package contents had been stripped (partial cleanup or
snapshot restore). npm trusts the hidden lockfile and never repairs such
entries, so every 'hermes update' skipped npm ci, the TUI build died on the
missing esbuild manifest, and the stale receipt kept the loop closed (#128935).
Probe each locked entry's package.json — the same file product builders
resolve tools through — so a gutted tree falls back to a real npm ci.
Fleet telemetry shows built-in memory writes failing 34-47% and automatic
compression failing ~15%, and several merged fixes each target one cause
(#133038 for memory, #133054/#133040/#133051/#133032 for compression). Both
metrics only carried outcome, so the fleet could not tell which cause a fix
moved.
- hermes.memory.op.count and hermes.compression.count gain one closed-enum
dimension, failure_class; `none` on success.
- Memory: every refusal/failure return in tools/memory_tool.py and
tools/memory_tool_store.py names its class (no_match, ambiguous,
over_budget, would_empty, drift, read_failed, retry_cap, scan_blocked,
missing_content, missing_old_text, invalid_args, stale_entry, disabled,
staged, gate_refused) through a context variable; the result the model
sees is unchanged. Plugin providers report provider_error / exception.
- Compression: the attempt log's own failure_class, collapsed onto the
compressor's closed names (exception:<Type> / rollback:<Type> keep only the
prefix, anything else reads other, a classless failure unknown). Skipped
rows carry the skip reason, and the skip set is now defined once in the
contract.
- Package schema v4 / event schema v4; v3 field sets of both metrics stay
valid so pending rows drain. Doc table rows and the consent copy (CLI
setup and every Desktop locale's collectedActivity) name the new reason.
A Desktop built before the blobless fix keeps walking history during
`hermes update`. On a treeless (tree:0) checkout every walk downloads trees
again, and #129514 filled a 434 GB disk during the minutes of dependency work.
The one-time conversion (#133016) ran at the very end of the update, after that
window. Both update hand-offs (update_completion._prepare and an older
updater's _update_takeover.prepare) now run it first, in the bootstrap
interpreter, and the late call is gone.
Re-running install.sh / install.ps1 over a treeless checkout now converts it
too. The filter is recorded only after the refetch succeeds, so a failure is
retried by the next update instead of leaving a blob:none filter over a
history that has no trees.
The pack cleanup also sweeps tmp_* transfer files that killed fetches left
behind. They are more than an hour old, a storm leaves gigabytes of them,
and git itself only prunes them after two weeks. The docs gain the recovery
steps for a disk that the storm already filled.
Co-authored-by: Brooklyn Nicholson <brooklyn.bb.nicholson@gmail.com>
Co-authored-by: Yuan Li <dskwelmcy@163.com>
skills.sh rows in the hermes-index carry the skills.sh slug as `path` and,
since mid-September, never a `resolved_github_id`: the scheduled index build
resolves paths AFTER the ClawHub walk, ~80 min into the job, when the 1 h
GitHub App token has expired. Every tree fetch 401s and the log says
"Resolved 0/20000 paths" (last good run 2026-09-15: 17917/20000). Installs
then work only where the repo dir happens to equal the slug, and everything
else is reported as a "stale index entry".
- build_skills_index: resolve skills.sh paths straight after the skills.sh
crawl, while the token is fresh.
- HermesIndexSource.fetch: when the indexed candidates miss, locate the skill
in the repo tree (slug-named dir, a lone SKILL.md in a generic dir such as
`skills/`, or the repo root) and install it under the slug, not as
`skills/`.
- _find_skill_in_repo_tree: a tree with exactly one non-root SKILL.md
resolves to it when no dir matches the slug (fails closed on 2+).
- GitHubAuth: once the credential chain came up empty, stay anonymous for the
instance instead of re-running `gh auth token` before every API call; with
gh installed but not logged in that cost up to 5 s per request and pushed an
anonymous install past four minutes.
Fixes#130129
Based on #130135 by @dskwe.
publishDirectory moved a finished product into place with one renameSync.
On Windows an antivirus scanner opens every freshly written file, so the
rename onto build/native-deps (right after hud-modifier-monitor.exe is
compiled) or hermes_cli/web_dist failed EPERM and threw away a build that
took minutes.
retryHeld retries EPERM/EACCES/EBUSY for about 3s and raises every other
code at once. All three publication renames use it, as does the HUD
helper's staging rename. The previous product's backup now sits beside
the output instead of inside withProduct's scratch tree, so a rollback
that also fails no longer has its only copy deleted by scratch cleanup;
the error names where it is.
Co-authored-by: dskwe <6286305+dskwe@users.noreply.github.com>
Co-authored-by: juliorobertoalvarez <39749815+juliorobertoalvarez@users.noreply.github.com>
Co-authored-by: bighamx <10262931+bighamx@users.noreply.github.com>
Co-authored-by: lzmuhioin <1814826+lzmuhioin@users.noreply.github.com>
Installers have cloned with --filter=tree:0 since c3e9ac87a7. A treeless
clone keeps commits only, and git requests a missing tree without telling
the server which trees it already has, so every checkout and every
path-filtered history walk re-downloads complete directory snapshots
(~0.58 MB per commit for this repo). GitHub's own guidance is that
treeless clones are for throwaway CI builds, not long-lived checkouts;
the blobless layout keeps every commit and tree local and fetches only
file contents on demand.
- install.sh / install.ps1 clone --filter=blob:none (same release tags,
--commit pins and ancestor checks the treeless switch was made for).
- hermes update (post-update step) converts an existing tree:0 checkout once with
`git fetch --refetch --filter=blob:none origin` (~120 MB, ~8 s
from GitHub) and records the new filter only after it succeeds, so a
failed refetch retries next update.
- The shallow-heal conversion unshallows as blob:none instead of tree:0.
- checkout_contains asks git with GIT_NO_LAZY_FETCH: a recorded SHA the
checkout lacks is "not contained", never a download of that commit's
whole history.
scripts/ci/check_agents_md_size.py (in scripts/check and lint.yml): root <= 12k, every
AGENTS.md plus its ancestors <= 30k (the joined root->cwd block on a 128k window, minus
the per-file labels), every area file <= _MAX_HINT_CHARS read from the source so the two
cannot drift. On origin/main it reports the root and all 11 root-to-area chains over.
The root file is project context and was 38.7k chars, past the context-file budget
(20k floor, 30,720 on a 128k window), so every session lost its middle: routing table,
testing rules, code-shape rules. It now keeps the two invariants, a condensed rubric
(every rule kept), environment, structure, one-line rules that apply everywhere and the
routing table (11.8k).
Moved verbatim to where they load: Testing -> new tests/AGENTS.md; dependency pinning and
PM ownership -> new pm/AGENTS.md; machine facts -> new hermes_platform/AGENTS.md;
session-scoped tool availability -> tools/AGENTS.md; the full rubric with its real-close
examples -> developer-guide/contributing.md. Pointers in lint, checker messages, docs
and test comments name the new homes.
On a case-insensitive filesystem (Windows, macOS) `scripts/check --staged`/`--commit`
and the hooks' judge() aborted with "checkout-index -a failed: pkg/Notes.md already
exists" when the judged tree held two paths that differ only by case, so the
case-collision guard never got to report them (exit 2 instead of FAILED:
case-collisions). Both snapshots now check out with -f into their empty temp dir; the
guards read the private index, which keeps both paths. Found on the Windows runner by
the case-collision test once that test staged its premise as blobs.
The pre-push hook judges each pushed branch tip, as CI judges the PR as a whole, not
every intermediate commit; CONTRIBUTING.md and the hook comments said "each commit you
push". CONTRIBUTING.md also said a moved function you lightly edit keeps its cap; that
holds only when no function of that name exists at the destination, and a rename plus
an edit is new code. Both limits are now stated.
bodyWithoutSelf treated a declaration of the function's name at any depth as a rebind,
so a renamed recursive function containing an unchanged `.map((legacy) => legacy)` kept
its old self-call text, no longer matched its base body, and lost its cap ("CC 22 >
target 20 (new code)"). Only the function's own scope (parameters, a named function
expression's name, declarations outside nested functions and classes) now stops the
normalization, and nested functions that bind the name keep their own reads, as the
Python measurer already does.
