The posix hand-off hands marker line 1 to a custodian subshell
(marker_refresher_start) and then exported HERMES_UPDATE_HANDOFF_PID="$$" —
the hand-off script's own pid.
`hermes update` adopts a live claim only when the owner is its own pid, the
HERMES_UPDATE_HANDOFF_PID value, or an ancestor
(update_lock.UpdateLock._is_partner). The custodian is a *sibling* of the
hand-off script, so it is none of the three and the ancestry walk can never
reach it: every desktop-initiated update refused the claim its own script had
just made and exited 2 ("Another Hermes update is already running ...
process <custodian>"), leaving the Desktop to relaunch and retry forever.
Export $MY_PID instead: the custodian once the handover succeeded, and the
script's own pid when it did not.
Fixes#134268
Review thread on #136050 (agent/conversation_compression_telemetry.py,
"Refusals inflate batch counts"). Since #135807 every exit of
`compress_context` (blocked by a guard, skipped, aborted, refused) logs one
`context compression attempt telemetry:` line, and
`scripts/micro_compaction_report.py` counted every such line as a batch
compaction. Keep only records whose `commit_status` is `committed`.
The builder shipped ~1.9k skills.sh rows per run with no resolved_github_id.
Most were rows whose repo 404s or whose fully-read tree has no matching
SKILL.md dir (renamed/removed upstream); every client install of one spent
~40 GitHub calls and failed as stale_index.
- fuzzy match ignores both '-' and '_' (pdb-database slug vs pdb_database dir)
- a single-skill repo (root or generic skills/ SKILL.md) resolves like the client does
- repo 404 or complete tree with no match: the row is dropped, counts logged
- truncated tree: bounded contents/<candidate>/SKILL.md probes, row kept if none hit
- transient errors keep the row unresolved, as before
A release that skipped bundles never writes the stable baseline pointer,
so its absence is normal at any point, not only before the first bundle
release. Drop the final-tag scan. A 404 on the default pointer now always
plans no upgrade arm. A supplied baseline-manifest that does not exist
still blocks, because that is a bad input.
No stable release has shipped signed packages yet. Every final tag so far
skipped bundles or predates the claim pipeline. So
releases/stable/release-candidates.json does not exist, and the three
"Pin actual OLD and NEW signed packages" jobs refused with "No published
stable package baseline".
transitions now treats a 404 as a first release only when no final stable
tag records a candidate manifest and no baseline-manifest input was given.
It still validates the receipt, emits empty matrices and baseline=none.
The packaged upgrade arms skip on that output. The gate accepts their skip
only when their own planner succeeded with baseline=none, so a missing
baseline after a real bundle release still blocks.
`_kill_tree` SIGKILLs the per-file pytest process group, which misses any
child that left the group with `start_new_session=True` (an auto-started
`gateway run`, a setsid server). Those are reparented to init and run
forever: one host accumulated 215 orphan gateways (~52 GB RSS, up to 21 h
old), and every `hermes update` listed each as "another Hermes home — left
alone".
After every attempt the runner now SIGKILLs any process whose environment
still points into that attempt's unique temp root (PYTEST_DEBUG_TEMPROOT /
TMPDIR, or a HOME / HERMES_HOME a fixture created under it). The root is
unique per attempt, so the match names only that attempt's processes.
Linux-only (/proc); the process-group kill stays the fallback elsewhere.
Pull requests and pushes to main no longer run any E2E suite: ci.yaml's detect gate
forces every e2e* lane off unless the run is a stable-release run (inputs.release) or a
manual dispatch, whatever the diff, the update-path classifier or the run-e2e label
selects. That covers the Python e2e and e2e-upgrade jobs, the Windows E2E and Windows
install + update journey in tests-os, and the Desktop core/update suites.
install-e2e.yml drops its pull_request (path-filtered `pr` route) and 12-hourly
schedule triggers; it runs from the stable-release gate (workflow_call) or by dispatch.
live-providers drops its nightly schedule and keeps the release-tag push + dispatch.
Why: a one-row change to left_core_migration.py held a PR for 40 minutes on the
Windows install + update E2E; the suites cost most of the larger-runner minutes and
the release gate already runs all of them strictly.
Vite inlines every VITE_* variable into the renderer and NODE_ENV selects its mode, but the
desktop receipt recorded only sources and prepared inputs. After `VITE_PERF_PROBE=1 npm run
build`, the next plain build reused (and productCurrent certified) the renderer that still
carries the perf probe; a NODE_ENV change had the same gap.
The desktop build inputs now record the renderer environment (VITE_* plus NODE_ENV, unset
read as vite's production default), so both rendererCurrent and productCurrent require it to
match. Reported by Greptile on #135092.
rendererCurrent and productCurrent share hostReceipt() for the schema/product/host checks. The stamped outputs come from bundleElectronMain's own electronOutputs list, so a renamed or added bundle can never be copied stale from the previous dist. The reuse copy skips node_modules/ and native/, which copyNativeTree recopies from nativeDeps right after, and a --typecheck build never reuses a renderer that skipped tsc.
With --no-checkout the file download moved out of the retried clone into the single unretried pin checkout, which ran after the empty clone was already published: a throttled pin left INSTALL_DIR with no index, and the next rerun stashed a 'delete every file' autostash. The pin now runs in the staged tree inside the deferred-checkout retry loop (which also stops materializing the tip first), and nothing is published until it succeeds.
Mirrors the install.sh change from #135092 on Windows. With -Commit (what
the desktop bootstrap passes), the fresh clone runs with --no-checkout and
the pin is checked out with retries inside the staging dir, so the branch
tip is never materialized first and a refused or failed pin never publishes
a checkout. A rerun over an existing checkout keeps pinning after the fetch.
The tagger-line lookup indexed into a single string, so the epoch was the
character 't'. A -Python path that does not exist left LASTEXITCODE at 0, so the
script printed done and listed a previous build's package.
Review findings on the local build scripts:
- Cleanup ran before desktop.py took the checkout lock, so a second run deleted
the outputs of a build that was still going. desktop.py gains --clean, which
removes this checkout's outputs after build_lock is held. Both scripts pass it
instead of deleting files themselves.
- build-bundle.ps1 left RELEASE_CLAIM_* and HERMES_PAYLOAD_* in the caller's
session. It now saves those variables and restores them in the finally block.
- The scripts said "unsigned" but inherited signing credentials. The ps1 removes
AZURE_* for the build. The sh removes CSC_* and APPLE_* and sets
CSC_IDENTITY_AUTO_DISCOVERY=false, the switch the desktop rebuild command uses.
The wording now says what is ignored.
- The Store bundle step needs the claim timestamp, and the script deletes the
claim. The ps1 prints it as HERMES_RELEASE_EPOCH, and BUILDING.md documents it.
scripts/build-bundle.sh (macOS, Linux) and scripts/build-bundle.ps1 (Windows)
build the checked-out commit through scripts/bundles/desktop.py --commit. That
path needs no tag and no push.
Both scripts refuse a dirty tree before any work starts. Both delete the
previous build output first. The staged Windows payload is read-only, so a
plain delete in a reused checkout fails with WinError 5.
build-bundle.ps1 -Store builds the Microsoft Store MSIX. The driver admits
Store packaging only for a stable tag. The script makes a local claim tag
rc.<N>-vX.Y.Z for the next patch after the newest published stable tag. It
counts local and remote attempt refs and starts at 900, so it cannot collide
with a real attempt. A finally block deletes the tag. The script never pushes it.
The payload pinned libc++ 29 while the build container's apt cmake needs 30,
so the two disagreed and the shim in the previous commit papered over it. The
pool has since retired 17 of our pins, so repin them all with
`hermes pm update --termux`, which fixes the mismatch at the source.
Repinned: libc++ 29 -> 30 plus 14 more runtime libs (fribidi, glib, harfbuzz,
libaom, libbluray, libexpat, libheif, libmp3lame, libpng, libvidstab, openssl,
pcre2, readline, vulkan-loader-generic) and the bionic lock rows for uv
0.12.15 -> 0.12.23 and ffmpeg 8.1.2-5 -> 8.1.3. The pool serves ffmpeg 8.1.3
only, so that row moves back from 9.0.1 as the pool has it.
Staging with the new table succeeds: 84 packages, 269 .so, same counts as
before. The libc++ SONAME is unchanged. The one unresolved DT_NEEDED
(libharfbuzz-cairo -> libcairo) is present on the old table too.
The shim is removed. What stays from it is the in-container
`cmake -E capabilities` check, which fails in 2 minutes if the table drifts
from the pool again, instead of after 30 minutes of compiling.
The wheelhouse build stage still compiled cmake from PyPI source (pillow ->
pybind11 -> cmake bootstrap, exit 11) after the Dockerfile and the apt install
began upgrading libc++ to 30. LD_LIBRARY_PATH put the payload's pinned libc++ 29
first, and a library found there beats the binary's own RUNPATH, so apt's cmake
died with 'cannot locate symbol _ZNSt6__ndk113__hash_memoryEPKvm' and
scikit-build-core read that as 'no cmake'.
Probed in the pinned termux-docker image under qemu with the same apt package
list: payload libc++ 29 first fails, a one-symlink dir to the system libc++ in
front of it starts cmake and clang. The build now makes that dir after apt runs
and fails fast if cmake cannot start, instead of burning 30 minutes compiling
cpython, cryptography and pillow first.
The ordered publication pass moves nousresearch/hermes-agent:stable/latest
only after it reads the R2 stable channel head. When that read fails
(v0.21.6: "Channel repository authority mismatch"), the release publishes
and the aliases never move.
`python -m scripts.releases.docker promote --version X.Y.Z` reads the
published vX.Y.Z receipt tag and runs the existing promote_stable with the
digest it binds (slim checked against the receipt, desktop from its own
attempt tag). docker-promote-stable.yml dispatches it with the Docker Hub
credentials only CI holds.
Two stable-release identity checks broke on the first pipeline cuts of v0.21.6:
- nix.yml "Verify release package runtime identity" took the LAST token of the
whole `hermes --version` output. That output ends with the OpenAI SDK line, so
the check compared 0.21.6 against 2.24.0 and failed every release. Parse the
first line ("Hermes Agent v<version> ...") instead.
- msix-shared.mjs releaseEpoch() still required the retired `vX.Y.Z-rc` claim
name. Attempt refs are `rc.<N>-vX.Y.Z` (versioning.parse_attempt_ref), so the
Windows universal bundle threw "Stable claim tag differs from its payload tag".
Accept exactly `rc.<N>-<tag>`; a different version (rc.3-v0.27.10 for v0.27.1)
still refuses.
The pinned termux base image ships libc++ 29. apt's cmake 4.4.4 needs a
newer libc++, but its Depends has no version bound, so apt keeps 29 and
cmake dies at load with an unresolved libc++ symbol. scikit-build-core
reads that as no cmake and compiles one from the PyPI sdist, whose
bootstrap exits 9. That killed the pybind11 build pillow needs, so every
termux .deb build since Oct 3 failed.
Name libc++ in both package lists so apt upgrades it.
abandon refused with 'no outstanding attempt' when a version had two, the
same branch as zero, despite its docstring promising to clear one. It now
clears all of them, oldest first.
release.py changelog prints the notes the next draft would carry without
claiming an attempt. When the body is over GitHub's limit, the refusal now
says to cut with --no-changelog, dump the changelog, edit it down by hand,
and paste it into the draft.
A local-only rc.* tag (a hand-made ref or a claim that never pushed) counted
as an outstanding attempt, so release refused and abandon reported 'no
outstanding attempt'. The remote is the only authority on claims now.
The pipeline seeded its version line at 0.21.4, but the legacy CalVer line
already published "Hermes Agent v0.21.5" as v2026.9.24. A first `--bump patch`
cut therefore derived 0.21.5 again: installs already reporting 0.21.5 would
compare equal and never see the new build as an update, and the release page
would carry two v0.21.5 entries.
Seed at 0.21.5 so the first pipeline patch is 0.21.6 and the first minor stays
0.22.0. Entrypoint fixtures now pin their own published head instead of
tracking the production seed.
#133931 made source_build.py import hermes_cli/web_build_limits.py but did not add it to
_UPDATE_DEPENDENCIES, so tests/ci/test_update_ci_routing.py has been red on main and on
every PR whose merge ref includes it: a change to the module would skip the
e2e_upgrade and e2e_desktop_update lanes that run the build it shapes.
The dashboard build (Vite 8 + Rolldown + Tailwind v4) saturates every CPU
on small hosts (200%+ top readings, OOMs and frozen SSH sessions on 2–4
vCPU VPS boxes). Rolldown exposes no JS-level worker knob, but its native
binding parallelizes with rayon, which honours RAYON_NUM_THREADS.
Cap both sides of the build, everywhere it runs:
- scripts/build/web.mjs (the npm run build / packaging entry): limit the
native bundler's rayon pool to half the available cores before vite is
imported, and re-exec once with a --max-old-space-size cap (sized from
the cgroup memory limit, 75%, floored at 1 GB, ceiling 4 GB) unless
the user already set one.
- hermes_cli/web_build_limits.py + build_source_web(): the same caps for
the install/update/dashboard-triggered source build path.
User env always wins (existing NODE_OPTIONS heap flag or
RAYON_NUM_THREADS is preserved); HERMES_WEB_BUILD_MAX_OLD_SPACE_SIZE /
HERMES_WEB_BUILD_THREADS tailor the caps, and HERMES_WEB_BUILD_LIGHT=1
tightens them (1 thread, 1 GB heap) for hosts that cannot spare full CPU
during the build — the light-build idea from the closed PR #78131.
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
- test_shared_metrics_update_stop_reasons: every stop token reads as its own class in-process
and through the parked copy (no free text kept), unknown tokens and post-apply tokens fall
back, a committed run that only owes parked changes is partial; the lock and Git-operation
refusals each record one row and leave the holder's latest.json byte-identical.
- test_shared_metrics_install_failures: a PermissionError stop reason now reads its own class
permission_denied (was os_error); ENOSPC reads disk_full; other OSErrors keep os_error.
- relay-shared-metrics.md: the new classes, partial outcome and parked fields.
- relay smoke: the failed update receipt names its fetch failure and the row asserts fetch_failed.
Nothing tested mac_bundle_recover(), the start-of-run repair that finishes or
rolls back a Hermes.app swap a killed hand-off left half done.
posix.sh --self-test-swap-interrupt (body in swap-selftest.sh) runs the REAL
mac_swap on a fixture bundle tree in its own process group. mac_swap now
names its step boundaries (swap_checkpoint start/staged/aside/installed); the
self-test makes the swapper SIGSTOP itself at one of them and SIGKILLs the
whole group there, or resumes it and SIGKILLs it mid-ditto / mid-cleanup rm.
It checks the wreck has the expected shape, runs the REAL mac_bundle_recover,
and asserts the target is the complete old or complete new bundle (whole-tree
checksum, version marker included), no .old/.new/.hermes-update-* is left,
and a second recovery changes and logs nothing. Seven cells: before-stage,
mid-stage, staged, aside, installed, mid-cleanup, complete (control).
The seams are inert in a real run: SWAP_PAUSE_AT is set only inside the
self-test, and MAC_DITTO stays /usr/bin/ditto unless the self-test runs where
ditto is absent (Linux: cp -R).
test_desktop_update_mac_swap_interrupt.py drives the arm under
platforms("posix"): the Linux suite and the macOS lanes of tests-os.yml
(real /usr/bin/ditto, macOS /bin/bash 3.2).
