`_kill_tree` SIGKILLs the per-file pytest process group, which misses any
child that left the group with `start_new_session=True` (an auto-started
`gateway run`, a setsid server). Those are reparented to init and run
forever: one host accumulated 215 orphan gateways (~52 GB RSS, up to 21 h
old), and every `hermes update` listed each as "another Hermes home — left
alone".
After every attempt the runner now SIGKILLs any process whose environment
still points into that attempt's unique temp root (PYTEST_DEBUG_TEMPROOT /
TMPDIR, or a HOME / HERMES_HOME a fixture created under it). The root is
unique per attempt, so the match names only that attempt's processes.
Linux-only (/proc); the process-group kill stays the fallback elsewhere.
Pull requests and pushes to main no longer run any E2E suite: ci.yaml's detect gate
forces every e2e* lane off unless the run is a stable-release run (inputs.release) or a
manual dispatch, whatever the diff, the update-path classifier or the run-e2e label
selects. That covers the Python e2e and e2e-upgrade jobs, the Windows E2E and Windows
install + update journey in tests-os, and the Desktop core/update suites.
install-e2e.yml drops its pull_request (path-filtered `pr` route) and 12-hourly
schedule triggers; it runs from the stable-release gate (workflow_call) or by dispatch.
live-providers drops its nightly schedule and keeps the release-tag push + dispatch.
Why: a one-row change to left_core_migration.py held a PR for 40 minutes on the
Windows install + update E2E; the suites cost most of the larger-runner minutes and
the release gate already runs all of them strictly.
Vite inlines every VITE_* variable into the renderer and NODE_ENV selects its mode, but the
desktop receipt recorded only sources and prepared inputs. After `VITE_PERF_PROBE=1 npm run
build`, the next plain build reused (and productCurrent certified) the renderer that still
carries the perf probe; a NODE_ENV change had the same gap.
The desktop build inputs now record the renderer environment (VITE_* plus NODE_ENV, unset
read as vite's production default), so both rendererCurrent and productCurrent require it to
match. Reported by Greptile on #135092.
rendererCurrent and productCurrent share hostReceipt() for the schema/product/host checks. The stamped outputs come from bundleElectronMain's own electronOutputs list, so a renamed or added bundle can never be copied stale from the previous dist. The reuse copy skips node_modules/ and native/, which copyNativeTree recopies from nativeDeps right after, and a --typecheck build never reuses a renderer that skipped tsc.
With --no-checkout the file download moved out of the retried clone into the single unretried pin checkout, which ran after the empty clone was already published: a throttled pin left INSTALL_DIR with no index, and the next rerun stashed a 'delete every file' autostash. The pin now runs in the staged tree inside the deferred-checkout retry loop (which also stops materializing the tip first), and nothing is published until it succeeds.
Mirrors the install.sh change from #135092 on Windows. With -Commit (what
the desktop bootstrap passes), the fresh clone runs with --no-checkout and
the pin is checked out with retries inside the staging dir, so the branch
tip is never materialized first and a refused or failed pin never publishes
a checkout. A rerun over an existing checkout keeps pinning after the fetch.
The tagger-line lookup indexed into a single string, so the epoch was the
character 't'. A -Python path that does not exist left LASTEXITCODE at 0, so the
script printed done and listed a previous build's package.
Review findings on the local build scripts:
- Cleanup ran before desktop.py took the checkout lock, so a second run deleted
the outputs of a build that was still going. desktop.py gains --clean, which
removes this checkout's outputs after build_lock is held. Both scripts pass it
instead of deleting files themselves.
- build-bundle.ps1 left RELEASE_CLAIM_* and HERMES_PAYLOAD_* in the caller's
session. It now saves those variables and restores them in the finally block.
- The scripts said "unsigned" but inherited signing credentials. The ps1 removes
AZURE_* for the build. The sh removes CSC_* and APPLE_* and sets
CSC_IDENTITY_AUTO_DISCOVERY=false, the switch the desktop rebuild command uses.
The wording now says what is ignored.
- The Store bundle step needs the claim timestamp, and the script deletes the
claim. The ps1 prints it as HERMES_RELEASE_EPOCH, and BUILDING.md documents it.
scripts/build-bundle.sh (macOS, Linux) and scripts/build-bundle.ps1 (Windows)
build the checked-out commit through scripts/bundles/desktop.py --commit. That
path needs no tag and no push.
Both scripts refuse a dirty tree before any work starts. Both delete the
previous build output first. The staged Windows payload is read-only, so a
plain delete in a reused checkout fails with WinError 5.
build-bundle.ps1 -Store builds the Microsoft Store MSIX. The driver admits
Store packaging only for a stable tag. The script makes a local claim tag
rc.<N>-vX.Y.Z for the next patch after the newest published stable tag. It
counts local and remote attempt refs and starts at 900, so it cannot collide
with a real attempt. A finally block deletes the tag. The script never pushes it.
The payload pinned libc++ 29 while the build container's apt cmake needs 30,
so the two disagreed and the shim in the previous commit papered over it. The
pool has since retired 17 of our pins, so repin them all with
`hermes pm update --termux`, which fixes the mismatch at the source.
Repinned: libc++ 29 -> 30 plus 14 more runtime libs (fribidi, glib, harfbuzz,
libaom, libbluray, libexpat, libheif, libmp3lame, libpng, libvidstab, openssl,
pcre2, readline, vulkan-loader-generic) and the bionic lock rows for uv
0.12.15 -> 0.12.23 and ffmpeg 8.1.2-5 -> 8.1.3. The pool serves ffmpeg 8.1.3
only, so that row moves back from 9.0.1 as the pool has it.
Staging with the new table succeeds: 84 packages, 269 .so, same counts as
before. The libc++ SONAME is unchanged. The one unresolved DT_NEEDED
(libharfbuzz-cairo -> libcairo) is present on the old table too.
The shim is removed. What stays from it is the in-container
`cmake -E capabilities` check, which fails in 2 minutes if the table drifts
from the pool again, instead of after 30 minutes of compiling.
The wheelhouse build stage still compiled cmake from PyPI source (pillow ->
pybind11 -> cmake bootstrap, exit 11) after the Dockerfile and the apt install
began upgrading libc++ to 30. LD_LIBRARY_PATH put the payload's pinned libc++ 29
first, and a library found there beats the binary's own RUNPATH, so apt's cmake
died with 'cannot locate symbol _ZNSt6__ndk113__hash_memoryEPKvm' and
scikit-build-core read that as 'no cmake'.
Probed in the pinned termux-docker image under qemu with the same apt package
list: payload libc++ 29 first fails, a one-symlink dir to the system libc++ in
front of it starts cmake and clang. The build now makes that dir after apt runs
and fails fast if cmake cannot start, instead of burning 30 minutes compiling
cpython, cryptography and pillow first.
The ordered publication pass moves nousresearch/hermes-agent:stable/latest
only after it reads the R2 stable channel head. When that read fails
(v0.21.6: "Channel repository authority mismatch"), the release publishes
and the aliases never move.
`python -m scripts.releases.docker promote --version X.Y.Z` reads the
published vX.Y.Z receipt tag and runs the existing promote_stable with the
digest it binds (slim checked against the receipt, desktop from its own
attempt tag). docker-promote-stable.yml dispatches it with the Docker Hub
credentials only CI holds.
Two stable-release identity checks broke on the first pipeline cuts of v0.21.6:
- nix.yml "Verify release package runtime identity" took the LAST token of the
whole `hermes --version` output. That output ends with the OpenAI SDK line, so
the check compared 0.21.6 against 2.24.0 and failed every release. Parse the
first line ("Hermes Agent v<version> ...") instead.
- msix-shared.mjs releaseEpoch() still required the retired `vX.Y.Z-rc` claim
name. Attempt refs are `rc.<N>-vX.Y.Z` (versioning.parse_attempt_ref), so the
Windows universal bundle threw "Stable claim tag differs from its payload tag".
Accept exactly `rc.<N>-<tag>`; a different version (rc.3-v0.27.10 for v0.27.1)
still refuses.
The pinned termux base image ships libc++ 29. apt's cmake 4.4.4 needs a
newer libc++, but its Depends has no version bound, so apt keeps 29 and
cmake dies at load with an unresolved libc++ symbol. scikit-build-core
reads that as no cmake and compiles one from the PyPI sdist, whose
bootstrap exits 9. That killed the pybind11 build pillow needs, so every
termux .deb build since Oct 3 failed.
Name libc++ in both package lists so apt upgrades it.
abandon refused with 'no outstanding attempt' when a version had two, the
same branch as zero, despite its docstring promising to clear one. It now
clears all of them, oldest first.
release.py changelog prints the notes the next draft would carry without
claiming an attempt. When the body is over GitHub's limit, the refusal now
says to cut with --no-changelog, dump the changelog, edit it down by hand,
and paste it into the draft.
A local-only rc.* tag (a hand-made ref or a claim that never pushed) counted
as an outstanding attempt, so release refused and abandon reported 'no
outstanding attempt'. The remote is the only authority on claims now.
The pipeline seeded its version line at 0.21.4, but the legacy CalVer line
already published "Hermes Agent v0.21.5" as v2026.9.24. A first `--bump patch`
cut therefore derived 0.21.5 again: installs already reporting 0.21.5 would
compare equal and never see the new build as an update, and the release page
would carry two v0.21.5 entries.
Seed at 0.21.5 so the first pipeline patch is 0.21.6 and the first minor stays
0.22.0. Entrypoint fixtures now pin their own published head instead of
tracking the production seed.
#133931 made source_build.py import hermes_cli/web_build_limits.py but did not add it to
_UPDATE_DEPENDENCIES, so tests/ci/test_update_ci_routing.py has been red on main and on
every PR whose merge ref includes it: a change to the module would skip the
e2e_upgrade and e2e_desktop_update lanes that run the build it shapes.
The dashboard build (Vite 8 + Rolldown + Tailwind v4) saturates every CPU
on small hosts (200%+ top readings, OOMs and frozen SSH sessions on 2–4
vCPU VPS boxes). Rolldown exposes no JS-level worker knob, but its native
binding parallelizes with rayon, which honours RAYON_NUM_THREADS.
Cap both sides of the build, everywhere it runs:
- scripts/build/web.mjs (the npm run build / packaging entry): limit the
native bundler's rayon pool to half the available cores before vite is
imported, and re-exec once with a --max-old-space-size cap (sized from
the cgroup memory limit, 75%, floored at 1 GB, ceiling 4 GB) unless
the user already set one.
- hermes_cli/web_build_limits.py + build_source_web(): the same caps for
the install/update/dashboard-triggered source build path.
User env always wins (existing NODE_OPTIONS heap flag or
RAYON_NUM_THREADS is preserved); HERMES_WEB_BUILD_MAX_OLD_SPACE_SIZE /
HERMES_WEB_BUILD_THREADS tailor the caps, and HERMES_WEB_BUILD_LIGHT=1
tightens them (1 thread, 1 GB heap) for hosts that cannot spare full CPU
during the build — the light-build idea from the closed PR #78131.
Windows-only tests job 112764655375 (run 37612501107, a YAML-only catalog PR)
went red with "667 passed, 0 failed": test_desktop_update_windows_marker.py
hit the runner's 300s per-file cap mid-file ("(300s exceeded; process tree
SIGKILL'd)"), so its 19 tests were never counted and it was listed under
"no tests ran". Run 37612584210 died the same way. Clean runs of this file
take 220-300s (arm64 hit 296.5s and 299.8s), so it is at the cap on every run.
Two defects:
1. The desktop_updater lane gate never worked. tests-os.yml passes
--ignore-glob='*test_desktop_update_windows_*.py' after `--`, and
run_tests_parallel.py forwards it to `pytest <file>`. pytest applies
--ignore/--ignore-glob only while recursing directories, never to an
explicit file argument, so the flag did nothing. These hand-off tests ran
on every PR, including ones that never touched the updater, even though
the job printed "desktop_updater lane off: skipping ...". The runner now
applies --ignore/--ignore-glob to its own file list (pytest's fnmatch on
the absolute path, relative patterns anchored at the repo root, which is
the per-file pytest's cwd) and says how many files it excluded.
2. test_desktop_that_never_exits_is_not_relaunched_over waited out the
production 150s Desktop-exit ceiling. That one wait is half the file's
budget. The script already honours HERMES_UPDATE_DESKTOP_EXIT_SECONDS
("so the self-tests need not sit it out"), and the posix twin sets it to
2. Set it to 5 and assert that the script's refusal names 5s, so the
override is proven to reach the wait. The verdict under test (refuse
with 4, never relaunch) is the same at any ceiling.
Repro (Linux, CI-shape args):
python scripts/run_tests_parallel.py --files "<windows_marker>:<windows_cwd>:<other>" \
-- --ignore-glob='*test_desktop_update_windows_*.py' -m "platforms and not integration"
base: "Running 3 test files", both gated files run
fixed: "note: --ignore/--ignore-glob excluded 2 test files", "Running 1 test files"
New test_passthrough_ignore_drops_files_the_runner_hands_pytest_explicitly
(--files and discovery x glob/glob-spaced/path) fails 6/6 on base and
passes 6/6 fixed.
- test_shared_metrics_update_stop_reasons: every stop token reads as its own class in-process
and through the parked copy (no free text kept), unknown tokens and post-apply tokens fall
back, a committed run that only owes parked changes is partial; the lock and Git-operation
refusals each record one row and leave the holder's latest.json byte-identical.
- test_shared_metrics_install_failures: a PermissionError stop reason now reads its own class
permission_denied (was os_error); ENOSPC reads disk_full; other OSErrors keep os_error.
- relay-shared-metrics.md: the new classes, partial outcome and parked fields.
- relay smoke: the failed update receipt names its fetch failure and the row asserts fetch_failed.
Nothing tested mac_bundle_recover(), the start-of-run repair that finishes or
rolls back a Hermes.app swap a killed hand-off left half done.
posix.sh --self-test-swap-interrupt (body in swap-selftest.sh) runs the REAL
mac_swap on a fixture bundle tree in its own process group. mac_swap now
names its step boundaries (swap_checkpoint start/staged/aside/installed); the
self-test makes the swapper SIGSTOP itself at one of them and SIGKILLs the
whole group there, or resumes it and SIGKILLs it mid-ditto / mid-cleanup rm.
It checks the wreck has the expected shape, runs the REAL mac_bundle_recover,
and asserts the target is the complete old or complete new bundle (whole-tree
checksum, version marker included), no .old/.new/.hermes-update-* is left,
and a second recovery changes and logs nothing. Seven cells: before-stage,
mid-stage, staged, aside, installed, mid-cleanup, complete (control).
The seams are inert in a real run: SWAP_PAUSE_AT is set only inside the
self-test, and MAC_DITTO stays /usr/bin/ditto unless the self-test runs where
ditto is absent (Linux: cp -R).
test_desktop_update_mac_swap_interrupt.py drives the arm under
platforms("posix"): the Linux suite and the macOS lanes of tests-os.yml
(real /usr/bin/ditto, macOS /bin/bash 3.2).
Get-DefaultBrowserExe read only the legacy UserChoice ProgId, which Windows
11 25H2 Settings no longer updates, so the update hand-off fell back to the
WinForms card (or a stale browser) on machines whose real default is
Chrome/Edge. Read UserChoiceLatest\ProgId first, legacy UserChoice second.
Fixes#108051. Follows #108062 / #108064.
Co-authored-by: KoNit-K <konit.block@protonmail.com>
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
Importing a/b/c.py runs a/__init__.py and a/b/__init__.py first. The
update-lane routing listed hermes_cli/local_runtime/processes.py (the
v2026.9.24 updater imports it after the pull) but not the package init
that import runs. #134116 changed what that init loads, its PR never ran
e2e-upgrade, and main went red until #134242.
classify_changes now routes the package inits above every module the
e2e_upgrade and e2e_desktop_update lanes own, and above each shared hub,
because a hub's inits run whenever the hub is imported. An init that is
itself a hub stays out. The routing guard's import walker counts the
package inits a module's imports run, so the next unrouted one fails it.
This routes eight inits that were missing: local_runtime, gateway, cron,
tools, tools/environments, plugins, hermes_platform,
hermes_cli/observability.
HERMES_PREVIEW_FULL_CONNECTORS=true|false makes the free-tier mint send
{"preview_full_connectors": <bool>} in the /api/anonymous/create body, so
the account service can record the cohort on the account and gate
toolkits from it. Any other value (unset included) keeps the body {}.
The variable joins the desktop bundle-env allowlist (release script,
banner writer, channel request decoder, BUILDING.md) so a bundle can
bake it in. It is self-reported and visible in the packaged JS: the
service must treat it as a preference, not proof.
tirith ran `tirith check` on every terminal command, default-on with fail-open, as a
pinned PM binary auto-downloaded at startup. Measured against the shipped 0.4.2 binary
it caught 14/16 attack commands but flagged 12/24 everyday agent commands
(`echo $((6*7))`, `"$VAR/x.sh"`, `gh api | python3`, LAN curls, `.dev` hosts,
`pip install` via analysis_incomplete), and in cron/-q deny mode each of those is a
hard block. It only ever saw terminal strings (execute_code, writes and the browser
bypass it), fail-open meant no protection wherever the binary could not install
(Windows, Termux, x64 musl), and it produced a steady issue/PR stream (fail_open
semantics four times, install loops filling disks, a SIGSEGV hang, session-approval
scoping, third-party promo text on approval cards).
Removed: tools/tirith_security.py, the tirith branches of the command guard (the
is_tirith tuple, session-only "always" downgrade, import-error/fail-open paths,
unattended tirith pass), the CLI and gateway startup installers, the PM package and
lock pins, the nix runtime dep, security.tirith_* defaults, the CLI warning string
and three tirith tips (gaps refilled from the tail so translations survive), plus
the gateway's "no automated assessor" warning that only existed to point at tirith.
Config v50 drops security.tirith_* and enables nothing in its place. Hermes ships
no tirith plugin; if tirith upstream publishes one, it goes through normal catalog
admission under their name.
Two ratchets this suite arms for its own siblings fired once they landed:
- the Windows pause (#132338) now imports gateway/status_inline_source.py
to recognise inline-bootstrapped gateways, so a change there starts the
e2e_upgrade suite like gateway/status.py (only two importers: not a hub)
- the marker-corpus consumers #132345 and #132354 ship are on main, so
_LANDS_IN_SIBLING_PR empties as its comment asks
`hermes_cli/_early_recovery_zip.py` (split out of `_early_recovery.py` on the
commit-point PR) is imported by the update path, so a change to it must start
the update pipeline and install E2E suites. Both lists and the install-e2e path
filter now match the `_early_recovery` prefix, which also routes any later
sibling. `remote-lifecycle-v2-marker.test.ts` reads the update-marker corpus
and is now a listed consumer; `remote-update-marker-programs.test.ts` exists in
this tree, so its sibling-PR allowance is gone.
Replaying the last 131 non-merge main commits (4787e4d56f) through the
classifier: main selects e2e_upgrade 5 / e2e_desktop_update 3, the CI branch
12 / 10. Four of the seven new hits per lane touched only plugins_cmd.py or
plugins_cmd_install.py. migrate_all_homes reaches them only when a home's
configured memory provider left core (the catalog install); no upgrade or
Desktop update journey configures one, so those suites cannot observe the
change, and the unit lane already runs on every .py edit. Measured on a
fresh home: memory_provider_migration, pm/plugins_state, agent/memory_provider
and plugins/memory run; plugins_cmd_install executes only its module body,
plugins_cmd is never imported.
Kept, because a fresh-home update runs them: memory_provider_migration.py
(migrate_all_homes on every source build), plugins/memory/__init__.py (the
post-update Honcho profile sync), pm/install.py (sealed() /
lazy_installs_allowed() gate every update's default-tool install), and
scripts/build/ (frontend-common.mjs is imported by freshness/web/tui/desktop).
After: 8 / 6. A guard test fails if the journey-home migration starts running
plugin-install code, the signal to route those modules again.
update_cmd_commit._admitted_python now resolves python3.N through hermes_platform.resolver
(cd471c9c55a), so the update path imports the resolver package and a change to it must start the
upgrade E2E, like hermes_platform.host already does.
What: scripts/ci/classify_changes.py names hermes_cli/git_pack_tidy.py in the update pipeline, so a
change to it runs e2e_upgrade.
Why: the derived routing guard (test_every_module_an_update_entry_point_imports_starts_its_suite)
found it imported by the update path and unrouted: main's partial-clone tidy (a0f47657b9) runs in
every update's pre-fetch via update_cmd_check.
