pm-bundle, install-e2e and termux-verify trigger on pm/**, so adding pm/AGENTS.md
(and every later edit to it) queued darwin-x64 / win32-arm64 bundle builds for a doc
change. Nothing reads .md files under pm/, so exclude them from the path filters.
scripts/ci/check_agents_md_size.py (in scripts/check and lint.yml): root <= 12k, every
AGENTS.md plus its ancestors <= 30k (the joined root->cwd block on a 128k window, minus
the per-file labels), every area file <= _MAX_HINT_CHARS read from the source so the two
cannot drift. On origin/main it reports the root and all 11 root-to-area chains over.
- `ENFORCEMENT` in scripts/code_health/config.py (blocking / advisory / off)
turns the ratchet down or off with a one-line commit to main. It is read
from the BASE revision, so it applies to fork PRs (repo variables are not
passed to fork-triggered workflows), CI and local hooks alike, and a PR
cannot relax its own check.
- `scripts/check --install-hook pre-push` judges each pushed commit the way
CI judges the PR (base = that commit's merge-base); `--commit REV` is the
underlying mode, `--uninstall-hook` removes our hooks and leaves foreign
ones. Pre-commit and pre-push share one tree snapshot that never touches
the user's worktree or index.
- A failing report names the local commands; CONTRIBUTING, AGENTS.md and the
PR template point at `python scripts/check`.
Fourth review round (F11-F18, N1-N4, kvnloo, Adolanium):
- Hits are old only where the base occurrence survives: a line the comment- and
whitespace-blind line diff keeps (blame semantics), or anything inside a unit whose
body is unchanged. An identical violation moved past unrelated code is new; removing
an allow comment or re-indenting is not. Hit fingerprints exclude comments (N4).
- Inline allows cover their own line; only a comment-only line covers the next (N3).
- Rules read a copy of the AST with import aliases resolved, so `sp.run`,
`from subprocess import run as execute`, aliased threading/asyncio/load_config are
seen (F12); deadline helpers must resolve to the real asyncio API, and
communicate()/wait() are judged only on receivers holding a spawned process (F13);
infinite deadlines are unbounded (F14).
- HERMES_* writes introduce a name too (N1); bool(os.environ[...]) is HX010 (N2);
HX009 needs the isinstance subject to come from gather(return_exceptions=True).
- Recursive self-references are normalised out of body hashes, so a recursive rename
keeps its cap, py and ts (F15). Decorators belong to their function's scope and
classes are matchable units, so their hits follow proven moves (F16).
- An over-cap file may not grow (F17, Teknium: strict improvement-only).
- A file the checker cannot parse or measure (ast/ruff/TS parse error) fails.
- scripts/check: --staged always lints an index snapshot (ignored files are invisible
to git status, not to guards) (F18); create the --prefix dir for Git for Windows
(F11); --only builds just the selected jobs, so it needs no ruff for non-ruff checks.
- CODEOWNERS covers scripts/check + scripts/code_health; frontend-only PRs skip the
Python-only lint jobs again and run only the ratchet.
`python scripts/check` runs ruff, the Windows-footgun/shebang/no-tmp/
config-writer/OS-marker guards and the code-health ratchet in parallel
with CI's pinned ruff (read from pyproject's dev group; a different local
ruff is never used silently, `uvx ruff@<pin>` is the fallback).
`--install-hook` makes it the pre-commit hook; the hook skips branches
that predate scripts/check, since hooks are shared by every worktree.
lint.yml gains the blocking `code health ratchet` job: it compares the PR
merge commit with its first parent (the base branch tip), so a PR is held
to main as it is now. The lint lane also runs for frontend-only changes
because the ratchet covers TypeScript, and the existing ruff installs read
the same pin.
pinned-source-validate checked out depth 2. Hermes resolves its own
version from release tags merged into HEAD, so the shallow checkout
reported 'unknown' and plugins that gate on the host version refused to
load (omh, #122578: requires Hermes >=0.21.1). A blobless full clone
carries the commit graph and tags for about the cost of the shallow one.
Adds package.json and pyproject.toml at any depth, flake.lock, every
*package-lock.json (incl. nix/node-gyp-11-4-0-package-lock.json), .npmrc
(the min-release-age quarantine), pm/artifact-mirror.json, the shell
installers and their pin generator, scripts/ci/, SECURITY.md and
plugin-catalog/. Drops the header claim that the ruleset's
required-reviewers rule already covers the manifests.
uv.lock, package-lock.json (any depth, incl. pm/uv.lock), pm/lock.json
and the root Dockerfile now request core review directly instead of
relying only on the ruleset's required-reviewers rule.
Path-based review (workflows, composite actions, eslint/prettier config,
the MCP catalog, install-hook files) moves to .github/CODEOWNERS. GitHub
requests and enforces that review, so CI no longer classifies those paths,
checks a label, or reruns failed jobs when the label is added.
Removes review-labels.yml, emit_review_status.py, the rerun-review-labels
job, and the ci_review / ci_review_files / mcp_catalog classifier outputs.
The run-e2e rerun job in label-rerun.yml stays.
The supply-chain scan keeps its content patterns (base64 decode + exec,
obfuscated subprocess), which CODEOWNERS cannot express. It fails on its
own again, as it did before #68833 routed it through the label gate. Its
.pth and install-hook path checks are dropped in favour of CODEOWNERS.
stable-release.yml runs `jobs: win32-bundle` as its own call after the
per-arch candidate calls. In that call validate selects only win32-bundle, so
assemble-win32-bundle's gate (win32-x64/win32-arm64 selected AND their build
jobs succeeded) was always false and the job skipped, which failed the stable
bundle phase with `assemble-win32-bundle=skipped`.
A build selected in the same call must still succeed; an unselected one is
fetched from the R2 handoff, as the assemble steps already do.
The desktop DMG's volume icon defaulted to the packager's icns, which is
actool's 256px fallback whenever mac.icon is the Icon Composer package, and
dmgbuild's badge option can only paste onto the stock removable-drive icon.
Ship our own drive-with-the-girl artwork instead: assets/dmg-volume.png
(hand-made, one artwork since volume icons have no appearance variants) is
emitted as apps/desktop/packaging/dmg-volume.icns, beside the DMG background
so the `files` whitelist keeps it out of the app bundle, and dmg.icon points
at it.
Tauri always uses the app icns as the volume icon and offers no override, so
the bootstrap workflow runs scripts/bundles/set_dmg_volume_icon.sh on the
built DMG before signing (the same .VolumeIcon.icns + custom-icon flag
dmgbuild sets), and the verify step checks the icon is on the mounted image.
Verified on macOS 26.6.2: both DMGs render the identical volume icon.
Tauri's bundler (cli 2.11.4) compiles an Icon Composer package listed in
bundle.icon into Assets.car with actool >= 26 and sets CFBundleIconName, so
the bootstrap installer gets an unbranded twin of the desktop package under
src-tauri/icons/icon.icon (generator target; the bundle stays the plain
brand because flavours only apply under apps/desktop/). Tauri only logs when
actool is too old, so the macOS job selects Xcode 26 and the verify step
asserts Assets.car and the plist key are in the built .app.
The desktop DMG's volume icon defaulted to the packager's icns, which is
actool's 256px fallback whenever mac.icon is the .icon package; point
dmg.icon at the full-resolution assets/icon.icns. Tauri's DMG already takes
--volicon from the bundle's .icns, which the regenerated outputs cover.
electron-builder (app-builder-lib 27.0.0-alpha.6) compiles a `.icon` given as
`mac.icon` into Contents/Resources/Assets.car and sets CFBundleIconName, but
only with actool >= 26 (Xcode 26); on any other host it throws and the build
fails. scripts/mac-icon.cjs picks the Icon Composer package when that actool
is present and the legacy .icns otherwise, so a dev Mac on Xcode 16 keeps
building the app it always built.
The release runners keep Xcode 16.4 selected although 26.x is installed, so
the darwin job now selects the newest Xcode 26 via DEVELOPER_DIR and asserts
the actool version before packaging; a silent fallback cannot ship from CI.
electron-builder's `.icon` path also replaces the bundled icon.icns with
actool's 256px fallback. macOS <= 15 shows that file, so after-pack restores
our 16..1024 assets/icon.icns.
The full suite was the last job to finish in 33 of 43 CI runs since it
moved from a 96-core to one 32-core runner (p50 666 s). That runner is
~90% packed, so only more machines help, and one bigger box is poor
value: on 96 cores the same files took 25-50% longer each and the suite
ran ~1.8x faster for 3x the price. Two 32-core slices scale linearly
(simulated from per-file CI timings: 262 s / 252 s vs 514 s on one box).
macOS (free 3-core runner, ~7.5 min) would become the slowest required
job once Linux splits, so it runs as two slices on the same free runner.
--slice now partitions from the checkout alone (sorted round-robin):
each slice job restores its own duration cache, and a main push saving
a newer cache between the two restores made timed LPT compute different
partitions per job, silently dropping or double-running files.
Round-robin lands within 5 s of timed LPT at this suite size. Each slice
saves its own cache key on main; the runner keeps the restored entries
for files outside its slice, so the cache stays complete.
Adding ci-reviewed starts a job that waits for the PR's CI run, then reruns
its failed jobs. A push during the wait cancels that run, the waiter sees
"completed", reruns it, and the rerun rejoins the per-PR concurrency group
and cancels the new head's run. The PR is left testing a stale commit (seen
on #131236 and #131246). Skip the rerun when the head moved: the new run's
own event already carries the label. Same guard on the run-e2e job.
- Run the data-only guard once, inside the changed-entries step of the
pinned-source job, and only when an entry YAML changed. README, policy,
docs and CI PRs that touch plugin-catalog/ without an entry no longer go
red.
- Allow contributors/emails/** beside plugin-catalog/**: the attribution
fix we push onto catalog PR branches is data the gate never executes
(29 of the last 134 merged catalog PRs carried one).
- Diff GitHub's merge commit against its first parent with fetch-depth: 2
instead of a full-history clone plus merge-base. The structural job,
which only validates the whole directory, goes back to a shallow
checkout with no diff at all.
- `subdir:` with no value means the repo root in both the parse step and
the structural validator, matching how the runtime loader has always
read it.
- Scan the whole clone for escaping symlinks, not just PLUGIN_DIR:
find does not descend into a symlinked directory, so an allowed
in-clone link (plugin/ext -> ../lib) hid a nested escape.
- Require catalog admission PRs to be data-only in both jobs; the
gate's enforcement code resolves from the PR checkout, so a diff
that also touches tooling could neuter the gate vetting it.
- persist-credentials: false on both checkouts: `hermes plugins
validate` runs pinned-repo code and must not see the job token.
- Broaden the self-updater patterns: releases/download and codeload
fetches, and the Node fs write sinks (writeFileSync, renameSync,
copyFileSync, cpSync, createWriteStream).
Follow-up to the admission-gate confinement:
- tests/ci: test_parser_imports_cannot_be_shadowed_by_checkout_files planted
`re.py`, but `re` is imported during interpreter startup, so a cwd copy never
shadows it and the test passed even with `-I` stripped from the workflow.
Plant `shlex.py` instead (the parser is its first importer): with a shadow
whose quote() swallows the rejection, the gate returns rc 0 / PASS without
`-I` and rc 1 with it, so the test now actually guards the fix.
- workflow: `timeout 120 git clone` -> 600. The clone is deliberately full
(the pin may not be the tip) and listed repos exceed 700 MB
(hermes-jackal-verified 1.43 GB, hindsight 741 MB); 120s would false-fail a
legitimate re-pin on a slow runner while 600s still bounds a hostile URL
well under the 30-minute job cap.
Attribution: beardthelion@users.noreply.github.com is already in the legacy
AUTHOR_MAP, so no mapping file is needed.
The pinned-source job joined a PR-controlled subdir onto the clone root
with no validation, accepted file:// repos and ref names as the "pin",
and followed committed symlinks out of the clone, so the gate could
validate attacker-planted files instead of the declared commit's
contents. Harden the step end to end:
- validate repo (https://, whitespace-free), sha (40-hex, commit object),
and subdir (plain relative path) in the parser before any use
- run the parser under python3 -I against installed ruamel.yaml so the
PR checkout cannot shadow its imports, capture parser output, and
reset shell variables per entry so a dead parser cannot replay the
previous entry's fields
- confine PLUGIN_DIR and every symlink beneath it to the clone root
- include renamed and typechanged entry files in the changed set
- feed entries over fd 3 and keep plugin-executed commands off stdin so
validate cannot drain the remaining list
- NUL-delimit both self-updater grep stages and sanitize the echoed
filename list so crafted names cannot forge workflow commands
- mirror the same repo/subdir rules in the runtime catalog loader and
the site extractor, and use \Z anchors so a trailing newline cannot
split the verdict between gates
`--branding stable` copies the identity from the published stable channel
record on R2, and refused when there was none. Production has no
releases/channels/stable.json yet: the channel pipeline merged after the
last stable release (v2026.9.24), and no stable release has published
through it since. So every stable-branded channel build failed at
allocation ("none is published"), including the first real dispatch.
When the stable record does not exist, use product_identity(), the
identity channel_publish already holds every stable release to. A stable
record without a head still refuses, so once the first stable release
publishes, its record is the only authority again and this path stops
running by itself.
The allocation step needs node for product_identity(), so the workflow's
identity-reader setup also runs for branding == 'stable'.
The E2E suites, the Windows E2E jobs, the Docker image build and the Nix
flake check took about 120 of the ~140 larger-runner minutes each PR push
spent, and fired on nearly every PR. The 32-core pool backed up behind
them.
On a pull request each slow lane now runs only when the diff touches the
suite, its harness, or the code it guards (classify_changes.py
_E2E_LANES, _DOCKER_PATHS, _NIX_LANE_PATHS), or when the PR carries the
run-e2e label. Every push to main still runs all of them, and a release
run still forces every lane on.
Adding run-e2e reruns the head commit's CI, Docker and Nix runs; detect
reads the live labels because a rerun replays the original event.
Replayed over the last 200 merged PRs, the modelled 32-core cost of one
push drops from about 120 to about 10 runner-minutes.
A preview channel always minted its own side-by-side identity (Hermes NAME,
ai.hermes.channel.h<token>), and a plain --build-commit carries
Hermes Agent <sha7> with a red icon, so there was no way to ship an exact
commit under the official name, icon and package ID.
--branding stable (with --channel) makes channel creation copy the identity
from the published stable channel record instead of reserving a new token.
Branding is fixed at creation like the rest of the identity: re-dispatching
with the other branding is refused, as is stable branding before a stable is
published or in a disposable namespace. --branding without --channel is
refused rather than silently falling back to a sha-branded one-off build.
The workflow gains a `branding` input, forwarded to the allocation step and
the dispatch log so the printed release.py replay matches what was sent.
atomic_config_write() now refuses deletion-by-omission at any mapping depth:
atomic_roundtrip_yaml_save treats keys absent from the payload as deleted
recursively, while the old docstring described the call as a merge, so a
partial dict could silently erase unrelated config (#125107). A new
atomic_config_replace() names the deliberate full-state replacement contract
explicitly, and the owners that intentionally remove state (save_config,
provider reset/switch, credential mirror scrub, doctor key migration, profile
channel stripping, TUI full-state saves) route through it. Additive writers
stay on the guarded atomic_config_write.
Salvages #125152 (identical patch, cherry-picked; conflict in
_write_user_config resolved to pass atomic_config_replace through the new
telemetry wrapper) and carries the executable bit main expects on
scripts/check_config_yaml_writers.py.
Fixes#125107
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
stable-release.yml called desktop-bundled-release.yml, docker.yml and
termux-verify.yml without `secrets: inherit`, so every job inside them that
declares `environment: release-signing` (or container-publish) attached its
deployment and read the environment's `vars`, but resolved `secrets.*` to the
empty string. The signed-candidate legs died at "Archive every pinned input"
with `missing env CLOUDFLARE_R2_ACCESS_KEY_ID`; the docker publish legs and
the termux packaging checks would have failed the same way.
A called workflow receives no secrets by default, and a job-level
`environment:` inside it is required but not sufficient: a caller job that
`uses:` a reusable workflow cannot declare `environment:` at all, so
`secrets: inherit` is the only mechanism that can carry them
(actions/runner#1490).
`secrets: inherit` is added only where the callee reads a protected
environment's secrets. ci.yaml, nix, pm-bundle, windows-venv-e2e and the
install-e2e chain reference none, and ci.yaml must stay secret-free by its own
header (it runs PR-controlled code).
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
* feat(docker): publish nousresearch/hermes-sandbox:desktop for terminal backends
The terminal backends (docker, modal, daytona, singularity) all default to
nikolaik/python-nodejs:python3.11-nodejs20, a bare Python+Node base. For Bot
Screen, computer_use and the browser to run INSIDE that sandbox instead of on
the gateway host, the sandbox image needs the display stack.
docker/sandbox-desktop.Dockerfile is that base plus:
- the everyday tools it lacked (jq, ripgrep, fd, tmux, less, nano, vim,
zip, rsync, tree, procps, htop, sudo for the base's uid-1000 `pn`)
- the exact package set the Hermes -desktop image installs (TigerVNC,
Xfce components, dbus, xauth, fonts)
- Playwright's headed Chromium (same build as the -desktop image)
- cua-driver 0.28.2 from its pinned release tarball
No Hermes inside; the default user stays root like the base so nothing
changes for people who just switch docker_image. Desktop processes run as
`pn`. 4.27 GB on amd64.
docker.yml gains a `sandbox` variant with its own cache scope and repository
(nousresearch/hermes-sandbox:desktop, :main-desktop, :<release>-desktop);
the docker-integration suite is skipped for it (no Hermes to test) and
docker/sandbox-desktop-smoke.sh runs instead: as `pn`, every launcher and
cua-driver binary resolves, the real launcher.sh publishes :20, the RFB
socket completes the 3.8 handshake relayed over `docker exec -i` stdio, and
a headed Chromium maps a window on that display. hadolint lints the new
Dockerfile in docker-lint.yml.
* ci(docker): build the sandbox image on release/dispatch, not every main push
Leaves docker.yml exactly as on main. The sandbox image carries no Hermes code,
so two 4 GB multi-arch builds per merge bought nothing. sandbox-image.yml builds
and smokes on a PR that edits its own Dockerfile/smoke, and publishes only on a
release or a manual dispatch with publish=true. Stable tag stays :desktop.
* feat(docker): sandbox desktop base on python3.13-nodejs26
Matches the Hermes image (Python 3.13 / Node 26) and the top of requires-python;
the default docker_image tag it inherited was Python 3.11 / Node 20. Same pn
uid 1000, Debian 13; smoke (launcher, RFB relay, headed Chromium) passes.
* feat(docker): sandbox desktop image bakes agent-browser and Pillow
The runtime PR wraps the browser tools' agent-browser invocation in the sandbox
and grabs the Screen pane thumbnail inside it (the X socket and its cookie live
there). Both need to be in the image the default tag publishes, or the runtime
finds a tag-compatible image without them. agent-browser is pinned to the range
the gateway resolves (tools/browser_tool.py AGENT_BROWSER_NPX_SPEC), scripts off,
so the first browser_navigate in a fresh sandbox never waits on npm. The smoke
checks both as pn.
* docker(sandbox-desktop): login sessions find the browser (PLAYWRIGHT_BROWSERS_PATH via /etc/environment)
Dockerfile ENV reaches docker exec only. As the target of the ssh backend
(sshd on top of this image) a login session gets its environment from PAM,
so agent-browser reported 'Chrome not found' over ssh while working under
docker exec. Write the browser location to /etc/environment as well; the
smoke test asserts a clean login environment resolves it.
install.ps1 now runs under the same watchdog as `hermes update` (40 min),
and the watchdog adds a py-spy stack for every Python process of the leg
before it stops them. The workflow installs py-spy beside pywinpty.
`windows: desktop-installer@latest -> hermes-update (HEAD -> NEXT)` and
`installer-script+desktop -> hermes-update (HEAD -> NEXT)` failed with
"source launcher publication failed". install.ps1 and `hermes update` ran
with setup-pm's HERMES_RUNTIME_DIR, while the desktop smoke launched the app
without it (as a user's app runs) and settled the checkout onto a second
store at <HERMES_HOME>\tools. The update then re-pointed its own running,
locked hermes.exe.
windows-e2e.ps1 now drops HERMES_RUNTIME_DIR, HERMES_PYTHON and VIRTUAL_ENV
on entry, so every product step of a leg (installer, app, update, chat)
resolves the one store a user's machine has. The driver keeps setup-pm only
through $DriverPython (and PATH, which older installers need for uv and
ripgrep).
`hermes update` also runs under a 45-minute watchdog. Past it, the driver
writes every process (pid, parent, start time, command line) to
logs\update-hang-processes.txt, stops this leg's processes and fails with
that table, instead of being cancelled blind at the job cap.
windows: desktop-installer@latest -> hermes-update (v2026.9.24 -> HEAD)
was cancelled at the 60-minute cap (job 108542725043) with nothing wrong:
its hermes update ran the historical venv->PM takeover, npm ci and a
Desktop rebuild in 24 minutes (one npm ci stalled 11 of them), exited 0 and
passed every post-update check, and the desktop smoke was cut off. Green
Windows legs already take up to ~40 minutes and the app-update driver may
wait 30. Real hangs stay bounded by pty-run.py and the per-step budgets.
The Result chart job lists the run's jobs and artifacts through the repo's
GITHUB_TOKEN, which shares one hourly budget with every other workflow. In
a busy hour it answered "API rate limit exceeded for installation" and
failed a run whose legs were all green (36283843710, 36285499760). Retry
after 30/60/90 s. If the API still refuses, write a warning and a summary
line instead of failing. Each leg's own conclusion still stands.
A red scheduled run blocked nothing and told nobody. install-e2e-red.yml
runs after every scheduled "Install & Update E2E" run: red opens one issue
labelled install-e2e-red (or rewrites the open one's body in place) with the
red legs grouped by failure class and linked; the first green run closes
it. No per-run comment, never a second issue.
It is its own workflow_run workflow because install-e2e.yml is also called
by stable-release.yml with read-only permissions, and a nested job asking
for issues: write would fail that call at startup. workflow_dispatch with a
dry-run default previews the change for any run id.
install-e2e.yml only ran on the clock, so nothing in front of a merge
installed a release and updated it on a real OS. A pull_request trigger,
path-filtered to the install/update surface (derived from 60 days of
update/install/pm commits), runs the new `pr` route of
generate-e2e-matrix.mjs with only the newest release tag sampled:
linux installer-script -> hermes-update (newest release -> PR)
linux installer-script -> hermes-update (PR -> NEXT)
windows installer-script -> hermes-update (PR -> NEXT)
macos installer-script -> hermes-update (newest release -> PR)
The bundle-manifest validation job is skipped on PRs (bundled legs need
dispatch-only manifests). The full matrix stays on schedule and release.
Trail from CI: 78+ nested 'git fetch origin --filter=blob:none --stdin'
lazy fetches, each with its own upload-pack on the seeded origin, until
the runner is OOM-killed. The origin serves full clones, so nothing is
legitimately missing; GIT_NO_LAZY_FETCH=1 for the installer run only.
The second CI run showed the OOM cause: hundreds of concurrent
upload-pack/pack-objects processes on the seeded origin.git, reaching
105 GB RSS about 90 s into seeding. Log the client-side git command and
its parent chain to find what drives them.
tests/ci/test_stable_release_graph.py requires every reusable-workflow
job in all-checks-pass; keep it there and let a red shard show without
failing the gate. Log a resource trail during the run: every shard's
runner died ~2 minutes into seeding on the first CI run with only
'The runner has received a shutdown signal'.
Drive the packaged app a real scripts/install.sh + hermes desktop install
built, against a local bare origin, a local api.github.com stand-in and a
scripted model: first run, Update now hand-off/relaunch/chat, a failed
Desktop build inside that update, and a second update while one runs.
Open bugs are message-gated (gated on #122991).
tests/e2e/core/upgrade/pm/: 22 cells in 7 files, one per failure class, each
driving the real entry points (install.sh, hermes update, hermes pm
repair/status/install, hermes doctor, hermes gateway run/install --force,
hermes kanban dispatch, hermes -z through the loopback provider) inside the
existing bwrap sandbox, seeded with an N-1 install and a dependency-changing
release:
- configured features survive update, repair and legacy-venv migration
- SIGKILL at the stage/publish boundary of a dependency update recovers
- stamps, receipts and doctor tell the truth (healthy, drifted, failed update)
- no stray in-tree venv / repo-local 3.11 .venv reaches any interpreter;
execute_code and Kanban workers spawned after an update boot and import deps
- repeated updates/repairs/restarts don't accumulate generations
- gateway install --force keeps the install bootable
Every non-gated cell has a recorded sabotage proof; open bugs are
message-gated with known_failure (gated on #124228, #122627, #122425,
#124214, #124075, #124049, #124668).
Harness: make_origin clones --single-branch and allows filters, so a
blobless developer checkout can serve the installer's clone.
CI: e2e-upgrade is sharded per suite directory (an e2e-upgrade-plan job
lists "core" plus every subdirectory holding test_*.py); a shard that
collects no files fails; HERMES_TEST_WORKERS=6.
