Every gate and every candidate now needs only admit, so the signed macOS
and Windows builds and the docker image stop waiting behind the full CI
run. acceptance stays the one join: it still needs ci, docker and all six
candidates, so what can be promoted is unchanged.
The four gates that skipped under --skip-tests only because ci skipped
(nix, termux-checks, windows-live, install-e2e) and pm-bundle needed their
own condition. SKIPPED_BY requires the gate to observe 'skipped', so
dropping the edge alone would have left them running and blocked the
release instead of failing it.
27df3b8847 dropped the exclusion 65e79880c8 added, so the 30-minute e2e
job (shallow checkout, no bwrap, 900s per file) ran tests/e2e/core/upgrade
next to its own e2e-upgrade job. Its install/update files then timed out
and the job was cancelled at the step limit on main and every Python PR.
gh v2.99 gained a native --attach flag for issues, PRs and comments, so
the custom trusted-publisher chain (gh-image extension + GH_IMAGE_SESSION_TOKEN
workflow_run job + attachment-upload script) is superseded.
Remove:
- .github/workflows/publish-e2e-evidence.yml (workflow_run publisher)
- scripts/ci/publish_e2e_evidence.py + its tests
- e2e-evidence-* artifact upload + evidence staging in e2e-desktop.yml /
e2e_screenshot_status.py, incl. the 'inline evidence is publishing...'
marker placeholder in the CI review comment status
Keep: the review-comment screenshot/diff counts and artifact links produced
by e2e_screenshot_status.py.
The extra removal left CI, the Docker image and the nix package still
requesting `hindsight`. Once the extra is gone, `--extra hindsight` and
extraDependencyGroups = [ "hindsight" ] ask for something that no longer
exists. Drop them the same way 73c598e319 originally did: remove it from
the CI extras lists, the Docker sealed-venv build and the nix default
groups, and point the nix examples/check at honcho. Also remove the
stray blank line left in the exclude-newer table.
Hermes-Setup has never had a CI build; every published copy was built by
hand. This workflow_dispatch lane builds the Windows x64 exe (signed via
the desktop MSIX's Azure Trusted Signing path, batch-sign-binaries.mjs)
and the macOS arm64 dmg (Developer ID signed, app + dmg notarized and
stapled) from any ref, and uploads them as run artifacts. Nothing is
published. No caches: no actions/cache or setup-node cache, and fresh
npm/cargo/electron-builder cache dirs.
User installs failed with 'resvg-py is missing' because the web/desktop
source builds rendered icons on whatever python was on PATH. The default
brand outputs are now committed; source_build, apps/desktop build.mjs and
the npm/docusaurus pre-hooks consume them directly. Flavored release
bundles (canary/commit) still render into their own product dir.
icons-freshness-check now regenerates and fails on any byte diff.
The e2e lane at 4 workers still starved the PTY turn and serve-SIGTERM
deadlines intermittently; drop it to 2. The upgrade lane had no cap and
ran cpu_count real-updater trees at once; cap it at 4.
The ubuntu-latest-32-* larger runners boot ubuntu-22.04 (glibc 2.35), and
the prebuilt llama-server in the native payload needs GLIBC_2.38, so the
staged-entry verification fails. Restore the two entries once the runner
images are updated.
The ubuntu-latest-32-* larger runners boot ubuntu-22.04 images, whose
system python3 is 3.10. The bundle bootstrap imports tomllib (3.11+), so
linux-arm64 failed before doing any work. Pin the interpreter instead of
trusting the image.
The dashboard PTY test failed on this branch with "node: not installed and
lazy installs are disabled". On this branch _make_tui_argv asks PM for node
(hermes_cli/main_tui_launch.py _tui_node_bin). The E2E sandbox forbids lazy
installs (HERMES_DISABLE_LAZY_INSTALLS=1 in tests/e2e/core/dashboard
hermetic_env). It gets node only when the runner's PM store has it
(tests/e2e/core/_pm_dependencies.py). The e2e job got node from
actions/setup-node, so node was on PATH but not in the PM store. main passes
the test because main's TUI launch takes node from PATH.
Install the locked toolchain (setup-pm toolchain: all) before npm ci, in
place of setup-node. The locked Node 26 also satisfies the boot-contract
suite. When HERMES_E2E_REQUIRE_TUI=1, the fixture now fails at once if the
store has no node, not later with an opaque 1011 close.
Contributors had to run `python -m pm.build_env --source . --lock-only`, then
re-source activate, then call sync_venv for opt-in extras, while `hermes pm
lock` only pinned tool artifacts in pm/lock.json. Now `hermes pm lock` with no
arguments is the one step: it runs the same PM check_project_lock /
lock_project operations as build_env's --check-lock / --lock-only (same
exclude-newer quarantine), writes nothing when the lock is current, changes
no environment, and prints the activate command to run next. Activation
syncs [all] plus recorded extras only, and --test-extras replaces the
default, so a newly added extra outside [all] gets
`source ./activate --test-extras all,NAME` (`-TestExtras` on PowerShell).
`hermes pm lock --bump NAME VERSION` keeps writing pm/lock.json; NAME and
VERSION are now only accepted together. build_env --lock-only is unchanged
for scripts. Contributor docs, AGENTS.md, CONTRIBUTING.es.md, the pyproject
comment, and the uv.lock CI remediation text point at `hermes pm lock`.
release.py was 2,804 lines, 2,076 of them the frozen legacy author dict.
The map and its resolver now live in scripts/releases/: authors.py holds
the directory loader, the merged AUTHOR_MAP and resolve_author, and
authors_legacy.py holds only the frozen LEGACY_AUTHOR_MAP literal (same
1,895 entries, same order). release.py drops to 707 lines.
The contributor-check job and audit_pr_attribution.py grep the legacy
file for quoted emails, so both now read authors_legacy.py. The
importers (contributor_audit.py, add_contributor.py), contributors/README
and the tests read the defining modules. No behaviour change.
`release.py release` gains two flags. They can be used together.
--skip-bundles ships only the claim, the GitHub release, the final tag
and the Docker image. No desktop, Termux or PM bundle job runs. The
final tag records candidateManifestSha256: null. Publication moves only
the Docker stable/latest aliases. The R2 stable head, feeds, APT, the
downloads page, the signed-package baseline and the Store stay on the
previous bundle release.
--skip-tests builds, signs and publishes every artifact and runs no
test job: source CI, Nix, PM bundle check, Termux, Windows live,
install/update E2E, bootstrap identity, native smokes, upgrade
acceptance, tests/docker and the in-build vitest step. The candidate
manifest records each smoke as skipped, never as passed.
The flags live in the claim message (skipBundles, skipTests), next to
autopublish. They are not workflow inputs, so a rerun cannot change
them. admit emits them, and every job condition and gate reads them.
stable.validate_claim and stable.validate_final are now the one shape
check for stable.py and the sequencer.
The gates stay strict. SKIPPED_BY in stable.py maps each job to the
flags that remove it. `gate` requires those jobs to report skipped and
every other gated job to report success. A job that ran although a flag
removes it blocks the release.
A release that skipped bundles never moves the R2 stable head. Two
readers depended on that head:
- The next version was derived from it, so the next cut would reuse the
version. It now takes the newer of the R2 head and the newest
published non-prerelease GitHub release with a vX.Y.Z tag. Bare v*
tags do not count, because those refs are not protected yet.
- The sequencer used it to decide which published releases still need
their publication pass, so a bundle-less release would re-advance
every 15 minutes. The head is now the newer of the R2 head and the
published release whose final tag binds the Docker stable alias
digest.
`release` also refuses a cut when its next version already has a final
tag. That closes the window between the final tag and the public
release, where the published identity still names the old version.
Tests: 42 release test files, 546 passed. Three tests fail on this
Windows host, and they fail the same way on a clean HEAD worktree:
- test_stable_release_graph::test_docker_recovery_refuses_to_replace_a_divergent_version_tag
- test_release_artifacts::test_windows_metadata_is_read_from_package_and_stale_stamp_is_rejected
- test_tag_builds_summary::test_admitted_failure_publishes_tag_info_without_promoting_channel[True]
Not verified: no real Stable Release dispatch ran with either flag, and
actionlint is not installed on this host. The workflow changes are
checked by the graph tests and by running the phase-result step script.
tools/lazy_deps.py was not deleted; it survives as an old-updater stub
that raises or stops for relaunch. The job display name and the
checker/test prose claimed it was deleted, which sends readers looking
for a missing file. The display name is not a required status check
(main requires only "All required checks pass", which keys on job ids),
so rename it to "No production imports of the tools.lazy_deps stub".
Stable Release Publication ran every 15 minutes (96 runs a day, each
checking out full history, setting up node and buildx, logging into
Docker Hub, and taking the release-signing environment) only because the
sequencer held a failed run for a 15-minute backoff that the failure
event could never satisfy, so the cron was what actually retried.
Drop the backoff: the reconcile pass started by a failed Stable Release
reruns its failed jobs right away. MAX_ATTEMPTS burning, oldest-first
retry ordering, the attempt-entry check, and the needs_retarget repair
stay. The schedule trigger goes; workflow_run and workflow_dispatch
remain the recovery paths.
The shared stable-release concurrency group cannot deadlock: the rerun
waits as pending behind this job, and the sequencer only confirms the
new attempt is queued before it exits and frees the group.
Upstream's e2e-desktop-core job provisioned setup-node + uv sync and pointed the
packaged smoke at a checkout .venv. Provision through setup-pm, hand its selected
interpreter to Desktop via HERMES_DESKTOP_PYTHON, and read the packager contract
from electron-builder.config.cjs, where the build config now lives.
The workflow edit trips the review-label gate. The new test is marked
windows_only, so tests-os.yml's '-m windows_only' job on windows-latest
already collects it.
The 32-file Linux lane runs many more child processes than CPU cores. Hosted children remained alive without readiness output and exhausted per-child deadlines across unrelated suites; four concurrent files complete the focused tenancy, terminal model and compaction cases locally without weakening their assertions.
Same runner class as the os-tests Windows row. Seven files run in
parallel, each spawning process trees; measured on the fixed suite the
test step drops from 153-171 s to 124 s and the job from ~5 min to 3.1.
The serve tree-kill test snapshotted process_tree(first.pid) and then
checked that same tree after `taskkill /T /F` - the snapshot is exactly
what taskkill /T kills, so the check could not fail. Windows never
re-parents: a grandchild spawned detached (cmd /c start /b ...) keeps a
dangling ppid and is invisible to both Process.children() and
taskkill /T. Reviewer sabotage (web_server spawns two detached sleepers
before READY) passed while both orphans survived.
Survivors are now every live process created since the spawn whose
HERMES_HOME, cwd or argv points into the test's scratch profile
(owned_processes), with a positive control that the scan sees the
backend itself before the kill. Same check for gateway stop. Cleanup
kills everything the profile owns.
Also: state-db guard matches the holder PID as a whole number, and the
e2e-windows job ends with a scan that fails on any python.exe /
hermes.exe left running.
Adds tests/e2e/core/windows (windows_only + integration; 18 tests, 5 strict-xfail KNOWN
entries for #120504#121150#121015#121114#120205) and an e2e-windows job in
tests-os.yml running it on windows-latest, one pytest process per file, no retries.
route already means "which legs run"; a leg name is the most specific
route. Presets keep their meaning (all, the linux trio, windows-desktop,
macos-desktop, the bundled three); any other value selects legs by name,
so a leg name, a fragment of one, or the job name GitHub shows
("<leg> / e2e") runs just those legs. A route that selects nothing fails
instead of producing a green empty run.
The generator is now the one interpreter of route for source legs: the
linux/windows/macos jobs run when it selected legs for them, replacing
three hand-kept preset lists. Dispatch route becomes a string input (a
choice cannot take a leg name) and reaches the gen step through env, never
interpolated into the script.
A push-triggered canary rebuilt the whole desktop matrix on every main
push. A daily schedule caps automatic canaries at one per 24h; release.py
already exits clean when HEAD carries the last canary tag, so a quiet day
ships nothing. workflow_dispatch fires one on demand, gated to the default
branch so a feature-branch dispatch cannot tag unmerged code.
Every leg installed a release tag and updated to HEAD, so nothing ran
HEAD's installer on an empty machine (where all four 2026-09-23
install.ps1 breaks lived) and nothing exercised the updater we ship
today -- tag legs run the OLD build's updater handing off to HEAD.
The generator appends a HEAD -> NEXT start after the sampled tags, so the
column runs wherever the update legs run (dispatch and stable-release).
NEXT is a reserved update ref: the drivers mint a child of the install
commit that adds one marker file, written to the object store only, which
the local bare clone carries into serve.git.
On Windows the HEAD leg takes every git.exe dir off PATH and installs no
remote get-url shim: Get-PinnedGit returns any git on PATH, so either one
skipped pinned-git staging. launch-from-spec's HEAD observer now uses the
driver's real git so it cannot poll '' forever on that leg.
The published image had no Xvnc/Xfce because nothing set the Dockerfile's
HERMES_BOT_DESKTOP argument, and a hosted instance (unprivileged, no sudo,
sealed /opt/hermes) cannot install at run time. The image layer is the only
delivery path.
- docker.yml: variant axis [slim, desktop]. :latest / :main / :v* stay the
image they are today; :latest-desktop / :main-desktop / :v*-desktop carry
the packages plus Playwright's headed Chromium. Slim owns the build cache
scope; one manifest per variant so a desktop publish failure never skips
slim's :latest.
- Dockerfile / stage2-hook.sh: XDG_RUNTIME_DIR=/tmp/hermes-runtime seeded
0700 as hermes (containers have no logind; the $HOME/.cache fallback was
the shared /opt/data volume), refused when foreign-owned; deterministic
Chromium discovery exporting the headless shell for ordinary browsing.
- bot_desktop: memory gate reads the cgroup working set (usage minus
inactive_file) so it cannot tighten over uptime and refuse to restart a
screen idle-stop just stopped; installable() gives three distinct dead-end
messages instead of a sudo line nobody there can run; env_for_agent
replaces a headless-shell pin so agent and dock share one Chromium.
Squash of IAvecilla/hermes-agent:bot-desktop-cloud-image (#112381, 13
commits), which GitHub auto-closed when its base branch merged as #108914.
Review fixes from pefontana (cache scope, per-variant merge, red browser
test) are included.
Co-authored-by: pefontana <pefontana@users.noreply.github.com>
Conflict resolutions and semantic fixups:
- tools/environments/base.py: main's hard-exit kill fence (kill a spawn the
fence missed, deregister from _live_foreground in a finally) wrapped around
pm-clean's output collector.
- pyproject.toml: pm-clean's marker list plus main's new `live` marker.
- hermes_cli/main.py: pm-clean runs startup recovery from hermes_bootstrap, so
the old early-recovery block stays gone; main's interrupted-pull restore
(auto-merged above it) runs right after bootstrap, as on main.
- hermes_cli/update_cmd.py: main's interrupted-pull marker now guards
pm-clean's first tree mutation (release-tag detach, ff-only, or reconcile)
and is cleared once git is done. The marker's target is the ref git actually
moves to (a release tag, not always origin/<branch>), since the restore
compares against it.
- hermes_cli/_early_recovery.py: restore `import subprocess`, which pm-clean
had dropped and main's auto-merged restore needs (NameError on the first
launch after a killed update; test_update_interrupted_pull red -> green).
- apps/desktop/src/i18n/{de,es,fr}.ts: main's new locales carry the full
settings.about block; trim it to `updates` as pm-clean's type and the other
overlays do (tsc: 27 errors -> 0).
- main's new e2e tests: `import yaml` -> hermes_yaml; wake-word import table
names pyopen_wakeword (pm-clean's wake-openwakeword extra); the anthropic
key-leak switch leg needs the SDK, and the api_server two-tenant test needs
aiohttp, both PM runtime extras the test env does not carry.
The terminal job only needs the Ink TUI, not every workspace (desktop/electron).
The e2e-upgrade bwrap check used different flags from _bwrap_usable (no --proc,
no --die-with-parent), so it could pass while the suite fell back to running the
real updater unsandboxed; it now asserts _helpers.BWRAP_OK itself.
The six provider keys sat in job-level env, so every step saw them: uv sync
(and any sdist build backend it runs), setup-uv, checkout and the retry
action. The job now carries only secrets.X != '' booleans for the gate, and
the values are set on 'Run live canaries' alone. actionlint clean.
tests/e2e/core/terminal drives the real `hermes --tui` over a PTY, so the e2e
job now installs the Node workspaces and builds ui-tui, and
HERMES_E2E_REQUIRE_TUI=1 makes a missing build fail instead of skip.
tests/e2e/core/upgrade runs a real N-1 -> HEAD `hermes update`: it needs full
history + tags, bubblewrap (every updater runs sandboxed so it can never reach
a real gateway or systemd), the warm uv cache, and up to ~15 min for one file.
It gets its own 60-minute job instead of stretching the e2e job.
Runs tests/e2e/core/live (`-m live`) nightly, on `v*` tags and on
workflow_dispatch (optional -k filter). Secrets-gated on LIVE_*_API_KEY
repo secrets (each case skips without its key; the job no-ops when none
are configured), main-repo only, one run per ref (never cancels a release
gate), 25-minute timeout. Uses direct pytest because scripts/run_tests.sh
starts from `env -i` so no credential can reach a test. Publishes a
usage/cost table to the step summary and uploads junit + usage JSONL.
(cherry picked from commit 75c5656ff3905512bf93c1fd887bbd5e85396f29)
A stalled stream burned the 600 s per-test timeout until the 30-min job
timeout cancelled the lane, and the failure()-only upload then saved no
traces. The CLI --reporter flag also dropped the config's html report.
The onboarding spec is a smoke test; #120005 needs a non-default profile.
The legacy visual Playwright lane stays disabled; the core suite gets its own
reusable workflow (retries 0, one worker, failure-only artifacts) gated on the
same python_prod/frontend classification and counted by All required checks
pass. The legacy config ignores e2e/core so the specs never run twice.
