rc.11-v0.21.7's Windows x64 MSIX smoke failed with 'Artifact checksum
mismatch' on the 2.6 GB package. The stored object was correct (re-downloaded:
size and SHA256 match the receipt exactly), so the runner received bad bytes.
_download_url retried HTTP and connection errors but raised on a completed
body that failed its hash, so one bad transfer failed the job and its
dependents until a whole-run rerun.
A size/SHA256 mismatch now counts as a bad transfer and is fetched again
within the same three attempts. A stored object that really differs from its
receipt fails all three the same way, and the verified destination is never
replaced.
Review follow-ups: `providers.<id>.models.<model>.timeout_seconds` now
matches (a second <...> segment no longer ends the pattern), the YAML
warning filter is scoped with catch_warnings, and the slash-command list
drops a `hidden` filter CommandDef never had.
DEFAULT_CONFIG changed 183 times in 30 days and nothing checked that a new
key reached the docs; /pause (the global emergency stop) went undocumented
for two months.
scripts/ci/check_docs_drift.py lists every DEFAULT_CONFIG leaf and every
registered slash command and finds the ones the docs never mention. A key is
documented by its dotted path, a placeholder path in inline code
(`auxiliary.<task>.timeout`, `display.pet.*`), or a YAML example nesting it
under its parents; a slash command by a `/name` row in
reference/slash-commands.md.
The 132 keys undocumented today are baselined in docs_drift_baseline.json;
tests/ci/test_docs_drift.py fails only on NEW gaps, so it never blocks an
unrelated PR, and documenting a baselined key never fails. Deterministic,
stdlib + repo imports, ~1s.
/pause is documented here, so the slash-command baseline starts empty.
contributing.md says what to do when the test fails.
The admission gate ran the install scanner (dangerous fails) but nothing refused
a compiled binary, committed bytecode, or an archive hiding either — the shapes
the scanner cannot read. This adds that enforcement to pinned-source-validate.
- scripts/scan_plugin_payloads.py (stdlib-only, so the no-install validator and
third-party CI can run it): flags compiled binaries by magic header (ELF,
Mach-O, PE/MZ, WebAssembly — not by extension, which a rename defeats),
.pyc/.pyo bytecode (incl. sourceless/hash-forged caches that run instead of
the .py beside them), and archives whose members are not reviewable source.
Plain-text source bundles and inert assets (fonts, images) pass.
- plugin-catalog-ci.yml: after `hermes plugins validate`, each changed entry is
scanned; a finding fails it. An entry may list `attested_binaries:` paths,
each verified with `gh attestation verify --repo <repo> --source-digest <sha>`
against the pinned commit; verified paths are dropped before the scan, any
other binary still fails. Repo-controlled filenames are newline-stripped in
the log so none can forge a workflow command.
- validate_plugin_catalog.py: `attested_binaries` is a known key, shape-checked
as a list of relative paths.
- Tests: unit tests for the detector and two new workflow-driven bypass cases
(bundled .exe, committed .pyc) in the confinement harness, which runs the real
`run:` block.
Full-content re-sweep of all 543 pins: only tabro carries a native binary (it is
delisted in the companion PR). Three entries (crawl4ai, hermes-vision-inline,
mermail) commit __pycache__/*.pyc build litter and would fail rule 17 on their
next pin bump — pre-existing, not touched here.
rc.10-v0.21.7's "Stage verified stable candidate artifacts" job failed with
ValueError: Built package provenance does not match the release
assemble() ran stamp_matches() on each metadata row. stamp_matches() is the
check for an install-stamp.json and requires baseVersion, but a metadata row
is what record() writes (platform, arch, tag, commit, identity, version, ...)
and never carries baseVersion. So every bundled stable release failed here
once all packages were built. record() already verified each package's own
stamp (including baseVersion) when it wrote the row.
The row check is now what the row actually contains: commit and tag. The
staged-candidate fixture wrote a baseVersion into its rows, which is how the
mismatch stayed hidden; it now writes exactly the keys record() does. With
the corrected fixture, main's code fails 8 tests with the production error.
The prompt-surface-review ruleset was meant to require one other reviewer;
it is now set to one hermes-agent-core approval. CODEOWNERS, the CI comment,
the staleness-test message and the developer docs said two.
A one-line code change can rewrite what every session sends to the model -- a
guidance constant moving tiers, a toolset gaining a tool, a dynamic rewriter
editing a description -- without the diff showing it. The rendered prompt
surface is now committed and reviewed like code.
- scripts/ci/prompt_surface.py renders the system prompt
(build_system_prompt_parts) and the agent's tools[] through the production
code for nine surfaces (cli x claude/gpt/gemini/hermes, cli in a code
workspace, desktop, telegram, discord, cron) into
tests/fixtures/prompt_surface/. A fresh home in a child process; clock,
host OS, network, tool availability, env probe and machine paths pinned,
so a laptop and CI produce the same bytes. Each tool schema is stored once,
with a per-surface copy only where a surface serves a different schema.
- tests/ci/test_prompt_surface.py fails when the committed snapshot is stale,
so a surface change must carry the regenerated snapshot in the same PR.
- prompt-surface-diff.yml (advisory, PR-only, stdlib-only, no PR code runs)
posts per-surface token deltas, prompt hunks and per-tool
description/param changes into the unified review comment.
- CODEOWNERS requests hermes-agent-core on the snapshot; a new
prompt-surface-review ruleset requires two of their approvals.
- Docs: prompt-assembly.md (how it works), contributing.md (what to do when
the test fails).
Review on #136434 (@jquesnelle): discover() sorted records by version only,
so attempts inside a version kept git ls-remote's NAME order (rc.10 between
rc.1 and rc.2). reconcile indexes records by version with the last one
winning, so once a version reached rc.10 it handed rc.9 (burned, no release)
to publish_attempt, which refuses with "Final tag release database ID is
invalid". v0.21.7 is at rc.9, so its next cut is exactly rc.10.
- Sort by (version, attempt).
- The test fakes list refs sorted by name, as git does; insertion order hid this.
- rc.10 test (fails without the sort), and a same-commit retry test (rc.1
abandoned, rc.2 re-cut at the same commit wins).
Every Stable Release Publication run since v0.21.6 has crashed in
sequencer.discover():
ValueError: v0.21.6 points at a different commit than rc.1-v0.21.6
v0.21.6 took four attempts: rc.1-3 were abandoned (each has its
abandoned-rc.<N> marker) and rc.4 won and became v0.21.6. discover() tested
"does the final tag exist" before "is this attempt abandoned", so for rc.1 it
compared v0.21.6's commit (rc.4's) with rc.1's and raised; the abandoned
branch was only reachable for versions that never got a final tag. Any
version whose winner was not rc.1 broke every later reconcile, including the
next release's publish.
The abandoned marker is now checked first: an abandoned attempt is burned
whatever its version became. A NON-abandoned attempt whose final tag points
elsewhere still raises.
The posix hand-off hands marker line 1 to a custodian subshell
(marker_refresher_start) and then exported HERMES_UPDATE_HANDOFF_PID="$$" —
the hand-off script's own pid.
`hermes update` adopts a live claim only when the owner is its own pid, the
HERMES_UPDATE_HANDOFF_PID value, or an ancestor
(update_lock.UpdateLock._is_partner). The custodian is a *sibling* of the
hand-off script, so it is none of the three and the ancestry walk can never
reach it: every desktop-initiated update refused the claim its own script had
just made and exited 2 ("Another Hermes update is already running ...
process <custodian>"), leaving the Desktop to relaunch and retry forever.
Export $MY_PID instead: the custodian once the handover succeeded, and the
script's own pid when it did not.
Fixes#134268
Review thread on #136050 (agent/conversation_compression_telemetry.py,
"Refusals inflate batch counts"). Since #135807 every exit of
`compress_context` (blocked by a guard, skipped, aborted, refused) logs one
`context compression attempt telemetry:` line, and
`scripts/micro_compaction_report.py` counted every such line as a batch
compaction. Keep only records whose `commit_status` is `committed`.
The builder shipped ~1.9k skills.sh rows per run with no resolved_github_id.
Most were rows whose repo 404s or whose fully-read tree has no matching
SKILL.md dir (renamed/removed upstream); every client install of one spent
~40 GitHub calls and failed as stale_index.
- fuzzy match ignores both '-' and '_' (pdb-database slug vs pdb_database dir)
- a single-skill repo (root or generic skills/ SKILL.md) resolves like the client does
- repo 404 or complete tree with no match: the row is dropped, counts logged
- truncated tree: bounded contents/<candidate>/SKILL.md probes, row kept if none hit
- transient errors keep the row unresolved, as before
A release that skipped bundles never writes the stable baseline pointer,
so its absence is normal at any point, not only before the first bundle
release. Drop the final-tag scan. A 404 on the default pointer now always
plans no upgrade arm. A supplied baseline-manifest that does not exist
still blocks, because that is a bad input.
No stable release has shipped signed packages yet. Every final tag so far
skipped bundles or predates the claim pipeline. So
releases/stable/release-candidates.json does not exist, and the three
"Pin actual OLD and NEW signed packages" jobs refused with "No published
stable package baseline".
transitions now treats a 404 as a first release only when no final stable
tag records a candidate manifest and no baseline-manifest input was given.
It still validates the receipt, emits empty matrices and baseline=none.
The packaged upgrade arms skip on that output. The gate accepts their skip
only when their own planner succeeded with baseline=none, so a missing
baseline after a real bundle release still blocks.
`_kill_tree` SIGKILLs the per-file pytest process group, which misses any
child that left the group with `start_new_session=True` (an auto-started
`gateway run`, a setsid server). Those are reparented to init and run
forever: one host accumulated 215 orphan gateways (~52 GB RSS, up to 21 h
old), and every `hermes update` listed each as "another Hermes home — left
alone".
After every attempt the runner now SIGKILLs any process whose environment
still points into that attempt's unique temp root (PYTEST_DEBUG_TEMPROOT /
TMPDIR, or a HOME / HERMES_HOME a fixture created under it). The root is
unique per attempt, so the match names only that attempt's processes.
Linux-only (/proc); the process-group kill stays the fallback elsewhere.
Pull requests and pushes to main no longer run any E2E suite: ci.yaml's detect gate
forces every e2e* lane off unless the run is a stable-release run (inputs.release) or a
manual dispatch, whatever the diff, the update-path classifier or the run-e2e label
selects. That covers the Python e2e and e2e-upgrade jobs, the Windows E2E and Windows
install + update journey in tests-os, and the Desktop core/update suites.
install-e2e.yml drops its pull_request (path-filtered `pr` route) and 12-hourly
schedule triggers; it runs from the stable-release gate (workflow_call) or by dispatch.
live-providers drops its nightly schedule and keeps the release-tag push + dispatch.
Why: a one-row change to left_core_migration.py held a PR for 40 minutes on the
Windows install + update E2E; the suites cost most of the larger-runner minutes and
the release gate already runs all of them strictly.
Vite inlines every VITE_* variable into the renderer and NODE_ENV selects its mode, but the
desktop receipt recorded only sources and prepared inputs. After `VITE_PERF_PROBE=1 npm run
build`, the next plain build reused (and productCurrent certified) the renderer that still
carries the perf probe; a NODE_ENV change had the same gap.
The desktop build inputs now record the renderer environment (VITE_* plus NODE_ENV, unset
read as vite's production default), so both rendererCurrent and productCurrent require it to
match. Reported by Greptile on #135092.
rendererCurrent and productCurrent share hostReceipt() for the schema/product/host checks. The stamped outputs come from bundleElectronMain's own electronOutputs list, so a renamed or added bundle can never be copied stale from the previous dist. The reuse copy skips node_modules/ and native/, which copyNativeTree recopies from nativeDeps right after, and a --typecheck build never reuses a renderer that skipped tsc.
With --no-checkout the file download moved out of the retried clone into the single unretried pin checkout, which ran after the empty clone was already published: a throttled pin left INSTALL_DIR with no index, and the next rerun stashed a 'delete every file' autostash. The pin now runs in the staged tree inside the deferred-checkout retry loop (which also stops materializing the tip first), and nothing is published until it succeeds.
Mirrors the install.sh change from #135092 on Windows. With -Commit (what
the desktop bootstrap passes), the fresh clone runs with --no-checkout and
the pin is checked out with retries inside the staging dir, so the branch
tip is never materialized first and a refused or failed pin never publishes
a checkout. A rerun over an existing checkout keeps pinning after the fetch.
The tagger-line lookup indexed into a single string, so the epoch was the
character 't'. A -Python path that does not exist left LASTEXITCODE at 0, so the
script printed done and listed a previous build's package.
Review findings on the local build scripts:
- Cleanup ran before desktop.py took the checkout lock, so a second run deleted
the outputs of a build that was still going. desktop.py gains --clean, which
removes this checkout's outputs after build_lock is held. Both scripts pass it
instead of deleting files themselves.
- build-bundle.ps1 left RELEASE_CLAIM_* and HERMES_PAYLOAD_* in the caller's
session. It now saves those variables and restores them in the finally block.
- The scripts said "unsigned" but inherited signing credentials. The ps1 removes
AZURE_* for the build. The sh removes CSC_* and APPLE_* and sets
CSC_IDENTITY_AUTO_DISCOVERY=false, the switch the desktop rebuild command uses.
The wording now says what is ignored.
- The Store bundle step needs the claim timestamp, and the script deletes the
claim. The ps1 prints it as HERMES_RELEASE_EPOCH, and BUILDING.md documents it.
scripts/build-bundle.sh (macOS, Linux) and scripts/build-bundle.ps1 (Windows)
build the checked-out commit through scripts/bundles/desktop.py --commit. That
path needs no tag and no push.
Both scripts refuse a dirty tree before any work starts. Both delete the
previous build output first. The staged Windows payload is read-only, so a
plain delete in a reused checkout fails with WinError 5.
build-bundle.ps1 -Store builds the Microsoft Store MSIX. The driver admits
Store packaging only for a stable tag. The script makes a local claim tag
rc.<N>-vX.Y.Z for the next patch after the newest published stable tag. It
counts local and remote attempt refs and starts at 900, so it cannot collide
with a real attempt. A finally block deletes the tag. The script never pushes it.
The payload pinned libc++ 29 while the build container's apt cmake needs 30,
so the two disagreed and the shim in the previous commit papered over it. The
pool has since retired 17 of our pins, so repin them all with
`hermes pm update --termux`, which fixes the mismatch at the source.
Repinned: libc++ 29 -> 30 plus 14 more runtime libs (fribidi, glib, harfbuzz,
libaom, libbluray, libexpat, libheif, libmp3lame, libpng, libvidstab, openssl,
pcre2, readline, vulkan-loader-generic) and the bionic lock rows for uv
0.12.15 -> 0.12.23 and ffmpeg 8.1.2-5 -> 8.1.3. The pool serves ffmpeg 8.1.3
only, so that row moves back from 9.0.1 as the pool has it.
Staging with the new table succeeds: 84 packages, 269 .so, same counts as
before. The libc++ SONAME is unchanged. The one unresolved DT_NEEDED
(libharfbuzz-cairo -> libcairo) is present on the old table too.
The shim is removed. What stays from it is the in-container
`cmake -E capabilities` check, which fails in 2 minutes if the table drifts
from the pool again, instead of after 30 minutes of compiling.
The wheelhouse build stage still compiled cmake from PyPI source (pillow ->
pybind11 -> cmake bootstrap, exit 11) after the Dockerfile and the apt install
began upgrading libc++ to 30. LD_LIBRARY_PATH put the payload's pinned libc++ 29
first, and a library found there beats the binary's own RUNPATH, so apt's cmake
died with 'cannot locate symbol _ZNSt6__ndk113__hash_memoryEPKvm' and
scikit-build-core read that as 'no cmake'.
Probed in the pinned termux-docker image under qemu with the same apt package
list: payload libc++ 29 first fails, a one-symlink dir to the system libc++ in
front of it starts cmake and clang. The build now makes that dir after apt runs
and fails fast if cmake cannot start, instead of burning 30 minutes compiling
cpython, cryptography and pillow first.
The ordered publication pass moves nousresearch/hermes-agent:stable/latest
only after it reads the R2 stable channel head. When that read fails
(v0.21.6: "Channel repository authority mismatch"), the release publishes
and the aliases never move.
`python -m scripts.releases.docker promote --version X.Y.Z` reads the
published vX.Y.Z receipt tag and runs the existing promote_stable with the
digest it binds (slim checked against the receipt, desktop from its own
attempt tag). docker-promote-stable.yml dispatches it with the Docker Hub
credentials only CI holds.
Two stable-release identity checks broke on the first pipeline cuts of v0.21.6:
- nix.yml "Verify release package runtime identity" took the LAST token of the
whole `hermes --version` output. That output ends with the OpenAI SDK line, so
the check compared 0.21.6 against 2.24.0 and failed every release. Parse the
first line ("Hermes Agent v<version> ...") instead.
- msix-shared.mjs releaseEpoch() still required the retired `vX.Y.Z-rc` claim
name. Attempt refs are `rc.<N>-vX.Y.Z` (versioning.parse_attempt_ref), so the
Windows universal bundle threw "Stable claim tag differs from its payload tag".
Accept exactly `rc.<N>-<tag>`; a different version (rc.3-v0.27.10 for v0.27.1)
still refuses.
The pinned termux base image ships libc++ 29. apt's cmake 4.4.4 needs a
newer libc++, but its Depends has no version bound, so apt keeps 29 and
cmake dies at load with an unresolved libc++ symbol. scikit-build-core
reads that as no cmake and compiles one from the PyPI sdist, whose
bootstrap exits 9. That killed the pybind11 build pillow needs, so every
termux .deb build since Oct 3 failed.
Name libc++ in both package lists so apt upgrades it.
abandon refused with 'no outstanding attempt' when a version had two, the
same branch as zero, despite its docstring promising to clear one. It now
clears all of them, oldest first.
release.py changelog prints the notes the next draft would carry without
claiming an attempt. When the body is over GitHub's limit, the refusal now
says to cut with --no-changelog, dump the changelog, edit it down by hand,
and paste it into the draft.
A local-only rc.* tag (a hand-made ref or a claim that never pushed) counted
as an outstanding attempt, so release refused and abandon reported 'no
outstanding attempt'. The remote is the only authority on claims now.
