The extra removal left CI, the Docker image and the nix package still
requesting `hindsight`. Once the extra is gone, `--extra hindsight` and
extraDependencyGroups = [ "hindsight" ] ask for something that no longer
exists. Drop them the same way 73c598e319 originally did: remove it from
the CI extras lists, the Docker sealed-venv build and the nix default
groups, and point the nix examples/check at honcho. Also remove the
stray blank line left in the exclude-newer table.
Same runner class as the os-tests Windows row. Seven files run in
parallel, each spawning process trees; measured on the fixed suite the
test step drops from 153-171 s to 124 s and the job from ~5 min to 3.1.
The serve tree-kill test snapshotted process_tree(first.pid) and then
checked that same tree after `taskkill /T /F` - the snapshot is exactly
what taskkill /T kills, so the check could not fail. Windows never
re-parents: a grandchild spawned detached (cmd /c start /b ...) keeps a
dangling ppid and is invisible to both Process.children() and
taskkill /T. Reviewer sabotage (web_server spawns two detached sleepers
before READY) passed while both orphans survived.
Survivors are now every live process created since the spawn whose
HERMES_HOME, cwd or argv points into the test's scratch profile
(owned_processes), with a positive control that the scan sees the
backend itself before the kill. Same check for gateway stop. Cleanup
kills everything the profile owns.
Also: state-db guard matches the holder PID as a whole number, and the
e2e-windows job ends with a scan that fails on any python.exe /
hermes.exe left running.
Adds tests/e2e/core/windows (windows_only + integration; 18 tests, 5 strict-xfail KNOWN
entries for #120504#121150#121015#121114#120205) and an e2e-windows job in
tests-os.yml running it on windows-latest, one pytest process per file, no retries.
The catalog plugin declares hindsight-client in its own plugin.yaml and the
plugin installer resolves it (into HERMES_LAZY_INSTALL_TARGET on the sealed
Docker image), so the pyproject extra, its exclude-newer entry and every
consumer that pre-installed it — the Dockerfile, the nix full package /
module examples / check override, and the CI `uv sync` matrices — stop naming
it. uv.lock regenerated with `uv lock` (only the hindsight-client package and
the extra leave the lock; the exclude-newer block is re-sorted by uv).
Provision ripgrep through setup-pm for the OS-specific test matrix. Make the
dashboard stopper fixture publish its completion atomically and wait on that
completion instead of racing backend death. Allow the real process-registry
signal test to finish killing a snapshotted child after macOS reparents it.
Native Windows collection called POSIX-only APIs, and the lane omitted its
Telegram dependency. Several live tests also encoded obsolete runtime identity,
release-channel, and process-launch contracts.
Refresh the tests against the current production seams. Give hosted PowerShell
probes enough time under contention and install the Telegram extra in the OS
lane.
Validated on native Windows 3.14.7:
- 12 passed and 3 skipped across the six files that failed in run 35672259145
Validated on Linux:
- 116 passed and 12 skipped across the same six files
- Ruff, actionlint, YAML parsing, and Windows marker discovery passed
The 6-target cold->warm smoke existed to prove setup-pm still cold-boots
when its code changes without a lock bump, and to reach linux-arm64,
darwin-x64 and win32-arm64. Both are now covered without a dedicated
workflow:
* The tools cache key hashes pm/**, the action itself and
scripts/ci/setup_toolchain.py, not only pm/lock.json. A provisioning
change misses the cache on every lane that uses the action, so the
cold path runs where the tests already are.
* tests-os gains a windows-11-arm leg running the same windows-marked
files (tests/pm carries ten of them). It needs the ARM64 build deps
because several extras build from sdist, and fewer workers on the
4-core runner.
The Windows SDK adapter test is the one thing left that no other lane
ran natively; it keeps its two Windows runners under
windows-bundle-sdk.yml, path-triggered on the signing scripts. The
run-scoped cache cleanup workflow and its script only served the smoke
and go with it.
`scripts/run_tests.sh tests/<dir>/` is how a change gets its regression
coverage run, so a test filed under the wrong directory is a test nobody
runs when that code changes. Two kinds of drift had accumulated.
Parallel directories for one source package, folded into the mirror:
tests/acp -> tests/acp_adapter (its __init__/conftest move with it)
tests/cli -> tests/hermes_cli (prompt_toolkit fixture merged into
hermes_cli/conftest.py)
tests/run_agent -> tests/agent (backoff fixture becomes
agent/conftest.py)
tests/relay -> tests/gateway/relay
tests/state -> tests/hermes_state
246 loose files at tests/ root, routed by the package they import/patch:
hermes_cli, hermes_state, agent, gateway, tools, plugins, tui_gateway, cron.
Installer and desktop-update script tests go to tests/scripts/{install,
desktop_update}/. 43 tests of root-level modules (batch_runner, utils,
hermes_constants, packaging) stay at the root.
Filenames drop their issue numbers (95 files: test_89315_x.py -> test_x.py);
the number stays in the module docstring where it has context.
Collisions: test_cli_skin_integration.py existed in both tests/ and tests/cli
with different subsets — merged into one (10 tests, all kept);
run_agent/test_pre_compress_memory_context.py -> agent/..._handoff.py;
tests/test_account_usage.py -> agent/test_account_usage_fetch.py;
tests/test_web_server.py -> hermes_cli/test_web_server_ws_ping.py.
Deleted: test_minisweagent_path.py (empty since PR #2804),
test_model_picker_scroll.py (tested a private copy of the logic, imported
nothing), test_process_loop_event_loop_warning.py (asserted asyncio behaviour,
imported nothing from Hermes).
Repo-root path arithmetic (Path(__file__).parents[N], dirname chains) is
bumped for the 202 files that changed depth and verified by evaluating every
such expression against the new location. classify_changes' desktop-updater
lane prefix, tests-os.yml's ignore glob and every in-tree path comment follow
the moves. tests/test_tests_tree_layout.py keeps the tree from drifting back.
Bring in the Python 3.14 runtime pins and wake-engine changes while
preserving the staged stable-release gate and review fixes.
The merge has no conflicts. Targeted tests on the existing Python 3.11
dev environment passed: 130 passed, 8 skipped. The lock check passed
with Python 3.14.7. Workflow lint and shell syntax checks also passed.
Full Python 3.14 runtime and native release acceptance remain for CI.
Run the entire CI workflow before Docker build and tests. Require Nix,
native payload smoke tests, install/update E2E and signed-package upgrade
acceptance before publishing. Keep Desktop Playwright E2E deferred.
Archive tested Docker images and signed bundle candidates with provenance
and hashes. Publishers consume those exact artifacts without rebuilding.
Advance stable channels only after all required publications succeed.
Keep canaries on their separate path and reject direct stable-builder
publication that bypasses the gate.
Move shared release transport, manifests and gates to Python. Keep native
Electron adapters in JS and share feed/MIME facts as JSON. Replace the
R2/feed JS implementation and move its protocol tests to Python.
Verified targeted Python and JS tests, real loopback transport and CLI
execution, temporary Git admission, workflow graph lint, and typechecks.
No live stable release was run. Native signing, package upgrades and real
registry/Store promotion still need their release-run receipts. Separate
services cannot promote atomically. A promotion failure keeps the run red.
pm python node: 3.14.7+20260901 (freshest python-build-standalone 3.14
build) for the 6 desktop targets; the bionic row moves from the third-party
TUR python3.11 deb to the official termux-main python_3.14.6-1 deb (which
lags PBS by one patch — pinned manually, documented). All 7 digests fetched
from the live sources (PBS release API + termux-main Packages index).
pm/packages.py: main_bin_rel python3.14, deb_package python, bionic fetch
constant, latest_versions guards bionic (no PBS build exists).
termux lane: PYTHON_ABI cp311->cp314, python3.11->python3.14 paths,
libpython3.11.so->3.14, TARGET_ENV 3.11.15->3.14.6 AND sys_platform
linux->android (CPython 3.13+ reports 'android', docs-verified) — linux-
gated markers no longer admit the termux target. runtime_libs.json needs no
change: every python 3.14.6-1 Depends is already staged.
CI: python-version/--python 3.11->3.14 across all 11 workflows incl. the
uv lockfile-check lane. Installers derive the minor from the lock already;
fallbacks bumped. Sandbox images nikolaik/python-nodejs:python3.11-nodejs20
-> python3.14-nodejs22 (tag exists). runtime_repair fall-forward cap now
tracks the <3.15 requires-python window. Docs/README python version claims
updated.
Real Windows child tests sent an interrupt but the text reader never delivered it. Keep caller-supplied streams intact and use the stdin buffer for JSONL. Exercise the real child on all native OS lanes and limit Windows nested-process oversubscription.
Use the native path separator for explicit test files. Copy the stdlib runtime owners before Docker invokes PM. Preserve the isolated home in config tests and patch current receipt/backup owners. Remove stale Arabic navigation keys introduced by the merge.
pytest-xdist is gone: run_tests.sh no longer dispatches on host, the
Windows xdist (--dist loadfile) arm is deleted, and pytest-xdist is
dropped from the dev extra (uv lock removes it + execnet). The
per-file subprocess runner (run_tests_parallel.py) is THE runner on
every host — the shape the CI linux lane already used.
- tests-os.yml: the macOS/Windows lanes now run scripts/run_tests.sh
like every other lane, passing the marker-narrowed file list via
--files and -m after --. The runner natively tolerates per-file
empty collections (exit 5, platform-gated file) and fails the run
when NOTHING collected — replacing the hand-rolled exit-5 branch.
- tests/gateway/conftest.py: the hasattr(config, 'workerinput')
controller-guard was xdist-only dead code; the file-locked cache
already handled the per-file model. Removed.
- tests/tools/test_browser_supervisor.py: the port came from xdist's
worker_id (fixed 9225 under per-file isolation — concurrent files
collide). Binds an ephemeral port instead (bind 0, read back).
- ~35 comment sites named xdist as the isolation mechanism; they now
describe the shared-process hazard they actually guard against
(bare pytest runs, same-file ordering) without naming a runner that
no longer exists.
list_os_marked_tests.py's stdout, when redirected on Windows, carries
CRLF line endings (Python text-mode \n -> \r\n). The workflow's
'set -- ' word-split on the \r too, leaving it attached to
every path — pytest then failed 'file or directory not found:
tests/.../test_foo.py\r' on the first file. macOS passed because LF-only
there. Strip \r via 'tr -d' before word-splitting (no-op on POSIX).
The windows-tests merge made tests.yml a 3-OS matrix with the FULL
suite on Windows (pytest-xdist loadfile) — whose worker-crash flake
(INTERNALERROR on a random file every run) has been red-flagging the
branch. Per ethie: don't rip the windows-tests merge's test fixes, just
match upstream/main's CI shape. tests.yml is back to the Linux-only
96-core full-suite lane (per-file subprocess isolation); tests-os.yml
restored for the macOS/Windows marked-test lanes (plain pytest, small
set, no xdist — adapted 2 our platforms() marker convention with
'-m platforms and not integration'); ci.yaml wires tests-os back with
its desktop_updater input gate + all-checks-pass dependency.
`Start-UiServer` printed the -SelfTestUi URL (and opened the browser window)
as soon as the TcpListener was bound, but the runspace that answers /progress
starts asynchronously — BeginInvoke returns before the pipeline is open and
the script block is JIT'd, which is seconds on a loaded runner. The kernel
accepted connections into the backlog during that gap and nobody answered
them. The self-test hit it three times (#90371 and two follow-ups each
widened a timeout instead of removing the race) and it just failed an
unrelated hermes_state.py PR (run 33591547099, two 5s stale-backlog
timeouts = red).
- windows.ps1: readiness handshake after BeginInvoke — one /progress
round-trip must succeed (≤15s) before the server is returned; on failure
tear the listener down and continue without UI. The URL now means
"serving", not "bound". Also fixes the browser opening to a page that never
loads on a slow machine.
- test: 1s per-attempt probe timeout so a single dead backlog socket cannot
consume half the readiness budget.
- CI: new `desktop_updater` classifier lane. tests/test_desktop_update_windows_*.py
spawn the real PowerShell script; the Windows-only job now runs them only
when scripts/desktop-update/**, the Electron updater launcher, conftest,
pyproject, or those tests change (push/dispatch fail open). A PR that
never touched that surface cannot be failed by its process timing.
Fold the OS-specific lanes into tests.yml's test job as a three-OS matrix.
Linux and Windows run the full suite (conftest skips foreign-OS markers).
macOS keeps macos_only for now. Delete tests-os.yml.
Every Linux lane that does real work ran on a 4-core `ubuntu-latest`. The
Python suite and the JS checks were split into many small jobs to make that
size usable. Each split job repeated the full setup. In most of the JS jobs
the repeated setup cost more than the work.
The work lanes move to larger runners. Then the splits that existed only to
make small runners usable go away.
Python tests: 12 slices become 1 job on a 96-core runner. Slicing cost a
matrix job, a duration cache, a per-slice artifact and a merge job. 96 cores
clear the floor that the slowest single test file sets, which is about 82s. A
second slice divides work that is already at that floor, and adds a second
setup. Duration data from run 32522943054 gives the numbers behind this: 3178
files, 11645s in series.
The worker count is explicit, because `run_tests.sh` defaults to twice the
core count. A later commit sets it from a measurement on this hardware.
JS checks: 14 jobs become 1. The matrix paid about 371s of repeated setup to
spread about 612s of work. One larger runner installs one time. The three UI
shard scripts and `run-ui-shard.mjs` are therefore removed, because the
unsharded `test:ui` covers the same tests.
The unit of parallel work inside that job is a CHECK, and not a workspace.
apps/desktop is most of the payload, and its own `check` is a serial && chain.
A spread across workspaces alone therefore leaves that chain as the long pole.
A package that declares `check:*` sub-scripts gives one unit for each
sub-script. That is the same selection rule the matrix used.
The loop lives in `.github/scripts/run-workspace-checks.mjs`, so the same
sequence runs on a laptop. It runs 11 units together, buffers the output of
each one, and fails at the end with the full list. Children that share one
stdout interleave their lines and make a failure hard to read.
`npm run --ws check` stops at the first workspace that fails.
`check:test:plugins` joins the desktop `check` script. The matrix prefers
`check:*` sub-scripts over the plain `check` script, so `check:test:plugins`
ran only as its own leg. Without this change the merge drops that suite and
the job stays green.
node_modules is cached on the lockfile, and `npm ci` is skipped on an exact
hit. The `cache: npm` option of `setup-node` caches only the ~/.npm tarball
cache, which leaves the extract and the postinstalls to pay again.
The arm64 image build stays on a native arm64 runner. A build of linux/arm64
on an x64 host uses emulation.
The docker test lane caps its workers at the core count. Each of those tests
drives a container, so the docker daemon sets the limit and not the processor.
`.github/actionlint.yaml` declares the runner labels. actionlint knows the
GitHub-hosted labels only, and an undeclared label reads as an error that
hides the real findings.
The `detect` job checks out one file through a sparse checkout, and its
timeout drops to 1 minute. It reads
`scripts/ci/classify_changes.py` and nothing else.
Verification:
- actionlint reports 9 findings across all workflows. An unmodified HEAD with
the same config reports the same 9. This change adds none.
- A wrong label still fails. actionlint reports `ubuntu-latest-32-cor` and
`ubuntu-latest-32-arm-cores`.
- Every changed workflow parses, and `name` parses as a string.
- A replay of the `save-durations` merge step against a three-artifact layout
returns all 3178 entries.
- An expansion of the npm script graph gives the same leaf commands for the
parallel units and for a plain `npm run check`, in both directions. Against
the 13-leg matrix the count is 13 to 11, and the whole difference is the
three UI shards that collapse into one unsharded `check:test:ui`.
- `--list` reports the 11 units, and a full local run completes and reports
the time of each unit.
- The runner labels cannot be verified here. The first real run is the test.
`shell: bash` runs the step with -e injected, and `set -uo pipefail` does
not clear it. A non-zero pytest exit killed the script before `status=$?`,
so the -eq 5 branch and its ::error message never ran. The job still failed
red, but the diagnostic that names the cause never printed.
the markers from the previous commit skip off-host. without a host to
run them on, every marked test is a silent skip. this commit adds the
hosts.
- tests-os.yml runs -m macos_only on macos-latest and -m windows_only
on windows-latest. ci.yml requires both lanes in all-checks-pass.
- a lane fails on pytest exit code 5 (zero tests selected). a renamed
marker cannot produce a green job that ran nothing.
- each lane repeats 'not integration' because a command-line -m
replaces the addopts filter.
- scripts/ci/list_os_marked_tests.py selects which files each lane
imports. -m filters after collection, and collection imports every
module. without this helper, one unrelated ImportError on the
foreign host fails a job whose own tests passed. the helper exits
non-zero when a marker matches no file, and writes bytes with
explicit lf so windows crlf translation cannot corrupt the bash
file list. it has its own tests in tests/ci/.
- the local runner now reports the skipped count and prints a note:
macos_only/windows_only tests were skipped on this host, and this
ci lane runs them. a green local run on linux no longer reads as
coverage of the other hosts.
- the runner default job count is now #cpu, not #cpu*2.
