ef4a82eed0
scripts/run_tests.sh now runs pytest-xdist -n <N> --dist loadfile as the single canonical path on every OS (Linux and Windows CI lanes both use it). The per-file subprocess model (run_tests_parallel.py, and the interim run_xdist.sh experiment) is deleted along with its two self-tests: persistent xdist workers pay the interpreter+import wall once per worker instead of once per file (~0.5-1.5s x ~3400 files was a ~6-minute floor on Windows), and --dist loadfile pins a file's tests to ONE worker, bounding state pollution to co-scheduled files — which is exactly the class of flake we are now committing to fix properly. The serial process-killer quarantine phase is dropped too. It existed to keep process-tree-sweep tests from killing sibling xdist workers; the durable fix belongs in those tests (sweeps must target their own children, not enumerate every python process), and keeping a divergent two-phase path would hide that work. Kept from the old wrapper: hermetic env -i scrubbing, Windows location-var forwarding, venv probing, bytecode pre-compile, -m 'not integration', and the HERMES_TEST_IMAGE docker-knob allowlist. HERMES_TEST_FILE_TIMEOUT/FILE_RETRIES/SLICE go away with the runner they parameterized; -j/HERMES_TEST_WORKERS now map to xdist -n (Linux CI pins 96, Windows 32, default auto). Docs updated to match: AGENTS.md (runner contract, flake policy, isolation section), CONTRIBUTING.md, tests/conftest.py comments, classify_changes docstring + its lane expectation (a .sh runner no longer trips the supply-chain scan lane), comfyui README, hermes-agent contributor guide, debugpy skill and its website doc.
275 lines
12 KiB
YAML
275 lines
12 KiB
YAML
name: Tests
|
|
|
|
on:
|
|
workflow_call:
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
# Cancel in-progress runs for the same ref
|
|
concurrency:
|
|
group: tests-${{ github.ref }}
|
|
cancel-in-progress: true
|
|
|
|
jobs:
|
|
test:
|
|
# One matrix covers three host OSes. Linux and Windows run the full suite.
|
|
# macOS runs only its ``macos_only`` tests for now (the suite is not yet
|
|
# macOS-clean). tests/conftest.py's ``pytest_collection_modifyitems`` hook
|
|
# skips foreign-OS markers on each host, so an un-gated full run selects
|
|
# "generic + this host's own marker" — no ``-m`` filter is needed for the
|
|
# full-suite lanes. Nothing slices by platform now: every lane runs the
|
|
# whole discoverable suite and lets the markers do the gating.
|
|
name: Python tests (${{ matrix.os }})
|
|
runs-on: ${{ matrix.runner }}
|
|
# The windows lane needs a bigger budget than the others: the full suite
|
|
# there has a long tail of files that spawn servers/sockets/subprocesses
|
|
# and run minutes-per-file on the CI runner (they pass in seconds on an
|
|
# idle dev box) — measured 88.6% completion at 60 minutes on run
|
|
# 33338310764.
|
|
timeout-minutes: ${{ matrix.timeout_min }}
|
|
strategy:
|
|
fail-fast: false
|
|
matrix:
|
|
include:
|
|
- os: linux
|
|
runner: ubuntu-latest-96-core
|
|
marker: ""
|
|
workers: "96"
|
|
timeout_min: 30
|
|
- os: windows
|
|
runner: windows-latest-32-core
|
|
marker: ""
|
|
workers: "32"
|
|
timeout_min: 150
|
|
- os: macos
|
|
runner: macos-latest
|
|
marker: macos_only
|
|
workers: ""
|
|
timeout_min: 30
|
|
steps:
|
|
- name: Disable Windows Defender real-time scanning
|
|
# Process-spawn-heavy test files pay Defender's real-time scan on
|
|
# every python.exe spawn / temp write, which inflated the lane's tail
|
|
# to minutes-per-file (seconds on an idle dev box). The runner is
|
|
# ephemeral and single-purpose; scanning it protects nothing. Best
|
|
# effort — some runner images may refuse, and the lane still passes,
|
|
# just slower.
|
|
if: matrix.os == 'windows'
|
|
shell: pwsh
|
|
run: |
|
|
try {
|
|
Set-MpPreference -DisableRealtimeMonitoring $true
|
|
Write-Host "Defender real-time monitoring disabled."
|
|
} catch {
|
|
Write-Host "Could not disable windows defender"
|
|
}
|
|
|
|
- name: Put Git bash ahead of the WSL stub on PATH
|
|
# windows-2025 ships C:\Windows\System32\bash.exe — the WSL launcher
|
|
# stub with no distro installed. PATH puts System32 before Git, so
|
|
# every test that resolves "bash" spawns the stub and gets its UTF-16
|
|
# "Windows Subsystem for Linux" banner with exit 1. Prepend Git's bin
|
|
# so "bash" resolves to the real MSYS bash the harnesses expect.
|
|
if: matrix.os == 'windows'
|
|
shell: pwsh
|
|
run: |
|
|
$gitBin = "C:\Program Files\Git\bin"
|
|
if (Test-Path "$gitBin\bash.exe") {
|
|
Add-Content $env:GITHUB_PATH $gitBin
|
|
Write-Host "Prepended $gitBin to PATH"
|
|
} else {
|
|
Write-Host "::warning::Git bash not found at $gitBin"
|
|
}
|
|
|
|
- name: Checkout code
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Install ripgrep (prebuilt binary)
|
|
if: matrix.os == 'linux'
|
|
run: |
|
|
set -euo pipefail
|
|
RG_VERSION=15.1.0
|
|
RG_SHA256=1c9297be4a084eea7ecaedf93eb03d058d6faae29bbc57ecdaf5063921491599
|
|
RG_TARBALL=ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl.tar.gz
|
|
curl -sSfL --retry 3 --retry-delay 5 -o "$RG_TARBALL" \
|
|
"https://github.com/BurntSushi/ripgrep/releases/download/${RG_VERSION}/${RG_TARBALL}"
|
|
echo "${RG_SHA256} ${RG_TARBALL}" | sha256sum -c -
|
|
tar -xzf "$RG_TARBALL"
|
|
sudo mv "ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl/rg" /usr/local/bin/rg
|
|
rm -rf "$RG_TARBALL" "ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl"
|
|
rg --version
|
|
|
|
- name: Install uv
|
|
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # 8.2.0
|
|
with:
|
|
# Pin the uv version: unpinned, setup-uv resolves "latest" by
|
|
# fetching a manifest from raw.githubusercontent.com on EVERY job —
|
|
# a transient fetch failure fails the whole job (2026-07-28 slice-5
|
|
# incident). Pinned, the binary downloads directly; no manifest hop.
|
|
version: "0.9.28"
|
|
# Persist uv's download/wheel cache (~/.cache/uv) across runs.
|
|
# Keyed on the dependency manifests, so the cache is reused until
|
|
# pyproject.toml or uv.lock changes. `uv sync` still runs every
|
|
# time, but resolves from the warm cache instead of re-downloading
|
|
# and re-building wheels.
|
|
enable-cache: true
|
|
cache-dependency-glob: |
|
|
pyproject.toml
|
|
uv.lock
|
|
|
|
- name: Set up Python 3.11
|
|
uses: ./.github/actions/retry
|
|
with:
|
|
command: uv python install 3.11
|
|
|
|
- name: Install dependencies
|
|
# `uv sync --locked` installs the exact pinned set from uv.lock (and
|
|
# fails if the lock is out of sync with pyproject.toml), giving a
|
|
# reproducible env. It also creates .venv itself, so no separate
|
|
# `uv venv` step is needed.
|
|
#
|
|
# The trailing extras beyond all/dev are the lazy-install features
|
|
# (tools/lazy_deps.py) that tests exercise for real: provider.anthropic,
|
|
# stt/tts.mistral, image.fal, terminal.modal, terminal.daytona,
|
|
# memory.hindsight, search.parallel. The hermetic test env forbids
|
|
# mid-run pip installs (HERMES_DISABLE_LAZY_INSTALLS=1 in
|
|
# tests/conftest.py), so the SDKs those tests need must be in the
|
|
# venv up front — resolved from uv.lock like everything else, which
|
|
# also honors the exact supply-chain pins these extras carry.
|
|
uses: ./.github/actions/retry
|
|
with:
|
|
command: uv sync --locked --python 3.11 --extra all --extra dev --extra anthropic --extra mistral --extra fal --extra modal --extra daytona --extra hindsight --extra parallel-web
|
|
|
|
- name: Minimize uv cache
|
|
# Optimized for CI: prunes pre-built wheels that are cheap to
|
|
# re-download, keeping the persisted cache small and fast to restore.
|
|
run: uv cache prune --ci
|
|
|
|
- name: Run tests
|
|
# Two shapes:
|
|
#
|
|
# * Full-suite lanes (Linux + Windows): scripts/run_tests.sh —
|
|
# pytest-xdist with --dist loadfile behind a hermetic env.
|
|
# Persistent workers pay the interpreter+import wall once per
|
|
# worker instead of once per file (a ~6-minute floor on
|
|
# Windows under the old per-file subprocess model). loadfile
|
|
# keeps each file's tests on ONE worker, so the remaining
|
|
# hazard is state shared by files co-scheduled on a worker.
|
|
# Failures from that are stateful-test bugs to fix, not runner
|
|
# bugs.
|
|
# * macOS (marker set): plain pytest over the files that carry the
|
|
# marker. list_os_marked_tests.py narrows WHICH FILES are
|
|
# imported (collection otherwise drags ~900 unrelated modules
|
|
# through import on the macOS host), and `-m` stays the
|
|
# authoritative selector. Passing `-m` REPLACES pyproject's
|
|
# ``-m 'not integration'`` addopts, so ``not integration`` is
|
|
# repeated or the integration suite returns through the side
|
|
# door. The lane FAILS on pytest exit 5 (zero tests selected) so
|
|
# a renamed/broken marker can never report green while running
|
|
# nothing.
|
|
shell: bash
|
|
run: |
|
|
set -uo pipefail
|
|
|
|
|
|
if [ -n "${{ matrix.marker }}" ]; then
|
|
LIST="${RUNNER_TEMP:-.}/selected-tests.txt"
|
|
|
|
if ! uv run --no-sync python scripts/ci/list_os_marked_tests.py \
|
|
"${{ matrix.marker }}" > "$LIST"; then
|
|
echo "::error::could not enumerate ${{ matrix.marker }} test files"
|
|
exit 1
|
|
fi
|
|
if [ ! -s "$LIST" ]; then
|
|
echo "::error::empty ${{ matrix.marker }} file list"
|
|
exit 1
|
|
fi
|
|
|
|
# Deliberately NOT `mapfile`: that is a bash 4 builtin and the
|
|
# macOS runner's /bin/bash is 3.2. Word-splitting is safe here
|
|
# because the helper emits repo-relative paths with no spaces.
|
|
# shellcheck disable=SC2046
|
|
set -- $(cat "$LIST")
|
|
echo "selected $# file(s) for ${{ matrix.marker }}:"
|
|
cat "$LIST"
|
|
|
|
status=0
|
|
uv run --no-sync python -m pytest \
|
|
"$@" \
|
|
-m "${{ matrix.marker }} and not integration" \
|
|
-v --tb=short || status=$?
|
|
if [ "$status" -eq 5 ]; then
|
|
echo "::error::No tests matched -m ${{ matrix.marker }}. Either the" \
|
|
"marker was renamed/dropped or selection is broken — this job" \
|
|
"must never pass without running its OS's tests."
|
|
exit 1
|
|
fi
|
|
exit "$status"
|
|
fi
|
|
|
|
# Linux full suite. run_tests.sh locates the venv itself (both the
|
|
# POSIX bin/ and the Windows Scripts/ layout), so no per-OS
|
|
# activation.
|
|
scripts/run_tests.sh
|
|
env:
|
|
# xdist worker count (-n). Linux pins 96 (the measured whole-suite
|
|
# sweep — one worker per core is fastest on the 96-core runner,
|
|
# and the curve is shallow); Windows pins 32. macOS leaves this
|
|
# unset (it runs the marked-file lane, not the full suite).
|
|
HERMES_TEST_WORKERS: ${{ matrix.workers }}
|
|
# Ensure tests don't accidentally call real APIs
|
|
OPENROUTER_API_KEY: ""
|
|
OPENAI_API_KEY: ""
|
|
NOUS_API_KEY: ""
|
|
|
|
e2e:
|
|
runs-on: ubuntu-latest
|
|
timeout-minutes: 15
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
|
|
|
- name: Install ripgrep (prebuilt binary)
|
|
run: |
|
|
set -euo pipefail
|
|
RG_VERSION=15.1.0
|
|
RG_SHA256=1c9297be4a084eea7ecaedf93eb03d058d6faae29bbc57ecdaf5063921491599
|
|
RG_TARBALL=ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl.tar.gz
|
|
curl -sSfL --retry 3 --retry-delay 5 -o "$RG_TARBALL" \
|
|
"https://github.com/BurntSushi/ripgrep/releases/download/${RG_VERSION}/${RG_TARBALL}"
|
|
echo "${RG_SHA256} ${RG_TARBALL}" | sha256sum -c -
|
|
tar -xzf "$RG_TARBALL"
|
|
sudo mv "ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl/rg" /usr/local/bin/rg
|
|
rm -rf "$RG_TARBALL" "ripgrep-${RG_VERSION}-x86_64-unknown-linux-musl"
|
|
rg --version
|
|
|
|
- name: Install uv
|
|
uses: astral-sh/setup-uv@fac544c07dec837d0ccb6301d7b5580bf5edae39 # 8.2.0
|
|
with:
|
|
version: "0.9.28"
|
|
enable-cache: true
|
|
cache-dependency-glob: |
|
|
pyproject.toml
|
|
uv.lock
|
|
|
|
- name: Set up Python 3.11
|
|
run: uv python install 3.11
|
|
|
|
- name: Install dependencies
|
|
uses: ./.github/actions/retry
|
|
with:
|
|
command: uv sync --locked --python 3.11 --extra all --extra dev --extra anthropic --extra mistral --extra fal --extra modal --extra daytona --extra hindsight --extra parallel-web
|
|
|
|
- name: Minimize uv cache
|
|
run: uv cache prune --ci
|
|
|
|
- name: Run e2e tests
|
|
run: |
|
|
source .venv/bin/activate
|
|
python -m pytest tests/e2e/ -v --tb=short
|
|
env:
|
|
OPENROUTER_API_KEY: ""
|
|
OPENAI_API_KEY: ""
|
|
NOUS_API_KEY: ""
|