a10620a669
Review fixes on the lifecycle-verbs PR. Three of them were escape hatches that
looked implemented and were dead code, and one turned a boot race into a
permanently parked unit.
- ATTACH now requires a LIVE `identify` answer. The claim-time record is
published with NO served set (the runner settles multiplex a moment later),
and `host_gateway()` reports `served_known=False` when nothing answers. An
owner whose served set is unknown yields a TRANSIENT refusal, never an
attach: previously `default`'s claim published "default,other" before its
socket bound, `other`'s systemd unit read that as "I am served", exited 78,
and systemd parked it for good.
- `served_profiles()` honours the actual `gateway.multiplex_profiles` setting
instead of forcing `multiplex=True`, so a standalone gateway stops claiming
the whole roster.
- `--replace` is threaded through the CLI guard into `start_gateway`, and
`--force` into `_host_attach_or_none`. Both previously exited in the guard
before the code that implements them ever ran ("nothing to start", rc=0).
- A supervised attach exits 75 (EX_TEMPFAIL), not 78. 78 is the PERMANENT
config refusal every supervisor parks on; "someone else serves me right now"
is a runtime observation that ends when that process does. No unit files
change: systemd already has RestartForceExitStatus=75/RestartSec=5, the s6
finish script passes 75 through, launchd relaunches a non-78 failure. Exit 0
would not do — s6 parks a clean exit too.
- `restart --all` retracts the stopped owner's record (`discard_dead_record`)
and re-enters with `replace=True`, so it can no longer attach to the corpse
it just stopped and exit 0.
- Rendezvous hardening: the dir is created/repaired 0o700, a record whose
`st_uid` is not ours is ignored, liveness is proven BEFORE we dial the home
it names, and a live `identify` must agree about `hermes_home`.
- `-p X gateway restart --all` reaches the `--all`-aware branch instead of the
generic guard's `hermes -p default gateway restart` one-liner.
- `host_gateway()` is memoized (2s TTL, invalidated on every record write), so
`gateway status`/doctor across N profiles pays one probe, not N.
Tests: the two new files build the record as raw JSON, so they COLLECT and RUN
against a tree without the `home` field and fail on the outcome. A/B against
the PR head: 9 failed / 6 passed → 15 passed. conftest's per-test
HERMES_GATEWAY_LOCK_DIR now defers to a caller-supplied value (and
run_tests.sh forwards it through `env -i`), and the per-process dir is a
deterministic self-sweeping per-PID path instead of an atexit-only mkdtemp.
`test_runner_startup_failures.py` stubs the new attach gate and releases the
host role it claims.
185 lines
8.8 KiB
Bash
Executable File
185 lines
8.8 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Canonical test runner for hermes-agent. Run this instead of calling
|
|
# `pytest` directly to guarantee your local run matches CI behavior.
|
|
#
|
|
# What this script enforces:
|
|
# * Per-file isolation via scripts/run_tests_parallel.py — each test
|
|
# file runs in its own freshly-spawned `python -m pytest <file>`
|
|
# subprocess. No xdist, no shared workers, no module-level leakage
|
|
# between files.
|
|
# * TZ=UTC, LANG=C.UTF-8, PYTHONHASHSEED=0 (deterministic)
|
|
# * Env vars blanked (conftest.py also does this, but this
|
|
# is belt-and-suspenders for anyone running pytest outside our
|
|
# conftest path — e.g. on a single file)
|
|
# * Proper venv activation (probes .venv, venv, then ~/.hermes/...)
|
|
#
|
|
# Usage:
|
|
# scripts/run_tests.sh # full suite
|
|
# scripts/run_tests.sh -j 4 # cap parallelism
|
|
# scripts/run_tests.sh tests/agent/ # discover only here
|
|
# scripts/run_tests.sh tests/agent/ tests/acp_adapter/ # multiple roots
|
|
# scripts/run_tests.sh tests/foo.py # single file
|
|
# scripts/run_tests.sh tests/foo.py -q # path + bare pytest flag
|
|
# scripts/run_tests.sh tests/foo.py -v --tb=long # bare flags "just work"
|
|
# scripts/run_tests.sh -k 'pattern' # value flags pass through too
|
|
# scripts/run_tests.sh tests/foo.py -- --tb=long # explicit '--' still works
|
|
#
|
|
# Bare pytest flags (anything starting with '-' that isn't one of this
|
|
# runner's own options: -j/--jobs, --paths, --slice, --file-timeout, etc.)
|
|
# are forwarded to each per-file pytest invocation automatically — no '--'
|
|
# separator required. The explicit '--' form still works and stacks with
|
|
# bare flags. Positional path arguments override the default discovery
|
|
# root (tests/).
|
|
|
|
set -euo pipefail
|
|
|
|
# ── Locate repo root ────────────────────────────────────────────────────────
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
# ── Locate python ───────────────────────────────────────────────────────────
|
|
# Probe local venvs first; fall back to the Nix devShell's editable venv
|
|
# (HERMES_PYTHON is exported by the devShell hook and ships [dev] extras:
|
|
# pytest, pytest-asyncio, pytest-timeout, ruff, ty).
|
|
#
|
|
# A candidate must have pytest INSTALLED, not merely exist. The release venv
|
|
# at ~/.hermes/hermes-agent/venv has bin/activate but no pytest, so an
|
|
# existence-only probe selected it in checkouts/worktrees without a local
|
|
# .venv — every file then died with "No module named pytest" and the run
|
|
# reported "0 tests passed" (which reads green at a glance even though the
|
|
# exit code is 1). Skip such a venv and keep probing instead.
|
|
VENV=""
|
|
VENV_PYTHON=""
|
|
SKIPPED_VENVS=""
|
|
for candidate in "$REPO_ROOT/.venv" "$REPO_ROOT/venv" "$HOME/.hermes/hermes-agent/venv"; do
|
|
if [ -f "$candidate/bin/activate" ]; then
|
|
if "$candidate/bin/python" -c 'import pytest' 2>/dev/null; then
|
|
VENV="$candidate"
|
|
VENV_PYTHON="$candidate/bin/python"
|
|
break
|
|
fi
|
|
SKIPPED_VENVS="$SKIPPED_VENVS $candidate"
|
|
fi
|
|
# Native Windows venv layout: python.exe and activate live under
|
|
# Scripts/, and there is no bin/. Anyone running this script from
|
|
# Git Bash / MSYS with a `python -m venv`- or uv-created venv hits
|
|
# this branch — without it the canonical runner refuses to start.
|
|
if [ -f "$candidate/Scripts/activate" ]; then
|
|
if "$candidate/Scripts/python.exe" -c 'import pytest' 2>/dev/null; then
|
|
VENV="$candidate"
|
|
VENV_PYTHON="$candidate/Scripts/python.exe"
|
|
break
|
|
fi
|
|
SKIPPED_VENVS="$SKIPPED_VENVS $candidate"
|
|
fi
|
|
done
|
|
|
|
if [ -n "$SKIPPED_VENVS" ]; then
|
|
for skipped in $SKIPPED_VENVS; do
|
|
echo "▶ skipping venv without pytest: $skipped" >&2
|
|
done
|
|
fi
|
|
|
|
if [ -n "$VENV" ]; then
|
|
PYTHON="$VENV_PYTHON"
|
|
elif [ -n "${HERMES_PYTHON:-}" ] && [ -x "$HERMES_PYTHON" ] \
|
|
&& "$HERMES_PYTHON" -c 'import pytest' 2>/dev/null; then
|
|
# Guard with an import check: HERMES_PYTHON may point at the RELEASE
|
|
# venv (no pytest) when inherited from a wrapped `hermes` binary rather
|
|
# than the devShell hook.
|
|
PYTHON="$HERMES_PYTHON"
|
|
echo "▶ no local venv — using Nix dev venv via HERMES_PYTHON: $PYTHON"
|
|
else
|
|
echo "error: no virtualenv with pytest found in $REPO_ROOT/.venv or $REPO_ROOT/venv," >&2
|
|
echo " and HERMES_PYTHON is not a python with pytest (enter the Nix devShell or create a venv)" >&2
|
|
if [ -n "$SKIPPED_VENVS" ]; then
|
|
echo " (skipped for missing pytest:$SKIPPED_VENVS — install dev extras there, or create $REPO_ROOT/.venv)" >&2
|
|
fi
|
|
exit 1
|
|
fi
|
|
|
|
|
|
# ── Live-gateway plugin (computed before we drop env) ───────────────────────
|
|
EXTRA_PYTHONPATH=""
|
|
EXTRA_PYTEST_PLUGINS=""
|
|
if [ -f "$HOME/.hermes/pytest_live_guard.py" ]; then
|
|
EXTRA_PYTHONPATH="$HOME/.hermes"
|
|
EXTRA_PYTEST_PLUGINS="pytest_live_guard"
|
|
fi
|
|
|
|
|
|
# ── Windows location variables (computed before we drop env) ───────────────
|
|
# `env -i` forwards HOME, which is enough on POSIX. Native Windows CPython
|
|
# resolves Path.home() from USERPROFILE (or HOMEDRIVE+HOMEPATH), stdlib
|
|
# platform paths come from LOCALAPPDATA/APPDATA, ssl/sockets need SYSTEMROOT,
|
|
# and tempfile needs TEMP/TMP. Dropping them breaks collection on native
|
|
# Windows (issues #67385, #70813). These are location variables, not
|
|
# credentials, so forwarding them keeps the isolation intent intact. Each is
|
|
# only forwarded when actually set, so POSIX runs are byte-for-byte unchanged.
|
|
WIN_ENV=()
|
|
for _win_var in USERPROFILE HOMEDRIVE HOMEPATH LOCALAPPDATA APPDATA SYSTEMROOT TEMP TMP; do
|
|
if [ -n "${!_win_var:-}" ]; then
|
|
WIN_ENV+=("$_win_var=${!_win_var}")
|
|
fi
|
|
done
|
|
|
|
# ── Test-runner knobs (computed before we drop env) ────────────────────────
|
|
# The runner's own documented environment knobs must survive the hermetic
|
|
# `env -i` below, or they are silent no-ops for anyone invoking this script:
|
|
#
|
|
# * HERMES_TEST_WORKERS / PATHS / FILE_TIMEOUT / FILE_RETRIES / SLICE are
|
|
# read by run_tests_parallel.py at argparse-default time — inside the
|
|
# stripped environment.
|
|
# * HERMES_TEST_IMAGE is read by tests/docker/conftest.py to skip its
|
|
# session-scoped `docker build`. CI's docker.yml sets it to the image
|
|
# the build step just loaded; stripping it made every per-file pytest
|
|
# subprocess rebuild the 5GB image from a cold builder cache instead
|
|
# (~4 min per worker per run, and the rebuilt image lacked the
|
|
# HERMES_GIT_SHA build-arg the workflow bakes in).
|
|
#
|
|
# These are test-infrastructure knobs, not credentials — same class as the
|
|
# HERMES_RUN_SLOW_PET_TESTS / HERMES_E2E_BROWSER opt-ins already forwarded.
|
|
# Keep this an explicit allowlist (no HERMES_TEST_* glob) so the "no
|
|
# credential can leak" property stays auditable at a glance.
|
|
TEST_ENV=()
|
|
for _test_var in HERMES_TEST_IMAGE HERMES_TEST_WORKERS HERMES_TEST_PATHS \
|
|
HERMES_TEST_FILE_TIMEOUT HERMES_TEST_FILE_RETRIES HERMES_TEST_SLICE \
|
|
HERMES_GATEWAY_LOCK_DIR; do
|
|
if [ -n "${!_test_var:-}" ]; then
|
|
TEST_ENV+=("$_test_var=${!_test_var}")
|
|
fi
|
|
done
|
|
|
|
# ── Run in hermetic env ──────────────────────────────────────────────────────
|
|
# env -i: start with empty environment, opt-in only what we need.
|
|
# No credential var can leak — you'd have to explicitly add it here.
|
|
echo "▶ running per-file parallel test suite via run_tests_parallel.py"
|
|
echo " (TZ=UTC LANG=C.UTF-8 PYTHONHASHSEED=0; clean env)"
|
|
|
|
cd "$REPO_ROOT"
|
|
|
|
# ── Pre-compile .pyc bytecode cache ─────────────────────────────────────────
|
|
# Each test file runs in its own subprocess via run_tests_parallel.py.
|
|
# Pre-building the bytecode cache once here (instead of each subprocess
|
|
# compiling on first import) avoids redundant work across ~2000 processes.
|
|
# Uses git to list tracked .py files (skips venv, node_modules, etc).
|
|
echo "▶ pre-compiling bytecode cache"
|
|
"$PYTHON" -m compileall -q -j 0 -- $(git ls-files '*.py') >/dev/null 2>&1 || true
|
|
|
|
echo "▶ launching test runner"
|
|
exec env -i \
|
|
PATH="$PATH" \
|
|
HOME="$HOME" \
|
|
${WIN_ENV[@]+"${WIN_ENV[@]}"} \
|
|
${TEST_ENV[@]+"${TEST_ENV[@]}"} \
|
|
TZ=UTC \
|
|
LANG=C.UTF-8 \
|
|
LC_ALL=C.UTF-8 \
|
|
PYTHONHASHSEED=0 \
|
|
PYTHONUTF8=1 \
|
|
${HERMES_RUN_SLOW_PET_TESTS:+HERMES_RUN_SLOW_PET_TESTS="$HERMES_RUN_SLOW_PET_TESTS"} \
|
|
${HERMES_E2E_BROWSER:+HERMES_E2E_BROWSER="$HERMES_E2E_BROWSER"} \
|
|
${EXTRA_PYTHONPATH:+PYTHONPATH="$EXTRA_PYTHONPATH"} \
|
|
${EXTRA_PYTEST_PLUGINS:+PYTEST_PLUGINS="$EXTRA_PYTEST_PLUGINS"} \
|
|
"$PYTHON" "$SCRIPT_DIR/run_tests_parallel.py" "$@"
|