438d10200c
The SSH backend's real-host tests skipped on every CI run (nothing set TERMINAL_SSH_HOST), and every Docker backend test mocked subprocess or the docker CLI, so no CI run had executed either backend end to end. - remote-backends.yml: starts a throwaway sshd container on 127.0.0.1 with a per-run keypair (scripts/ci/start_test_sshd.sh) and runs test_ssh_environment.py plus the new test_docker_backend_real.py against it and the runner's Docker daemon with a digest-pinned debian:12-slim. A file that skips fails the job, so the lane cannot go green over zero coverage. - New classifier lane `remote_backends` (tools/environments/, tools/terminal_tool*, the real tests, the sshd harness, run_tests.sh). ci.yaml calls the workflow only when it fires, plus every push to main; required_results makes a selected lane mandatory, like the update consumers. - The real tests read HERMES_TEST_SSH_* / HERMES_TEST_DOCKER_IMAGE (forwarded by run_tests.sh), because the hermetic conftest blanks TERMINAL_SSH_KEY as a credential before any test can read it. - New real SSH cases: exit code/stderr, cwd persistence with spaces vs per-call workdir, quoting across the ssh hop, SendEnv passthrough through a real AcceptEnv, file tools write/read on the remote. - New real Docker cases: exec + exit codes, cwd vs workdir, exported env persistence, docker_env/docker_forward_env reach the container and nothing else does, quoting, file tools write/read, /root surviving a container replacement. - The existing one-shot test is xfail(strict): TERMINAL_SSH_PERSISTENT=false is read into ssh_config["persistent"] but SSHEnvironment never consumes it, so every SSH session persists exported env. Its _cleanup also cleaned the raw task id, not the resolved env key, so one test's env leaked into the next.
206 lines
11 KiB
Bash
Executable File
206 lines
11 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Canonical test runner for hermes-agent. Run this instead of calling
|
|
# `pytest` directly to guarantee your local run matches CI behavior.
|
|
#
|
|
# What this script enforces:
|
|
# * Per-file isolation via scripts/run_tests_parallel.py — each test
|
|
# file runs in its own freshly-spawned `python -m pytest <file>`
|
|
# subprocess. No xdist, no shared workers, no module-level leakage
|
|
# between files.
|
|
# * TZ=UTC, LANG=C.UTF-8, PYTHONHASHSEED=0 (deterministic)
|
|
# * Env vars blanked (conftest.py also does this, but this
|
|
# is belt-and-suspenders for anyone running pytest outside our
|
|
# conftest path — e.g. on a single file)
|
|
# * The checkout's test environment (via scripts/run-in-hermes-env when needed)
|
|
#
|
|
# Usage:
|
|
# scripts/run_tests.sh # full suite
|
|
# scripts/run_tests.sh -j 4 # cap parallelism
|
|
# scripts/run_tests.sh tests/agent/ # discover only here
|
|
# scripts/run_tests.sh tests/agent/ tests/acp_adapter/ # multiple roots
|
|
# scripts/run_tests.sh tests/foo.py # single file
|
|
# scripts/run_tests.sh tests/foo.py -q # path + bare pytest flag
|
|
# scripts/run_tests.sh tests/foo.py -v --tb=long # bare flags "just work"
|
|
# scripts/run_tests.sh -k 'pattern' # value flags pass through too
|
|
# scripts/run_tests.sh tests/foo.py -- --tb=long # explicit '--' still works
|
|
#
|
|
# Bare pytest flags (anything starting with '-' that isn't one of this
|
|
# runner's own options: -j/--jobs, --paths, --slice, --file-timeout, etc.)
|
|
# are forwarded to each per-file pytest invocation automatically — no '--'
|
|
# separator required. The explicit '--' form still works and stacks with
|
|
# bare flags. Positional path arguments override the default discovery
|
|
# root (tests/).
|
|
|
|
set -euo pipefail
|
|
|
|
# ── Locate repo root ────────────────────────────────────────────────────────
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
REPO_ROOT="$(cd "$SCRIPT_DIR/.." && pwd)"
|
|
|
|
# ── Locate python ───────────────────────────────────────────────────────────
|
|
# The suite runs under the checkout's isolated test environment (pm.testenv),
|
|
# and only there: unless the inherited environment is current for this
|
|
# checkout, this script re-executes itself under scripts/run-in-hermes-env,
|
|
# which syncs and applies it. That holds however the script is launched
|
|
# (`scripts/run_tests.sh`, `bash scripts/run_tests.sh`, from CI or a shell), so
|
|
# a branch switch or lock edit never runs the suite against the previous
|
|
# dependency set.
|
|
#
|
|
# Without an activation, an explicit HERMES_PYTHON that has pytest is honored:
|
|
# the Nix devShell's editable venv and CI's minimal installer lanes provide
|
|
# one on purpose. The import check matters: a wrapped `hermes` binary exports
|
|
# HERMES_PYTHON pointing at a release venv without pytest.
|
|
_has_pytest() { [ -n "$1" ] && [ -x "$1" ] && "$1" -c 'import pytest' 2>/dev/null; }
|
|
# shellcheck source=scripts/_activation.sh
|
|
. "$SCRIPT_DIR/_activation.sh"
|
|
if [ -z "${__HERMES_ACTIVATED:-}" ] && _has_pytest "${HERMES_PYTHON:-}"; then
|
|
PYTHON="$HERMES_PYTHON"
|
|
echo "▶ not activated — using HERMES_PYTHON: $PYTHON"
|
|
else
|
|
if ! hermes_activation_current "$REPO_ROOT"; then
|
|
if [ -n "${__HERMES_TESTS_REEXEC:-}" ]; then
|
|
echo "error: run-in-hermes-env did not produce a current environment for $REPO_ROOT" >&2
|
|
exit 1
|
|
fi
|
|
echo "▶ environment missing or stale for $REPO_ROOT — re-running under run-in-hermes-env" >&2
|
|
export __HERMES_TESTS_REEXEC=1
|
|
exec "$SCRIPT_DIR/run-in-hermes-env" "$BASH" "${BASH_SOURCE[0]}" "$@"
|
|
fi
|
|
PYTHON="${__HERMES_TEST_PYTHON:-}"
|
|
if ! _has_pytest "$PYTHON"; then
|
|
echo "error: activation provided no test interpreter with pytest (__HERMES_TEST_PYTHON=${PYTHON:-unset})" >&2
|
|
exit 1
|
|
fi
|
|
fi
|
|
|
|
|
|
# ── Live-gateway plugin (computed before we drop env) ───────────────────────
|
|
EXTRA_PYTHONPATH=""
|
|
EXTRA_PYTEST_PLUGINS=""
|
|
if [ -f "$HOME/.hermes/pytest_live_guard.py" ]; then
|
|
EXTRA_PYTHONPATH="$HOME/.hermes"
|
|
EXTRA_PYTEST_PLUGINS="pytest_live_guard"
|
|
fi
|
|
|
|
|
|
# ── Windows location variables (computed before we drop env) ───────────────
|
|
# `env -i` forwards HOME, which is enough on POSIX. Native Windows CPython
|
|
# resolves Path.home() from USERPROFILE (or HOMEDRIVE+HOMEPATH), stdlib
|
|
# platform paths come from LOCALAPPDATA/APPDATA, ssl/sockets need SYSTEMROOT,
|
|
# and tempfile needs TEMP/TMP. Dropping them breaks collection on native
|
|
# Windows (issues #67385, #70813). PATHEXT is also required: without .EXE,
|
|
# PowerShell opens a native child as a document without waiting for its exit.
|
|
# These are location variables, not
|
|
# credentials, so forwarding them keeps the isolation intent intact. Each is
|
|
# only forwarded when actually set, so POSIX runs are byte-for-byte unchanged.
|
|
WIN_ENV=()
|
|
for _win_var in USERPROFILE HOMEDRIVE HOMEPATH LOCALAPPDATA APPDATA SYSTEMROOT TEMP TMP \
|
|
ComSpec PATHEXT PROGRAMFILES ProgramFiles PROGRAMDATA ProgramData; do
|
|
if [ -n "${!_win_var:-}" ]; then
|
|
WIN_ENV+=("$_win_var=${!_win_var}")
|
|
fi
|
|
done
|
|
# Native build toolchain (Windows arm64 has no wheels for every pinned C extension, so
|
|
# `uv sync` inside a PM test compiles ruamel-yaml-clib and friends). The MSVC developer
|
|
# environment is exported by scripts/build/windows-deps.ps1 into the job env; without
|
|
# INCLUDE/LIB/VSINSTALLDIR the build backend reports "Visual C++ 14.0 or greater is
|
|
# required". These describe compiler locations, not credentials.
|
|
for _tool_var in INCLUDE LIB LIBPATH VSINSTALLDIR VCINSTALLDIR VCToolsInstallDir VCToolsVersion \
|
|
VCToolsRedistDir WindowsSdkDir WindowsSDKVersion WindowsSdkBinPath WindowsSdkVerBinPath \
|
|
WindowsLibPath UCRTVersion UniversalCRTSdkDir VSCMD_ARG_HOST_ARCH VSCMD_ARG_TGT_ARCH VSCMD_VER \
|
|
DevEnvDir ExtensionSdkDir Platform CARGO_HOME RUSTUP_HOME RUSTUP_TOOLCHAIN \
|
|
CARGO_TARGET_AARCH64_PC_WINDOWS_MSVC_LINKER CC_aarch64_pc_windows_msvc CC CXX AR \
|
|
VCPKG_ROOT OPENSSL_DIR OPENSSL_STATIC OPENSSL_LIB_DIR OPENSSL_INCLUDE_DIR; do
|
|
if [ -n "${!_tool_var:-}" ]; then
|
|
WIN_ENV+=("$_tool_var=${!_tool_var}")
|
|
fi
|
|
done
|
|
# setuptools locates the compiler through vswhere under "%ProgramFiles(x86)%\Microsoft Visual
|
|
# Studio\Installer"; without that variable a primed INCLUDE/LIB still reads as "Visual C++ 14.0
|
|
# or greater is required". The parenthesised name cannot be read with ${!var}.
|
|
_pf86="$(env | sed -n 's/^ProgramFiles(x86)=//p' | head -n1)"
|
|
[ -z "$_pf86" ] || WIN_ENV+=("ProgramFiles(x86)=$_pf86")
|
|
|
|
# ── Test-runner knobs (computed before we drop env) ────────────────────────
|
|
# The runner's own documented environment knobs must survive the hermetic
|
|
# `env -i` below, or they are silent no-ops for anyone invoking this script:
|
|
#
|
|
# * HERMES_TEST_WORKERS / PATHS / FILE_TIMEOUT / FILE_RETRIES / SLICE are
|
|
# read by run_tests_parallel.py at argparse-default time — inside the
|
|
# stripped environment.
|
|
# * HERMES_TEST_IMAGE is read by tests/docker/conftest.py to skip its
|
|
# session-scoped `docker build`. CI's docker.yml sets it to the image
|
|
# the build step just loaded; stripping it made every per-file pytest
|
|
# subprocess rebuild the 5GB image from a cold builder cache instead
|
|
# (~4 min per worker per run, and the rebuilt image lacked the
|
|
# HERMES_GIT_SHA build-arg the workflow bakes in).
|
|
# * HERMES_E2E_REQUIRE_TUI turns a missing Ink TUI build into a failure in
|
|
# tests/e2e/core/terminal instead of a skip (set by the e2e CI job).
|
|
# * CI / GITHUB_ACTIONS tell suites they run on a disposable runner (e.g.
|
|
# tests/e2e/core/upgrade runs the real updater unsandboxed only there).
|
|
# * HERMES_E2E_WINDOWS_INSTALL opts tests/e2e/core/windows_update into running
|
|
# the real install.ps1 (it writes HKCU PATH); HERMES_E2E_MACHINE_ROOT,
|
|
# HERMES_E2E_PROFILES_ROOT and HERMES_E2E_ARTIFACTS place its fake machines,
|
|
# their user profiles and the transcripts CI uploads.
|
|
# * HERMES_E2E_STRICT_ACCEPTANCE makes tests/e2e/core/_pending_fixes.py's
|
|
# known gaps hard failures in an acceptance run (workflow dispatch input).
|
|
# * HERMES_TEST_SSH_HOST / _USER / _PORT / _KEY and HERMES_TEST_DOCKER_IMAGE
|
|
# arm the real-backend tests (tests/tools/test_ssh_environment.py,
|
|
# tests/tools/test_docker_backend_real.py); remote-backends.yml points them at
|
|
# a throwaway sshd container and a small public image. The key is a
|
|
# per-run throwaway, never a credential.
|
|
#
|
|
# These are test-infrastructure knobs, not credentials — same class as the
|
|
# HERMES_RUN_SLOW_PET_TESTS / HERMES_E2E_BROWSER / HERMES_RUN_E2E opt-ins
|
|
# forwarded below.
|
|
# SSL_CERT_FILE/DIR are trust-store locations: the pinned interpreter's
|
|
# OpenSSL has no compiled-in bundle path on NixOS, so network tests (PM
|
|
# downloads, channel reads) need the host's pointer to verify TLS.
|
|
# Keep this an explicit allowlist (no HERMES_TEST_* glob) so the "no
|
|
# credential can leak" property stays auditable at a glance.
|
|
TEST_ENV=()
|
|
for _test_var in HERMES_TEST_IMAGE HERMES_TEST_WORKERS HERMES_TEST_PATHS \
|
|
HERMES_TEST_FILE_TIMEOUT HERMES_TEST_FILE_RETRIES HERMES_TEST_SLICE \
|
|
SSL_CERT_FILE SSL_CERT_DIR HERMES_GATEWAY_LOCK_DIR HERMES_E2E_REQUIRE_TUI CI GITHUB_ACTIONS \
|
|
HERMES_E2E_WINDOWS_INSTALL HERMES_E2E_MACHINE_ROOT HERMES_E2E_PROFILES_ROOT HERMES_E2E_ARTIFACTS \
|
|
HERMES_E2E_STRICT_ACCEPTANCE HERMES_TEST_SSH_HOST HERMES_TEST_SSH_USER HERMES_TEST_SSH_PORT \
|
|
HERMES_TEST_SSH_KEY HERMES_TEST_DOCKER_IMAGE; do
|
|
if [ -n "${!_test_var:-}" ]; then
|
|
TEST_ENV+=("$_test_var=${!_test_var}")
|
|
fi
|
|
done
|
|
|
|
# ── Run in hermetic env ──────────────────────────────────────────────────────
|
|
# env -i: start with empty environment, opt-in only what we need.
|
|
# No credential var can leak — you'd have to explicitly add it here.
|
|
echo "▶ running per-file parallel test suite via run_tests_parallel.py"
|
|
echo " (TZ=UTC LANG=C.UTF-8 PYTHONHASHSEED=0; clean env)"
|
|
|
|
cd "$REPO_ROOT"
|
|
|
|
# ── Pre-compile .pyc bytecode cache ─────────────────────────────────────────
|
|
# Each test file runs in its own subprocess via run_tests_parallel.py.
|
|
# Pre-building the bytecode cache once here (instead of each subprocess
|
|
# compiling on first import) avoids redundant work across ~2000 processes.
|
|
# Uses git to list tracked .py files (skips venv, node_modules, etc).
|
|
echo "▶ pre-compiling bytecode cache"
|
|
"$PYTHON" -m compileall -q -j 0 -- $(git ls-files '*.py') >/dev/null 2>&1 || true
|
|
|
|
echo "▶ launching test runner"
|
|
exec env -i \
|
|
PATH="$PATH" \
|
|
HOME="$HOME" \
|
|
${WIN_ENV[@]+"${WIN_ENV[@]}"} \
|
|
${TEST_ENV[@]+"${TEST_ENV[@]}"} \
|
|
TZ=UTC \
|
|
LANG=C.UTF-8 \
|
|
LC_ALL=C.UTF-8 \
|
|
PYTHONHASHSEED=0 \
|
|
PYTHONUTF8=1 \
|
|
${HERMES_RUN_SLOW_PET_TESTS:+HERMES_RUN_SLOW_PET_TESTS="$HERMES_RUN_SLOW_PET_TESTS"} \
|
|
${HERMES_E2E_BROWSER:+HERMES_E2E_BROWSER="$HERMES_E2E_BROWSER"} \
|
|
${HERMES_RUN_E2E:+HERMES_RUN_E2E="$HERMES_RUN_E2E"} \
|
|
${EXTRA_PYTHONPATH:+PYTHONPATH="$EXTRA_PYTHONPATH"} \
|
|
${EXTRA_PYTEST_PLUGINS:+PYTEST_PLUGINS="$EXTRA_PYTEST_PLUGINS"} \
|
|
"$PYTHON" "$SCRIPT_DIR/run_tests_parallel.py" "$@"
|