fix(tests): close leaked SessionDB handles suite-wide and cap pytest memory
Root cause of the 2026-08-16 OOM incidents (three runs of `python -m pytest -o addopts= -q tests/hermes_cli/` ballooning to 16-25 GB RSS and getting killed): ~40 files under tests/hermes_cli/ construct SessionDB() directly and never close it. Each instance keeps the writer connection (state.db + -wal fds), up to _READ_POOL_MAX pooled readers with their SQLite page caches, and — once token accounting has run — an atexit registration that pins the instance alive until interpreter exit. In one process over 637 files those accumulate without bound; the sanctioned per-file runner masks it, so CI never saw it. Fix the class, not the sites: * hermes_state: register every successfully constructed SessionDB in a test-only WeakSet (populated only when HERMES_TEST_ISOLATION is set, i.e. under this test suite; production never touches it). * tests/conftest.py: autouse _close_leaked_session_dbs teardown closes everything left in the registry after each test. close() is idempotent and unregisters the pinning atexit hook, so instances become collectable. * tests/conftest.py: session-scoped _pytest_memory_cap applies a defensive RLIMIT_AS of 12 GiB (Linux only) so any future in-process leak fails fast with MemoryError instead of eating the box. Overridable/disable-able via HERMES_PYTEST_MEM_CAP (documented in scripts/run_tests_parallel.py). * tests/hermes_state/test_session_db_leak_sweep.py: behavior contract for registration, idempotent close, and the cross-test sweep. Measured (capped single-process `pytest -o addopts= -q tests/hermes_cli/`): peak RSS 4.16 GiB before -> 1.67 GiB after; per-test open .db fd count previously climbed monotonically (0 -> 12 -> 17 -> 104 within the SessionDB-heavy files), now stays bounded (<= 5, transient). Sanctioned runner over the affected 35 files: 495 passed, 0 failed, no FLAKY. Incident evidence: ~/.hermes/logs/oom-incidents/20260816-202114 (fd dumps show 100+ open state.db/state.db-wal handles across pytest tmpdirs; 3rd recurrence that day).
This commit is contained in:
@@ -35,6 +35,12 @@ Environment:
|
||||
HERMES_TEST_PATHS Override discovery roots (colon-sep; on Windows
|
||||
';' also works and drive letters are handled;
|
||||
default: 'tests')
|
||||
HERMES_PYTEST_MEM_CAP
|
||||
Per-pytest-process RLIMIT_AS cap in GiB applied by
|
||||
tests/conftest.py (Linux only; default 12). Any
|
||||
runaway in-process accumulation dies with
|
||||
MemoryError instead of eating the box. Set to 0 /
|
||||
'off' to disable, or another integer to change it.
|
||||
|
||||
Exit code: 0 if every file's pytest exited 0; 1 otherwise.
|
||||
"""
|
||||
|
||||
Reference in New Issue
Block a user