fix(tests): close leaked SessionDB handles suite-wide and cap pytest memory
Root cause of the 2026-08-16 OOM incidents (three runs of `python -m pytest -o addopts= -q tests/hermes_cli/` ballooning to 16-25 GB RSS and getting killed): ~40 files under tests/hermes_cli/ construct SessionDB() directly and never close it. Each instance keeps the writer connection (state.db + -wal fds), up to _READ_POOL_MAX pooled readers with their SQLite page caches, and — once token accounting has run — an atexit registration that pins the instance alive until interpreter exit. In one process over 637 files those accumulate without bound; the sanctioned per-file runner masks it, so CI never saw it. Fix the class, not the sites: * hermes_state: register every successfully constructed SessionDB in a test-only WeakSet (populated only when HERMES_TEST_ISOLATION is set, i.e. under this test suite; production never touches it). * tests/conftest.py: autouse _close_leaked_session_dbs teardown closes everything left in the registry after each test. close() is idempotent and unregisters the pinning atexit hook, so instances become collectable. * tests/conftest.py: session-scoped _pytest_memory_cap applies a defensive RLIMIT_AS of 12 GiB (Linux only) so any future in-process leak fails fast with MemoryError instead of eating the box. Overridable/disable-able via HERMES_PYTEST_MEM_CAP (documented in scripts/run_tests_parallel.py). * tests/hermes_state/test_session_db_leak_sweep.py: behavior contract for registration, idempotent close, and the cross-test sweep. Measured (capped single-process `pytest -o addopts= -q tests/hermes_cli/`): peak RSS 4.16 GiB before -> 1.67 GiB after; per-test open .db fd count previously climbed monotonically (0 -> 12 -> 17 -> 104 within the SessionDB-heavy files), now stays bounded (<= 5, transient). Sanctioned runner over the affected 35 files: 495 passed, 0 failed, no FLAKY. Incident evidence: ~/.hermes/logs/oom-incidents/20260816-202114 (fd dumps show 100+ open state.db/state.db-wal handles across pytest tmpdirs; 3rd recurrence that day).
This commit is contained in:
@@ -6,6 +6,7 @@ state.db; env-based so subprocess children are protected too."""
|
||||
import os
|
||||
import sys
|
||||
import threading
|
||||
import weakref
|
||||
from pathlib import Path
|
||||
from typing import Any, Optional
|
||||
|
||||
@@ -125,6 +126,30 @@ def _is_production_state_db(resolved: Path, root: Path) -> bool:
|
||||
return len(parts) == 3 and parts[0] == "profiles"
|
||||
|
||||
|
||||
# Test-only SessionDB instance registry. Under the hermetic suite every
|
||||
# successfully constructed SessionDB is added to this WeakSet so the autouse
|
||||
# teardown in tests/conftest.py (_close_leaked_session_dbs) can close whatever
|
||||
# a test forgot to close. Dozens of tests build SessionDB() directly and never
|
||||
# close it; each instance holds a writer connection plus pooled readers, and a
|
||||
# single-process run over tests/hermes_cli/ accumulated 16-25 GB RSS (OOM
|
||||
# incident 20260816). The per-file runner masks this in CI; the registry fixes
|
||||
# the class at the source instead of patching ~40 test files.
|
||||
#
|
||||
# Population is gated on the isolation marker (exported by tests/conftest.py
|
||||
# before any test module imports). Production processes never populate it;
|
||||
# do not "simplify" the gate away. WeakSet membership never pins an instance.
|
||||
_test_instance_registry: "weakref.WeakSet[Any]" = weakref.WeakSet()
|
||||
|
||||
|
||||
def _register_test_instance(db: Any) -> None:
|
||||
"""Track *db* for suite-level teardown closing (test-isolation runs only)."""
|
||||
if os.environ.get(_TEST_ISOLATION_MARKER_ENV):
|
||||
try:
|
||||
_test_instance_registry.add(db)
|
||||
except Exception: # pragma: no cover — registry must never break init
|
||||
pass
|
||||
|
||||
|
||||
# Last SessionDB() init error, per-process; surfaced by /resume-style slash
|
||||
# commands so users know WHY. Only SessionDB.__init__ writes it.
|
||||
_last_init_error: Optional[str] = None
|
||||
|
||||
Reference in New Issue
Block a user