fix(tests): close leaked SessionDB handles suite-wide and cap pytest memory

Root cause of the 2026-08-16 OOM incidents (three runs of
`python -m pytest -o addopts= -q tests/hermes_cli/` ballooning to
16-25 GB RSS and getting killed): ~40 files under tests/hermes_cli/
construct SessionDB() directly and never close it. Each instance keeps
the writer connection (state.db + -wal fds), up to _READ_POOL_MAX pooled
readers with their SQLite page caches, and — once token accounting has
run — an atexit registration that pins the instance alive until
interpreter exit. In one process over 637 files those accumulate without
bound; the sanctioned per-file runner masks it, so CI never saw it.

Fix the class, not the sites:

* hermes_state: register every successfully constructed SessionDB in a
  test-only WeakSet (populated only when HERMES_TEST_ISOLATION is set,
  i.e. under this test suite; production never touches it).
* tests/conftest.py: autouse _close_leaked_session_dbs teardown closes
  everything left in the registry after each test. close() is idempotent
  and unregisters the pinning atexit hook, so instances become
  collectable.
* tests/conftest.py: session-scoped _pytest_memory_cap applies a
  defensive RLIMIT_AS of 12 GiB (Linux only) so any future in-process
  leak fails fast with MemoryError instead of eating the box.
  Overridable/disable-able via HERMES_PYTEST_MEM_CAP (documented in
  scripts/run_tests_parallel.py).
* tests/hermes_state/test_session_db_leak_sweep.py: behavior contract
  for registration, idempotent close, and the cross-test sweep.

Measured (capped single-process `pytest -o addopts= -q tests/hermes_cli/`):
peak RSS 4.16 GiB before -> 1.67 GiB after; per-test open .db fd count
previously climbed monotonically (0 -> 12 -> 17 -> 104 within the
SessionDB-heavy files), now stays bounded (<= 5, transient). Sanctioned
runner over the affected 35 files: 495 passed, 0 failed, no FLAKY.

Incident evidence: ~/.hermes/logs/oom-incidents/20260816-202114
(fd dumps show 100+ open state.db/state.db-wal handles across pytest
tmpdirs; 3rd recurrence that day).
This commit is contained in:
Teknium
2026-08-16 22:25:05 -07:00
parent 31cde4e6ae
commit d070e480a3
5 changed files with 198 additions and 1 deletions

View File

@@ -6,6 +6,7 @@ state.db; env-based so subprocess children are protected too."""
import os
import sys
import threading
import weakref
from pathlib import Path
from typing import Any, Optional
@@ -125,6 +126,30 @@ def _is_production_state_db(resolved: Path, root: Path) -> bool:
return len(parts) == 3 and parts[0] == "profiles"
# Test-only SessionDB instance registry. Under the hermetic suite every
# successfully constructed SessionDB is added to this WeakSet so the autouse
# teardown in tests/conftest.py (_close_leaked_session_dbs) can close whatever
# a test forgot to close. Dozens of tests build SessionDB() directly and never
# close it; each instance holds a writer connection plus pooled readers, and a
# single-process run over tests/hermes_cli/ accumulated 16-25 GB RSS (OOM
# incident 20260816). The per-file runner masks this in CI; the registry fixes
# the class at the source instead of patching ~40 test files.
#
# Population is gated on the isolation marker (exported by tests/conftest.py
# before any test module imports). Production processes never populate it;
# do not "simplify" the gate away. WeakSet membership never pins an instance.
_test_instance_registry: "weakref.WeakSet[Any]" = weakref.WeakSet()
def _register_test_instance(db: Any) -> None:
"""Track *db* for suite-level teardown closing (test-isolation runs only)."""
if os.environ.get(_TEST_ISOLATION_MARKER_ENV):
try:
_test_instance_registry.add(db)
except Exception: # pragma: no cover — registry must never break init
pass
# Last SessionDB() init error, per-process; surfaced by /resume-style slash
# commands so users know WHY. Only SessionDB.__init__ writes it.
_last_init_error: Optional[str] = None