d070e480a3
Root cause of the 2026-08-16 OOM incidents (three runs of `python -m pytest -o addopts= -q tests/hermes_cli/` ballooning to 16-25 GB RSS and getting killed): ~40 files under tests/hermes_cli/ construct SessionDB() directly and never close it. Each instance keeps the writer connection (state.db + -wal fds), up to _READ_POOL_MAX pooled readers with their SQLite page caches, and — once token accounting has run — an atexit registration that pins the instance alive until interpreter exit. In one process over 637 files those accumulate without bound; the sanctioned per-file runner masks it, so CI never saw it. Fix the class, not the sites: * hermes_state: register every successfully constructed SessionDB in a test-only WeakSet (populated only when HERMES_TEST_ISOLATION is set, i.e. under this test suite; production never touches it). * tests/conftest.py: autouse _close_leaked_session_dbs teardown closes everything left in the registry after each test. close() is idempotent and unregisters the pinning atexit hook, so instances become collectable. * tests/conftest.py: session-scoped _pytest_memory_cap applies a defensive RLIMIT_AS of 12 GiB (Linux only) so any future in-process leak fails fast with MemoryError instead of eating the box. Overridable/disable-able via HERMES_PYTEST_MEM_CAP (documented in scripts/run_tests_parallel.py). * tests/hermes_state/test_session_db_leak_sweep.py: behavior contract for registration, idempotent close, and the cross-test sweep. Measured (capped single-process `pytest -o addopts= -q tests/hermes_cli/`): peak RSS 4.16 GiB before -> 1.67 GiB after; per-test open .db fd count previously climbed monotonically (0 -> 12 -> 17 -> 104 within the SessionDB-heavy files), now stays bounded (<= 5, transient). Sanctioned runner over the affected 35 files: 495 passed, 0 failed, no FLAKY. Incident evidence: ~/.hermes/logs/oom-incidents/20260816-202114 (fd dumps show 100+ open state.db/state.db-wal handles across pytest tmpdirs; 3rd recurrence that day).
61 lines
2.4 KiB
Python
61 lines
2.4 KiB
Python
"""Suite-wide SessionDB leak-closing contract (OOM incident 20260816).
|
|
|
|
A raw single-process ``pytest tests/hermes_cli/`` used to accumulate every
|
|
SessionDB a test constructed and forgot to close — writer connection,
|
|
pooled read connections, and (once token accounting ran) an ``atexit``
|
|
registration pinning the instance alive — ballooning to 16-25 GB RSS.
|
|
|
|
The fix is two-sided:
|
|
|
|
* ``hermes_state_guard._register_test_instance`` adds every successfully
|
|
constructed SessionDB to a WeakSet registry when the
|
|
``HERMES_TEST_ISOLATION`` marker is set (test-isolation runs only).
|
|
* the autouse ``_close_leaked_session_dbs`` fixture in ``tests/conftest.py``
|
|
closes everything in the registry at each test's teardown.
|
|
|
|
These tests pin the *behavior contract*: instances register under pytest,
|
|
close() empties them idempotently, and a leaked instance from an earlier
|
|
test is actually closed by the suite-level sweep.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import hermes_state_guard
|
|
from hermes_state import SessionDB
|
|
|
|
# Deliberate cross-test handoff: test_leaked_instance_* leaks an instance;
|
|
# the later test (pytest runs file order deterministically without a
|
|
# randomizer plugin, and the sanctioned runner executes whole files in one
|
|
# process) asserts the autouse teardown sweep closed it.
|
|
_leaked: list[SessionDB] = []
|
|
|
|
|
|
def test_constructed_sessiondb_is_registered(tmp_path):
|
|
db = SessionDB(db_path=tmp_path / "state.db")
|
|
try:
|
|
assert db in hermes_state_guard._test_instance_registry
|
|
finally:
|
|
db.close()
|
|
# close() must fully release the writer connection…
|
|
assert db._conn is None
|
|
# …and be idempotent: a second close (the suite sweep will call it
|
|
# again at teardown) must not raise.
|
|
db.close()
|
|
|
|
|
|
def test_leaked_instance_stays_open_within_the_test(tmp_path):
|
|
db = SessionDB(db_path=tmp_path / "state.db")
|
|
db.create_session(session_id="leak-probe", source="cli", model="m")
|
|
# Intentionally NOT closed — the suite-level sweep owns cleanup.
|
|
assert db._conn is not None
|
|
_leaked.append(db)
|
|
|
|
|
|
def test_previously_leaked_instance_was_closed_by_the_sweep():
|
|
assert _leaked, "expected the previous test to have leaked an instance"
|
|
db = _leaked.pop()
|
|
# The autouse _close_leaked_session_dbs teardown between the two tests
|
|
# must have closed the leaked instance (writer conn released), which is
|
|
# what bounds fd/RSS growth in single-process runs.
|
|
assert db._conn is None
|