fix(web): join the state.db eager-reconcile worker at lifespan shutdown

The dashboard lifespan started `_eager_reconcile_own_session_db` on a
daemon thread and never joined it. Under pytest each TestClient context
spawned one; the fresh tmp HERMES_HOME store makes every worker take the
bootstrap path, so on a slow CI runner ten of them were still queued on
`_session_db_bootstrap_lock` when the test's autouse leaked-DB sweep ran
`SessionDB.close()` on the connection the live worker was stepping in
`_open_probed` -> cross-thread `sqlite3_close` on an active statement ->
`Fatal Python error: Segmentation fault` after every test had passed
(PR #113430 run 35153362037, attempt 1; ~1/4 locally).

The worker is now a regular (non-daemon) thread that the lifespan
`finally` joins, so its connection is only ever closed by the thread that
opened it and it cannot outlive the server or the interpreter. Startup is
unchanged (the open still happens off the ready-probe path); the join is
bounded by SessionDB's write patience, so shutdown cannot hang on it.
This commit is contained in:
teknium1
2026-09-16 15:20:12 -07:00
committed by Teknium
parent c8f0f6b136
commit c15286f44d
2 changed files with 39 additions and 5 deletions

View File

@@ -152,13 +152,18 @@ async def _lifespan(app: "FastAPI"):
# Bring state.db schema current BEFORE the first session-list poll
# (#79531/#80037): a store left behind by `hermes update` otherwise 500s
# every poll while the read-probe heal loses to sibling lock contention.
# Daemon thread so a locked store never delays the socket (Desktop
# ready-probe times out at 10s, GH-73083).
threading.Thread(
# Off-thread so a locked store never delays the socket (Desktop
# ready-probe times out at 10s, GH-73083). NOT a daemon, and joined at
# shutdown: its sqlite connection must be closed by the thread that is
# stepping it. A daemon copy that outlived the lifespan had its
# connection closed from the main thread mid-probe (pytest's leaked-DB
# sweep) and segfaulted the interpreter. The worker is time-bounded by
# SessionDB's lock patience, so the join cannot hang shutdown.
eager_reconcile_thread = threading.Thread(
target=_eager_reconcile_own_session_db,
daemon=True,
name="statedb-eager-reconcile",
).start()
)
eager_reconcile_thread.start()
# Import hermes_cli.gateway *before* the yield: on Windows + 3.11 the
# import holds the GIL, so run_in_executor still froze the loop 15-22s and
@@ -274,6 +279,7 @@ async def _lifespan(app: "FastAPI"):
pass
if os.getenv("HERMES_DESKTOP") == "1":
_terminate_desktop_managed_gateway()
eager_reconcile_thread.join()
def _app_state_default(app: "FastAPI", name: str, factory):

View File

@@ -108,6 +108,34 @@ def test_hosted_room_recovery_cannot_block_or_abort_backend_startup(monkeypatch)
release.set()
def test_lifespan_shutdown_joins_statedb_reconcile_worker(monkeypatch):
"""The eager state.db reconcile runs off the startup path but never outlives
the lifespan: shutdown joins it, so its sqlite connection is only ever closed
by the thread stepping it (a daemon copy left running had its connection
closed cross-thread by teardown and segfaulted the interpreter)."""
from fastapi.testclient import TestClient
started = threading.Event()
finished = threading.Event()
def slow_reconcile():
started.set()
time.sleep(SLOW_SECONDS)
finished.set()
monkeypatch.setattr(web_server_mod, "_warm_gateway_module", lambda: None)
monkeypatch.setattr(web_server_mod, "_eager_reconcile_own_session_db", slow_reconcile)
before = time.perf_counter()
with TestClient(web_server_mod.app, raise_server_exceptions=False):
assert started.wait(timeout=1.0)
# Off the startup path: the socket is up long before the worker is done.
assert time.perf_counter() - before < SLOW_SECONDS * 0.8
assert finished.is_set(), "lifespan shutdown returned before the reconcile worker finished"
assert not any(t.name == "statedb-eager-reconcile" for t in threading.enumerate())
# ---------------------------------------------------------------------------
# Test 2 — get_status run_in_executor keeps event loop free for other requests
# ---------------------------------------------------------------------------