fix(state): second-process maintenance on state.db refuses ANY foreign holder

`hermes doctor --fix`'s WAL checkpoint and `repair_state_db_schema`'s
preflight documented themselves as fail-OPEN: `live_writer_holds_db` only
refused on unknown/deleted/uninspectable holders and then trusted a
`BEGIN IMMEDIATE` probe, which is blind to a `journal_mode=DELETE` reader
(SHARED only) and cannot run on a malformed file — exactly the states repair
and checkpoint get invoked in. A repair in a second process then REINDEXed /
VACUUMed a file the gateway still held (#103339 item 2).

- `hermes_state_holders.live_writer_holds_db`: any foreign holder of the DB or
  a sidecar is a live holder; the probe is only an additional positive signal.
- doctor `--fix`: the checkpoint runs on `_exclusive_repair_db_guard`'s
  connection instead of a bare writable `sqlite3.connect`, so an opener
  arriving after the scan is refused, not joined; `_session_count` is a
  `mode=ro` reader.
- Normal SessionDB writers are untouched: gateway + dashboard in two processes
  both keep writing (a process-wide flock on the write path — PR #109270's
  shape — would break that).

Tests: the two-process repair race test releases the test process's own
header-probe fd (it is a genuine holder now); the mid-repair writer fixture
opens its connection after staging starts (a pre-existing holder is refused up
front, which is the point).

Refs #103339 #100896
This commit is contained in:
teknium1
2026-09-14 07:20:38 -07:00
committed by Teknium
parent 4da41c92b4
commit 12173db5b7
7 changed files with 162 additions and 41 deletions

View File

@@ -320,15 +320,14 @@ def live_writer_holds_db(
*,
connect_repair_durable: Callable[..., sqlite3.Connection],
) -> bool:
"""Return whether repair lacks proven exclusive ownership of ``db_path``."""
foreign_holders = foreign_state_db_holders(db_path)
if any(
pid < 0
or path.startswith("uninspectable holder:")
or path.startswith("uninspectable descriptor:")
or path.endswith(" (deleted)")
for pid, path in foreign_holders
):
"""Return whether repair lacks proven exclusive ownership of ``db_path``.
ANY foreign process holding the DB or a sidecar is a live holder (#103339): the lock probe below
cannot see a DELETE-mode reader (SHARED only) and cannot run at all on a malformed file, and those
are exactly the states repair/VACUUM/checkpoint get invoked in. The holder scan is the authority and
fails closed on its own failures (unknown/uninspectable sentinels); the probe only adds a positive
lock signal on top."""
if foreign_state_db_holders(db_path):
return True
probe = None
@@ -342,8 +341,7 @@ def live_writer_holds_db(
lowered = str(exc).lower()
return "locked" in lowered or "busy" in lowered
except sqlite3.DatabaseError:
return False
except Exception:
# Malformed/unreadable with no holder on the scan: nobody else has it open, so repair may run.
return False
finally:
if probe is not None: