fix: name the process holding the state.db write lock when a writer times out

"database is locked (another Hermes process held the state.db write lock for
over 60s)" identified the victim only. The open-descriptor scan cannot single
out the writer because every Hermes process (gateway, CLI sessions, worktree
agents, cron) has the DB open, so an operator hit repeatedly by
session_persistence_failed:locked had nothing to act on.

SQLite's unix VFS takes fcntl byte-range locks whose offset encodes the lock
kind (state.db-shm byte 120 = WAL write, 121 = checkpoint; the pending-byte page
on state.db = PENDING/RESERVED), and the kernel exports them with the owning pid
in /proc/locks. hermes_state_lockowners reads that table at the moment the
patience budget runs out and logs one WARNING per write-class holder with
describe_holder_pid()'s argv summary, for both the transcript write path and
open+init lock patience. The holder stays out of the exception text on purpose:
classify_persistence_error() buckets by phrase and a holder argv such as a
worktree named fix-corrupt-db would flip the bucket.

Docs: the Write Contention section still described attempt-counted retries
(_WRITE_MAX_RETRIES = 15); updated to the time budgets in force and the new log line.
This commit is contained in:
teknium1
2026-09-19 23:36:26 -07:00
committed by Teknium
parent b851bd1a23
commit 2182f51d7c
4 changed files with 211 additions and 5 deletions

View File

@@ -52,6 +52,7 @@ from hermes_state_profile_repair import SessionProfileRepairMixin
from hermes_state_schema import SessionSchemaMixin
import hermes_state_holders as _state_holders
import hermes_state_lockguard as _lockguard
from hermes_state_lockowners import log_write_lock_holders
from hermes_state_dbfile import (
_connect_tracked_db, _fd_is_truly_unlinked, _prepare_connection_retirement,
_read_sqlite_application_id, _stat_sqlite_sidecar_identity,
@@ -798,6 +799,7 @@ class SessionDB(
self._close_connection_quietly(self._conn)
now = time.monotonic()
if now >= deadline:
log_write_lock_holders(self.db_path, self._WRITE_PATIENCE_S)
raise
jitter = random.uniform(self._WRITE_RETRY_SLOW_MIN_S, self._WRITE_RETRY_SLOW_MAX_S)
time.sleep(min(jitter, max(deadline - now, 0.001)))
@@ -1016,7 +1018,10 @@ class SessionDB(
if "locked" in err_msg or "busy" in err_msg:
if self._sleep_before_write_retry(deadline, patience_s):
continue
# Say what actually happened, not disk/permission damage.
# Say what actually happened, not disk/permission damage. The holder goes to
# the log, not the message: classify_persistence_error() buckets by phrase and
# a holder's argv (a worktree named fix-corrupt-db) would flip the bucket.
log_write_lock_holders(self.db_path, patience_s)
raise sqlite3.OperationalError(
f"database is locked (another Hermes process held the "
f"state.db write lock for over {patience_s:.0f}s — "