fix: name the process holding the state.db write lock when a writer times out
"database is locked (another Hermes process held the state.db write lock for over 60s)" identified the victim only. The open-descriptor scan cannot single out the writer because every Hermes process (gateway, CLI sessions, worktree agents, cron) has the DB open, so an operator hit repeatedly by session_persistence_failed:locked had nothing to act on. SQLite's unix VFS takes fcntl byte-range locks whose offset encodes the lock kind (state.db-shm byte 120 = WAL write, 121 = checkpoint; the pending-byte page on state.db = PENDING/RESERVED), and the kernel exports them with the owning pid in /proc/locks. hermes_state_lockowners reads that table at the moment the patience budget runs out and logs one WARNING per write-class holder with describe_holder_pid()'s argv summary, for both the transcript write path and open+init lock patience. The holder stays out of the exception text on purpose: classify_persistence_error() buckets by phrase and a holder argv such as a worktree named fix-corrupt-db would flip the bucket. Docs: the Write Contention section still described attempt-counted retries (_WRITE_MAX_RETRIES = 15); updated to the time budgets in force and the new log line.
This commit is contained in:
@@ -52,6 +52,7 @@ from hermes_state_profile_repair import SessionProfileRepairMixin
|
||||
from hermes_state_schema import SessionSchemaMixin
|
||||
import hermes_state_holders as _state_holders
|
||||
import hermes_state_lockguard as _lockguard
|
||||
from hermes_state_lockowners import log_write_lock_holders
|
||||
from hermes_state_dbfile import (
|
||||
_connect_tracked_db, _fd_is_truly_unlinked, _prepare_connection_retirement,
|
||||
_read_sqlite_application_id, _stat_sqlite_sidecar_identity,
|
||||
@@ -798,6 +799,7 @@ class SessionDB(
|
||||
self._close_connection_quietly(self._conn)
|
||||
now = time.monotonic()
|
||||
if now >= deadline:
|
||||
log_write_lock_holders(self.db_path, self._WRITE_PATIENCE_S)
|
||||
raise
|
||||
jitter = random.uniform(self._WRITE_RETRY_SLOW_MIN_S, self._WRITE_RETRY_SLOW_MAX_S)
|
||||
time.sleep(min(jitter, max(deadline - now, 0.001)))
|
||||
@@ -1016,7 +1018,10 @@ class SessionDB(
|
||||
if "locked" in err_msg or "busy" in err_msg:
|
||||
if self._sleep_before_write_retry(deadline, patience_s):
|
||||
continue
|
||||
# Say what actually happened, not disk/permission damage.
|
||||
# Say what actually happened, not disk/permission damage. The holder goes to
|
||||
# the log, not the message: classify_persistence_error() buckets by phrase and
|
||||
# a holder's argv (a worktree named fix-corrupt-db) would flip the bucket.
|
||||
log_write_lock_holders(self.db_path, patience_s)
|
||||
raise sqlite3.OperationalError(
|
||||
f"database is locked (another Hermes process held the "
|
||||
f"state.db write lock for over {patience_s:.0f}s — "
|
||||
|
||||
Reference in New Issue
Block a user