In rollback-journal (DELETE) mode a sibling process can take the write lock
between schema load and the messages_fts probe. FTS5's xConnect then fails its
%_config read and SQLite reports SQLITE_BUSY with the text "vtable constructor
failed: messages_fts". Every state.db lock classifier matched on the words
"locked"/"busy", so:
- a writable SessionDB() failed after 1s instead of waiting out the lock with
_WRITE_PATIENCE_S, and callers disabled persistence for the run;
- a read-only open (dashboard, `hermes sessions list`, cross-profile readers)
failed on the first busy timeout with no retry at all;
- the error read as not transient (dashboard 500, not 503) and as persistence
cause "unknown" instead of "locked".
Add hermes_state_errors.is_sqlite_lock_error: SQLITE_BUSY/SQLITE_LOCKED by
result code when SQLite supplies one, text only when it does not (our own
re-raised messages, RPC-wrapped strings). Route the writer open patience loop,
the _execute_write retry, the reconcile re-raise, the WAL->DELETE flip, the
maintenance holder probe, is_transient_sqlite_error and
classify_persistence_error through it. The read-only open retries a lock
inside its existing bounded retry budget, next to the transient IOERR case.
`database.journal_mode: delete` can never self-apply to a store that is already WAL:
apply_wal_with_fallback deliberately never live-downgrades (#68545 — other gateway/cron/worker
connections may hold uncheckpointed WAL commits), so operators applying the containment for the
multi-writer corruption class saw one ERROR per process forever and the only escape hatch was an
undocumented hand-run PRAGMA on the file.
The new pre-DB `sessions set-journal-mode` verb is the sanctioned offline path: it refuses while ANY
foreign process holds the file or a sidecar (the same foreign_state_db_holders scan doctor/repair
admission uses, naming each PID), flips through _set_journal_mode_no_wait (busy_timeout=0, so an
opener appearing mid-way makes SQLite refuse instead of racing it), verifies header bytes 18/19,
and reminds the operator when config.yaml disagrees. `--db PATH` covers kanban.db / cron stores
that log the same ERROR. The never-live-downgrade invariant is untouched; the ERROR, doctor hints
and docs now name the command instead of the raw PRAGMA.
apply_wal_with_fallback returns via _apply_delete_for_wal_reset_bug on
WAL-reset-vulnerable SQLite builds (Debian 12 / Ubuntu 22.04 system Pythons,
the reporter's pinned image) before the #110848 existing-WAL cross-VM check,
so exactly the deployment class the fix targets still got zero startup signal
while doctor flagged it. Share one helper between both early-return paths and
key the once-per-process dedupe on the DB path instead of the label so a
gateway serving several profiles hears about each database. The docker doc
remedy now uses the image's python3 (it ships libsqlite3 but no sqlite3 shell).
Review finding: vulnerable-SQLite early return skipped the cross-VM ERROR; dedupe per label; sqlite3 shell absent from image.
d8dcdfd620 (v2026.9.14) made apply_wal_with_fallback refuse to ENABLE WAL
on a fresh database whose directory is on a cross-VM bind mount (virtiofs/9p),
but a database that was already WAL on such a mount kept WAL — correctly, we
never live-downgrade under other openers — and emitted nothing. The operator
in #110848 ran exactly that shape (Podman applehv virtiofs bind mount) and got
"database disk image is malformed" within a minute with no prior signal.
- apply_wal_with_fallback: in the on-disk-WAL branch, log a once-per-process
ERROR ('cross_vm_fs_existing_wal') when the DB file is on a cross-VM
filesystem, naming the two remedies (offline PRAGMA journal_mode=DELETE
after stopping every process + database.journal_mode: delete, or move the
database to a native/named volume). The fresh-DB refusal is unchanged.
- hermes doctor: _report_database_journal_modes flags a WAL database on a
cross-VM filesystem with check_warn and the same remedy (ranked above the
WAL-reset exposure warning; the exposure bookkeeping is kept).
- docs: docker.md gains "Filesystem requirements for state.db in containers";
configuration.md's database comment no longer implies operators must set
delete by hand on virtiofs.
Detection stays /proc/self/mountinfo-based (runs inside the Linux container
on macOS/Windows hosts). locking_mode=EXCLUSIVE is deliberately not adopted:
gateway, cron and workers open state.db concurrently.
Fixes#110848
Port from openclaw/openclaw#120597: Docker Desktop, OrbStack, and Podman
expose host bind mounts as fuse.virtiofs or 9p. SQLite WAL shared-memory
is not coherent across the VM boundary and fails SILENTLY under write
pressure (zero-filled pages), so the existing reactive marker fallback
(_WAL_INCOMPAT_MARKERS) never fires — the mode must be refused before
the pragma.
apply_wal_with_fallback now checks the DB path against
/proc/self/mountinfo (longest mount-point prefix wins; per-directory
cached) and applies journal_mode=DELETE with a one-shot WARNING naming
the fix (native volume instead of a bind mount). On-disk WAL databases
are still never live-downgraded; require_wal callers get
WalUnsupportedError. Non-Linux hosts and unreadable mount tables
conservatively skip detection (behavior unchanged).
Sabotage-verified: disabling only the wiring makes
test_fresh_db_on_cross_vm_fs_gets_delete fail.
Follow-up to the probe-unknown guard:
- `require_wal=True` now raises WalUnsupportedError when the on-disk mode
cannot be read instead of reporting an unverified "wal" (the function's
contract is "the mode actually set").
- Drop `_apply_wal_companions` from the branch: journal_size_limit /
synchronous pragmas were being applied to a file whose mode is unknown,
contradicting the "touch nothing" rule the branch exists for.
- Cut the retracted #104596 mechanism narrative from comments, log text and
docstrings; the guard is hardening (same rule the DELETE branch already
applied), not a root-cause fix.
- Tests: one binding test per file (no pragma of any kind reaches the
connection; require_wal raises); drop the log-dedupe test (`_log_once`
is already covered). test_captures_cause_on_failed_init's double now
fails only set-pragmas, matching the read-only-mount it simulates.
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
