The FTS fail-open detach now waits up to the caller's write budget (20 s /
60 s) for the write lock, so the one-time quarantine check before the loop
left a long window: a sibling that quarantined the file meanwhile still got
its triggers dropped and the stale breadcrumb committed on the quarantined
handle. Re-check the handle flag and the process-wide storage latch at the
top of every attempt, via the same _raise_if_db_corrupt(storage=True) that
_execute_write runs per attempt.
Classify the retryable lock error with is_sqlite_lock_error (result code
first) instead of a locked/busy substring match, matching #120488.
When a canonical write trips a corrupt FTS index, SessionDB detaches the derived
indexes (breadcrumb + trigger drop) and retries the write. The detach ran one
BEGIN IMMEDIATE on the writer connection, whose busy timeout is only 1 s, and gave
up on "database is locked" — so the canonical write escaped as "database disk
image is malformed". The usual lock holder is a sibling writer (gateway + TUI)
detaching the same index, so under load the second writer's turn was lost.
The detach now waits out lock contention on the caller's write budget with the
same jittered retry as _execute_write (default _WRITE_PATIENCE_S for the search
fail-open callers).
Repro: a second process takes BEGIN IMMEDIATE the instant the corruption error
surfaces and holds it 2.5 s. Base: append raises after 1.02 s (3/3). Fixed: the
row lands after the holder releases, FTS detached (3/3). Found by the E2E sqlite
torture chamber (fts_corruption_fail_open) at load ~200.
closes#116244
serialize _raise_if_db_replaced inside self._lock and handle reopen after close across optimize_fts, rebuild_fts, vacuum, _enter_fts_fail_open, and _execute_write error branch so a clean close never falsely poisons handles with deletedwalgenerationerror.
gateway/run.py bridges the LAUNCH profile's sessions.cjk_fts / search_slow_ms
into HERMES_CJK_FTS / HERMES_SEARCH_SLOW_MS at import (and re-bridged them per
turn), and hermes_state_fts / hermes_state_search read os.getenv — so a served
secondary always got the default profile's values.
hermes_state_common.routed_sessions_setting() reads the routed profile's
config.yaml under a HERMES_HOME override and the env bridge when unscoped; both
consumers use it. The per-turn re-bridge is skipped inside a secondary's scope
so it can no longer write the default's slots from a routed turn.
classify_persistence_error bucketed every _DB_CORRUPTION_MARKERS hit as "corrupt",
so an error SQLite itself scoped to the FTS5 index layer (SQLITE_CORRUPT_VTAB, or an
`fts5: corrupt structure record for table "messages_fts"` report) that escaped the
write path — the detach in _enter_fts_fail_open refused (generation/lock check),
or a read/search path with no fail-open at all — reached the turn boundary and the
gateway startup notice as structural corruption: the turn ended with `.recover` /
restore-backup advice on a file whose canonical tables were provably healthy.
One provenance rule, hermes_state_errors.is_fts_scoped_corruption_error, now feeds
both the write-repair gate (SessionDB._is_fts_write_corruption_error delegates to it,
so the gateway transcript retry inherits it) and the classifier: a known result code
outranks prose (only SQLITE_CORRUPT_VTAB is FTS-scoped; bare SQLITE_CORRUPT/NOTADB
and any contradictory code fail closed), and without a code the text must both carry
a corruption marker and name a messages_fts* object. The new "fts_index" cause
renders index-scoped guidance (doctor --fix / restart, do not run recovery) in the
turn explainer and the home-channel notice. The structural fail-close is untouched:
bare malformed / not-a-database still quarantine and still classify "corrupt".
Salvaged from PR #97843 (SulthanZahran1), trimmed: the quick_check-backed
"corrupt_unconfirmed" tier is dropped — on a live handle that just observed an
unscoped SQLITE_CORRUPT, PRAGMA quick_check on a damaged shadow b-tree raises rather
than reports on 3.53.1, so the probe could never downgrade the exact shape it was
built for, and a verdict that softens quarantine guidance on prose alone weakens the
fail-close. #97841 (Finn763) reached the same fts_index cause via text markers
only; its LIKE-degradation intent already lives in _search_messages_impl (_fts_stale).
Fixes#97794
Co-authored-by: finn763 <165816600+finn763@users.noreply.github.com>
`sqlite3 state.db .recover` re-emits the FTS5 shadow tables (messages_fts_data,
_idx, _docsize, _config, _content) as ordinary tables but cannot re-emit the
CREATE VIRTUAL TABLE row. The next SessionDB open ran the FTS DDL in
_ensure_fts_schema and died with "fts5: error creating shadow table
messages_fts_data: table 'messages_fts_data' already exists", so a recovered
database was unusable until someone hand-dropped the shadows.
_init_fts now runs _drop_orphan_fts_shadow_tables before any FTS DDL. It is
per-family and exact-name scoped: a family's shadows are dropped only when its
own vtable row is absent from sqlite_master (type='table' AND sql LIKE
'CREATE VIRTUAL TABLE%'), so a healthy messages_fts_trigram survives a
base-family repair untouched. A repaired base/trigram family is then treated
like a missing-trigger repair and rebuilt from the canonical messages table
under the cross-process rebuild admission; the shadows are derived index
state, nothing is lost.
Live repro: real `sqlite3 x.db .recover | sqlite3 y.db` on sqlite 3.50.4
keeps the vtable rows (the shell emits CREATE VIRTUAL TABLE), so the
deterministic fixture removes the vtable row via writable_schema leaving the
shadows behind: BEFORE OperationalError on open; AFTER opens, fts_enabled,
MATCH returns every message, trigram sqlite_master rowids unchanged.
Salvaged from PR #56824 (intent applied onto the current hermes_state_fts /
hermes_state_schema siblings). The ownership-safety point (never touch a live
family's shadows) was raised by @ggoldani in #103868 / #103840.
Refs #103840
Refs #56815
Co-authored-by: ggoldani <ggoldani@users.noreply.github.com>
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
