fix(doctor): name structural state.db corruption honestly and route it to sessions recover

`hermes doctor` reported every write-health-probe failure as "state.db FTS write
corruption" and `--fix` ran the FTS repair ladder — rebuild, REINDEX, sqlite_master
surgery + VACUUM — on the damaged file. When the damage is structural (canonical
tables/indexes), none of those rungs can fix it, each one writes to the torn file in
place, and the operator is then told to "restore from the backup copy beside
state.db": a `.malformed-backup` that is a snapshot of the same corrupt image.
`hermes sessions recover`, the tool that actually rebuilds canonical rows into a
fresh file, was never mentioned (#88587; the 1.7 GB field incident lost days to it).

Discriminate before mutating. hermes_state_repair.integrity_damage_is_structural
maps `PRAGMA integrity_check` output onto the file: a `Tree N` id resolved through
sqlite_master.rootpage, an index named in `row N missing from index X`, or a
`Freelist:` line is structural unless the object is a Hermes-owned messages_fts*
table/shadow (full-matched, so a user lookalike such as archive_fts_data is never
swept into the rebuildable set). state_db_has_structural_damage runs it read-only on
a fresh connection; an integrity_check that RAISES under the walk (torn root page)
is structural too — no FTS-only fixture does that while sessions/messages read
cleanly. doctor's state check consults it first: structural damage becomes a
manual issue naming `hermes [-p <profile>] sessions recover --source <this db>
--inspect-only` (profile pinned, #105887) and explicitly warning off the
.malformed-backup; nothing is mutated and no backup is written. FTS-only damage
keeps the existing in-place repair path.

Verified against real fixtures: a torn `sessions` root page (before: "FTS write
corruption", --fix wrote a 1:1 malformed-backup and failed; after: structural,
recover guidance, no writes) and the 16-byte DEADBEEF messages_fts_data stomp
(still repaired in place via rebuild_fts).

Salvaged from PR #88604 (liuhao1024) onto the split doctor_state.py; the
classifier lives beside the repair ladder in hermes_state_repair so the ladder
itself can consult it next.

Fixes #88587
This commit is contained in:
liuhao1024
2026-09-11 03:06:35 -07:00
committed by Teknium
parent 38adfe90a4
commit 3d167a8271
4 changed files with 177 additions and 1 deletions

View File

@@ -12,6 +12,7 @@ import itertools
import json
import logging
import os
import re
import shutil
import sqlite3
import stat
@@ -684,6 +685,62 @@ def _schema_not_built(exc: BaseException) -> bool:
return any(m in str(exc).lower() for m in ("no such table", "no such column"))
# Hermes-owned FTS5 objects: the virtual tables and their shadow b-trees. Full-matched, so a
# user-created lookalike (``archive_fts_data``) is not swept into the rebuildable set.
_FTS_OBJECT_RE = re.compile(
r"messages_fts(_trigram|_cjk)?(_data|_idx|_content|_docsize|_config|_segdir|_segments)?"
)
_INTEGRITY_TREE_RE = re.compile(r"\bTree (\d+)\b")
_INTEGRITY_MISSING_INDEX_RE = re.compile(r"missing from index (\S+)")
def integrity_damage_is_structural(integrity_lines, master_rows) -> bool:
"""True when any damaged object named by ``PRAGMA integrity_check`` output lies outside
the FTS shadow set: a ``Tree N`` id mapped through ``sqlite_master.rootpage``, an index in
``row N missing from index X``, or the file's own freelist. Unparseable lines are not
counted (the FTS wording stays, which is incomplete rather than wrong). #88587: an FTS
rebuild cannot repair a canonical b-tree, and the ``.malformed-backup`` it leaves behind
is a snapshot of the same damage."""
name_by_rootpage = {int(rp): name for rp, _type, name in master_rows if rp}
for line in integrity_lines:
text = str(line)
if text.startswith("Freelist"):
return True
tree = _INTEGRITY_TREE_RE.search(text)
if tree:
name = name_by_rootpage.get(int(tree.group(1)), "")
if name and not _FTS_OBJECT_RE.fullmatch(name):
return True
missing = _INTEGRITY_MISSING_INDEX_RE.search(text)
if missing and not _FTS_OBJECT_RE.fullmatch(missing.group(1)):
return True
return False
def state_db_has_structural_damage(db_path: Path) -> bool:
"""Read-only ``integrity_check`` + ``sqlite_master`` rootpage map on a fresh connection;
``integrity_damage_is_structural`` over the result. A check that RAISES instead of
reporting (a torn page under the walk) is structural too: no FTS-only fixture does that
while ``messages``/``sessions`` read cleanly, and the FTS rebuild ladder cannot help.
Cannot-open / locked stays False so the caller keeps the FTS path."""
try:
conn = sqlite3.connect(f"file:{db_path}?mode=ro", uri=True, timeout=1.0)
except sqlite3.Error:
return False
try:
master_rows = [tuple(r) for r in conn.execute(
"SELECT rootpage, type, name FROM sqlite_master WHERE rootpage > 0").fetchall()]
lines = [str(r[0]) for r in conn.execute("PRAGMA integrity_check").fetchall()]
except sqlite3.OperationalError:
return False
except sqlite3.DatabaseError:
return True
finally:
conn.close()
return integrity_damage_is_structural(
itertools.chain.from_iterable(line.splitlines() for line in lines), master_rows)
def _db_opens_cleanly(db_path: Path) -> Optional[str]:
"""Probe a DB on a fresh connection. Returns None if healthy, else a reason.