SQLite protects a WAL generation with per-PROCESS POSIX locks (SHARED on
state.db, DMS byte on -shm). Any in-process open()/close() of either file
cancels both (sqlite.org/howtocorrupt.html §2.2); the next last-connection
close in ANY process then checkpoints and unlinks -wal/-shm, and the holder
sticky-halts with DeletedWalGenerationError. #109841 removed one such
close (mode tightening) but the class is open-ended: raw header probes,
plugins, tool reads of ~/.hermes, any library that touches the files.
hermes_state_lockguard re-holds the same two ranges as OFD locks
(F_OFD_SETLK) on private descriptors for as long as a writer handle is
open. OFD locks belong to the open file description, so a stray close()
cannot cancel them, and they conflict with the EXCLUSIVE a sibling needs
for the close-time reset exactly like SQLite's own. Released before the
handle's own close so a true last close still ends the generation; the
descriptors are closed only once no connection to the path remains, so a
holder scan from another process never counts them. Works on Python 3.11
(where sqlite3 cannot arm SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE) and on macOS
(F_OFD_SETLK=90 per XNU bsd/sys/fcntl.h); no-op on Windows.
Live repro (Linux, Python 3.11.15, SQLite 3.53.1): holder = SessionDB
writer; in-process os.open/os.close of state.db and -shm; then a foreign
sqlite3.connect()+close(). Before: -wal unlinked, holder write raises
DeletedWalGenerationError. After: -wal keeps its inode, holder writes.
CLI `_rewind_persisted_user_turn`, TUI `_rewind_active_session_history` and gateway
`rewind_session` each re-ran get_active_message_ids -> get_messages_as_conversation ->
split_user_originated_turn -> rewind_to_message with their own warm/durable comparison
helpers and three different out-of-range contracts (RuntimeError / ValueError / None).
The durable transcript is the authority for a rewind, so the implementation now lives
with the data: `SessionDB.rewind_user_turn` (hermes_state_rewind.py) with one typed
out-of-range error (`RewindTargetUnavailableError`). Surfaces keep only lock, eviction
and rendering glue and map that error to their own message.
Twelve modules each carried their own sqlite3.connect + PRAGMA + `with conn:`
stack. The #69567 fd-leak fix (a `with conn:` commits but never closes, so each
call leaked a connection and its WAL/SHM fds until GC) was pasted as code plus
docstring into six of them and hosted_room_policy_checkpoint never received
it; plugins/plugin_storage.plugin_db was the only production caller issuing a
raw `PRAGMA journal_mode=WAL`, bypassing the network-FS fallback, the
WAL-reset-bug gate and the never-live-downgrade invariant that
hermes_state_wal.apply_wal_with_fallback carries.
hermes_cli/sqlite_util.py (already home to add_column_if_missing/write_txn,
imported by cron, gateway and hermes_cli alike) gains `open_db(path, *,
db_label, busy_timeout_ms, wal, foreign_keys, synchronous_full, row_factory,
check_same_thread, wal_lock_retries, initialize)` and `transaction(conn,
immediate=)`; cron/ledger.py is deleted and hosted_rooms_common's
open_sqlite/connect/transaction become 1-3 line forwarders. Migrated:
agent/verification_evidence, cron/{executions,incidents,notepad,
delivery_queue}, gateway/{delivery_ledger,hosted_room_policy_checkpoint,
hosted_rooms_common (-> hosted_rooms, hosted_room_driver)}, hermes_cli/
projects_db, tools/async_delegation, plugins/plugin_storage.
Behavior changes (each module keeps its effective PRAGMA set otherwise):
- hosted_room_policy_checkpoint: connection now closed after every use and
on init failure (was leaked per call), busy_timeout PRAGMA set explicitly.
- projects_db: gains busy_timeout=5000 (was the sqlite3 default 5 s connect
timeout with no PRAGMA); explicit and observable.
- delivery_ledger / async_delegation: busy_timeout PRAGMA now mirrors the
10 s connect timeout they already had.
- plugin_storage.plugin_db: WAL through apply_wal_with_fallback (DELETE on
network filesystems / WAL-reset-vulnerable builds instead of raw WAL);
busy_timeout=5000.
- cron/incidents._redact_error: redact_sensitive_text(force=True) — the
error text is persisted to disk.
- delivery_ledger's private duplicate-column guard and the unguarded
`ALTER TABLE ADD COLUMN` sites (shared_metrics, api_server_run_idempotency,
holographic store, kanban model_override) go through add_column_if_missing.
- hermes_state.py::_scrub_surrogates: dead byte-copy of
hermes_state_messages._scrub_surrogates (0 callers) deleted.
create_main used O_WRONLY|O_CREAT on the main file and closed the fd, which drops this process's POSIX locks whenever state.db already exists (the gateway's own async_delegation import path). O_EXCL restricts the descriptor to a brand-new inode; existing files take the chmod(2) path.
Refs #109786#109687
POSIX fcntl locks are owned per (process, inode): closing any descriptor
for state.db releases every lock the process holds on that inode,
including the locks of an already-open SQLite connection. The
owner-only hardening cycle opened the live database and its -wal/-shm
read-only, fchmod'ed, and closed, so any process that already held a
connection (gateway, desktop hermes serve, dashboard share one) dropped
its live locks on every SessionDB init. A sibling process then took the
shared-memory DMS exclusively at its own close, checkpointed, and
unlinked the sidecars while long-lived holders kept the deleted inodes
open, tripping the deleted-WAL generation guard.
chmod(2) on the path never opens the file, so it cannot disturb locks.
The descriptor path remains only for first-time main-db creation, where
no locks can exist yet.
The owner-only pre-create helper ran before sqlite3.connect() and turned
a directory-as-state.db misconfiguration into IsADirectoryError instead
of the sqlite OperationalError the open path (and its lock-patience
classifier) expects. A directory leaks no row data, so skip it and let
sqlite fail canonically. Also map the salvage carry-commit author email
for the attribution gate.
SessionDB._open_writer did db_path.parent.mkdir(parents=True), so a multiplexer
or Desktop backend still holding a moved-away profile's route re-scaffolded
profiles/<name> (then cache/, cron/, logs/, SOUL.md...) on its next turn. Route
the mkdir through mkdir_under_hermes_home, which refuses a deleted or missing
named profile home — the same guard config/logging/cron already use.
Since 0.21.0 reads go through mode=ro pooled connections. A read-only OPEN already
rides out the millisecond WAL transition window (checkpoint / WAL reset / frame flush
by a sibling process; the ro reader cannot rewrite the -shm index) with a bounded retry
(#100436), but a WARM pooled reader hitting the same window while its SELECT executes
propagated `disk I/O error` straight out of get_session(): 37 identical tracebacks on a
multi-process WSL2 ext4-on-vhdx install, each followed by "compression session recovery
failed", with quick_check=ok (#100871). The reporter's A/B shows the operator
workaround (journal_mode=delete) collapses read throughput ~30000x, so the flake has to
be absorbed on the read path.
_read_one/_read_all now replay the idempotent statement within the existing read-only
IOERR budget (3 x 50 ms) on the SAME connection -- close+reopen would cancel this
process's POSIX locks for every sibling connection -- and a persistent IOERR still
propagates. No quarantine: EIO on a read is busy, not broken. Every SELECT in the
SessionDB siblings (63 call sites) reaches the pool through these two helpers, so the
class is covered without a wrapper type.
Same-connection retry per #100882's analysis (@fangliquanflq); #100883
(@Sahilvishnaliya) diagnosed the missing recovery in the 0.21.0 read pool.
Fixes#100871.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: Sahilvishnaliya <222165401+Sahilvishnaliya@users.noreply.github.com>
apply_wal_with_fallback() reports "wal" in two indeterminate cases -- the vulnerable-
SQLite gate (_apply_delete_for_wal_reset_bug) and the non-vulnerable probe-unknown path
(a7f2a593d1) -- meaning "touched nothing, the connection inherits the header's mode".
SessionDB turned that assumption into `_wal_active=True`, which enables the mode=ro read
pool that skips `self._lock`. On a file that is really in rollback-journal mode those
readers race the writer with a 5s busy timeout and no retry: random SQLITE_BUSY read
failures for the instance's lifetime (#86515).
Confirm the header on the freshly opened connection before enabling the pool. When the
probe is still blocked, reads queue on the writer connection under the lock -- slower,
never wrong. Every other apply_wal_with_fallback caller ignores the return value, so the
consumer is the right place to gate; changing the return contract to Optional across
15 call sites (#87044's shape) is not needed.
Live repro: DELETE-mode file, sibling holding BEGIN EXCLUSIVE during open ->
before: _wal_active=True and _checkout_read_conn() hands out a pooled mode=ro conn;
after: _wal_active=False, reads take the locked writer path.
Fixes#86515. Based on the analysis in #87044.
Co-authored-by: QDung210 <dqdung205@gmail.com>
A lock-free _raise_if_db_replaced() probe at the top of the _execute_write
retry loop raced a concurrent close(). close() runs under the same _lock and
ends the WAL generation: it checkpoints, closes the connection (SQLite
unlinks the -wal/-shm sidecars), nulls _conn and clears
_db_sidecar_identity. The probe could observe the mid-teardown state —
sidecars already unlinked while _db_sidecar_identity was not yet cleared —
and misclassify this process's OWN clean close as an externally deleted WAL
generation, raising a sticky DeletedWalGenerationError that permanently
refused every later write on that handle (#105567).
Move the live probe inside the lock, ahead of the close-race reopen
decision, so it only ever observes the stable post-close state (identity
cleared -> the existing adopt/reopen path). The corrupt flag check stays on
the lock-free fast path; external file/generation replacement detection is
unchanged, just serialized with teardown.
Synthetic repro (100 rounds x 40 writes, direct SessionDB handles): before
~9 failing rounds / ~360 DeletedWalGenerationError; after 0 failures,
4000/4000 writes persisted across repeated runs. tests/state (181) plus the
generation/replaced/corrupt guard suites (55) pass.
Fixes#105567
st_nlink == 0 alone cannot distinguish a genuine orphan from one that
still has a surviving hard link (e.g. a backup) after the watched
sidecar path itself was removed or replaced — that left st_nlink >= 1
on a truly orphaned generation, letting a new opener through while a
live writer still owned the old one. Compare (st_dev, st_ino) between
the fd and the current watched sidecar path instead: only an exact
match means they're the same live file, so any mismatch or unstattable
watched path still fails closed.
iter_deleted_sqlite_sidecar_holders() and SessionDB._wal_generation_was_lost()
both treated a `` (deleted)`` suffix on a /proc/<pid>/fd/* target as proof that
state.db-wal or state.db-shm was unlinked. On OpenZFS that suffix is not proof:
a live, still-linked file whose dentry was unhashed is reported the same way,
with st_nlink still 1 and the same (dev, ino) as the path. The guard then fires
permanently and the gateway falls back to JSONL forever, because the WAL was
never actually deleted.
Add _fd_is_truly_unlinked(), which confirms via os.stat(fd_path).st_nlink == 0
before a target counts as an orphaned generation. An unstattable descriptor
still counts as deleted, so the guard keeps failing closed. _iter_proc_fd_targets()
and _proc_fd_targets() now also yield the /proc fd path itself so both call
sites (open-path and the sticky write-path probe) can run the check.
Every read-write SessionDB open issued three writes that usually change
nothing, and a write statement takes the database write lock even when it
matches no rows:
- _ensure_db_file_generation's INSERT OR IGNORE into state_meta. The stamp
is minted once per FILE, so every open after the first inserted nothing.
- the NULL-`active` heal, UPDATE messages SET active = 1 WHERE active IS
NULL, which matches nothing on a healthy database.
- the fts_storage_version stamp, which re-wrote the same value on every
open of an already-optimized database.
The connection is opened with timeout=1.0, so each blocked write costs a
full busy timeout while a sibling process holds the write lock, and the
open path's patience loop can ultimately give up and raise.
Gate all three on a read. Measured on a real 99 MB state.db (239
sessions, 6470 messages) with a sibling holding the write lock: 2117-2136
ms -> 3.6-7.1 ms. On an already-optimized database the unpatched open does
not merely stall, it raises `database is locked`; patched it completes in
3.7-6.8 ms. With a sibling running 200 ms write transactions in a loop
(n=20 opens): p50 894.6 -> 9.6 ms, p90 1094.7 -> 13.9 ms. A settled
database now issues zero main-database write statements to open.
The reads cost nothing measurable: the state_meta probe is a primary-key
seek (2.0 us), the messages probe is 1.7 us on the modern NOT NULL column
(unsatisfiable constraint, short-circuited) and 0.4-0.6 us at 300k rows on
a legacy default-less column via the partial index that already exists for
exactly this predicate. An uncontended open is unchanged.
Semantics are preserved. INSERT OR IGNORE still resolves the first-opener
race inside SQLite and racers still converge on the winner's token via the
re-read; the application_id gate and the PASSIVE-only checkpoint are
untouched; the heal is still considered on every startup, as #60108
deliberately made it, with only the write now conditional on a read
proving there is something to repair.
Read-first also fixes a correctness bug. Under contention the generation
block was abandoned by its `except sqlite3.Error` handler, so a process
ended up with no generation token at all even though the value was already
on disk and a plain read would have returned it -- and that token feeds
the deleted-WAL and replaced-file guards added by #101221. The heal's
`except OperationalError: pass` likewise skipped the repair silently, so
the unconditional form did not even deliver the unconditional repair it
advertised whenever it mattered most.
The probe deliberately does not use INDEXED BY: that hint raises
OperationalError("no query solution") against the modern NOT NULL column,
and the existing handler would swallow it, disabling the repair forever.
- capture_retired_wal_generation: drop the main_image_max_bytes kwarg (no caller passes it; read the module constant).
- replace _write_json_durably with utils.atomic_json_write (late import keeps the capture dependency-light).
- clean the abandoned .partial staging dir on capture failure instead of leaving it for the retry to work around.
- _disable_close_time_checkpoint: document that it is the per-instance twin of _close_time_checkpoint_configurable and must agree with __init__'s capability decision.
- _close_quietly kept on the late import (module-level import from hermes_state_dbfile risks future cycles).
- hermes backup excluded *.db-wal by suffix but not the retired-wal capture dirs, so it would ship the capture's main-image copy while dropping the captured -wal that is the artifact's point; exclude <name>.retired-wal-* dirs whole (they must move as manifest+image+wal unit).
- _copy_range treated a short read as success, yielding a truncated copy with a valid manifest while the unlinked inode still dies at exit; raise RetiredGenerationCaptureError and clean the .part file.
- _quarantine_reason docstring no longer claims close() checks replaced before generation loss (close evaluates loss first and skips quarantine when lost).
_pin_connection took the shared connection via self._conn, so the lexical
lock audit flagged it (then its caller) as unlocked use. Pass the conn in
from the lock-held close paths and whitelist _settle_lost_generation_locked
as a close()-only lifecycle helper.,
Where SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE exists no retention capability is bound, so a
setconfig() that raises means close() will run SQLite's close-time checkpoint over the
newer generation. That fallthrough was a DEBUG line; an operator has to see it.
Every production close of a SessionDB goes through hermes_state_registry.release_or_close,
whose teardown swallows any exception from close() at DEBUG. With the capture in place that
meant a RetiredGenerationCaptureError (disk full, permissions) left the handle open silently
and, on Python 3.11, skipped the retention pin: the raise happened before the pin, so the
interpreter's exit still ran sqlite3_close's implicit checkpoint and wrote the stale frames
over the newer generation (probe: 302 rows -> 4 after exit, worse than main).
- close() now decides retention first and takes the pin BEFORE attempting the capture on
runtimes without setconfig; the capture failure is logged at ERROR and re-raised, so a
later close() retries it while the pin already protects the newer generation.
- The registry logs RetiredGenerationCaptureError at ERROR instead of DEBUG (other teardown
errors stay quiet). Same treatment on the release_or_close fallback path.
- The lost-generation settlement moves out of close() into _settle_lost_generation_locked();
the sticky _close_checkpoint_disabled attribute and the dead "setconfig exists but failed"
late-bind block are gone (_disable_close_time_checkpoint returns the call's outcome).
Regression test drives release_or_close with a failing capture: no raise, error text in the
log, pin taken exactly once (3.11), retry closes cleanly. Red on the salvaged head on both
3.11 and 3.12, green here.
With the retired generation captured durably, closing a lost-generation
handle is safe wherever SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE took effect: the
frames are preserved and sqlite3_close no longer checkpoints them into the
newer generation. On Python 3.11, where sqlite3 has no setconfig, closing
still runs SQLite's internal checkpoint over the newer main file, so only
there the exact quarantined connection is retained instead of closed: one
public Py_IncRef reference via ctypes.pythonapi (no struct-layout access),
bound before the writer opens, taken in close() after the capture. Runtimes
with setconfig never touch ctypes; a writable SessionDB requires CPython
with ctypes only where retention is the sole guard.
Read-only handles and every other quarantine reason close as before.
Regressions cover both branches: where retention applies, the retired inode
stays readable after close() and GC and an independent deleted WAL under
the same pathname is untouched; elsewhere the capture is the surviving copy.
A separate process's newer generation survives close, GC and normal exit
in both cases. The mock-based close-time regression originally written for
Refs #105670
Co-authored-by: fangliquanflq <fangliquan@qq.com>
After DeletedWalGenerationError the retired frames exist only in an
unlinked -wal inode that this process keeps open. #106315 stops them from
being checkpointed under wrong page numbers, but the canonical remediation
("stop the gateway, dashboard and cron writers, then reopen") lets the
kernel drop the inode with them: committed transactions whose disposition
is unknown were destroyed by the prescribed recovery path itself.
Capture the exact retired generation next to the database at the first
halt, or at close() if that sees the loss first. The WAL is located by the
(st_dev, st_ino) recorded at open among this process's own descriptors,
never by pathname, so a sibling's deleted WAL or a newer sidecar minted at
the same path cannot be mistaken for it; it is read with pread and no
descriptor is closed, moved or truncated. The artifact holds the WAL, the
-shm when still ours, the main image (or its header past a size cap) and a
manifest with identities, digests and the sidecar generation found at the
path at capture time. Nothing is merged: whether the frames belong on top
of the file now at the path stays an operator decision. close() refuses to
settle without the capture: it raises and leaves the handle open.
Regressions: capture at halt and at close, refusal to settle on capture
failure, inode-not-pathname selection with a second deleted WAL under the
same name, refusal to guess without a recorded identity, and a subprocess
control where rows committed only in the retired WAL are recovered from
the capture alone after the writer process has exited.
Refs #105670
`is_zeroed_state_db` and `has_invalid_sqlite_header_preopen` shared the
is_file / stat / live-connection / read_header preamble; `_preopen_header`
owns it now (zeroed is the NUL subset of "no SQLite header"). The
quarantine message also names `hermes sessions recover --source <bak>` so
the preserved bytes are actionable, not just parked.
Startup only quarantined a 0-byte / all-NUL state.db. A file whose first page
was clobbered with record bytes (#102198) went straight to sqlite3.connect,
which raised "file is not a database" and deleted the -wal sidecar — the one
piece of evidence that could have been recovered.
`has_invalid_sqlite_header_preopen` generalises the zeroed probe (zeroed is a
subset of "no SQLite header"; same live-connection contract, never raises).
`quarantine_invalid_state_db` moves the file AND its -wal/-shm aside as
`state.db.<zeroed|notadb>-<ts>-<pid>.bak` before anything opens it; a fresh
DB is created as before.
Re-authored on current main (the quarantine helpers moved to
hermes_state_dbfile.py in d15c61b5dc); one regression test proves the
notadb case is quarantined with its sidecars and the new DB passes
integrity_check.
Refs #102198 (the write-after-SIGTERM that clobbers page 0 is not addressed
here; this preserves the evidence instead of destroying it).
retry_deferred_fts_recovery gated only on _db_corrupt ("mirrors _try_wal_checkpoint /
close") — after this PR it no longer mirrored them: on a replaced/lost-generation handle
the periodic housekeeping tick still ran FTS DDL/DML + commit, the same split-brain write
class as the #105670 checkpoint. One SessionDB._quarantine_reason() now decides for the
periodic checkpoint, close(), and the FTS retry, with the halt path's precedence
(replaced before generation loss) and the operator wording in one place.
Test: the periodic-checkpoint case folds into the close test (same setup), which now
also proves the FTS retry returns False without touching the file; the mutation with
main's schema sibling swapped in returns True (a rebuild ran).
The cherry-picked commit re-wrapped hermes_state.py wholesale (+527/-131 for a
fix of about twenty lines). Restore main's layout and re-apply only the fix:
disable the close-time checkpoint on both generation-loss halts, gate the
periodic checkpoint on the sticky generation flags, and name the quarantine
reason at close.
- close() and _try_wal_checkpoint() now skip when _db_replaced or _db_wal_generation_lost
(previously only _db_corrupt was checked) — prevents checkpointing stale-generation frames
into the main DB, which is the shutdown-time damage reported in #105670
- _halt_if_db_generation_changed() calls _disable_close_time_checkpoint() alongside the flag
set (3.12+: disables SQLite internal last-connection checkpoint too)
- Regression tests: halted handle must not run explicit PRAGMA checkpoint on close(),
halt must call setconfig(NO_CKPT_ON_CLOSE), periodic _try_wal_checkpoint() must skip
Every PLUGIN-COMPAT __getattr__ now calls hermes_cli.plugin_compat.warn_once(facade, name, target) before
resolving, emitting a HermesPluginCompatWarning (FutureWarning) once per process per name: old path, new
path, removal target. Importing a facade for its live API stays silent; only resolving a moved name warns.
COMPAT_MANIFEST.md documents the warning and how to silence it during migration.
Verified the runtime never routes through a pointer: every entry point (run_agent, cli, hermes_cli.main,
gateway.run, tui_gateway.server, web_server, model_tools + tool discovery, hermes_state, cron.scheduler,
browser_tool, mcp_tool, kanban, auth) imports clean and `hermes doctor` runs end to end with the warning
promoted to an error.
Also restores the check_compat_pointers CI step to .github/workflows/lint.yml, which a0be177aac dropped
when the compat layer was regenerated (the lint script itself was present; the workflow step was not).
hermes_cli/plugin_compat.py, tests/test_plugin_compat_warning.py and the two-line insert per facade are
part of the compat layer and go away with it.
The Sep 2026 decomposition (PR #102117) makes internal import paths a non-API: names now live in
the focused modules that define them. This commit is the ONLY thing keeping the old paths alive,
so external plugins have time to update. It is deliberately a single, unsquashed commit:
git revert <this sha>
removes every shim, stub and manifest at once on the announced date. Nothing in-tree may depend on
these pointers: scripts/check_compat_pointers.py (wired into lint.yml) fails CI if it does.
What it adds (see COMPAT_MANIFEST.md, compat_manifest.json):
- 332 facade modules get one delimited `PLUGIN-COMPAT` block appended at the end of the file
- 1,172 moved names resolved lazily via a module `__getattr__` (PEP 562) — never a top-level import,
so no import cycles; facades that already had `__getattr__` get a chained one
- 592 third-party/stdlib names the old modules used to expose, with their original import statements
- 266 public definitions that had been deleted as unused, restored byte-for-byte from the pre-decomposition
tree (+40 private helpers and 16 imports pulled in only because a restored definition needs them)
- 3 deleted modules recreated as re-export stubs (gateway/startup_watchdog, hermes_cli/observability/
relay_runtime, tools/environments/modal_utils)
- private names (`_x`) get no pointer: they were never API (3,792 skipped)
Verified: all 335 touched modules import under a fresh HERMES_HOME and every manifest name resolves;
the lint reports zero in-tree uses; ruff clean; targeted suites unchanged.
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
Keep structured tool_calls searchable through the standard FTS index while
removing their repetitive JSON from the trigram projection. Reuse the
existing optimize-storage rebuild path for deployed v1 layouts.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Replace the requires_wal-gated unit assertion (which never runs on WAL-reset-
vulnerable runtimes such as macOS 3.46) with an end-to-end test through
repair_state_db_schema that traces every _connect_repair_durable call against
the guard's enter/exit and fails on main's shape (a connect after guard-exit).
Runs on every platform. Docstring now names the actual hazard.
Review blocker on e62940d: every state-db guidance site printed
hermes sessions recover --source <db>
but cmd_sessions rejects that shape with exit 2 ("--output is required
unless --inspect-only is used") before any snapshot is taken — the user
follows the instruction during a corruption incident and gets nothing.
All five state-db sites now print the established two-stage operator
contract (the same shape `sessions repair` failure output and
docs/state-db-recovery.md already use):
hermes sessions recover --source <db> --inspect-only
hermes sessions recover --source <db> --output recovered-state.db
with the stop-the-gateway precondition stated for the gateway/turn
banners, and --inspect-only leading in the hermes_state refusal strings
(inspection before writing anything).
New TestEmittedCommandsSatisfyCliContract dispatches the exact emitted
flag shapes through the real cmd_sessions and asserts they pass the
contract gate (rc != 2) on a scratch DB, plus a premise test pinning
that the v1 no-flag shape is still rejected with rc 2 — so a guidance
string can never again pass a source-substring test while the command
it prints deterministically fails.
Noted for merge order: #101423 and #101168 also touch
hermes_cli/session_recovery.py. They are complementary recovery-integrity
work, not duplicates of this guidance/gate fix; whichever lands second
should rebase and rerun the lost_and_found + session-recovery suites.
(cherry picked from commit 34dc59a284509e76a0342c36d03a2a437aa8a3b9)
Refs #100368. The forensics thread established that a sqlite3 CLI with
the WAL-reset opener bug (fixed 3.51.3+ / backports 3.50.7 / 3.44.6;
Debian/Ubuntu system shells 3.45.1/3.46.1 are in the vulnerable band)
unlinks the live -wal/-shm pair when pointed at a live state.db whose
writer's DMS lock has been cancelled, splitting the store into two
concurrent generations whose acknowledged writes vanish while both
report integrity_check ok. Hermes' own corruption banners instructed
exactly that command.
- gateway corruption broadcast, run_agent corrupt-cause explanation,
hermes_state repair-budget and forensic-backup refusals, and the
kanban manual-recovery hint now route operators to
`hermes sessions recover --source <db>` (which snapshots the damaged
bundle before any shell touches it) and warn against a raw sqlite3
shell on the live file
- find_sqlite3_cli() now refuses a WAL-reset-vulnerable shell for the
page-level salvage lane even on the snapshot, reusing the canonical
gate from hermes_cli.sqlite_runtime so the embedded runtime and the
salvage shell can never disagree
- find_sqlite3_cli_refusal() records why a shell was refused so the
lost_and_found lane can tell the operator exactly what to install
instead of a generic "not found"
- regression tests cover the version gate (vulnerable/fixed matrix, the
mirror check), every refusal reason, and each guidance site
Test plan:
- scripts/run_tests.sh tests/hermes_cli/test_sqlite3_cli_salvage_gate.py
tests/test_state_db_repair_loop_cap.py
tests/run_agent/test_corruption_recovery_guidance.py
tests/hermes_cli/test_session_recovery_lost_and_found.py
tests/hermes_cli/test_session_recovery.py tests/test_sqlite_wal_reset_gate.py
tests/hermes_cli/test_sqlite_runtime.py - 91 passed, 1 skipped locally
(cherry picked from commit e62940d1021e80e9b7d6423ced1cbdfe7dd0c37d)
