Importing hermes_state loaded ctypes and built the Restart Manager structures
on every POSIX process that never scans with Restart Manager; build them once
on first scan instead. Bind argtypes by function name rather than zipping dict
order against a positional tuple.
`_windows_restart_manager_holders` rebuilt two ctypes Structure classes and
called `ctypes.POINTER()` on the fresh class every scan. POINTER memoises in
the process-global `ctypes._pointer_type_cache`, so each call pinned another
class set forever: probe measured 3 -> 221 cache entries over 200 calls. The
scan runs from auto-VACUUM admission and the FTS-rebuild guard in a long-lived
gateway, so the growth was monotonic. Structures, constants and argtypes now
live at module level (ctypes.wintypes imports on every platform); the
`ctypes.WinDLL("rstrtmgr")` load stays per call so the unit test can still
inject a fake rstrtmgr. Same probe after: 22 -> 22.
While here: one `_sqlite_family(base)` helper replaces the fourth hand-rolled
(db, -wal, -shm) triple in this module and the three existing ones; the
Windows lane computes abspath once instead of per holder; the RmGetList
sizing call and the retry loop collapse into one bounded loop (pass 1 sizes
with a NULL buffer, later passes re-size), which keeps the existing fake's
`apps is None` contract.
In rollback-journal (DELETE) mode a sibling process can take the write lock
between schema load and the messages_fts probe. FTS5's xConnect then fails its
%_config read and SQLite reports SQLITE_BUSY with the text "vtable constructor
failed: messages_fts". Every state.db lock classifier matched on the words
"locked"/"busy", so:
- a writable SessionDB() failed after 1s instead of waiting out the lock with
_WRITE_PATIENCE_S, and callers disabled persistence for the run;
- a read-only open (dashboard, `hermes sessions list`, cross-profile readers)
failed on the first busy timeout with no retry at all;
- the error read as not transient (dashboard 500, not 503) and as persistence
cause "unknown" instead of "locked".
Add hermes_state_errors.is_sqlite_lock_error: SQLITE_BUSY/SQLITE_LOCKED by
result code when SQLite supplies one, text only when it does not (our own
re-raised messages, RPC-wrapped strings). Route the writer open patience loop,
the _execute_write retry, the reconcile re-raise, the WAL->DELETE flip, the
maintenance holder probe, is_transient_sqlite_error and
classify_persistence_error through it. The read-only open retries a lock
inside its existing bounded retry budget, next to the transient IOERR case.
Independent review found the previous commit could make the unbounded-growth
symptom it fixes PERMANENT, and that its argv narrowing re-opened #92401 inside
a single install.
- other_generations_for_path() counts only LIVE generations. A retired one is
already write-fenced (StateDbReplacedError, close-time checkpoint disabled)
and leaves the registry only when its holder releases -- which a gateway
handle does not do before shutdown. One inode replacement (repair swap,
backup restore, snapshot) therefore skipped auto-VACUUM for that path for the
whole process lifetime.
- _argv_scoped_to_other_home is ranked evidence now. argv[0] is the SHARED
install binary for every profile on a host, so it is neutral, never proof of a
hold; the process's own --hermes-home / HERMES_HOME= / --profile / -p
selection decides which home it serves, and a token naming ANOTHER profile's
store is tested before any own-prefix check. argv[0] also stops dismissing a
holder of a store whose home is not part of an install layout (a custom
HERMES_HOME is served BY the binary under ~/.hermes).
- install_root comes from hermes_constants.named_profile_home, not
basename(parent) == "profiles": an arbitrary <X>/profiles/<n>/ tree no longer
promotes all of <X> to "ours".
- The launch profile keeps its gateway.sessions_dir override when its own store
is pruned; every other served profile prunes under its own <home>/sessions.
Pruning under the wrong dir orphaned transcripts forever.
- glob.escape on the request_dump_<id>_* sweep (pre-existing).
Tests: the retired-generation test asserted the starvation mechanism; it is
replaced by the invariant (a live sibling defers VACUUM) plus a red-on-base test
that a retired, write-fenced generation does not. New argv cases cover the
shared binary with -p other, another profile's store token, and a non-Hermes
<X>/profiles/ tree; the housekeeping fixture now asserts the unpinned store
still resolves inside the sandbox before yielding.
Three silent data-correctness bugs on a host where ONE process serves every
profile:
- The gateway constructor ran auto-archive and auto-prune/VACUUM once, on a
handle pinned to the construction-time launch home, so a served secondary
profile's state.db was never pruned or vacuumed by anybody. Both now run per
SERVED profile, inside that profile's runtime scope, against its own
SessionDB, its own `sessions:` config and its own transcript dir.
- The dashboard/`hermes serve` auto-archive sweep archived an arbitrary
profile's store but read the config through the PROCESS HERMES_HOME, so one
profile's sessions.auto_archive_days governed every other profile's
retention. It now loads the config of the home whose store it sweeps.
- Auto-VACUUM admission read "no FOREIGN holder" as "the store is quiet", but
the /proc scan skips our own pid by design. VACUUM plus its TRUNCATE
checkpoint therefore retired a WAL generation another live SessionDB in THIS
process still held. The path-keyed registry now answers the in-process half.
Also: argv naming only the install root no longer dismisses a multiplexer as
"another instance" when the store is a profile store beneath that root — under
one-process-per-host that argv is exactly what the holder looks like.
`hermes sessions optimize`, `optimize-storage` and `prune` now run the same fail-closed
holder scan doctor and repair use before rewriting the store. While a gateway, Desktop,
dashboard or cron process holds state.db (or a WAL sidecar) they print each holder as
`PID N (command)` with the stop remedy and exit 1; `--force` overrides with a warning,
`--dry-run` previews are never gated. The Desktop console's `sessions optimize` gets the
same refusal. Why: a user ran `optimize-storage` under a fleet of eight live gateways and
every agent answered every turn with the retired-WAL refusal until all writers were
stopped by hand (#110054, maintainer follow-up 09-20).
The DeletedWalGenerationError text is now two layers: a first sentence for the person
reading a chat bubble or banner (what happened, nothing is lost, quit every Hermes
process on the profile, `hermes doctor` names the holders, never `doctor --fix` or delete
files while they run, docs link), then the operator detail. The classifier fingerprint
"deleted state.db-wal or state.db-shm" is unchanged. The cause table
(`hermes_state_user_copy`, feeding the CLI banner, TUI/Desktop RPC error and the gateway
home-channel notice) and the chat explainer carry the same first steps; the gateway
notice no longer hardcodes `doctor --fix` + `gateway restart` for every non-corrupt cause,
which for a held retired generation is the second-writer trap.
New user-guide page `session-storage-recovery.md` (registered in sidebars, linked from the
guard text, the developer state-db-recovery page and the sessions guide): the three steps,
the do-nots, why maintenance refuses, and what the files beside state.db are
(retired-wal captures + manifest.json, pre-update-emergency backups, corrupt backups,
snapshots).
doctor_platform already rendered "PID N (cmd)" inline for the journal-mode holder report;
the retired-WAL warning needs the same text, so the loop moves to
hermes_state_holders.describe_holder_pid and both sites call it. The issue line now names
the same writers the DeletedWalGenerationError message names (gateway, dashboard, cron)
and repeats "do not delete the WAL yourself", so doctor and the guard tell one story.
The test also spies _state_db_stats, the probe that printed the literal
"0 process(es) holding the DB open" line the issue is about.
hermes_state_common pulls in agent.* at import, so the URI builder moves to
hermes_state_holders (errno/os/sqlite3/pathlib only) where the gateway
readiness probe and backup can adopt it in a follow-up sweep. The doctor
structural-damage branch is one helper instead of two copies, the holder
scan goes through hermes_state_repair._live_writer_holds_db, the migration
hint uses _schema_not_built (the startswith("no such ") check also matched
"no such module: fts5"), and the hermes_state import is hoisted so an import
failure cannot mask itself as UnboundLocalError.
Review findings on #110914 (@ehz0ah): the psutil leg compared watched
abspath against the kernel-resolved path psutil reports, so a symlinked
HERMES_HOME on macOS returned no holders and let the fallback probe mint
replacement sidecars under a live writer. Both sides now realpath.
doctor's `file:{path}?mode=ro` truncated at '?'/'#' in a home name;
build the URI with as_uri().
`hermes doctor --fix`'s WAL checkpoint and `repair_state_db_schema`'s
preflight documented themselves as fail-OPEN: `live_writer_holds_db` only
refused on unknown/deleted/uninspectable holders and then trusted a
`BEGIN IMMEDIATE` probe, which is blind to a `journal_mode=DELETE` reader
(SHARED only) and cannot run on a malformed file — exactly the states repair
and checkpoint get invoked in. A repair in a second process then REINDEXed /
VACUUMed a file the gateway still held (#103339 item 2).
- `hermes_state_holders.live_writer_holds_db`: any foreign holder of the DB or
a sidecar is a live holder; the probe is only an additional positive signal.
- doctor `--fix`: the checkpoint runs on `_exclusive_repair_db_guard`'s
connection instead of a bare writable `sqlite3.connect`, so an opener
arriving after the scan is refused, not joined; `_session_count` is a
`mode=ro` reader.
- Normal SessionDB writers are untouched: gateway + dashboard in two processes
both keep writing (a process-wide flock on the write path — PR #109270's
shape — would break that).
Tests: the two-process repair race test releases the test process's own
header-probe fd (it is a genuine holder now); the mid-repair writer fixture
opens its connection after staging starts (a pre-existing holder is refused up
front, which is the point).
Refs #103339#100896
Field evidence (2026-09-07, production host): the host runs two independent
Hermes instances - a main gateway (user ubuntu, HERMES_HOME=/home/ubuntu/.hermes)
and a demo gateway (user demo, HERMES_HOME=/home/demo/.hermes). Because the
demo process is owned by another user, its /proc/<pid>/fd table is unreadable
and foreign_state_db_holders() falls back to cmdline + _looks_like_hermes().
The demo argv matches Hermes patterns exactly, so it was flagged as an
uninspectable holder of the MAIN instance's state.db even though lsof proved
0 open handles on it. Result: _recover_stale_fts was deferred 42 times across
6 gateway restarts, the fts_stale breadcrumb never cleared, and FTS
self-repair stayed permanently disabled.
#92419 removed substring false positives (journalctl/grep mentioning hermes);
a genuine second instance with a DIFFERENT HERMES_HOME was still misjudged.
Add _argv_scoped_to_other_home(): when the argv of an uninspectable Hermes
process proves it lives under a different /.hermes home (or a state.db
sidecar under a different parent) AND no token references our state.db,
sidecars, or home, do not count it as our holder. Applied to all three
uninspectable branches (holder + two descriptor paths). Ambiguous argv
without absolute-path tokens remains fail-closed, preserving the
conservative intent.
References #92401
Follow-up on the #97330 salvage (#97329): main grew `_canonical_sqlite_path`
after the PR's base; the holder module re-implemented it as a closure. Export
`canonical_sqlite_path` from hermes_state_holders and alias the legacy name in
hermes_state. Drop the unused `_read_proc_argv` / `_looks_like_hermes`
re-exports (no in-tree callers).
Move live-holder inspection into a bounded helper and make repair fail closed when a process still owns the state database, including deleted WAL/SHM descriptors and ambiguous procfs reads.
Preserve the contributor lineage from the original four-commit review train while presenting one coherent release object on current main.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: ciabata <296402666+ciabata-git@users.noreply.github.com>
