Gateway /resume rewrites sessions.source to the resuming platform, which is
correct routing behavior (source feeds find_latest_gateway_session_for_peer,
Telegram listings, and the TUI ghost pruner) but destroys creation
provenance. Add a separate immutable created_source column stamped once at
session creation and preserved by all upsert paths; surface it in
hermes sessions list as '<created>→<current>' when the two diverge.
Fixes#56439
whole_lineages only adds "AND s.id NOT IN (<continued ancestors>)", and
every id in that set is a parent with end_reason='compression'. Every
writer sets ended_at and end_reason together (and every reopen clears
both), so an open row is never in it: the flag could not change
count_open_prune_matches, it only added a recursive CTE and doubled the
bound params on each prune preview.
Remove the parameter and both call sites (CLI prune, dashboard prune),
and move the dashboard "preview lists what it deletes" comment to the
list_prune_candidates call it describes. Prune behaviour is unchanged.
_note_pinned_skipped diffed count_prune_matches(include_pinned=True/False)
with whole_lineages on for prune. Excluding a pinned tip turns it into an
unmatched continuation, so its unpinned compression ancestors drop out of
the "without" count too and the note reported them as pinned (chain
a -> a-2 -> a-3 with only a-3 pinned said "3 pinned sessions").
Count plain filter matches for the note; the delete itself is unchanged.
count_prune_matches has no other caller that needs whole_lineages, so the
keyword goes back to base's signature.
_prune_where popped whole_lineages out of **filters and prune_sessions
injected it by mutating that dict, so a filter-shaped kwarg silently
changed the SQL shape and a typo'd flag would be ignored rather than
rejected. It is now a keyword-only argument of _prune_where and the
list/count prune helpers; prune_sessions passes whole_lineages=True and
the CLI and dashboard previews pass it explicitly. Behaviour unchanged.
Retention prune (the default-on startup auto-prune, `hermes sessions prune`
and the dashboard prune) aged every session row on its own. A conversation
that rotating compression split into segments has an ended, old root by
construction, so once that root passed retention_days it was deleted while
the conversation's live tip was still being written: the pre-compression
turns vanished from the resume/Desktop history and from session_search, and
the tip was orphaned.
Prune now deletes a compression ancestor only together with every
continuation after it (`whole_lineages`), so a lineage ages through its
newest segment and goes as a unit once the whole conversation qualifies.
Branch, delegate, reset and tool children do not count as continuations.
The CLI and dashboard prune previews pass the same flag, so they list what
prune deletes; bulk export keeps its current selection.
(cherry picked from commit b082fc6ffa02607cba219a5dfa361d4f3bf4c266)
An explicit SESSION_ID whose stored prompt is already NULL used to print
"No degraded stored system prompts found.", which reads like a scan
result. Say the row already has no stored prompt and there is nothing to
clear. The SESSION_ID override assertions (memory-only clear, empty-row
message, unknown id -> exit 1) move into the memory-only scan test so the
stack keeps two invariant tests.
The helper now tests a single thing - whether a non-empty stored prompt
lacks the Skill Safety heading - so _repair_prompts_missing_skills_markers
(plural, "skills markers") misdescribed it. Rename to
_repair_prompts_lacks_skill_safety.
A memory-only tools[] pin self-heals: restore_agent_tool_prefix appends every
fresh tool to the pin and persists it (merged != pinned) on the next turn. The
clear_pin flag was also unreachable in scan mode (findings need skill_manage).
So drop clear_pin/_HYGIENE_PIN_TOOLS and clear_system_prompt_for_rebuild, and
reuse update_system_prompt(sid, None), which already nulls prompt+hash and GCs
in one write.
The detector now parses each pin and checks the marker once per row, keyed on
prompt_builder.SKILL_SAFETY_HEADING instead of a hand-copied literal. The scan
classifies each compact_rows page as it arrives, keeps only finding dicts, and
dedupes ids that OFFSET paging can re-serve during concurrent inserts.
The repair-prompts detector cleared two legitimate prompts:
- a pin with skills_list/skill_view but no skill_manage and zero skills
installed: build_skills_system_prompt returns '' and SKILLS_GUIDANCE is
only emitted with skill_manage, so the healthy prompt has neither marker;
- an exact memory-only pin, which is also a user toolsets=[memory] config;
the healthy rebuild was re-flagged on every run (not idempotent).
Key the decision on the missing '## Skill Safety' guidance, which is
unconditional when skill_manage is in the pin, and require skill_manage in
the pin. Memory-only rows are now reported as unverifiable; the explicit
SESSION_ID override still clears them and their pin.
Docs: describe the tightened rule and note that a running gateway keeps
cached prompts in memory, so it must be restarted after --apply.
The "display index not backfilled" probe was spelled twice in
hermes_state_messages.py, and the copy in the delete fence tested only
display_order while _ensure_display_order tests display_order OR
display_identity. Hoist one _DISPLAY_INDEX_MISSING_SQL beside
_DISPLAY_ACTIVE_CLAUSE and use it at both sites. The fence now refuses
whenever the read path would have backfilled instead of projecting; on
every reachable delete the two probes agree (the read path backfills both
halves before the export snapshot exists), so this only tightens fail-closed.
hermes_state_timeline keeps its own probe: it carries a role slot.
_cmd_export re-derived "which formats are human-read transcripts" as
`format == "html" or only`; use SAVE_TRANSCRIPT_FORMATS instead. Equivalent
on every reachable path: md/qmd without --only never reach _collect_sessions
(they route to _export_markdown), and --only forces the transcript view.
Bind include_compacted once in a local `_one` instead of threading it.
The IOERR test now injects on "WITH page AS", the display CTE's own opener,
so reshaping the SQL cannot let _ensure_display_order's SELECT probe absorb
the failure and make the case pass vacuously. Comments that narrated
removed guards now state the current reason.
--delete-after-verified had three guards after the merge: #119935's pre-delete
message re-count, #120065's caller-side `previous != snapshot` compare between
exported items, and #120065's expected_display_messages compare inside
delete_session's BEGIN IMMEDIATE. Only the last one is race-free: it compares
the exact display rows against the store at DELETE time, so a same-count
rewrite, an append after the re-count, or inter-item drift is refused there.
The two caller-side checks run outside the transaction and can only ever
duplicate a verdict the fence already gives, so drop them and merge the
snapshots straight into expected_messages.
Verified with the S1 probe (--race): compacted_fires false, same_count_fires
false with the caller-side guards removed.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
Co-authored-by: John Paul Soliva <soliva.johnpaul@icloud.com>
In-place compaction is the default. It soft-archives every earlier row of
a session under the same id (active = 0, compacted = 1), and Desktop and
the dashboard still show those turns. The md/qmd export read the session
through export_session -> get_messages with the default live-only clause,
so it wrote only the compaction summary and the carried tail.
verify_export_file then compared the file with that same dict, and
delete_session removed every row of the session, including the archived
turns that never reached the file.
The md/qmd export now reads the display history (include_compacted), for
a single session and for --lineage logical. export_session and
export_session_lineage take include_compacted, off by default:
import_sessions inserts every message as live context, so the JSON export
and stranded-session adoption keep reading live rows only.
The other transcripts people read had the same hole without the delete:
`/save md` and `/save html` (CLI and gateway), `sessions export --format
html` (one session or all of them) and `--only user-prompts` each held 1
of 6 answers on a six-turn session after one compaction. They now read
the display history too (SAVE_TRANSCRIPT_FORMATS; export_all gains
include_compacted and reads per session then, since the display read
dedupes per session). `/save json`, JSONL and the dashboard's JSON export
stay live-only for the import reason above.
Before deleting, the verify step also re-counts the store's display rows
for every session the file covers and refuses on a mismatch. A message
that lands while the files are written, or a later export change that
reads a narrower view, now refuses the delete instead of being removed
unseen. Like the adoption retire loop, the re-count runs just before
delete_session, not inside its transaction. Rewind rows (undone turns,
the superseded originals of a carried tail) are still deleted without
being exported, as `hermes sessions delete` does: they are not part of
the history the session shows.
Measured through the real CLI on a session with 6 turns and one default
in-place compaction (15 rows, 13 shown): before, 3 messages were exported
and all 15 rows deleted, with answers 1-5 missing from the file; after,
13 messages are exported in display order, then deleted.
(cherry picked from commit adeaff1e33f1ae2b8a066ef2374bb29ade50b3b8)
The admission gate read `db.db_path`, which made the refusal depend on whatever object
`SessionDB` resolved to; the CLI tests substitute a lightweight double and CI went red with
AttributeError: 'FakeDB' object has no attribute 'db_path'. The path now comes from
`_default_db_path()` — the exact resolver `SessionDB()` itself uses two lines above — so the
scan targets the same file in production and stays reachable regardless of the db object.
`hermes sessions optimize`, `optimize-storage` and `prune` now run the same fail-closed
holder scan doctor and repair use before rewriting the store. While a gateway, Desktop,
dashboard or cron process holds state.db (or a WAL sidecar) they print each holder as
`PID N (command)` with the stop remedy and exit 1; `--force` overrides with a warning,
`--dry-run` previews are never gated. The Desktop console's `sessions optimize` gets the
same refusal. Why: a user ran `optimize-storage` under a fleet of eight live gateways and
every agent answered every turn with the retired-WAL refusal until all writers were
stopped by hand (#110054, maintainer follow-up 09-20).
The DeletedWalGenerationError text is now two layers: a first sentence for the person
reading a chat bubble or banner (what happened, nothing is lost, quit every Hermes
process on the profile, `hermes doctor` names the holders, never `doctor --fix` or delete
files while they run, docs link), then the operator detail. The classifier fingerprint
"deleted state.db-wal or state.db-shm" is unchanged. The cause table
(`hermes_state_user_copy`, feeding the CLI banner, TUI/Desktop RPC error and the gateway
home-channel notice) and the chat explainer carry the same first steps; the gateway
notice no longer hardcodes `doctor --fix` + `gateway restart` for every non-corrupt cause,
which for a held retired generation is the second-writer trap.
New user-guide page `session-storage-recovery.md` (registered in sidebars, linked from the
guard text, the developer state-db-recovery page and the sessions guide): the three steps,
the do-nots, why maintenance refuses, and what the files beside state.db are
(retired-wal captures + manifest.json, pre-update-emergency backups, corrupt backups,
snapshots).
`database.journal_mode: delete` can never self-apply to a store that is already WAL:
apply_wal_with_fallback deliberately never live-downgrades (#68545 — other gateway/cron/worker
connections may hold uncheckpointed WAL commits), so operators applying the containment for the
multi-writer corruption class saw one ERROR per process forever and the only escape hatch was an
undocumented hand-run PRAGMA on the file.
The new pre-DB `sessions set-journal-mode` verb is the sanctioned offline path: it refuses while ANY
foreign process holds the file or a sidecar (the same foreign_state_db_holders scan doctor/repair
admission uses, naming each PID), flips through _set_journal_mode_no_wait (busy_timeout=0, so an
opener appearing mid-way makes SQLite refuse instead of racing it), verifies header bytes 18/19,
and reminds the operator when config.yaml disagrees. `--db PATH` covers kanban.db / cron stores
that log the same ERROR. The never-live-downgrade invariant is untouched; the ERROR, doctor hints
and docs now name the command instead of the raw PRAGMA.
`hermes sessions archive --older-than N` (SessionDB.archive_sessions) selected
every ENDED row matching the filters, which includes the compression ancestors
of a long conversation: they are ended (`end_reason='compression'`) and old by
construction. `set_session_archived` then flipped `archived` across the whole
lineage — including the OPEN, actively written, lease-holding live tip — and
the default `archived=exclude` listing (Desktop sidebar, `hermes sessions
list`, /resume) lost the chat while messages kept flowing.
A lineage is now matched through its tip only: `_prune_filter_where` gains
`lineage_tips_only`, which `archive_sessions` always sets and the CLI sets in
archive mode so the dry-run preview and the confirmation count show exactly
the rows the archive will touch. An idle, ended tip still archives its whole
chain (the lineage stays one unit — the listed row is the root, projected to
the tip, so sparing only the tip would leave the chat hidden anyway).
Prune is unchanged. Docs: the bulk-archive section says how compacted
conversations are matched.
Direction from #115500 by @whyyagswhy (automatic archives must not hide the
open live tip); the mechanism differs because the listing keys on the root.
Co-authored-by: whyyagswhy <166958865+whyyagswhy@users.noreply.github.com>
The per-profile store model (#88734), the parent-inheritance fence (#88381),
profile-stamped topic rows (#76423) and profile-prefixed voice keys (#75198)
are all forward-only: they put NEW state under the right profile and refuse to
widen existing damage, but nothing walks the stores and settles what earlier
releases left crossed. #113884 found 246 sessions stranded that way and could
only warn.
`hermes sessions repair-profiles` scans every profile's state.db plus the
gateway's voice-mode and sessions.json files and names six kinds of crossing:
1. `profile_name` disagreeing with the row's own session key -> relabel;
2. rows physically in another profile's store -> move (all message
generations, usage rows, system prompt) to the owning store, parents before
children so lineage survives, copy-then-delete so a crash leaves a duplicate
the next run settles;
3. `parent_session_id` crossing namespaces -> sever (own identity kept);
4. routing rows outside the default store under multiplexing -> move (an
existing row wins); routing rows for a profile that no longer exists -> drop;
5. Telegram topic bindings and voice-mode entries missing their bot's profile
-> relabel from the sessions that hold the chat (ambiguous chats reported);
6. sessions.json mirror entries for an unclaimed namespace -> drop (the legacy
import re-injects them into routing every boot).
Report-only by default. `--apply` refuses while a gateway owns any store, takes
a quick snapshot of every store first, and is idempotent. Two cases are
reported but never guessed: rows keyed to a profile that does not exist, and
`agent:main` rows inside a named profile's store (`--legacy-main rekey|move`
says which of the two histories they are).
Storage side lives in `hermes_state_profile_repair.py` (SessionDB mixin);
orchestration across stores in `hermes_cli/sessions_repair_profiles.py`; the
CLI face in `hermes_cli/sessions_cmd_repair_profiles.py` (pre-DB handler: it
opens every store itself).
Part of #88715 (PR-6). Closes the remediation gap #113884 only warns about.
`hermes sessions list` applied --limit inside the SQL query and rendered
whatever came back, so a user with 35 sessions saw 20 rows and a prompt
and had no way to tell the page was cut. The lister now asks for one row
past the cap, drops that probe row, and ends a truncated page with
"… more not shown (use --limit N to see more)". `--limit 0` is LIMIT 0
(no rows), so the copy never suggests it.
The footer goes through one shared helper, `cli_output.print_truncated`,
and the three "... N more" / "… N more" / "… and N more" variants already
in sessions_cmd.py (export dry-run preview, never-active cleanup, prune/
archive preview) now use it too, so every capped listing in the command
reads the same. Migrating checkpoints / curator / skills search / console
listers is follow-up work.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Squashed integration of the user-facing message audit for this surface set.
Full per-finding receipts: /tmp/ux-audit/lanes/*-receipt.md (campaign artifacts).
hermes_state_common pulls in agent.* at import, so the URI builder moves to
hermes_state_holders (errno/os/sqlite3/pathlib only) where the gateway
readiness probe and backup can adopt it in a follow-up sweep. The doctor
structural-damage branch is one helper instead of two copies, the holder
scan goes through hermes_state_repair._live_writer_holds_db, the migration
hint uses _schema_not_built (the startswith("no such ") check also matched
"no such module: fts5"), and the hermes_state import is hoisted so an import
failure cannot mask itself as UnboundLocalError.
read_only_db_uri() replaces four inline mode=ro URI sites (two of which
still used the raw f-string that truncates on ?/# in the home path:
state_db_has_structural_damage and collect_state_db_stats). The doctor
write probe now applies the live-holder gate in both modes: a quiet store
is probed in place as on main, a held store is probed through a read-only
snapshot, and a held store over 1 GB is skipped with an info line unless
--fix is given (the unconditional copy cost one full DB write per plain
doctor run). Connect/backup failures propagate to the existing
classification instead of being reported as FTS write-health failures.
Observational sessions commands print a migration hint instead of a raw
traceback when a read-only opener meets an older schema.
Co-authored-by: Ahmett101 <Ahmett101@users.noreply.github.com>
- Keep (a) list/stats/pinned open SessionDB(read_only=True) — one parametrized test — and (b) a missing store prints empty results and is never created.
- Drop the insights read-only test (already covered on main), the status test, the mutating-action/live-writer/doctor-isolation tests, and the two doctor factory tests.
- Replace _EmptyObservationalStore + error-string sniffing with a plain '_default_db_path() does not exist' branch printing each action's empty output; the fake-store tests in test_sessions_pin keep working because the branch only runs when the open fails.
## What does this PR do?
Makes observational CLI commands open `state.db` in read-only mode, so they can inspect a live Hermes installation without participating in writable WAL lifecycle handling.
### Symptom
Running `hermes status`, `hermes doctor` without `--fix`, `hermes sessions list`, `hermes sessions stats`, or `hermes insights` while a gateway owns the store could open another writable session handle. The live turn could then lose its WAL generation and stop.
### Impact
Users inspecting status or session history during an active turn could lose that in-flight turn and leave the gateway halted until recovery.
### Bug Cause
**Trigger:** observational CLI helpers constructed `SessionDB()` with its writable default.
**Causal chain:**
1. A live gateway holds the `state.db` WAL generation.
2. A nested observational CLI command opens a second writable handle.
3. Writable-handle close behavior can participate in WAL lifecycle work and retire the generation used by the live writer.
**Why it is wrong:** these commands only query state and should not have writer privileges.
**Working sibling / contrast:** repair and mutating session commands still use writable access intentionally.
**Ruled out:** no state schema, migration, or WAL checkpoint implementation changes are included.
### Fix
Routes status, non-fixing doctor state inspection, sessions list/stats, and both insights entrypoints through `SessionDB(read_only=True)`. Repair and mutating paths remain writable, and regression tests cover WAL preservation with a live writer.
## Related Issue
Fixes#110173
## Type of Change
- ✅ Bug fix (non-breaking change that fixes an issue)
## Changes Made
- `hermes_cli/status.py`, `hermes_cli/doctor_state.py`, and insights helpers — open observational state readers read-only.
- `hermes_cli/sessions_cmd.py` — make only `list` and `stats` read-only; retain writable access for mutations.
- `tests/hermes_cli/test_observational_sessiondb_modes.py` — verify access modes and a live writer's WAL remains usable.
## How to Test
- ✅ `scripts/run_tests.sh tests/hermes_cli/test_observational_sessiondb_modes.py tests/hermes_cli/test_cli_insights_command.py` — 9 passed.
- ✅ `scripts/run_tests.sh tests/hermes_cli/test_doctor.py tests/hermes_cli/test_doctor_structural_corruption.py tests/hermes_cli/test_sessions_error_exit_codes.py` — 75 passed; two sandbox-only failures came from blocked host process/symlink operations.
- ✅ A live `SessionDB` writer remains able to create and retrieve a session after `sessions stats` reads the store.
## Checklist
### Code
- ✅ I've read the Contributing Guide
- ✅ My commit messages follow Conventional Commits
- ✅ I searched for existing PRs to make sure this isn't a duplicate
- ✅ My PR contains only changes related to this fix
- ✅ I've run relevant tests locally (see How to Test)
- ✅ I've added tests for my changes
- ✅ I've tested on my platform: macOS
### Documentation & Housekeeping
- ✅ Documentation update: N/A
- ✅ `cli-config.yaml.example`: N/A
- ✅ `CONTRIBUTING.md` or `AGENTS.md`: N/A
- ✅ Cross-platform impact considered
- ✅ Tool descriptions/schemas: N/A
`hermes sessions export --session-id X <dir>/` crashed with IsADirectoryError
because jsonl/html/trace opened the positional as a file while --help called it
an "output path" and md/qmd really do take a directory. An existing directory
(or one spelled with a trailing separator) now receives a default-named file
(`hermes_session_<id>.<fmt>`), and the help text spells out per-format what
OUTPUT means.
`hermes sessions repair --check-only` printed the corruption reason and
exited 0, so scripts and the console wrapper gating on the status read a
broken state.db as healthy. Return 1 from the CLI handler (main.py already
sys.exits a truthy return) and from the console handler, where
`_capture_output` turns the status into a ConsoleCommandError carrying the
printed reason.
Salvaged from PR #103321 (the check-only reporting part only; the probe
rewrite and connection-tracking changes were not taken). Refs #63386.
SQLite dynamic typing lets a TEXT cell ('not-a-timestamp'), inf/nan or a
garbage double (8.4e252 salvaged from a damaged page) sit in a REAL
timestamp column. Every reader called datetime.fromtimestamp()/float
arithmetic on the raw cell, so ONE bad row raised TypeError/OverflowError
out of the row loop and took down the whole `hermes sessions list`/browse
table (#102399), all three exporters — JSONL/MD, QMD, HTML (#102352) —
and `hermes insights` (#99959).
Fix the class with ONE helper, hermes_cli.timefmt.coerce_epoch(): a
stored cell becomes float epoch seconds inside a sane 1970..2103 window
or None after a WARNING that names the session id. Every reader routes
through it — relative_time (list/browse/resume picker), format_epoch
(prune/candidates tables), the three exporters' timestamp formatters,
insights' _get_sessions/_day/period range — so a bad row renders as
'?'/'N/A'/raw text for that one cell and the command completes.
Write side: hermes_state_messages._coerce_timestamp (append_message,
append_messages_batch, import) and the import path's started_at now use
the same window, so a new out-of-range timestamp falls back to now()
instead of being persisted — new bad rows cannot be written by Hermes.
Reported-by: #102399, #102352, #99959 reporters; kokhlo's insights
analysis pointed at every reporting site, not just line 860.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
cmd_sessions used 'with db:' which breaks test doubles lacking the context
manager protocol (13 reds in test_sessions_pin/delete/export). Restore the
explicit try/finally db.close() (same semantics for real SessionDB). Add
get_session/count_prune_matches to the FakeDB doubles in test_sessions_delete
instead of re-adding getattr guards to production code.
