Gateway /resume rewrites sessions.source to the resuming platform, which is
correct routing behavior (source feeds find_latest_gateway_session_for_peer,
Telegram listings, and the TUI ghost pruner) but destroys creation
provenance. Add a separate immutable created_source column stamped once at
session creation and preserved by all upsert paths; surface it in
hermes sessions list as '<created>→<current>' when the two diverge.
Fixes#56439
Prune's _CONTINUATION_EDGE_SQL re-implemented SessionDB's
_NON_CONTINUATION_CHILD_FILTER_SQL clause by clause. If a new fork marker
lands in one copy only, prune and compression disagree about which
children continue a lineage and prune can again delete the start of a
live chat. Both constants are now built from
hermes_state_common._non_continuation_child_sql (same pattern as
_legacy_reset_child_sql); the class constant is byte-identical and the
prune edge differs only in whitespace.
Review P1 on #123186 (ehz0ah): the N=3 sustained-overload budget was
object-local. The gateway binds a fresh compressor to the same session on
every turn / cache eviction, and restart / API-server requests construct
one too, so each fresh instance restarted the budget at zero and a
sustained summary-provider outage walked the session back into
compression_exhausted and auto-reset — the exact wipe the PR exists to
prevent. Reproduced by the reviewer with three fresh compressors bound to
one session: every attempt ended at counter=1.
Persist the streak as sessions.compression_overload_streak through the
same durable channel the fallback streak and recovery deadline already
use (#100185): SessionCompressionMixin get/set pair over the sessions
row, loaded in the compressor's durable-load block, written on every
change (overload abort, successful summary, committed boundary, runtime
model switch).
- Compression rotation carries the streak to the child row at the
boundary, same as the fallback streak: the parent's value is read
before the bind, re-applied, and persisted onto the fresh child row.
- A completed compaction boundary settles the budget to 0 — including
the committed degraded fallback, so a recovered provider regains a
full budget instead of the session staying degraded forever.
- Schema event appended to SCHEMA_HISTORY["sessions"] (seq 28, after
compression_recovery_deadline) so salvage replay maps the new column.
- Fresh-instance regression: two aborts on one compressor, then a fresh
compressor bound to the same session inherits streak=2 and the third
session-wide attempt commits the fallback; plus a rotation carry-over
test. Both fail red against the memory-only implementation.
(cherry picked from commit 3742fee8d03d7077d23b0dac3a3c79fe48e102ac)
A session whose started_at is corrupt and has no in-window activity or
message timestamp still returned the raw cell as last_active, and the
order_by_last_active fallback sorted it above every healthy session.
Both fallbacks now go through the same window as the UNION ALL values;
a session with no trusted timestamp gets NULL.
The session tools pin (sessions.tool_names) stored names only, so every fresh
process re-materialized the bytes from its own surface and every surface hop
of one durable session was a full prompt-cache miss:
* tool_search's deferred catalog is built per process ("Search 6 additional
tools" in the TUI gateway vs 5 in -q);
* a pinned tool missing from the fresh build (skill_manage under the -q
footprint) came back from the static registry schema, without its
dynamic_schema_overrides;
* a -q --resume that rebuilt the stored prompt (model switch, cwd drift)
persisted its own pruned array over the pin.
The pin now stores the full definitions and restore replays a pinned tool that
is still available byte-for-byte (deregistered tools drop, new ones append at
the tail, legacy name-only pins still work). A continuing session whose prompt
is rebuilt applies the pin before building it, matching the freeze policy
(tools[] only changes on /new, /reload-mcp, compaction). The array is
content-addressed in the existing system_prompts store like the prompt itself,
so identical arrays across sessions are stored once; get_session resolves it.
After a restart the routing index rebuilt every lane from `SessionEntry.origin`,
which carries the runtime profile (key namespace) but not the bot that received
the conversation. Delivery then fell to `_is_shared_bot_satellite`: a lane owned
by a secondary bot whose runtime profile is ALSO a satellite of the default bot
was handed to the default bot, and authorization read the wrong allowlist.
- `SessionEntry.transport_profile` (routing JSON) + nullable
`sessions.transport_profile` (SCHEMA_SQL, reconciled by the existing column
path; `agent:main` keys untouched, standalone gateways write nothing). Stamped
from the pinned `RoutingIdentity` at create, reset/switch, DB recovery and
every peer refresh; compression forks inherit it like the other routing columns.
- `session_identity.restore_identity()` re-pins a `RoutingIdentity(transport=None)`
from the persisted transport profile; `authz_mixin._restored_source(entry)` is
the one seam every revive path uses (auto-resume, heartbeat restore, plugin
injection, background-process events).
- `_adapter_for_source` / `_adapter_profile_for_source` honour a restored identity:
the persisted bot's adapter or None — never the default bot by heuristic.
Entries written before the column exist keep the old chain.
Phase 5 of #88715.
Follow-up to the aligned-projection salvage (#114191):
- the realign migration deletes the now-meaningless state_meta key
fts_tool_full_content_high_water instead of leaving inert residue, and the
empty-store and populated-store branches share one do_align() body;
- _fts_tool_prefix_migration_requires_rebuild is dead after the realign
(the shape probe it guarded is false once do_align ran, and a deferred
align marks the index stale before that branch is reached) -> removed;
- the legacy inline triggers use _FTS_NEW_INDEXED_CONTENT_SQL again instead
of four inlined copies of the same CASE expression;
- tests trimmed to two invariants (strict-probe survives churn; a raw-messages
index realigns once on open, retired marker gone).
`messages_fts` declared `content='messages'` while its triggers indexed only a
bounded prefix of every long tool row, with the boundary held in a
`fts_tool_full_content_high_water` state_meta marker that the migration, the
rebuild seeding and the in-place rebuild path all re-stamped. FTS5's strict
integrity check re-reads the external content source and compares it with the
stored token stream, so the two could never agree: one long tool row is enough to
make
INSERT INTO messages_fts(messages_fts, rank) VALUES('integrity-check', 1)
fail with `fts5: checksum mismatch for table "messages_fts"`. The delete/update
triggers re-evaluated the *new* marker, so they sent full content for a row whose
index held a prefix and left tokens behind that survived deleting the row.
Fix the class by removing the moving part. `messages_fts` now reads a view,
`messages_fts_src`, that computes exactly what the writers index - tool rows
truncated to FTS_TOOL_CONTENT_PREFIX_CHARS, everything else verbatim - through a
fixed per-row expression with no state_meta lookups, shared by the triggers, the
boundary sweep and the chunked insert.
- `messages_fts_src` view added; `messages_fts` external content points at it
- triggers, boundary sweep and chunked backfill all read that one projection
- `_stamp_fts_tool_high_water`, the marker seeding in `_seed_fts_rebuild_markers`
and the in-place rebuild stamp are gone, with
FTS_TOOL_FULL_CONTENT_HIGH_WATER_KEY
- FTS_STORAGE_VERSION 2 -> 3: an existing index is re-pointed and rebuilt ONCE by
`_migrate_misaligned_fts_source` under the shared cross-process rebuild
admission, because a v2 index holds token streams its old source cannot read
back; an empty index swaps shape in place, and legacy inline DBs are untouched
- search behaviour is unchanged: tool rows were already prefix-indexed, and
explicit tool search already used the stored-content LIKE path
Thread logical session cwd through deferred Desktop/TUI builds, normalize absent cwd during construction, and share title provenance constants between SessionDB and Honcho.
(cherry picked from commit 2693f4f27c776ac819d92c9b52e8a03ad2a985d8)
hermes_state_common pulls in agent.* at import, so the URI builder moves to
hermes_state_holders (errno/os/sqlite3/pathlib only) where the gateway
readiness probe and backup can adopt it in a follow-up sweep. The doctor
structural-damage branch is one helper instead of two copies, the holder
scan goes through hermes_state_repair._live_writer_holds_db, the migration
hint uses _schema_not_built (the startswith("no such ") check also matched
"no such module: fts5"), and the hermes_state import is hoisted so an import
failure cannot mask itself as UnboundLocalError.
read_only_db_uri() replaces four inline mode=ro URI sites (two of which
still used the raw f-string that truncates on ?/# in the home path:
state_db_has_structural_damage and collect_state_db_stats). The doctor
write probe now applies the live-holder gate in both modes: a quiet store
is probed in place as on main, a held store is probed through a read-only
snapshot, and a held store over 1 GB is skipped with an info line unless
--fix is given (the unconditional copy cost one full DB write per plain
doctor run). Connect/backup failures propagate to the existing
classification instead of being reported as FTS write-health failures.
Observational sessions commands print a migration hint instead of a raw
traceback when a read-only opener meets an older schema.
Co-authored-by: Ahmett101 <Ahmett101@users.noreply.github.com>
docs/ was not the documentation site; it was a grab bag of long-form
design notes, wire contracts and observability guides that landed with
feature PRs because their authors needed somewhere to put them. Root
AGENTS.md already says long-form dev docs live in
website/docs/developer-guide/; this moves the 14 living documents there
(or to the matching user-guide section) so they are published, searchable
and linked from the sidebar instead of being found by grep only.
Developer guide: micro-compaction, gateway-session-lifecycle (was
session-lifecycle), state-db-recovery, multiplexing-gateway,
chronos-managed-cron-contract, relay-connector-contract, observer-hooks
(was observability/README), gateway-monitoring (observability/monitoring),
relay-shared-metrics, middleware, streaming-tts, billing-lifecycle.
User guide: egress/network-isolation (was security/network-egress-
isolation), features/kanban-multi-gateway (was kanban/multi-gateway).
Each page got title/description frontmatter and a sidebar entry; repo-
relative links became site links or GitHub blob URLs; two MDX brace
hazards escaped. Every in-tree pointer (module docstrings, config
comments, the relay conformance test's Path, the monitoring-doc test,
gateway-internals, cron-internals, kanban docs, .dockerignore, AGENTS.md)
now names the new location. `docusaurus build` passes with no unresolved
links on the moved pages.
gateway/run.py bridges the LAUNCH profile's sessions.cjk_fts / search_slow_ms
into HERMES_CJK_FTS / HERMES_SEARCH_SLOW_MS at import (and re-bridged them per
turn), and hermes_state_fts / hermes_state_search read os.getenv — so a served
secondary always got the default profile's values.
hermes_state_common.routed_sessions_setting() reads the routed profile's
config.yaml under a HERMES_HOME override and the env bridge when unscoped; both
consumers use it. The per-turn re-bridge is skipped inside a secondary's scope
so it can no longer write the default's slots from a routed turn.
The _sql_json_extract wrapper removed the `COALESCE(model_config` text the
alias string-replace keyed on, so the deferred-backfill SELECT joined an
unqualified `model_config`. Build the predicate from the alias directly and
derive the unaliased constant from it, so the two can never disagree.
Sibling widening of the #101726 salvage: FTS_TRIGRAM_SESSION_SQL (trigram view/triggers/backfill),
the v16 delegate-tagging data migration and reopen_session's legacy reset-child stamp still called
json_extract() on the raw model_config cell, so one malformed JSON row could still abort FTS
maintenance, a schema migration or /resume of a reset child. Zero raw model_config json_extract
reads remain in hermes_state_*.py.
`hermes sessions prune --source cron --older-than 14` on a store with ~60K
cron sessions (~50K matches) died with sqlite3.OperationalError: too many
SQL variables. SessionDB.delete_sessions, _collect_delegate_child_ids and
_delete_delegate_children each bound the full id list into a single
IN (?,?,...). SQLite caps bound parameters at SQLITE_MAX_VARIABLE_NUMBER
(999 on < 3.32, 32766 after), so any bulk delete above that failed outright.
Chunk every IN list (`_id_chunks` / `_SQL_IN_CHUNK` in hermes_state_common,
900 ids; the delegate walk binds each id twice so it chunks at half). Same
transaction, same cascade/orphan contract; only the parameter binding is
split.
Hand-ported from PR #102679 (targeted the pre-decomposition hermes_state.py
god file; the functions now live in hermes_state_sessions.py). Authored by
@mssteuer; ported under --author.
The delegation tool carries its own CREATE TABLE for async_delegations
(tools/async_delegation.py _initialize_schema) plus a lazy ALTER TABLE
ADD COLUMN for the tables it finds already existing. Its column list
had drifted ahead of the canonical SCHEMA_SQL: origin_session_id
(raw api_server session id of the originating request, the wake
self-post target) existed only through the tool's lazy path, so two
databases at the same schema_version had different
async_delegations shapes depending solely on whether the delegation
tool had ever run. Rebuild/replay pipelines that reconstruct state.db
from the canonical schema then hit the column with no version gate to
explain it (#94691).
Declare the column in SCHEMA_SQL with the same TEXT NOT NULL DEFAULT ''
shape the tool uses. Fresh installs now carry it canonically; the
declarative _reconcile_columns backfills it into legacy databases on
the next writable open (same pattern as earlier additive columns); the
tool's lazy ALTER keeps serving pre-reconciliation databases. The two
schema authorities now agree, pinned by a test that runs the tool's
initializer over a canonical database and asserts the shape is
unchanged.
Fixes#94691
* fix(tui-gateway): a seeded session is durable at create, and its seed is written once
session.create accepts opening messages. Three defects sat in that path:
- A seeded session without a parent was never persisted at create, so a
restart before the first prompt lost it and session.resume answered
4007. Only branch children (#93959) were persisted up front. The
same rationale applies to any seeded create: seeded content is
intent, not an abandoned draft. Parentless seeds now persist their
row, transcript and client title at create; empty drafts stay lazy.
- _coerce_seed_history dropped display_kind, so a seeded row tagged
"hidden" (model-facing scaffolding) rendered as a user bubble. The
coercion keeps "hidden" and only "hidden"; every other kind is
stamped by the gateway at turn time and is not accepted from the wire.
- A branch child's seed was written twice: _seed_branch_row copied it at
create but never marked it persisted, so the first prompt's
_persist_branch_seed appended the copy again. The create path now
sets _branch_seed_persisted, and the gate is a create-time `seeded`
stamp instead of parent_session_id, so a resumed session (whose
history comes from the DB) can never re-append its transcript.
Two invariant tests, both red on main: a parentless seed survives a
gateway restart with the hidden row kept out of the wire transcript and
not re-written by the first-submit path; a branch child's seed is stored
exactly once. The reasoning-fields fixture stamps `seeded`, the flag
session.create sets.
* fix(tui-gateway): a hidden seed row stays out of the list preview and the create count
Live-testing the seeded create on every surface showed two places where
the newly durable hidden row (display_kind="hidden") still surfaced:
- session.list built a session's preview from its first user row with no
display_kind filter, so a hidden opening row (model-facing scaffolding
the gateway never paints) became the sidebar preview. The preview
predicate now skips hidden rows, in every listing query that shares it.
- session.create reported message_count as the raw seed length while its
messages array already filtered the hidden row (2 vs 1). It now counts
what is on the wire, the same rule session.resume applies.
Both are covered by the existing seeded-create test: the create count
equals the wire transcript, and the preview of a session whose first
user row is hidden is its first visible user row.
* fix(tui-gateway): a live unpersisted resume counts the wire transcript
session.resume on a live session that has no row yet reported message_count as
the raw history length while its messages array was already filtered, the same
mismatch the previous commit fixed on session.create. Count the wire, as the
cold, deferred and reuse-live resume paths already do.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
Consolidated from PR #106543 (5 commits, final tree d2c4d908) by @Totoro-qaq.
publish_compression_child() fails closed on any non-automatic end stamp and
end_session() is first-stamp-wins, so a stale tui_close on a session the TUI
still routes turned every rotation into "compute the summary, then discard it"
(#106459). The host that still routes the session clears the stale explicit
close via SessionDB.reopen_if_explicitly_closed() before the turn starts;
publication never heals explicit closes. Review probes by @ehz0ah.
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
On a fan-out-heavy install state.db reached 3.4 GB; 70% of message bytes
belonged to subagent sessions, and every one of those rows was also
indexed into messages_fts_trigram, whose shadow tables are ~2.6x the
text they cover (1,029 MB trigram vs 350 MB standard FTS on that DB).
session_search already hides source='subagent' sessions, so the
substring/CJK index bought nothing for them.
Extend the v29 cron exclusion: the messages_fts_trigram_src view, the
three sync triggers, and both deferred-backfill INSERT...SELECTs now use
one shared predicate (FTS_TRIGRAM_SESSION_SQL / fts_trigram_session_sql)
that skips sessions with source IN ('cron','subagent') or the
$._delegate_from creation marker (children spawned under a gateway turn
inherit the gateway's source). Compression/branch continuations carry
parent_session_id without the marker and stay indexed. Child rows remain
canonical in `messages` and fully indexed in the standard messages_fts
word index; explicit source_filter=['subagent'] CJK searches route to
LIKE like cron already did.
The v29 migration gate becomes `< 30` and reuses the same view-swap +
admitted rebuild, so existing installs purge historical child postings
once on open. Fresh DB with 2,000 x 2 KB child messages: 22.4 MB ->
12.5 MB (trigram shadow 10.09 MB -> 0.02 MB).
Keep structured tool_calls searchable through the standard FTS index while
removing their repetitive JSON from the trigram projection. Reuse the
existing optimize-storage rebuild path for deployed v1 layouts.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Preselect indexed recent candidates before rich hydration, cap and deduplicate compression lineage traversal, and interrupt sustained SQLite work through a cooperative progress deadline. Fail closed when the bounded browse API is unavailable and cover legacy-schema reconciliation plus malformed lineage cases.
