Gateway /resume rewrites sessions.source to the resuming platform, which is
correct routing behavior (source feeds find_latest_gateway_session_for_peer,
Telegram listings, and the TUI ghost pruner) but destroys creation
provenance. Add a separate immutable created_source column stamped once at
session creation and preserved by all upsert paths; surface it in
hermes sessions list as '<created>→<current>' when the two diverge.
Fixes#56439
A completed /handoff transfers the source session to the gateway, but the
source interface's teardown can still stamp it with a terminal end_reason
(cli_close) before the destination's first reply. The stale-route self-heal
then finds a row that is neither recoverable nor a reset boundary, silently
drops the routing entry and mints a brand-new empty session — the handed-off
leg and its whole transcript are orphaned.
Field incident (QQ DM, one session_key, 2026-09-22):
21:33 qqbot session B (253 messages) starts
22:29 /handoff qqbot from the CLI; the gateway claims the row,
handoff_state='completed', synthetic turn dispatched
22:29 the CLI teardown stamps B end_reason='cli_close'
22:54 the user's next DM takes the stale-route self-heal
("routing key ... is ended in state.db but still live in
sessions.json; dropping stale entry ... (#54878)") and creates a
new session C; the /resume-able leg B is lost
A row a completed handoff owns is now recoverable whatever end_reason it
carries: handoff_state='completed' means ownership was transferred, and the
stamp left behind by the departing interface is not a user-facing close. Both
peer-recovery queries (exact key and peer tuple) share the widened predicate.
The existing reset-boundary fence still applies, so a later deliberate
boundary drops the row, and pending/failed/NULL handoffs are untouched.
The CLI-side guard for the teardown path is the other half of this seam and
is tracked separately (#118196 / PR #118202). This change makes losing the
handed-off leg impossible even when such a stamp lands.
Tests: tests/gateway/test_handoff_owned_recovery.py (incident case plus
controls: plain cli_close stays closed, a reset after the handoff still
fences, a failed/pending handoff does not widen recovery, and a live row still
outranks the stale handed-off row).
A negative older_than_days built last_active_before in the future, so
prune_sessions matched every ended session. sessions.retention_days fed
it from config at startup via maybe_auto_prune_and_vacuum on both the
CLI and gateway paths with auto_prune on by default, and
list_never_active_keyed_sessions had the same unguarded cutoff.
_prune_where and list_never_active_keyed_sessions now raise ValueError
on a negative bound, and maybe_auto_prune_and_vacuum warns and skips on
retention_days < 0 so a bad config never reaches a cutoff.
After a restart the routing index rebuilt every lane from `SessionEntry.origin`,
which carries the runtime profile (key namespace) but not the bot that received
the conversation. Delivery then fell to `_is_shared_bot_satellite`: a lane owned
by a secondary bot whose runtime profile is ALSO a satellite of the default bot
was handed to the default bot, and authorization read the wrong allowlist.
- `SessionEntry.transport_profile` (routing JSON) + nullable
`sessions.transport_profile` (SCHEMA_SQL, reconciled by the existing column
path; `agent:main` keys untouched, standalone gateways write nothing). Stamped
from the pinned `RoutingIdentity` at create, reset/switch, DB recovery and
every peer refresh; compression forks inherit it like the other routing columns.
- `session_identity.restore_identity()` re-pins a `RoutingIdentity(transport=None)`
from the persisted transport profile; `authz_mixin._restored_source(entry)` is
the one seam every revive path uses (auto-resume, heartbeat restore, plugin
injection, background-process events).
- `_adapter_for_source` / `_adapter_profile_for_source` honour a restored identity:
the persisted bot's adapter or None — never the default bot by heuristic.
Entries written before the column exist keep the old chain.
Phase 5 of #88715.
_COMPRESSION_LINEAGE_CTE (record_gateway_session_peer with
include_compression_ancestors=True on explicit resume) still stepped
through a _reset_from child of a compression-ended parent, so resuming
the reset sibling re-keyed the real lineage's ancestors onto the fork's
routing peer while get_compression_lineage() said the fork was alone.
Apply the same _RESET_CHILD_SQL rule as _CHAIN_STEP_SQL and the list
CTE; pin CTE == Python walk on the 5-row reporter lineage (#114271).
The delivery ledger adds adapter_profile lazily (add_column_if_missing) only
when a gateway opens it, so a store whose ledger predates that column and
whose gateway has not run since raised 'no such column: adapter_profile'
from purge_profile_state / rekey_profile_state. Probe the column like the
topic tables' profile_name: purge falls back to the session_key namespace
match, rekey skips the adapter_profile UPDATE.
Share the column inspection between purge_profile_state and rekey_profile_state
and say why identity settlement must not force the /topic migration (the
migration is opt-in by design, see apply_telegram_topic_migration). Collapse
the v3/legacy binding delete into one statement.
profile purge/rekey gate their profile_name SQL on the column actually being
present, so supported v1/v2 topic tables (never migrated because /topic was
never run) no longer abort identity settlement with
"no such column: profile_name". Binding cleanup/rekey still runs by exact
agent:<name>: session_key namespace on those shapes.
Fixes#113757
Co-authored-by: Yagna Vudathu <yagnavudathu@gmail.com>
The probe helpers had been appended to the gateway/run.py facade and the three watchers each
carried their own copy of the "probe → fail open → offload" shell, with gateway code parsing
loop/heartbeat rows itself. One sibling now owns the gate shape (_gate + off_loop_gate), and
hermes_cli.loops.store_has_active_loop / hermes_cli.heartbeat.store_has_active_heartbeat own
what an ACTIVE row is (heartbeat gains the _META_PREFIX loops already had). One fail-open layer
per read: has_pending_handoffs is a bare bounded query, the gate catches. The loop watcher calls
its executor hop directly — it already did so unconditionally for the scan, so the
"runner without a hop" fallback there was dead.
The handoff watcher (2s interval) and loop wakeup watcher (15s) entered
_async_profile_runtime_scope for EVERY served profile on EVERY tick — a
full config.yaml/.env parse plus secrets hydration per entry — even when
the profile's state.db held zero pending handoffs or active loops. On a
multiplex gateway this, not message traffic, was the idle CPU driver
(py-spy: ~75% of on-CPU time in YAML parsing from these two paths).
Both watchers now probe the profile's cached SessionDB first (only the
HERMES_HOME contextvar installed, off the loop thread via the runner's
executor hop) and enter the scope only when work exists:
- handoff watcher: new SessionDB.has_pending_handoffs() existence probe
- loop watcher: list_active_loops() under the probe context
All probes fail OPEN (error/unknown -> historical always-enter
behavior); runners without the executor hop (test stand-ins) skip the
gate, so existing watcher tests are unaffected. The startup
stale-handoff reclaim stays ungated: it runs once per boot and must
also see 'running' leftovers, which a pending-only probe would miss.
`hermes profile delete` removes the profile directory and tears its runtime down, but the name is
also baked into durable identity the delete path never touches — `agent:<name>:*` routing keys,
`gateway_heartbeats.profile` and `delivery_obligations`. An inbound event on a chat keyed to the
dead name then enters the routing index, resolves a profile whose directory is gone, and logs
`Profile '<name>' does not exist` on every event for the life of the store (the #111926 flood,
reached from a *deleted* rather than a renamed profile). The delete side is now symmetric with the
rename rekey (`rekey_profile_state` / `rekey_profile_routing` / `migrate-profile-identity`), with
the same ownership rule:
- `SessionDB.purge_profile_state(name)` — the mirror of `rekey_profile_state`, in one
`_execute_write` transaction. Routing keys, heartbeat rows and the telegram topic rows the rekey
also owns are hard-deleted (a binding is matched by `profile_name` OR its `session_key`
namespace, because the rename rewrites both); `delivery_obligations` rows are terminalized
(`state='abandoned'`) rather than dropped, so pending delivery state is not lost silently.
- `SessionStore.purge_profile_routing(name)` — the mirror of `rekey_profile_routing`: drops the
in-memory entries and persists the drop. Mandatory, not belt-and-braces — the owning process
writes its in-memory copy back, so a durable delete made elsewhere is undone by its next save.
- A delete-only control verb `purge-profile-identity`, deliberately NOT inside
`_unserve_profile()`: that hook also unserves a rename's old name, whose identity the rekey still
has to migrate. `hermes profile delete` requires the owner's `{"ok": true}` answer and reports a
partial settlement (naming the retry) instead of a clean success.
- The retry is the new `hermes profile purge-identity <name>`. It refuses a name that is a live
profile again: the purge keys off the name alone, so `delete foo` (settlement pending) →
`create foo` → `purge-identity foo` would otherwise delete the NEW incarnation's identity. The
delete path tombstones the directory before it purges, so the guard never blocks the delete.
- `sessions` rows are not deleted by the purge: it settles identity, not history. What a delete
leaves of a profile's conversation record is `delete_profile`'s business — it removes the
profile's own home, `state.db` included.
Tests (`scripts/run_tests.sh`, red on base → green): `tests/hermes_state/test_purge_profile_state.py`,
`tests/gateway/test_purge_profile_routing.py`, `tests/gateway/test_profile_identity_purge.py`,
`tests/hermes_cli/test_profile_identity_purge_cmd.py` and `TestDeleteProfile` in
`tests/hermes_cli/test_profiles.py` — 95 passed, 0 failed across those five files.
Renaming a profile moved profiles/<old>/ to profiles/<new>/, so the row DATA
travelled with the directory, but the profile name is also baked into
keys/values the move left untouched: session keys (agent:<old>:* namespace),
sessions.profile_name (fail-closed owner ladder / Desktop sidebar scope /
@session: deep links), sessions.origin_json.profile,
gateway_heartbeats.profile, delivery_obligations (session_key +
adapter_profile), telegram_dm_topic_* profile_name bindings, and the
gateway_routing index. Left stale, every inbound event on a chat keyed to the
old name resolved to a profile that no longer exists — flooding errors.log
with "Profile <old> does not exist ... falling back to global HERMES_HOME"
every few seconds — and renamed sessions dropped out of the sidebar / broke
their deep links.
The routing index is held in memory by a live multiplexer and written back
periodically, so a CLI-side DB rewrite alone is clobbered. Fix in layers:
- SessionDB.rekey_profile_state: atomic durable rewrite of the state.db
tables, matching the agent:<name>: namespace by exact prefix (substr, not
LIKE — '_' is a legal profile-name character and a LIKE wildcard), rewriting
the profile inside routing/origin JSON, and REFUSING on a target collision
(routing rows or telegram bindings) instead of silently merging.
- SessionStore.rekey_profile_routing: rekey the in-memory routing index
(keys + origin.profile) then persist — the half a DB write cannot reach.
Raises on a target-key collision before mutating.
- Control verb migrate-profile-identity (params-carrying; the socket passes
params only to handlers that declare them, bare handlers unchanged) so a
live gateway rekeys its in-memory copy AND both durable stores (routing home
+ the renamed profile's own state.db).
- rename_profile calls the verb when a multiplexer is live and, if it fails,
does NOT fall back to a racing CLI-side write: it prints a warning telling
the operator to restart the gateway and retry. With no live gateway it
performs the durable rewrite itself (safe: nothing else holds the store
open).
Checkpoints keyed by the profile's workdir path are a known related gap,
tracked separately, not addressed here.
Tests: rekey_profile_state (all tables, routing/origin JSON, collisions,
idempotent, no-op), rekey_profile_routing (namespace + origin, no-op, no
overwrite), control verb param passing, and rename end-to-end for both the
live-gateway (delegates, refuses unsafe fallback) and no-gateway (durable
rewrite) paths.
_notif_gateway_owns_heartbeat decided by the immutable sessions.source column,
so a heartbeat on an ARCHIVED gateway-sourced row (Telegram /reset, idle/daily
auto-reset, compression rotation) was skipped by the Desktop poller and never
registered by the gateway either — restore_heartbeat_watches only claims a key
whose current session_id is that row. The tick belonged to nobody and stayed
due forever, where origin/main's Desktop fired it.
Ownership now uses the same predicate the gateway does: a gateway_routing entry
whose current session_id is this session, with an origin and not suspended
(SessionDB.gateway_routing_entry_for_session; both the session's profile store
and the launch store are consulted so multiplexed and per-profile gateways are
covered). No entry is fail-open, as on main. The check runs after the cheap
is_active/is_due gate so idle sessions never touch the DB per poll.
Probe (SessionStore telegram -> force_new, heartbeat on the archived sid):
before: desktop fired False / gateway watches [] / still due True;
after: desktop fired True / still due False; the current gateway sid is still
left to the gateway (desktop fired False) — the hijack fix stays intact.
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
