_insert_message_rows stamps _row_id and the stored-row digest onto the
caller's dicts inside the write transaction. Only append_messages_batch
restored that state on rollback. archive_and_compact, replace_messages
and the rotation handoff left the rolled-back id + digest on the dicts;
SQLite reuses the id, so a later flush found a digest mismatch on the
foreign row, adopted it and silently dropped the user's message.
Move the capture/restore into _execute_transcript_write, used by every
caller that inserts caller-owned dicts: each attempt starts from the
caller's state and a final failure restores it before re-raising.
(Rewind replacement and import insert dicts built inside the txn.)
Also: bind _message_row_params directly on insert instead of the
serialized-dict round-trip, import the public DB_ROW_SNAPSHOT /
CANONICAL_ROW names, set adopt=False once, and reuse target_row
instead of re-SELECTing when nothing was written.
A legacy (no-digest) dict over a non-blank assistant row adopted the whole
decoded DB row: tool_calls / reasoning* / codex_* were overwritten with the
stored JSON (which still holds the escaped lone surrogate the sanitizer just
fixed, re-injecting it into the provider payload) and live-only fields were
popped. Resumed dicts (_rows_to_conversation stamps _row_id without a
digest) and compaction clones hit this path. Adopt content only, as before
this stack, via a content-only canonical handled like the metadata-only one.
_insert_message_rows dropped a clone's parent digest but only the flush
path restamped it, so clones made by archive_and_compact / replace /
rotation handoff / import reached the legacy path and the first live edit
after a clone was not persisted. Stamp the stored-row digest inside
_insert_message_rows (one batched SELECT, cold paths only; the flush path
statement count is unchanged) and drop the duplicate call in
append_messages_batch.
Define the _db_row_snapshot / _canonical_row keys once in
agent/message_metadata.py and import them everywhere instead of repeating
the literals.
The row digest hashed every repair column, so a same-process metadata write
(reaction, display-kind stamp, api_content / codex reasoning backfill,
platform message id) made our own row look like a foreign winner. The
re-flush then adopted the stale DB row: a later live edit (the non-ASCII
strip recovery) was reverted, and unsanitized tool_calls/reasoning were
copied back onto the live dict.
The digest now covers only the owned (non-metadata) columns: it means "the
row is still what we last committed". Match -> write the live owned values
and hand over only presentation metadata the live dict lacks; mismatch ->
genuine other writer, adopt as before. The r3 "stored content equals the
durable form of live" special case is subsumed and removed.
Also:
- _insert_message_rows drops a carried digest when it assigns a new row id
(compaction/replace/import clones carried the parent's version).
- append_messages_batch restores each message's _row_id / digest /
timestamp and pops the adopted row at the top of every _execute_write
attempt, so a rolled-back attempt cannot resolve to a foreign row.
- message_id is no longer synced onto live (int -> str flip, spurious
platform_message_id).
- The JSONL divert strips both bookkeeping keys via one frozenset.
The CAS row snapshot was a full copy of each message's durable payload
riding on the live dict. The rough token estimator priced it (about 2x
estimates -> premature compaction) and it doubled transcript memory.
Replace it with a 16-byte blake2b digest of the repair columns. The
compare now runs in Python against the target row already read inside
the BEGIN IMMEDIATE transaction, followed by a plain UPDATE. Also:
- add _db_row_snapshot to PERSISTENCE_ONLY_MESSAGE_FIELDS so the
estimator and the outbound request builder both drop it
- derive _REPAIR_COLUMNS/_SYNC_FIELDS from _MESSAGE_WRITE_COLUMNS
- use hermes_state_common._placeholders
- drop the dead resume-path stamp (the SELECT has no token_count, so it
was always None) and the dead tool name assignment in
_decoded_repair_row
- keep the digest out of divert JSONL
The kept active-row test now pins estimate stability across a flush and
the survival of a concurrent writer's row. It goes red on the old
prod files and red when the digest compare is removed.
When the heal recreated the session row but its single retry write then
failed (database locked, turn lease, disk), the replay was carried only by
the per-call _replay_history argument. The next flush found a live row, hit
no FK error, ran no heal, and stamped the history prefix durable without
writing it: the durable transcript silently lost everything but the tail
(scratch probe: rows ['a2'] instead of ['one','a1','two','a2']).
Set _session_row_replay_pending in the heal branch right after the markers
are stripped, so any exit after a heal (recreate failed, retry failed)
leaves the replay pending until a write succeeds. That makes the separate
recreate-failed assignment and the _replay_history parameter redundant;
the retry is a plain _adoption_budget=0 call. The fails-closed test now
covers FK -> heal -> locked retry -> full replay on the next flush.
Co-authored-by: 赵桂雄 <daniel21436@hotmail.com>
When the session row is gone and the heal cannot recreate it, the flush
fails closed with the flush markers already stripped. The next flush then
recreates the row up front via _ensure_db_session, so no FK error fires,
no heal runs, and _db_flush_collect stamps the history prefix as durable
without writing it. The recreated row ends up holding only the new tail
(probe: rows ['a2'] instead of ['one', 'a1', 'two', 'a2']).
Remember the failed heal per session id and have the next flush for that
session replay the history prefix; clear it once a flush succeeds.
Co-authored-by: 赵桂雄 <daniel21436@hotmail.com>
The FK classifier behind session_row_missing also matches the sessions
table's own parent_session_id / system_prompt_hash FKs, and session
create is an upsert. Healing on the classification alone could replay
the whole transcript into a still-live row and duplicate it. Heal only
when get_session(session_id) confirms the row is gone.
The delegate-child branch's get_session(parent_id) was unguarded: a
store error there escaped the flush's error handler. Route both lookups
through one guarded helper (as the compression-tip helper does); a failed
lookup is not proof of deletion, so it falls through to the existing
"will retry next flush" return.
While here: _db_flush_failed now returns which retry to take
("adopted" / "healed" / None) instead of the caller string-comparing
_last_persistence_error_cause; its docstring covers the heal branch;
messages is required (the only caller always passes it); the marker
reset reuses _strip_persistence_markers. Tests drop the change-detector
assertion on the lingering cause after a successful heal and the Scenario
B docstring now says fail-closed, which is what it asserts.
The session-row heal retried with conversation_history=None so the
history prefix would be replayed onto the recreated row. On a muted
notification-reply turn that also made _db_flush_collect treat every
history row as new: each replayed row was written display_kind=hidden
and the flag was stamped onto the live in-memory history dicts, hiding
the user's whole past transcript from pollers from then on.
Keep the history set on the heal retry and pass an explicit
replay_history flag instead: history rows are written again (not stamped
durable) but keep their original visibility, and only this turn's new
rows get the mute treatment.
Deleting a parent session cascades to its delegate children. A live child
then can never be recreated: create_session(parent_session_id=...) hits
the parent FK on every flush for the agent's lifetime. When the recreate
fails and the parent row is gone, create once without the parent for that
call only; agent._parent_session_id is restored because the relay and
hooks still key on it.
The heal predicate (errno 787 or a 'foreign key constraint' substring)
and the classifier phrase were two separate definitions of the same
failure. Move the SQLITE_CONSTRAINT_FOREIGNKEY code check into
classify_persistence_error, document the bucket, and branch the heal on
agent._last_persistence_error_cause == 'session_row_missing' so the two
can't drift.
The heal retry passed the caller's conversation_history through, so
_db_flush_collect's id()-based history shortcut treated the prefix as
already durable and skipped it. The row was recreated with only the new
tail and the flush returned True: a silent partial restore. On the
session_row_missing retry, pass conversation_history=None so the full
in-memory transcript lands on the recreated row.
Co-authored-by: 赵桂雄 <daniel21436@hotmail.com>
`_ensure_db_session` trusts the cached `_session_db_created` flag as proof
the row exists, and the flush path only retries row creation while that
flag is False. Any store-side removal of the row under a live agent —
`hermes sessions delete`, the Desktop/web delete, bulk prune, a
profile-repair move, an in-place store rebuild — leaves the flag stale, so
every later turn's append fails the FK and is dropped with one WARNING per
turn. The agent keeps answering; the durable transcript silently stops
growing, and because the failed transaction leaves no rows behind there is
no post-hoc trace in the store.
Classification (`hermes_state_errors.py`): add the `session_row_missing`
cause — matched by `SQLITE_CONSTRAINT_FOREIGNKEY` (787) when the code
survives, else the RPC-wrapped phrase — so the turn-end explanation names
the real failure instead of "unknown".
Heal (`agent/session_persistence.py::_db_flush_failed`): on that cause,
drop the stale flag, reset the flush markers, call `_ensure_db_session()`,
and replay once within the flush's existing `_adoption_budget`. Because the
deletion erased the session's message rows too, the replay clears the
per-message persisted markers so the FULL in-memory transcript lands on the
recreated row, not just the current tail (mirrors
`_db_flush_adopt_compression_tip`). If row creation also fails, return
False without appending into a guaranteed rollback — fail-open, batch stays
unmarked for the next flush. No new fail-closed path.
(cherry picked from commit ad97047da0f9801a4fdca4478bfd68085f02c3a8)
handle_api_interrupt exits via break into finalize_turn, whose
_close_transcript_tail already closes the tail with the same final_response, so
its own close was a duplicate; the scaffold strip stays (the partial-text row
must follow the tool row). The scaffolding drop no longer returns a flag nobody
reads, and the give-up test relies on the autouse backoff stub in
tests/agent/conftest.py instead of re-stubbing it.
`_persist_session` closed the tool tail it uncovered after popping
empty-response scaffolding with the generic "Operation interrupted." row.
That close was dead on every path whose owner already shapes the tail
(finalizer, abort_turn_on_interrupt) and wrong where it did fire: a
non-interrupt terminal exit (retries exhausted, invalid response, rate
guard) and even the give-up path got an "interrupted" row the finalizer
then followed with a second closing row, and an in-flight Stop during the
nudge request lost its own "Operation interrupted: waiting for model
response (...)" reason because the finalizer's close no-ops on an
assistant tail.
The persist layer now only pops the scaffolding. `handle_api_interrupt`
strips it before appending its partial/placeholder row and, when it has no
partial text, closes the tail with its own reason, mirroring
`abort_turn_on_interrupt`. The give-up path is closed by
`_close_transcript_tail` with the delivered final_response, as before the
stack.
After an empty-response give-up or a Stop mid-recovery,
_drop_trailing_empty_response_scaffolding popped the flagged scaffolding
and then also rewound the trailing tool results and their
assistant(tool_calls) row. Those rows are saved before the tools run, so
the rewind only removed them from the live history: result["messages"]
(which CLI, TUI and ACP reuse as the next turn's history) lost a tool call
that had already executed, and a "continue" made the model run it again.
On a Stop, the finalizer then found no tool tail to close, so the saved
transcript ended on an unanswered tool result.
Drop only the flagged scaffolding rows. _persist_session closes a tool
tail the drop uncovers with close_interrupted_tool_sequence, which covers
the exits that return without the finalizer; _terminal_empty and the
finalizer already close the tail themselves.
(cherry picked from commit 9fe02fc7f27742a28ece5f789be78cfa4b4052ef)
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
`hermes peer dm` posts to POST /api/sessions/{id}/chat, the third Bot-DM
transport. The local (`tools.bot_mode_dm`) and relayed
(`tui_gateway.methods_bot_relay`) lanes both re-run a transiently failed turn
once under the shared policy (`tools.bot_failure_reasons.retry_action`) and
resume the row the failed attempt left as the transcript's unanswered tail;
this lane ran the turn once and handed the provider's 429 paragraph to the
sender as the reply (#115325).
The policy is asked about a result dict now, not two streams: `result_retry_action`
joins `error` + `failure_reason` (the turn loop's own typed verdict) so one
classifier serves every lane, and the server-error rule accepts the providers'
`server_error` / `overloaded_error` spellings — the codes the in-process lanes
key on instead of a status number.
The resume half is the CLI lane's rule extracted to `agent.session_persistence.
adopt_unanswered_turn`, which `quiet_single_query` (env-gated dispatcher re-run)
and the API lane (in-process re-run, on the agent it just built) now share.
The regression drives the real route and the real `_run_agent` over a real
store: a 429 re-runs the same DM once with the persisted row adopted as this
turn's user message (so no second copy), a 401 still reports one attempt.
(cherry picked from commit 8fe6d46ada5b8064bc7132ee956fda998244c10a)
Resolved toward the branch: PM provisions uv/python (main's install.ps1 uv-shim
salvage + its test and workflow steps dropped), the shim re-exec stays retired,
package.json carries no electron-builder block (afterExtract identity stamp wired
into electron-builder.config.cjs instead; after-pack.mjs keeps signing only),
Desktop workspace-deps helpers stay retired. Main's scratch-dir bootstrap
(export_scratch_tmp_env) is taken and re-run after profile resolution.
`Fixes #102374` was only true for the compacted transcript: the prune ran
inside local `compress()` / `salvage_grown_transcript`, and under native
compaction the server compacts LOCAL_TRIGGER_SAFETY_MARGIN below the local
trigger, so local compaction rarely fires. Every assistant response still
persisted its ~120 KB checkpoint to `messages.codex_reasoning_items` and the
older rows' shadowed copies were never rewritten — the field path in the
issue (1,967 checkpoint rows, 12.9 GiB in one lineage).
`SessionDB._insert_message_rows` (the one INSERT path: append batch,
replace, compact, import) now rewrites the older ACTIVE assistant rows of the
same session to drop their `type: "compaction"` items once a newer carrier
row lands, keeping every non-checkpoint item; wire-neutral because
`prune_pre_checkpoint_items` never replays a shadowed checkpoint. The flush
mirrors it on the live transcript (`drop_shadowed_checkpoints`) so forks and
compaction built from memory carry one checkpoint too, and the marker
contract holds: the dicts read exactly as their rows.
Squash of the 54 commits on victor-kyriazakos:feat/user-channel-warning-suppression
(PR #112302, head f45c640e55) so the contributor's authorship survives a rebase-merge;
the commits interleave with a cron delivery-ledger rework that the salvage removes in
follow-up commits, so per-commit cherry-picks were not practical.
Adds display.suppress_warning_notifications (global + per-platform, default false):
one resolver (gateway/warning_notifications.py), BasePlatformAdapter.emit_warning /
emit_media_warning / warning_text, a notification_category classification carried
through wakes, queues and persistence, and render/present boundaries for CLI/TUI.
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
Keep upstream's reviewed catalog as the only plugin name index.
Catalog pins and custom update sources share staged PM validation.
Publish code and dependencies with recovery after process death.
Reject a concurrent enablement change before publishing disabled code.
Use the manifest loader's supported version in the installer. Keep
probe cooldowns for timeouts, not TLS failures that a CA change fixes.
Preserve the backup, uninstall, browser and memory-provider repairs.
Verified with the canonical runner on native Windows ARM64, real Git
repositories, local TLS endpoints and UV dependency generations.
Desktop catalog tests and both TypeScript checks pass. The full suite
and native release builds were not run. No remote push.
The turn-start stamp had grown its own copy of the "what does the current
user row hold" rule (persist override = clean transcript, live content =
wire bytes = sidecar when they differ) that _db_flush_row already
implements. Two copies drift; extract durable_user_row_content() in
session_persistence and call it from both.
Also: reuse _persist_lock() instead of a third open-coded lock/nullcontext
ladder; drop the hasattr guard on set_latest_user_api_content (it predates
this fix and exists on every SessionDB); cut the comment to the WHY;
trim the new test file from 18 cases to the 7 invariants (real close
flush E2E, repeated-"ok" positional protection, API-only pre-flushed
turn, normal path writes nothing, compaction keeps positional, store
guards). Still 3 red / 4 green when agent/turn_context.py is swapped
for main's copy.
Merge upstream b1f003e186 while preserving PM runtime ownership and
Python 3.14 worker startup, Windows signing, and macOS wait recovery.
Keep retired runtime modules deleted. Port upstream updater preflight
checks into the checkout strategy and preserve live build logging.
Carry checkpoint filename handling and process recovery into the current
module layout. Regenerate locks and adapt incoming platform test markers.
Focused Python and JavaScript tests, desktop and root-test typechecks,
conflict-path lint checks, lock validation, and retired-import checks pass.
The full test suite and packaged release builds were not run.
The footgun class-fix left 23 files with unquoted encoding=utf-8-sig
(and main.py an f-string with nested double quotes) — SyntaxError /
NameError 'utf' at every import. Quoted all encoding values, re-verified
py_compile across every staged .py.
hermes_state.py: delete every '# noqa: F401 (re-exported...)' import block (hermes_state_common/errors/guard/
readpool/sessions/fts/dbfile/wal/repair/registry + agent.context_compressor _DB_PERSISTED_MARKER_KEY); keep
only the names hermes_state.py itself uses, without noqa.
hermes_state_registry.py: drop get_shared_session_db/release_shared_session_db/close_shared_session_dbs
aliases; every caller (gateway/, tools/, tui_gateway/, cron/, mcp_serve, run_agent, tests) now imports
acquire/release/close_all/release_or_close from hermes_state_registry.
hermes_state_titles.py: drop set_auto_title_if_empty shim (title_generator keeps its getattr fallback).
Re-remove shim-only names restored by 34abf954bd: latest_user_message_row_id (tests call
latest_message_row_id(key, role='user'); role-targeting assertions kept) and get_session_activity (tests
build the snapshot via agent.session_activity.build_activity_snapshot over db.get_session(sid)).
hermes_state_wal._log_once resolves its dedupe sets as module globals instead of via hermes_state;
hermes_state_repair helpers call module globals directly (tests patch hermes_state_repair.<name>).
Frozen updater surface untouched (update_cmd_maint imports only SessionDB from hermes_state).
Adolanium review §3 (Medium/Low). BASE shape at every steer/redirect/persist/agent-cache
lock site was: lock = getattr(obj, "_x_lock", None); if lock is not None: with lock: <direct
attribute read>; else: <getattr fallback for object.__new__ test stubs>. The refactor rewrote
several of these as 'with getattr(...) or nullcontext(): getattr(slot, None)', which (a) turned a
missing slot under the lock from a loud AttributeError into silent None and (b) in the gateway
peek helper read the cache without any lock when the lock attribute was absent.
Restored BASE semantics at:
- agent/agent_runtime_helpers.py::_requeue_pending_steer
- agent/interrupt_control.py: steer, redirect, clear_interrupt, _has_pending_redirect,
_drain_pending_redirect, _drain_pending_steer (new _ic_slot helper: direct read under lock)
- agent/session_persistence.py::_persist_lock (explicit None check + BASE rationale)
- gateway/slash_commands.py::_cached_agent_for (BASE callers read ONLY under the lock; no lock -> None)
- gateway/run_agent_cache.py::_evict_cached_agent (BASE: self._agent_cache direct under lock)
- agent/client_lifecycle.py::_is_openai_client_closed: BASE body + docstring verbatim (outer
is_closed first; inner _client.is_closed only when _client exists; else False)
- agent/stream_delivery.py::_ensure_stream_writer_state: restore BASE rationale that the lock is
created unconditionally in agent_init (_STREAM_STATE) and the lazy path is stub-only
A/B (/tmp/rf/rev/ab_lock_fallbacks.py, ab_client_closed.py) is byte-identical BASE vs HEAD on
the reviewer's inputs + edge cases. tests/agent/test_lock_fallback_base_semantics.py pins it
(7 of 25 cases fail on the pre-fix tree).
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
