A legacy (no-digest) dict over a non-blank assistant row adopted the whole
decoded DB row: tool_calls / reasoning* / codex_* were overwritten with the
stored JSON (which still holds the escaped lone surrogate the sanitizer just
fixed, re-injecting it into the provider payload) and live-only fields were
popped. Resumed dicts (_rows_to_conversation stamps _row_id without a
digest) and compaction clones hit this path. Adopt content only, as before
this stack, via a content-only canonical handled like the metadata-only one.
_insert_message_rows dropped a clone's parent digest but only the flush
path restamped it, so clones made by archive_and_compact / replace /
rotation handoff / import reached the legacy path and the first live edit
after a clone was not persisted. Stamp the stored-row digest inside
_insert_message_rows (one batched SELECT, cold paths only; the flush path
statement count is unchanged) and drop the duplicate call in
append_messages_batch.
Define the _db_row_snapshot / _canonical_row keys once in
agent/message_metadata.py and import them everywhere instead of repeating
the literals.
Gateway/TUI/CLI callers pass their live dicts straight to
append_messages_batch, so a concurrent-winner adoption leaves the
decoded durable row (_canonical_row) on a dict that may later be sent
to the model. Treat it as persistence-only like _row_id and the digest
so the outbound builder and token estimator both drop it.
The CAS row snapshot was a full copy of each message's durable payload
riding on the live dict. The rough token estimator priced it (about 2x
estimates -> premature compaction) and it doubled transcript memory.
Replace it with a 16-byte blake2b digest of the repair columns. The
compare now runs in Python against the target row already read inside
the BEGIN IMMEDIATE transaction, followed by a plain UPDATE. Also:
- add _db_row_snapshot to PERSISTENCE_ONLY_MESSAGE_FIELDS so the
estimator and the outbound request builder both drop it
- derive _REPAIR_COLUMNS/_SYNC_FIELDS from _MESSAGE_WRITE_COLUMNS
- use hermes_state_common._placeholders
- drop the dead resume-path stamp (the SELECT has no token_count, so it
was always None) and the dead tool name assignment in
_decoded_repair_row
- keep the digest out of divert JSONL
The kept active-row test now pins estimate stability across a flush and
the survival of a concurrent writer's row. It goes red on the old
prod files and red when the digest compare is removed.
Edit tool rows now carry display_metadata.tool_result_metadata.inline_diff
(~9KB of ANSI per edit) on the live message dict. build_api_messages strips
display_kind/display_metadata/_row_id before the wire, but the local
estimator's wire shadow only dropped PERSISTENCE_ONLY_MESSAGE_FIELDS
({"timestamp"}), so estimate_messages_tokens_rough and
estimate_request_tokens_rough priced the diff: one edit row went from 32 to
2527 estimated tokens (100 edits: 3100 -> 252600). That inflates compaction
preflight, post-tool checks and turn-overflow scoring, compacts early and
breaks the prompt cache.
PERSISTENCE_ONLY_MESSAGE_FIELDS (agent/message_metadata.py) is now the single
set of local-only fields: timestamp, display_kind, display_metadata, _row_id.
build_api_messages pops exactly that set and the estimator shadow drops it,
so the two can no longer drift. The iteration-limit summary path
(_iteration_summary_api_messages) hand-builds its wire messages and stripped
timestamp but not display_*; its key set now unions the same constant, so
the inline diff never reaches the provider there either (strict gateways
reject unknown keys). The tail-budget walk (_estimate_msg_budget_tokens) is
allowlist-based and already ignored these fields.
