A gap below the newest held id, and turns appended above an unpersisted current
turn, were summarized away without being read. Name those held ids and clone
the rest.
A compaction landing during the micro summary call made _sync_micro_compact_to_db
skip its write, but _micro_compact still returned the spliced list, advanced the
cursor and reported "absorbed". finalize_turn's _persist_session flush then
appended the unmarked summary row as live, beside the winning generation.
The commit-time re-check also never fired in the common case: held_archive_watermark
returned early when newest_held >= the start watermark, which is always true when
nothing was appended. A lease-less caller (stale_raises=True) now checks the newest
held row's liveness regardless; the in-place lease path is unchanged.
_sync_micro_compact_to_db now reports a stale abort, and _micro_compact then returns
the original list, restores the rolling summary/cursor (and a defragged marker), and
emits stale_generation telemetry. _micro_start_watermark returns
(skip_outcome, watermark) so the two identical pre-flight skip branches collapse.
Prune and micro-compaction hold no compression lease. When the newest exact row of the history they
hold is no longer active, another compaction (a /compress on this or another surface, or an earlier
pass) has already committed. `held_archive_watermark` then falls back to the start watermark, which is
right for the in-place commit (its lease rules out overlap) but for a lease-less caller archives the
winner's rows and clones them back as a "concurrent tail": two summary generations live in one session.
`_archive_watermark_for` now asks `held_archive_watermark` to raise `StaleHeldHistory` on that branch.
The prune commit returns its input unchanged (`prune:stale_generation`), micro-compaction checks before
the slow summary call and skips the pass (telemetry `stale_generation`), and its commit re-checks so a
compaction that lands during the summary call is not overwritten either. The in-place commit keeps its
fallback.
Raised on #120634 by @ehz0ah (agent/conversation_compression.py:3623 thread) and independently by
@JoaoMarcos44 in #120821.
Co-authored-by: ehz0ah <haozhe4547@gmail.com>
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
_micro_start_watermark collapsed two different answers into None: "this
store has no watermark API", where the commit keeps today's archive-
everything behaviour as it always has, and "the read raised". None means
archive every active row, so a transient store failure committed exactly
the unbounded archive this watermark was added to prevent — a turn another
surface appended vanished from active model history when the read failed
and the write then succeeded.
The helper now returns (status, watermark), and a failed read skips the
pass rather than committing without a bound. The proactive prune path was
already safe: its watermark is resolved inside the try that wraps the
commit, so a raised read aborts the commit instead of widening it.
The new test drives a real SessionDB whose watermark read raises: the pass
must not run and the foreign turn must stay live. Red without the guard.
(cherry picked from commit dd5179e3c72552e6e7737575cc305459728fc549)
Both commits call archive_and_compact with no watermark, which archives
every active row. A turn another surface appended to the same session
after this process loaded it (a Desktop session continued from Telegram)
was archived with the rest as compacted (active=0, compacted=1): marked
summarized away though no summary holds it. The display, REST with
include_compacted and session_search still show it, but it is gone from
the model's history, so the agent forgets a turn the user can still see.
Micro-compaction also runs a slow aux summary call before its commit, so
a turn that arrived during it went the same way.
Both now cap the archive at the newest row the process held, the rule the
in-place compaction commit applies, so those rows take the concurrent-
append path and are cloned after the new set. Micro-compaction captures
the held history and the store's watermark before the summary call and
caps at commit. The cap moves into held_archive_watermark(session_db,
session_id, ...), which the compressor paths can call; _held_watermark
stays as the in-place commit's wrapper.
A store without get_active_message_watermark or get_message_role keeps
today's archive-everything commit, as a store without archive_and_compact
already skips the prune. Micro-compaction's DB sync runs under a broad
except, so an unguarded call there would have failed silently.
Held lists without row ids (the gateway's replay dicts) keep today's
behaviour, as in the in-place commit. Both features are opt-in
(compression.proactive_prune_tokens > 0, compression.micro_compact).
(cherry picked from commit 8d67bee760db0fc65d4330df15198ddfd5f24480)
Superseding a stale micro marker joins the now-adjacent user turns into one
model-facing row, while the originals stay in display history as compacted
rows, so resumed display history painted every merged input twice (and one
more time per later pass). Flag the join display_metadata.model_only and skip
it in every display projection: resume dedupe, the indexed and legacy
get_messages pages, and the prompt timeline. The model payload is unchanged.
Micro-compaction carries a non-contiguous prefix and suffix around its summary marker. Using tail_count=len(result)-1 incorrectly marked summarized assistant/tool rows as rewind-only active=0, compacted=0.
Pass the exact unchanged carried messages instead and resolve their durable originals transactionally by row id or unique identity+timestamp, preserving summarized rows as compacted history.
Fixes#118481
The compound refusal check `_response_refusal_text(response) or
_is_summary_refusal(content)` was spelled out verbatim at both call sites
(agent/context_compressor.py:3639 in `_call_summary_llm` and
agent/micro_compaction.py:157 in `_micro_summarize_one`, the latter through
two separate `_cc()` facade lookups). `_classify_summary_failure` already
describes it as one concept ("refusal content (prose or provider `refusal`
field)"), and the ordering invariant — an explicit provider `refusal` field
wins over summary-shaped content — lived in two callers instead of one place.
Add `_is_refusal_response(response, content)` next to the detectors and call
it from both sites (micro: one `_cc()` lookup). `_is_summary_refusal` and
`_response_refusal_text` stay as-is for the unit tests. Behaviour unchanged.
Proof: guard + micro test files green (62 passed); mutating the new predicate
to `return False` fails exactly the batch refusal-body test, the provider
refusal-field test and the micro refusal test (3 failed, 59 passed).
OpenAI-style structured-output refusals arrive in
`choices[0].message.refusal` with `content` empty or filler, so the
prose detector never sees the refusal and the filler would be committed
as the compaction checkpoint. Read that field (str, or the dict shapes
some proxies emit) in both the batch (`_call_summary_llm`) and micro
(`_micro_summarize_one`) paths and route it through the same
"refusal content" failure so it can never become `_previous_summary`.
Also widen the prose opener with #118406's extra shapes — `we` as
subject, `refuse to`, `could not` / `couldn't`, `apologise`, and a bare
`I'm unable` / `I am not able` opener. #118406's `could(?:not|['’]t)`
missed "couldn't" (it only matched "couldnot"/"could't"); spelled out
here as `could\s*not|couldn['’]t`.
Co-authored-by: Mohamad Kanso <91088196+MohamadKanso@users.noreply.github.com>
The refusal guard in _micro_summarize_one already returns None, which
routes through _record_micro_failure like every other failed
summarization. Snapshotting and restoring _micro_compact_cursor around
that path changed the behaviour of EVERY summarize_failed outcome, not
just refusals, to satisfy one test's cursor assertion. The cursor write
in _resolve_compact_cursor is an idempotent re-resolution, so there is
nothing to undo.
The test now resolves the cursor via _next_exchange before snapshotting
it, so it still proves the source exchange survives a refusal without
needing production code to rewind state.
The _db_persisted marker asserts that a message dict's persisted row is
durable as written; any in-place mutation must pop it or the flush scan
identity-skips the dict and state.db keeps the stale row forever. Six
mutation sites violated the contract:
- micro_compaction._merge_adjacent_user_turns rewrote content on a
carried-forward dict after superseding a stale micro marker. On the
archive-failure path nothing re-stamps, so the merged text never
reached state.db.
- repair_message_sequence passes mutated stamped survivors in place:
_merge_assistant_into (tool_calls union, content join,
reasoning_content carry), _prune_unanswered_tool_calls (tool_calls
rewrite), _merge_consecutive_users (content join).
- sanitize_tool_call_arguments rewrote corrupted/blank
function.arguments and prepended the corruption marker onto stamped
resumed rows, leaving the corrupt bytes durable and self-perpetuating
across resumes.
- _sanitize_messages (surrogate and non-ASCII recovery) and
_strip_images_from_messages (image-rejection recovery) rewrote live
dicts on the recovery path.
Each site now pops the marker when a persisted field actually changes,
and the agent-aware callers (repair_message_sequence_with_cursor,
turn_iteration_prep, turn_recovery) invalidate the bounded flush-scan
prefix so repaired rows are rewritten on the next flush.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
The 15 rolling micro-compaction methods (cursor resolution, exchange
serialization, one-exchange summarization, defrag, splice, DB sync, telemetry)
move verbatim into MicroCompactionMixin, mixed into ContextCompressor
(MRO: ContextCompressor -> MicroCompactionMixin -> ContextEngine). Origin
constants and call_llm are imported lazily so tests patching
agent.context_compressor.X still apply; logger name parity is kept.
