A wedged Relay pipeline made every compaction block for SCOPE_OP_TIMEOUT
(10s) while it still held the compression lease, with no breaker. The
first stalled mark now trips a process-wide flag and later marks return
immediately; marks are observability and must never tax compaction.
Also: make relay_runtime.SCOPE_OP_TIMEOUT public (relay_compaction reads
it), fold the commit-time telemetry hold into _hold_attempt_telemetry
with one _ATTEMPT_TELEMETRY_FIELDS tuple, and drop the mapping-table
change-detector tests (the ATOF freshness and failure invariants stay).
The 50 ms publisher timeout could expire before its worker started, so
asserting that the worker entered the fake event made a correct timeout
look like a failure under load. Use an event to observe caller completion
and a two-second timeout consistent with the test timing rule.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit b25b6c3b0ce4b933b35d6cb5cee114b65ff097bf)
A stalled Relay event could hold the compression lease while a mark was
published. Use the same timeout as other Relay scope operations so the
agent can continue after a stalled publisher.
A committed proactive prune estimated both full transcripts even when
no Relay scope could receive the mark. Check for a target before doing
that work.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 808eab610aa161875ad07c79947282fe54da0999)
summarizer_provider and summarizer_model carry the resolved summary provider
and model ids as they are, but the module docstring and the docs said every
categorical field comes from a closed set, and the field table did not list
them. A consumer reading that contract would treat them as bounded labels.
List both in the field table as free-form and narrow the closed-set claim to
the other categorical fields.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit ce2020dbf354785d47c3002f431afe60c1046fa3)
The micro-compaction publisher resolved its Relay parent from the
compressor's current session only. After a rotating compaction earlier in
the turn that session is a child with no Relay scope, so the pass the
finalizer runs before the turn ends lost its mark, while the attempt, prune
and provider-native publishers already fall back to the session the turn
started in.
The finalizer now passes that session to the pass, and its mark takes the
same fallback.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 3519c942fad6ba2dce3b4475ca7ff56a02c7410f)
The attempt record now logs empty_transcript when a context engine returns no
messages, but the hermes.compaction failure_class set did not list it, so the
compaction.attempt mark read failure_class: other and lost the reason. Add it
to the attempt-only classes and the documented set.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 0bdf840844f35e95361ae89edcb1264d2767663c)
The Codex route now logs an attempt record when compact_thread() raises
or a required checkpoint refuses the route, and the record reaches the
compaction publisher like every other Codex exit. Pin that each raising
exit produces exactly one compaction.attempt mark (provider_native,
failed, failure_class exception) without the error text, so a later
change to the Codex route cannot drop these marks again.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 3a3b6b0bfeda336c0f1f118f8ff26eced9d67228)
overflow_reason is a v1 field, but it was always null: both overflow
callers passed only trigger="overflow", although each already holds the
error classifier's verdict for the provider rejection it recovers from.
A consumer could not tell a 413 from a context-length rejection or the
Anthropic long-context tier gate.
Both callers now pass the classified reason through _compress_context
into the attempt seed, so the attempt log line and the compaction mark
carry it. The mark keeps it in a closed set (context_overflow,
payload_too_large, long_context_tier, else other) and leaves it null for
every other trigger. The Relay docs list the set instead of calling the
field reserved.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit c3ad6f7fb8bfcc8f9c350f3a4272029d63333c7f)
With compression.in_place: false every commit moves the agent to a
child session, and that child has no Relay scope until a Relay-managed
LLM call or the next turn opens one. A multi-pass preflight rotates
twice before any LLM call, so the second attempt's record named a
session with neither a live turn nor a session scope, and its mark was
dropped at DEBUG.
The publisher now passes the session the agent's turn started in
(_inflight_turn_session_id, which the turn start already records because
compression can rotate mid-turn). _resolve_target keeps the same session's
live turn and then its session scope first; only when both are missing
does it fall back to that turn, and only while it is the live,
instrumented turn of that session. A child that already has a scope
keeps the mark there, because that scope owns the child's next LLM
starts and so must receive the freshness reset. The record's session_id
still names the child.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit af5c1c9bfde91105e88535a9235ceca6bd662887)
The Compaction Marks section lists the history and request rewrites
that emit no hermes.compaction mark, but missed one: when a gateway turn
ends with compression_exhausted, the gateway replaces the session with a
new, empty one through reset_session (gateway/run_turn.py,
_hmwa_compression_exhaustion_reset) instead of compacting it. Nothing
on that path calls compress_context or the other mark emitters, so
readers should not expect a mark for the reset.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit ded1094b77585eb9b4af910881cc98226de7f809)
The two tests assert outcome failed (summary_generation_aborted is the
class, not the outcome), but their names and the ATOF module docstring
said aborted, which is a different v1 outcome.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit be76bc7f4ffad4b0bb580ad932979f7773056236)
Review of the first cut of the marks found four places where the record
said something the attempt did not:
- A split that failed after it had stored the compacted history
(failed_not_indexed without a rollback) returns the compacted
transcript, so the model sees it next, but the record read
aborted/failed and was named compaction.attempt. Relay then never
reset freshness and the compacted history was never recorded. The
commit path now logs history_rewritten (whether the returned
transcript replaced the input), and such a record is a committed
compaction with failure_class session_split_failed and split_status
failed_not_indexed.
- in_place and session_rotated read false/false on a commit without a
session database. They are now set only when split_status is
in_place_committed or rotated_committed, and null otherwise.
- trigger_class read auto for a record with no trigger or an unlisted
one. It now reads unknown there; an unlabelled automatic caller
(trigger_source auto) keeps trigger unknown with trigger_class auto.
- The v1 sets listed values Hermes never emits (kind request_truncation
and window_reduction, scope request, trigger recovery, outcome noop,
token_count_method provider_usage, split_status pending and unknown).
Consumers treated them as live. They are dropped from the v1 sets and
documented as reserved for later versions.
session_ownership_unreadable joins the attempt-only failure classes.
The Compaction Marks docs now list the four sources exactly, what does
not emit a mark yet (OpenAI Responses native compaction and its
pre-checkpoint pruning, image shrinking after a provider rejection,
request-only truncation), the closed sets and null policy, why a Codex
app-server compaction keeps the compaction name (Relay records no LLM
start for those turns, so the freshness reset only affects a later
Hermes-side call) and is never marked twice, and two known gaps around
rotating compactions.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 6e6bc81fc0e074c88d1d632e8110d7490e203015)
A second concurrent turn on a live session (a background-review fork or
a concurrent gateway turn) runs with Relay instrumentation off, but its
compaction mark fell back to the live session's scope. The mark claimed
the live conversation was compacted when only the fork's throwaway
transcript was, and the reserved compaction name reset the live agent
scope's freshness. That also bypassed the opt-out resolve_execution_context
honors.
_resolve_target now returns nothing when relay_instrumentation_enabled()
is false. Publishing also skips persistence-detached agents
(_persist_disabled, set on background-review forks), as micro-compaction
already does in the turn finalizer.
The proactive prune mark moves from ContextCompressor to its caller in
turn_preflight, the one place that knows the agent. That lets it skip a
fork, name the agent's session (a compressor without a bound session id
used to drop the mark), and measure tokens with the same rough message
estimate the attempt record uses.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 37e177f1494570303f4a010534273a64760b30d3)
Hermes rewrites a session's history in several ways (batch summary,
deterministic fallback, micro-compaction, proactive tool-result prune,
Codex thread compaction) but never told Relay about any of them. A Relay
consumer could not see what was compacted, why, when, how long it took
or whether it went through compress_context, and Relay never reset
LLM-history freshness after a rewrite, so LLM starts after a compaction
stayed projected to the current user turn.
agent/compaction_events.py builds one flat, content-free
hermes.compaction v1 record per rewrite with closed enums (kind, scope,
official, method, trigger, trigger_class, outcome, failure_class) and an
other fallback, plus the effect counts and timings the attempt record
already carries. agent/relay_compaction.py emits it with the existing
scope.event API on the owning session's scope stack: under the live turn
of the same session, else under the session scope, else not at all.
A committed history or provider rewrite uses Relay's reserved
compaction mark name, which resets freshness for the session's agent
scope; every other outcome is a compaction.attempt mark. Records come
from the single attempt telemetry seam (so blocked, Codex-routed and
aborted attempts are covered), the micro-compaction telemetry record,
the proactive prune commit and Codex-native compactions Hermes observes.
No payload carries message, summary, focus-topic, error or path text,
and publishing never raises into compaction.
The marks are on whenever the Relay runtime is live; Relay exporters are
configured separately, so this adds no outbound path and no config key.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit dc5fd0875e7ec895b7826ee829bfc1b7f082de97)
Finalize effect counts from the transcript that crosses the commit boundary, including retained partial-compression tails and boundary anchors. Explicitly discard bypassed shared-metric state so a later refusal cannot consume stale attempt metadata.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit ddf3b229a9e2fdee651f36afbec974717d334f76)
The candidate guard refused an empty transcript from a context engine and
returned without an attempt record, so that compress_context call logged
nothing and the shared metric never counted it. Every other refusal in the
guard logs its own class.
Log it as an aborted attempt with failure_class empty_transcript; the shared
metric counts it once as failed under its closed-set "other" class.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 3f6a58fb3f2695a58d67fedb9f7410abff95ad58)
The attempt record read its identity from the agent's current-attempt fields,
which the next attempt overwrites when it begins. A stalled attempt that
unwound after its stall fallback had started therefore logged the fallback's
attempt_id, session_id and trigger_source, so two records claimed one attempt
and the stalled one disappeared.
Each attempt now carries its own seed on _Attempt and passes it to every
emit, so the record always describes the attempt that wrote it. Emits with no
attempt in hand (pool saturation) keep reading the agent's copy.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit a36db44d63888406702cab44d3ca29335727a6e2)
Every returning exit of the Codex route logs one attempt record, but
two exits raised after the attempt began and logged nothing: an
exception from compact_thread() only stopped the heartbeat and
re-raised, and compress_context raised CompressionCheckpointUnavailable
for a Codex-owned thread with compression.checkpoint_required set. A
consumer counting attempts saw neither.
Both exits now log one codex_app_server record, aborted with
exception:<Type>, the same class shape the Hermes route records when its
summary phase raises. The record carries the type name only, never the
error text, and the caller still receives the original exception object.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 17e55b05801624ea16f034b31e2408d386ee19f0)
#118580's regression test covers an in-place commit. A rotating commit
runs the same memory flush and then switches the agent to the child
session, so its record could lose the attempt's fields or name the child.
Drive both commit modes through a real AIAgent with a session database
and assert the record keeps this attempt's trigger, method, effect and
regions, and names the session the attempt started in. Both cases fail
without the telemetry hold around commit_memory_session.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit be2006f29a758e9f4f7c380396a997b6377bef02)
The attempt record learns trigger_source only from the trigger keyword
each caller passes to _compress_context. The existing test checked only
the shared-metric bucket mapping, so dropping a label from any of the
seven call sites passed every test.
Drive each real caller with an agent double that records the keyword:
idle, turn-start threshold, engine preflight, pre-API and post-tool in
the turn loop, and both gateway hygiene paths (the live Codex agent and
the detached hygiene agent). Removing any one label turns exactly its
case red.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 93604a6eb31b741605acedfa38ce21770f4cb187)
_adopt_if_parent_rotated returns the same sit-out result for two
different exits: the lease found the parent already rotated by another
path (ownership lost), and the ownership lookup itself raised
("skipping compression this cycle", ownership unknown). Both were logged
as session_ownership_lost.
Each exit now writes its own record from inside the helper: a rotated
parent is skipped / session_ownership_lost, and a failed lookup is
aborted / session_ownership_unreadable, the same shape as the existing
cooldown_state_unreadable exit. Both records name the session the
attempt started in, not the child an adoption switched the agent to.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 83dfe1eaf6e630198734a65d8c13a26945ff43d9)
An attempt record that is written before compress() seeds this
attempt's telemetry, or after a pre-commit restore, did not describe
that attempt:
- snapshot_stale and the early commit_fence_cancelled exit run after the
attempt-start clear, so the record had no trigger_source;
- a commit fence refused after the summary restores the compressor's
pre-attempt state, which puts the previous attempt's dict back, so the
record carried the previous attempt_id, trigger, method and reclaimed
tokens with this attempt's failure class;
- pool_saturated fires before any attempt begins, but still took the
previous attempt_id from the agent.
_begin_compression_attempt now keeps the attempt seed (attempt_id, the
session the attempt started in, trigger) on the agent as well, since a
restore replaces the compressor's copy. When the compressor's telemetry
is not this attempt's own, the emitter starts the record from that seed
with method none and no token counts, and ignores the compressor's
fallback flags, which belong to the same earlier attempt. A refusal with
no attempt begun gets a fresh id and trigger unknown. Bypassed exits read
the same seed, so they also name the session the attempt started in
rather than a child it adopted.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit c29dba73a4c07c833951666f517a237e1e0fc65d)
The content-free attempt record is the one place that says why a
compaction ran and what it did, but it could not answer either question:
- Every automatic caller passed no trigger, so idle, turn-start
threshold, engine preflight, pre-API, post-tool and gateway hygiene
attempts all logged trigger_source=auto. Each call site now passes its
own label; the shared metric still buckets them as auto.
- tokens after, message counts and how the summary was made were
computed and logged as prose but never reached the record. A finished
rewrite now records messages_before/after, tokens_before/after,
tokens_reclaimed (signed; a summary can grow a short transcript),
token_count_method=estimate_rough, method (llm_summary,
aux_fallback_main, deterministic_fallback), items_dropped, and the
tool_results_pruned / reasoning_items_pruned pass counts.
- Attempts that never reached the local compressor returned silently:
the automatic breaker gate (before and under the lease), every Codex
app-server exit, a lease that found the session already rotated, and
an unreadable durable cooldown. Each now logs one record with a
commit_status (blocked, skipped, failed, aborted, committed), a route,
and a class such as blocked:cooldown. These stay out of shared metrics,
which never counted them.
Test doubles for _compress_context gain the trigger keyword the call
sites now pass.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 1795993d1a01d91dd76e6ec824cb606f8a03b897)
context_compressor.py and conversation_compression.py are both past the
2,000-line cap, so the attempt-telemetry fixes that follow need room in
them. Move code that already has a topical home, unchanged:
- ContextCompressor's per-attempt telemetry methods (_begin_compression_telemetry,
_record_compression_regions, _record_aux_compression_call) and _safe_int
become CompressionTelemetryMixin in agent/context_compressor_telemetry.py.
- The Codex app-server compaction route (_compress_context_via_codex_app_server,
_record_codex_compaction_failure) joins its cooldown lookup in
agent/conversation_compression_codex.py and late-imports the facade's
shared helpers.
No behavior change.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 91d981495f6baee699316d5e843276b32d6a6b9e)
The commit-time hold keeps the telemetry trio for its own attempt's
success emit. Clear it at the start of the next attempt (and keep
pool-saturation refusal emits from inheriting the last attempt's
payload) so a later emit cannot report the previous attempt's numbers.
Follow-up on the #118580 review.
[Ported onto main, where _emit_compression_attempt_telemetry now lives in
agent/conversation_compression_telemetry.py: the include_last_telemetry
parameter moved there with the same behavior.]
(cherry picked from commit 1c05c6a6e713cd77aee3111d5a809572ad3f7d0c)
(cherry picked from commit 5a551af25a643d0d6c1f1ded2ce9d9c4a52276ba)
_commit_compaction calls commit_memory_session before the success emit.
The engine session-end notification answers with a full per-session
reset, nulling _last/_active_compression_telemetry, so the emit fell
back to {} and every committed line carried emitter defaults only.
Closes#118580
(cherry picked from commit b2242e0aa79dbc9ce2995e1a0cf8a88867869801)
(cherry picked from commit 865ace8293ec8138c309a02d8fc18b34f7e6fffb)
Restore the pre-compression transcript when a context engine raises. Evaluate growth against the immutable input snapshot and defer memory publication until the candidate passes every refusal gate, so in-place mutation cannot bypass refusal or leak side effects.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 969371c564c172046934b535b3488610171726de)
Context engines may mutate the message list they receive, and the
context-engine plugin guide says that mutation only becomes visible if
the pass commits. Callers that pass their own commit fence (gateway
session hygiene) or set compression.context_timeout_seconds to 0 hand
the live list to the engine without the facade's deep snapshot.
compress_context restored the pre-compression snapshot on the
no-progress and superseded refusals, but the compressor-abort,
empty-transcript and would-grow refusals returned the live list as the
engine left it. An engine that cleared its input and returned [] made
compress_context return an empty transcript while warning that the
conversation continues unchanged.
Restore the snapshot in compress_context whenever _candidate_rejected
refuses a candidate or the commit refuses a grown one, so every refusal
returns the original transcript. The no-progress and superseded
branches no longer restore on their own because the caller now does it
for every refusal. Warnings, telemetry and the commit path are
unchanged.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit d14fc7c96031d15f85acb00d433bcfdd39804554)
Teknium's 9198add5 mapped 'contributors/emails/ronny.pietschke@icloud.com',
but that is not the commit-author email on any of my commits (git log
--format='%ae' in iowap-org/iowap-hermes-integration shows ronny@kesuek.de
exclusively). The 20bc0333 sweep already added the correct file; this
removes the stale duplicate. Content unchanged: Kesuek.
