A wedged Relay pipeline made every compaction block for SCOPE_OP_TIMEOUT
(10s) while it still held the compression lease, with no breaker. The
first stalled mark now trips a process-wide flag and later marks return
immediately; marks are observability and must never tax compaction.
Also: make relay_runtime.SCOPE_OP_TIMEOUT public (relay_compaction reads
it), fold the commit-time telemetry hold into _hold_attempt_telemetry
with one _ATTEMPT_TELEMETRY_FIELDS tuple, and drop the mapping-table
change-detector tests (the ATOF freshness and failure invariants stay).
A stalled Relay event could hold the compression lease while a mark was
published. Use the same timeout as other Relay scope operations so the
agent can continue after a stalled publisher.
A committed proactive prune estimated both full transcripts even when
no Relay scope could receive the mark. Check for a target before doing
that work.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 808eab610aa161875ad07c79947282fe54da0999)
summarizer_provider and summarizer_model carry the resolved summary provider
and model ids as they are, but the module docstring and the docs said every
categorical field comes from a closed set, and the field table did not list
them. A consumer reading that contract would treat them as bounded labels.
List both in the field table as free-form and narrow the closed-set claim to
the other categorical fields.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit ce2020dbf354785d47c3002f431afe60c1046fa3)
The micro-compaction publisher resolved its Relay parent from the
compressor's current session only. After a rotating compaction earlier in
the turn that session is a child with no Relay scope, so the pass the
finalizer runs before the turn ends lost its mark, while the attempt, prune
and provider-native publishers already fall back to the session the turn
started in.
The finalizer now passes that session to the pass, and its mark takes the
same fallback.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 3519c942fad6ba2dce3b4475ca7ff56a02c7410f)
The attempt record now logs empty_transcript when a context engine returns no
messages, but the hermes.compaction failure_class set did not list it, so the
compaction.attempt mark read failure_class: other and lost the reason. Add it
to the attempt-only classes and the documented set.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 0bdf840844f35e95361ae89edcb1264d2767663c)
overflow_reason is a v1 field, but it was always null: both overflow
callers passed only trigger="overflow", although each already holds the
error classifier's verdict for the provider rejection it recovers from.
A consumer could not tell a 413 from a context-length rejection or the
Anthropic long-context tier gate.
Both callers now pass the classified reason through _compress_context
into the attempt seed, so the attempt log line and the compaction mark
carry it. The mark keeps it in a closed set (context_overflow,
payload_too_large, long_context_tier, else other) and leaves it null for
every other trigger. The Relay docs list the set instead of calling the
field reserved.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit c3ad6f7fb8bfcc8f9c350f3a4272029d63333c7f)
With compression.in_place: false every commit moves the agent to a
child session, and that child has no Relay scope until a Relay-managed
LLM call or the next turn opens one. A multi-pass preflight rotates
twice before any LLM call, so the second attempt's record named a
session with neither a live turn nor a session scope, and its mark was
dropped at DEBUG.
The publisher now passes the session the agent's turn started in
(_inflight_turn_session_id, which the turn start already records because
compression can rotate mid-turn). _resolve_target keeps the same session's
live turn and then its session scope first; only when both are missing
does it fall back to that turn, and only while it is the live,
instrumented turn of that session. A child that already has a scope
keeps the mark there, because that scope owns the child's next LLM
starts and so must receive the freshness reset. The record's session_id
still names the child.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit af5c1c9bfde91105e88535a9235ceca6bd662887)
Review of the first cut of the marks found four places where the record
said something the attempt did not:
- A split that failed after it had stored the compacted history
(failed_not_indexed without a rollback) returns the compacted
transcript, so the model sees it next, but the record read
aborted/failed and was named compaction.attempt. Relay then never
reset freshness and the compacted history was never recorded. The
commit path now logs history_rewritten (whether the returned
transcript replaced the input), and such a record is a committed
compaction with failure_class session_split_failed and split_status
failed_not_indexed.
- in_place and session_rotated read false/false on a commit without a
session database. They are now set only when split_status is
in_place_committed or rotated_committed, and null otherwise.
- trigger_class read auto for a record with no trigger or an unlisted
one. It now reads unknown there; an unlabelled automatic caller
(trigger_source auto) keeps trigger unknown with trigger_class auto.
- The v1 sets listed values Hermes never emits (kind request_truncation
and window_reduction, scope request, trigger recovery, outcome noop,
token_count_method provider_usage, split_status pending and unknown).
Consumers treated them as live. They are dropped from the v1 sets and
documented as reserved for later versions.
session_ownership_unreadable joins the attempt-only failure classes.
The Compaction Marks docs now list the four sources exactly, what does
not emit a mark yet (OpenAI Responses native compaction and its
pre-checkpoint pruning, image shrinking after a provider rejection,
request-only truncation), the closed sets and null policy, why a Codex
app-server compaction keeps the compaction name (Relay records no LLM
start for those turns, so the freshness reset only affects a later
Hermes-side call) and is never marked twice, and two known gaps around
rotating compactions.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 6e6bc81fc0e074c88d1d632e8110d7490e203015)
A second concurrent turn on a live session (a background-review fork or
a concurrent gateway turn) runs with Relay instrumentation off, but its
compaction mark fell back to the live session's scope. The mark claimed
the live conversation was compacted when only the fork's throwaway
transcript was, and the reserved compaction name reset the live agent
scope's freshness. That also bypassed the opt-out resolve_execution_context
honors.
_resolve_target now returns nothing when relay_instrumentation_enabled()
is false. Publishing also skips persistence-detached agents
(_persist_disabled, set on background-review forks), as micro-compaction
already does in the turn finalizer.
The proactive prune mark moves from ContextCompressor to its caller in
turn_preflight, the one place that knows the agent. That lets it skip a
fork, name the agent's session (a compressor without a bound session id
used to drop the mark), and measure tokens with the same rough message
estimate the attempt record uses.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 37e177f1494570303f4a010534273a64760b30d3)
Hermes rewrites a session's history in several ways (batch summary,
deterministic fallback, micro-compaction, proactive tool-result prune,
Codex thread compaction) but never told Relay about any of them. A Relay
consumer could not see what was compacted, why, when, how long it took
or whether it went through compress_context, and Relay never reset
LLM-history freshness after a rewrite, so LLM starts after a compaction
stayed projected to the current user turn.
agent/compaction_events.py builds one flat, content-free
hermes.compaction v1 record per rewrite with closed enums (kind, scope,
official, method, trigger, trigger_class, outcome, failure_class) and an
other fallback, plus the effect counts and timings the attempt record
already carries. agent/relay_compaction.py emits it with the existing
scope.event API on the owning session's scope stack: under the live turn
of the same session, else under the session scope, else not at all.
A committed history or provider rewrite uses Relay's reserved
compaction mark name, which resets freshness for the session's agent
scope; every other outcome is a compaction.attempt mark. Records come
from the single attempt telemetry seam (so blocked, Codex-routed and
aborted attempts are covered), the micro-compaction telemetry record,
the proactive prune commit and Codex-native compactions Hermes observes.
No payload carries message, summary, focus-topic, error or path text,
and publishing never raises into compaction.
The marks are on whenever the Relay runtime is live; Relay exporters are
configured separately, so this adds no outbound path and no config key.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit dc5fd0875e7ec895b7826ee829bfc1b7f082de97)
Finalize effect counts from the transcript that crosses the commit boundary, including retained partial-compression tails and boundary anchors. Explicitly discard bypassed shared-metric state so a later refusal cannot consume stale attempt metadata.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit ddf3b229a9e2fdee651f36afbec974717d334f76)
The candidate guard refused an empty transcript from a context engine and
returned without an attempt record, so that compress_context call logged
nothing and the shared metric never counted it. Every other refusal in the
guard logs its own class.
Log it as an aborted attempt with failure_class empty_transcript; the shared
metric counts it once as failed under its closed-set "other" class.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 3f6a58fb3f2695a58d67fedb9f7410abff95ad58)
The attempt record read its identity from the agent's current-attempt fields,
which the next attempt overwrites when it begins. A stalled attempt that
unwound after its stall fallback had started therefore logged the fallback's
attempt_id, session_id and trigger_source, so two records claimed one attempt
and the stalled one disappeared.
Each attempt now carries its own seed on _Attempt and passes it to every
emit, so the record always describes the attempt that wrote it. Emits with no
attempt in hand (pool saturation) keep reading the agent's copy.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit a36db44d63888406702cab44d3ca29335727a6e2)
Every returning exit of the Codex route logs one attempt record, but
two exits raised after the attempt began and logged nothing: an
exception from compact_thread() only stopped the heartbeat and
re-raised, and compress_context raised CompressionCheckpointUnavailable
for a Codex-owned thread with compression.checkpoint_required set. A
consumer counting attempts saw neither.
Both exits now log one codex_app_server record, aborted with
exception:<Type>, the same class shape the Hermes route records when its
summary phase raises. The record carries the type name only, never the
error text, and the caller still receives the original exception object.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 17e55b05801624ea16f034b31e2408d386ee19f0)
_adopt_if_parent_rotated returns the same sit-out result for two
different exits: the lease found the parent already rotated by another
path (ownership lost), and the ownership lookup itself raised
("skipping compression this cycle", ownership unknown). Both were logged
as session_ownership_lost.
Each exit now writes its own record from inside the helper: a rotated
parent is skipped / session_ownership_lost, and a failed lookup is
aborted / session_ownership_unreadable, the same shape as the existing
cooldown_state_unreadable exit. Both records name the session the
attempt started in, not the child an adoption switched the agent to.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 83dfe1eaf6e630198734a65d8c13a26945ff43d9)
An attempt record that is written before compress() seeds this
attempt's telemetry, or after a pre-commit restore, did not describe
that attempt:
- snapshot_stale and the early commit_fence_cancelled exit run after the
attempt-start clear, so the record had no trigger_source;
- a commit fence refused after the summary restores the compressor's
pre-attempt state, which puts the previous attempt's dict back, so the
record carried the previous attempt_id, trigger, method and reclaimed
tokens with this attempt's failure class;
- pool_saturated fires before any attempt begins, but still took the
previous attempt_id from the agent.
_begin_compression_attempt now keeps the attempt seed (attempt_id, the
session the attempt started in, trigger) on the agent as well, since a
restore replaces the compressor's copy. When the compressor's telemetry
is not this attempt's own, the emitter starts the record from that seed
with method none and no token counts, and ignores the compressor's
fallback flags, which belong to the same earlier attempt. A refusal with
no attempt begun gets a fresh id and trigger unknown. Bypassed exits read
the same seed, so they also name the session the attempt started in
rather than a child it adopted.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit c29dba73a4c07c833951666f517a237e1e0fc65d)
The content-free attempt record is the one place that says why a
compaction ran and what it did, but it could not answer either question:
- Every automatic caller passed no trigger, so idle, turn-start
threshold, engine preflight, pre-API, post-tool and gateway hygiene
attempts all logged trigger_source=auto. Each call site now passes its
own label; the shared metric still buckets them as auto.
- tokens after, message counts and how the summary was made were
computed and logged as prose but never reached the record. A finished
rewrite now records messages_before/after, tokens_before/after,
tokens_reclaimed (signed; a summary can grow a short transcript),
token_count_method=estimate_rough, method (llm_summary,
aux_fallback_main, deterministic_fallback), items_dropped, and the
tool_results_pruned / reasoning_items_pruned pass counts.
- Attempts that never reached the local compressor returned silently:
the automatic breaker gate (before and under the lease), every Codex
app-server exit, a lease that found the session already rotated, and
an unreadable durable cooldown. Each now logs one record with a
commit_status (blocked, skipped, failed, aborted, committed), a route,
and a class such as blocked:cooldown. These stay out of shared metrics,
which never counted them.
Test doubles for _compress_context gain the trigger keyword the call
sites now pass.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 1795993d1a01d91dd76e6ec824cb606f8a03b897)
context_compressor.py and conversation_compression.py are both past the
2,000-line cap, so the attempt-telemetry fixes that follow need room in
them. Move code that already has a topical home, unchanged:
- ContextCompressor's per-attempt telemetry methods (_begin_compression_telemetry,
_record_compression_regions, _record_aux_compression_call) and _safe_int
become CompressionTelemetryMixin in agent/context_compressor_telemetry.py.
- The Codex app-server compaction route (_compress_context_via_codex_app_server,
_record_codex_compaction_failure) joins its cooldown lookup in
agent/conversation_compression_codex.py and late-imports the facade's
shared helpers.
No behavior change.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 91d981495f6baee699316d5e843276b32d6a6b9e)
The commit-time hold keeps the telemetry trio for its own attempt's
success emit. Clear it at the start of the next attempt (and keep
pool-saturation refusal emits from inheriting the last attempt's
payload) so a later emit cannot report the previous attempt's numbers.
Follow-up on the #118580 review.
[Ported onto main, where _emit_compression_attempt_telemetry now lives in
agent/conversation_compression_telemetry.py: the include_last_telemetry
parameter moved there with the same behavior.]
(cherry picked from commit 1c05c6a6e713cd77aee3111d5a809572ad3f7d0c)
(cherry picked from commit 5a551af25a643d0d6c1f1ded2ce9d9c4a52276ba)
_commit_compaction calls commit_memory_session before the success emit.
The engine session-end notification answers with a full per-session
reset, nulling _last/_active_compression_telemetry, so the emit fell
back to {} and every committed line carried emitter defaults only.
Closes#118580
(cherry picked from commit b2242e0aa79dbc9ce2995e1a0cf8a88867869801)
(cherry picked from commit 865ace8293ec8138c309a02d8fc18b34f7e6fffb)
Restore the pre-compression transcript when a context engine raises. Evaluate growth against the immutable input snapshot and defer memory publication until the candidate passes every refusal gate, so in-place mutation cannot bypass refusal or leak side effects.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit 969371c564c172046934b535b3488610171726de)
Context engines may mutate the message list they receive, and the
context-engine plugin guide says that mutation only becomes visible if
the pass commits. Callers that pass their own commit fence (gateway
session hygiene) or set compression.context_timeout_seconds to 0 hand
the live list to the engine without the facade's deep snapshot.
compress_context restored the pre-compression snapshot on the
no-progress and superseded refusals, but the compressor-abort,
empty-transcript and would-grow refusals returned the live list as the
engine left it. An engine that cleared its input and returned [] made
compress_context return an empty transcript while warning that the
conversation continues unchanged.
Restore the snapshot in compress_context whenever _candidate_rejected
refuses a candidate or the commit refuses a grown one, so every refusal
returns the original transcript. The no-progress and superseded
branches no longer restore on their own because the caller now does it
for every refusal. Warnings, telemetry and the commit path are
unchanged.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
(cherry picked from commit d14fc7c96031d15f85acb00d433bcfdd39804554)
Local Ollama exposes cloud models both as "<model>:cloud" and as
"<model>:<size>-cloud" (gpt-oss:120b-cloud). Treat the second form like
the first, requiring a ":" so provider ids such as "gmi-cloud" do not
match. Update the require-side comment and docstrings that still listed
only DeepSeek/Kimi/MiMo or claimed every member 400s without the field.
Ollama Cloud users reported reasoning being sent back wrong. Since #50480
the replay policy strips reasoning_content for every provider outside the
DeepSeek/Kimi/MiMo echo families, so on ollama-cloud an assistant tool-call
turn went back with no reasoning at all. Ollama's /v1 maps the field onto
the chat template's thinking input, so thinking models (kimi-k3, glm-5.3)
lost their own chain of thought between tool calls.
Add an "ollama" echo family (provider ollama-cloud, host ollama.com, or a
local server's ":cloud" model). Purely local Ollama models stay on the
strip side.
#118661's note only fired on the stored-prompt restore path, which a long-lived agent (classic
CLI, TUI, Desktop) runs once per conversation: those surfaces never saw a mid-conversation skill
change. The check now runs from build_turn_context every turn against the prompt the turn
actually sends.
- What was told is kept on the agent for its prompt; a fresh agent (gateway, -q --resume) reads
the newest note back from the WHOLE transcript instead of the last 200 rows, so a long session
no longer re-announces an unchanged delta (review by @kvnloo).
- A compaction rebuild that now lists the announced skills stays quiet.
- Index rows split on ": ", so qualified plugin skills (plugin:skill) parse.
- The plugin go-live note drops its own skills list: the per-turn note covers plugin skills like
any other, so a plugin install no longer announces its skills twice.
The in-process skills index cache was keyed on config alone (dirs, tools, platform, disabled
names). In-process writers clear it (skill_manage, hub install, curator), but a write from
another process never did: `hermes skills install` in a second terminal, a git pull of an
external dir, or the gateway curator writing for a CLI kept every running session on the
pre-change index until restart (#92313). The key now carries each SKILL.md/DESCRIPTION.md's
file signature for every root, the same manifest the disk snapshot already validates against
(~1.6 ms for 168 files on a real 755-skill home), and the snapshot check reuses it instead of
walking the tree twice.
Co-authored-by: everm1nd <everm1nd.mail@gmail.com>
Co-authored-by: leon7609 <leon7609@users.noreply.github.com>
Co-authored-by: yellowdog01 <yellowdog01@users.noreply.github.com>
The persisted <available_skills> index is reused byte-for-byte across turns so
the provider prefix cache stays warm (#104414). A skill that lands after the
prompt was built (hub install, skill_manage from another session, org sync,
curator) therefore never reached the index the model actually routes on until
compaction rebuilt the prompt: skills_list is live, but the prompt tells the
model to scan the index, not to call it. cloudflare-os#267 found the same fault
in their chat-cached catalog ("a catalog states what a session can reach now,
which is not a fact a chat can hold").
Hermes' idiom for stored-prompt drift is a one-shot note behind the cached
prefix, not a rebuild, so agent/skills_index_delta.py stages the delta on the
same per-turn user-message channel the surface-switch note rides: the added
skills' own index lines (name + description, the routing signal), the removed
names, cumulative against the stored prompt and re-staged only when the delta
changes (read back from the api_content sidecar, so the gateway's fresh agent
per turn does not stack copies). An undone delta retires the stale note once.
The current index comes from the same cached builder the prompt did, so an
unchanged turn costs one LRU hit.
The inline-shell trust gate matched only the exact install dir from the
active profile's hub lock, and a lock read failure fell through to
expansion. So a nested SKILL.md inside a --force community bundle, the
same skill viewed by another profile through skills.external_dirs, and
any skill under a truncated or wrong-shape lock all ran their !`cmd`
snippets on view. Match descendants of community install dirs, also read
the lock that owns the skills tree the dir lives in, and fail closed when
an existing lock cannot be read.
#135348 ran ruff's UP037 autofix with ruff.strict.toml's target-version = "py314". On 3.14
annotations are lazy (PEP 649), so stripping quotes is safe there; on 3.11-3.13 a function or
variable annotation is evaluated when the def or statement runs. Every unquoted annotation that
names the enclosing class (`def clone_for_agent(self) -> ContextEngine`), a class bound later in
the module, or a TYPE_CHECKING-only import now raises NameError on import: 300 modules on main
(gateway.config, run_agent, hermes_state, cron.jobs, the Telegram adapter, ...) fail to import
under 3.11, the update path crashes mid-`hermes update` on any install whose interpreter is
older than 3.14, and the e2e-upgrade suite's retry leg fails with
`NameError: name 'ContextEngine' is not defined`.
- Re-quote exactly the 79 annotations (36 files) that name something unbound when they are
evaluated (including tui_gateway/server.py's `_TurnScopes`, which only reaches that module's
globals through register() after import); no other text changes.
- agent/turn_scripted_prelude.py: `Generator[...]` gets its third argument back. d76cce4f11
applied UP043 (default type args, valid only from 3.13) to a module-level alias, so
`Generator[Y, S]` raises TypeError on 3.11/3.12 and takes gateway.run, agent.conversation_loop
and tui_gateway.{server,ws,entry} down with it.
- ruff.strict.toml target-version = "py311", the requires-python floor. With it UP037 leaves
these quotes alone and only strips the two that are genuinely redundant, which this commit
strips; any future version-gated fix stays valid on every supported Python.
- tests/ci/test_strict_ruff_target.py pins the target to the requires-python floor (red on main).
Without the opt-in, external recall is kept whole up to
max(10 x hooks.output_spill.max_chars, 100k chars) and still spills above
that, so one runaway provider can't overflow the context window and wedge
the session. Normal provider-budgeted recall (~16 KB for Honcho) is
unaffected.
When the announcement carries no call_id and the done item does, the
call_id-exclusive lookup found nothing and left the announcement pending;
response.completed then settled it as a second call. Fall back to the
item/index match only when the matched announcement has no call_id of its
own, so a conflicting call_id still never evicts an unrelated call.
Match pending argument and completion events by output_index when item IDs change. Preserve authoritative arguments and successful output ordering without settling the same call twice.
Verification: independent code review passed; 75 focused tests passed; live Copilot Responses probe passed.
The persist / restore / carry logic for session YOLO had grown three copies
(the CLI mixin, tui_gateway, the gateway runner), each with its own key choice,
ordering and error handling. tools/approval_yolo.py now owns it:
- toggle_session_yolo: flip or set, persist BEFORE the live flip (a racing
restore never revives an OFF), stored-copy-aware so a toggle after a restart
turns it OFF; a failed write logs a warning and never blocks the flip.
- restore_session_yolo: re-arm a persisted flag in a fresh process; skipped
under the frozen process --yolo.
- transfer_session_yolo: move the live flag to a continuation id.
- with_session_yolo: a row being created records a live flag (lazy first row,
compression child, /branch).
Each surface keeps only its storage: the session row for CLI/TUI/Desktop, the
routing entry for the gateway. Removed: HermesCLI._transfer_session_yolo /
_persist_session_yolo, GatewayRunner._restore_session_yolo, the TUI-only
restore helper, and the TUI's post-publish set_session_yolo carry (the
compression publish now writes the child row with the flag).
Closes a gap the unification exposed: a CLI manual /compress that rotated the
session dropped YOLO (the next dangerous command prompted, and a --resume of
the continuation came back with it off).
