When the model answers beside a housekeeping tool call (memory, todo_list,
skill_manage, session_search) and the follow-up is empty, the turn ends on
that pre-tool answer (fallback_prior_turn_content, since #10472). The final
then repeats text the client already streamed before the tool, but nothing on
the wire says so: message.complete looks like a new response.
Record the reuse where it happens (the housekeeping fallback and the
budget-exhaustion reuse of a previewed verification candidate), report
response_reused only when the final still equals that text (a footer or
explanation appended later drops it), and carry it on message.complete.
Never inferred from equal text: "Done." / tool / "Done." are two responses.
The #131740 residual: a Desktop turn wedged between the 'Turn ended' log
line and run_conversation returning. The liveness watchdog fired ~600s
later naming 'API call #8 completed' as the last activity - a stamp from
the last provider response - so the stall surface read like a hung API
call, nothing pinned the blocking frame, and the abort hit a turn whose
loop had already finished.
- finalize_turn stamps the activity clock at 'Turn ended'
(_log_turn_exit): a stall in the post-loop tail is measured from when
the loop actually ended, and the watchdog's 'last activity' names the
finalizer instead of a stale API call. Guarded: a broken
activity-clock seam never raises into the finalizer.
- TurnLivenessWatchdog._surface_stall dumps every thread's stack (turn
thread marked, 40-thread cap) at the moment it fires, so the wedged
frame lands in errors.log on the next occurrence without py-spy.
Refs #131740
Five opt-in counters that tune the agent loop itself, recorded through
record_process_mark (disabled config => zero rows, zero files):
- hermes.file_edit.count: tool, mode, outcome, match_strategy. The fuzzy
matcher reports the strategy that landed (or no_match / ambiguous) into a
context-local probe opened only around the patch/write_file handlers, so we
learn which strategies earn their keep without counting internal callers.
- hermes.loop_guard.count: provider, model, signal, detector. Hooked where
the guardrail already warns/blocks/halts and at turn end for the iteration
budget; latched once per turn per (signal, detector).
- hermes.tool_recovery.count: provider, model, tool, next_tool,
next_outcome. One row per failed tool call, resolved against the model's
next round (same tool first) or the turn end (no_tool_call / gave_up).
- hermes.terminal.outcome.count: backend, command_kind, outcome. The kind is
a table lookup on the first program word, never the text. Hermes' own
deadline/interrupt now set hermes_timed_out / hermes_interrupted on the
backend result, so a command's own `exit 124` reads nonzero, not timeout.
Disjoint dims from hermes.execution_backend.count (served vs not).
- hermes.model_reply_issue.count: provider, model, issue. Refusal and
truncation only from the structured finish reason; empty/reasoning_only
from the normalized message; issue=none per response as the denominator
(model_route counts attempts and files empty replies as failures).
Background review / curator loops and _host_local / unmetered terminal
calls never count. Helpers live in shared_metrics_harness.py; call sites
are one or two lines each.
The summary-interrupt path added a second copy of the string the loop's
interrupt path builds; turn_explainers matches its prefix, so both now
share interrupted_during_api_call_reason().
An InterruptedError from the now-interruptible max-iteration summary was
swallowed by handle_max_iterations' broad except and delivered as the
max_iterations_no_summary fallback with interrupted=False, so finalize_turn
cleared the pending interrupt message and CLI/gateway had nothing to requeue.
Propagate the cancellation, drop the unanswered summary nudge, and surface it
in the finalizer exactly like an interrupted loop API call: interrupted=True,
interrupted_during_api_call exit reason, INTERRUPT_WAITING_FOR_MODEL_PREFIX
text, and result["interrupt_message"] preserved.
skill_view, read_file and the computer_use screenshot dedup answer a
repeat of unchanged content with an "unchanged" stub that points at the
earlier result. Compaction and a committed proactive prune already
advance all three to a fresh generation through
_reset_read_dedup_caches, so the first repeat after the boundary serves
content again (#32106, #84857, #112763).
Two rewrites never reached that reset:
- post-turn micro-compaction splices older exchanges into the rolling
summary (_micro_compact_after_turn), taking their tool results with
them;
- a replayable native Responses compaction checkpoint makes the next
request drop every item before it (build_assistant_message).
After either, a repeat read got a stub for a body the model could no
longer see. Both now call _reset_read_dedup_caches for the turn's task
and session. Micro-compaction resets only on a real splice: no-op and
defrag passes return the input list and leave the caches alone. The
checkpoint reset no longer depends on the context engine exposing
note_native_compaction_checkpoint, and it is skipped without a task id,
where the helper would clear every task's caches.
No message, tool schema or system-prompt bytes change.
(cherry picked from commit b7c8ecbbc20cd95e70003dce8120b02dbd2ab581)
A turn that falls out of the loop after a tool result with no follow-up
assistant text left the durable transcript ending at a raw tool row and
returned a silent result: Desktop/TUI showed a ready composer (or kept
spinning) with no final message. The pending_tool_result explainer copy
already existed but nothing minted the exit reason.
finalize_turn now detects the non-interrupted tool tail, fails the turn
with turn_exit_reason=pending_tool_result, and synthesizes the visible
assistant close before persistence so the durable tail is alternation-
safe. Stream-recovered turns (#95514) and interrupted tails keep their
existing paths.
Fixes#55316Fixes#54756
Co-authored-by: blakehermes9 <blakehermes9@users.noreply.github.com>
Fork turns (background review, side questions) reuse the parent's session_id
and, when unrouted, the model, so their `conversation turn:` and `Turn ended:`
lines are indistinguishable from live turns. In #118693 this made the fork's
own superseded exit read as a killed foreground stream.
- build_cache_parity_fork stamps _turn_origin on fork agents
- _log_turn_exit and the conversation-turn line append origin=<write_origin>
for fork turns only; live turns are byte-identical
- new invariant test: cancel_background_review_for_live_turn flags only the
fork (parent instance flags and thread bit untouched) via the real
InterruptControlMixin path
- new regression test: fork turn-exit lines carry origin=background_review
(red on base)
Related to #118693
Co-authored-by: CommandCodeBot <noreply@commandcode.ai>
A plugin's transform_llm_output replacement reached final_response only.
agent/turn_finalizer.py::finalize_turn fired the hook after the assistant
row had already been persisted — and the row is first written even earlier,
in agent/turn_final_response.py::finish_text_response's durable flush — so
messages[-1], the SQLite/JSON session, /resume and the next turn's replay all
kept the raw model text while the user had seen the rewritten one.
Writing the transformed text back after that flush cannot work: SQLite treats
a non-blank assistant row as settled (resolve_and_repair_transcript_batch
adopts the stored content instead of overwriting it), so the only correct seam
is BEFORE the row is first persisted. apply_llm_output_transform (new, in
turn_finalizer) fires the hook once per turn_id and records the outcome;
finish_text_response calls it ahead of append+flush and writes the result into
the row (api_content for the promoted-reasoning sidecar), finalize_turn's
_persist_step calls it ahead of the recovery-path tail close, and
_apply_output_hooks reads the recorded outcome (firing only when no earlier
seam saw a response) before post_llm_call. Only the current turn's not-yet-
written text changes — earlier turns and the system prompt are untouched.
post_llm_call is unchanged: it is an observer whose return is ignored by
contract, so there is nothing of it to persist (#14913/#44253's premise).
Fixes#44239
Slim redo of #44244 (AIalliAI, earliest; same sync-then-persist idea, moved to
the pre-flush seam) — also supersedes #65921 (SingleVirgin, sibling fix).
Co-authored-by: AIalliAI <285906080+AIalliAI@users.noreply.github.com>
(cherry picked from commit 5fe02a0aeecef422a4ffb4ff4385015f6a1528a0)
On a `custom` main route (llama.cpp, Ollama, vLLM, ...) whose
auxiliary.title_generation is not pinned elsewhere, the turn prologue fired the
`response_format: json_schema` title request on a daemon thread at the same
instant as the turn's own streaming request, against the same self-hosted
server. A single-slot server can decode the title grammar/completion into the
main reply: the user then receives `{"title": ...}` as the assistant turn, the
main loop persists it as a genuine assistant row, replays it, and the model
adopts the format (#117296). No Hermes writer routes the aux response into the
transcript; the leaked JSON is the main completion itself.
`maybe_auto_title` now returns the upgrade thread and leaves it UNSTARTED when
`title_upgrade_must_wait_for_turn(main_runtime)`; the prologue parks it on
`agent._deferred_title_upgrade` and `finalize_turn` starts it once the model
has answered. Hosted providers keep the turn-start timing. Usage accounting
(`task='title_generation'`) and `sessions.title` are unchanged.
What: a new opt-in gateway runtime-footer field `served_model` rendered as
`alias → served`. It is populated from the `x-litellm-model-id` response header
(fallback `x-litellm-model-api-base`) that routing proxies send on every chat
completion, captured through an httpx response hook installed on the agent's
OpenAI client (agent/served_model.py, wired in create_openai_client), and from
Hermes' own provider fallback (primary runtime model → active model) when no
header is present. The turn result carries `requested_model` / `served_model`;
gateway/run_turn.py passes them to the footer. Off unless listed in
`display.runtime_footer.fields`; the default field set renders exactly as before.
Why: behind a routing proxy (or during a silent Hermes fallback) every reply
shows the configured alias, so operators cannot see which deployment actually
served a request (#54864). The SDK's parsed objects drop response headers, so
the capture has to sit on the transport.
Squash of the 54 commits on victor-kyriazakos:feat/user-channel-warning-suppression
(PR #112302, head f45c640e55) so the contributor's authorship survives a rebase-merge;
the commits interleave with a cron delivery-ledger rework that the salvage removes in
follow-up commits, so per-commit cherry-picks were not practical.
Adds display.suppress_warning_notifications (global + per-platform, default false):
one resolver (gateway/warning_notifications.py), BasePlatformAdapter.emit_warning /
emit_media_warning / warning_text, a notification_category classification carried
through wakes, queues and persistence, and render/present boundaries for CLI/TUI.
In-process delegate_task children (and cron runs fired from a worker) inherit
the dispatcher worker's HERMES_KANBAN_TASK via os.environ. Three readers still
gated on the bare env var instead of is_dispatcher_owned_worker_context():
- agent/turn_finalizer.py: a child exhausting ITS iteration budget recorded
`timed_out` against the parent's task and released the parent's claim.
- tools/kanban_tools.py::inject_new_comments_from_env: a child polled operator
notes addressed to the worker, steered itself with them, and advanced the
shared per-task watermark so the worker never saw them.
- agent/title_generator.py::_kanban_task_title: a child's session was titled
after the parent's card.
Each now uses the single identity predicate. Dispatcher-owned workers are
unchanged (existing #87096 tests still pass).
Fixes#112817
format_session_db_unavailable consumes the now-pinned cause table but its own
two fallbacks (no recorded cause; network-drive gloss) still printed a bare
`hermes doctor`. Same class; one test covers both fallbacks.
Also: spell the corrupt action as plain concatenation instead of an f-string
with escaped braces, and fix the finalizer comment to state the invariant
(final_response is never rebound) rather than describe a local that no longer
exists.
hermes_state_user_copy.py is the one table feeding the CLI banner, gateway
warning and TUI/Desktop RPC errors, and its action strings were still bare
(`hermes doctor --fix`, `hermes gateway stop`, `hermes sessions recover`).
Substitute {profile_arg} once in describe_storage_failure so those surfaces
get the same pin as the turn explainer.
Also keep final_response unbound in the turn_finalizer error fallback: it
feeds the external-memory sync and the background-review gate, which must
still see an empty response on a persistence-failed turn. One test binds the
fallback (explainer stubbed empty) and the untouched final_response.
gateway/run_notifications.py::_send_session_db_warning_notifications already
computed profile_arg but left `hermes doctor` (fts_index branch) and
`hermes gateway restart` (default branch) bare; agent/turn_finalizer.py had a
bare `hermes doctor` in the error fallback used when the explainer produced
no text. Same class as the previous commit. Also reflows the explainer
strings so continuation lines break at clause boundaries.
The turn-end file-mutation verifier only sees write_file/patch receipts. It
asserted "N file(s) were NOT modified this turn" whenever a call had failed,
which is wrong when the file was in fact changed afterwards through a path
that leaves no receipt (terminal redirect, execute_code) or when the
successful retry used another spelling of the same path (relative vs
absolute, separator/case variants on Windows): the state dict was keyed on
the model's raw `path` argument, so the pop never matched.
- Header now says what the recorder knows: "N file edit(s) FAILED this turn",
and asks the user to confirm what actually landed.
- Failure entries carry the task-resolved, normcase'd on-disk identity plus a
(mtime_ns, size) snapshot; a later success clears every entry with the same
identity regardless of spelling.
- At turn end `_file_mutations_still_failed` re-stats each target and drops
entries whose file changed since the failed call, so a receipt-less
mutation no longer produces a false footer.
- `tool_executor` passes the effective task id so relative paths resolve the
way the file tools resolved them.
Kept the deliberate first-error-per-path semantics (the pinned test says why);
did not add an "unverified" bucket for receipt-less non-error results, since
the built-in tools always return a receipt on success and it would only add
noise.
Co-authored-by: KoNit. <124019182+KoNit-K@users.noreply.github.com>
An interrupted turn left `finalize_turn` with a diagnostic `final_response`
("Operation interrupted: waiting for model response") and no failure, so the
result said `completed=True` — the only producer that did; `turn_recovery`
and `codex_runtime` already return `completed=False` for an interrupt and the
gateway stream gate documents that contract. `completed` now also requires
`not interrupted`.
The API server then hard-coded the terminal status: the session chat stream
emitted `assistant.completed {completed: true, interrupted: false}` and
`run.completed` for every turn that did not raise, and `/v1/runs` booked any
non-`failed` result as `completed` — including an interrupt that did not come
through `/stop` and a turn that ran out of iteration budget. Automation that
reads the run status or the terminal event saw unfinished work as delivered,
and `partial: true` could sit next to `completed: true` in one payload.
`api_server_runs.terminal_run_status()` is now the single mapping for both
surfaces: interrupted -> `cancelled`, failed/partial/`completed=False` ->
`failed` (with `turn_exit_reason` and the fallback text as `output`),
otherwise `completed`; the terminal event is always `run.<status>` and a
late `pending_steer` rides on every terminal status instead of only on
`completed`.
CLI exit codes (`-q` quiet mode, `-z` one-shot) are deliberately unchanged
here: `hermes -z` returning 0 whenever text was produced was a stated design
choice (093f567f0d) and scripts depend on it, so that flip needs a
maintainer decision.
Fixes the gateway/producer half of #111770; slimmer redo of #111785 by
@KoNit-K (same mapping idea, one helper instead of three ladders).
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Squashed integration of the user-facing message audit for this surface set.
Full per-finding receipts: /tmp/ux-audit/lanes/*-receipt.md (campaign artifacts).
Hook callbacks are gated per (hook, callback, call identity). Two hooks
on the tool-loop path fired without any identity, so concurrent terminal
calls in one turn, or overlapping turns, still collapsed onto a single
gate key and the second invocation was skipped as if a callback had hung.
transform_terminal_output now forwards the tool_call_id bound in the
approval context around dispatch (only when set); transform_llm_output
forwards the turn_id already in scope. Payloads are additive: the
dispatcher withholds unknown fields from narrow-signature callbacks.
The corruption explainer filled `{db_path}` from `_default_db_path()`, the
process default. A Desktop `serve` backend launched on the root home hosts
named-profile sessions whose SessionDB is `profiles/<name>/state.db`, so the
operator was told to inspect/repair a different profile's database. Pass the
agent's own `_session_db.db_path` from the turn finalizer; the process
default remains the fallback for agents without a bound store.
Reported in #105887.
On BASE the #95514 stream-recovery rebound final_response inline, before
the tail-closing / persist-override / micro-compaction / _persist_session
calls inside the guarded persist try-block, so a raise in any of them left
the caller with the streamed text (cleanup_errors reported the failure).
HEAD's _close_transcript_tail returned the recovered value only on normal
completion; an exception in the same block dropped it and the user-visible
answer regressed to the pre-recovery '' (then rewritten by the explainer).
Split the helper into _drop_transcript_scaffolding / _recover_final_from_stream
/ _close_transcript_tail and rebind final_response in the guarded step as soon
as it is computed, restoring BASE ordering. Adds a regression test.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
Move the structural clone from the four call sites (auto review, codex
runtime, CLI /refine, gateway /refine) into AIAgent._spawn_background_review,
which every review path — immediate, idle-queue deferred, requeued — passes
through. Callers can no longer forget it, and the private helper is no longer
imported across hermes_cli/ and gateway/ package boundaries.
Tests now bind the real chokepoint (capturing at _spawn_background_review_now)
so they still fail if the clone is removed.
Detect same-inode cp via a generation stamp, halt FTS repair, and divert
unwritten transcripts to sessions/<id>.jsonl plus the gateway pending spool.
Co-authored-by: Cursor <cursoragent@cursor.com>
- Preserve streamed assistant text in Desktop UI when message.complete delivers empty text.
- Prevent destructive hydration in Desktop useMessageStream over rendered text on empty completion.
- Recover stream buffer in finalize_turn when final_response is empty on healthy turns.
- Unify in-place blank assistant repair, watermark clone resolution, non-blank concurrent winner adoption, and batch row appends into a single atomic guarded SessionDB transaction.
- Synchronize canonical committed content to live in-memory messages dicts and preserve all-or-nothing rollback semantics on persistence failure.
Independent review caught a compaction authority the gate missed:
post-turn micro-compaction (turn_finalizer -> _micro_compact) absorbs the
oldest exchanges into a rolling summary with no pre-compress checkpoint
hook in its path, and both compression.checkpoint_required and
compression.micro_compact could be enabled together — assistant evidence
could vanish into a summary the checkpoint filter later excludes, without
ever reaching the durable provider.
- agent_init: checkpoint_required forces micro-compaction off (warned),
mirroring the native-compaction suppression
- turn_finalizer: defense-in-depth guard at the call site (attribute is
plain mutable state a future path could flip on a live agent)
- behavioral regression test with a sabotage control (gate off proves the
harness reaches the call site; gate armed proves zero calls)
- docs + config example mention the suppression; stale v1 test header fixed
Review-pass follow-up on the load-time durability stamp:
- hermes_state.py: import the marker from agent.context_compressor instead
of a third synced literal (hermes_state already imports agent.* at module
level; only run_agent is circular). Old comment claimed otherwise.
- agent/turn_finalizer.py: replace the raw "_db_persisted" string at the
fill-empty-tail pop site with the shared constant (was outside the drift
guard).
- agent/conversation_compression.py: the no-op progress check now falls back
to a marker-insensitive comparison (_strip_marker_for_comparison). Loaded
rows are stamped at materialization time while compress() output is
marker-swept, so a semantically-identical no-op copy on a cold-resumed
session would previously compare unequal and take the progress branch.
Raw == still runs first so engine-returned list subclasses keep their
__eq__ semantics.
- test_marker_constant_in_sync extended to turn_finalizer + identity
assertions; new test_noop_progress_check_is_marker_insensitive
(mutation-checked: fails when the helper is neutered).
'database disk image is malformed' contains the word 'disk', so
classify_persistence_error bucketed SQLITE_CORRUPT / SQLITE_NOTADB
failures as 'disk' and the turn-completion explainer told users to
free disk space for a structurally damaged state.db (the #77386-family
misdiagnosis, reproduced in the v0.20.0 malformed-DB incident report).
- hermes_state: new 'corrupt' bucket in PERSISTENCE_ERROR_CAUSES,
matched via _DB_CORRUPTION_MARKERS BEFORE the locked/disk buckets
- run_agent: explainer text for 'corrupt' points at hermes doctor and
explicitly says freeing space will not help
- cron explainer-variant suppression picks the new variant up
automatically (it iterates PERSISTENCE_ERROR_CAUSES)
Adds a bounded fallback path in turn_finalizer.py that always records
a terminal timed_out outcome via _record_task_failure (CAS receipt path)
when the iteration budget is exhausted, regardless of whether the normal
fallback paths (interrupted/failed/anomalous exit_reason) were eligible.
Previously, a kanban worker whose budget was exhausted but whose turn was
interrupted, failed, or exited with an anomalous reason would silently
leave its task in an ambiguous lifecycle state — the dispatcher would
eventually detect it as a crashed or protocol-violation worker, but the
failure was not bounded and could take a full tick cycle to reconcile.
The CAS invariant in _end_run (WHERE ended_at IS NULL) guarantees
idempotence: if another path already closed the run, the call is a no-op.
Extracted the inline kanban-budget-exhausted recording into a shared
helper function (_record_kanban_budget_exhausted) used by both the
existing iteration_limit_fallback path and the new bounded fallback path.
Closes#87096
Refresh-loss interrupt is cooperative, so a stalled writer could still flush after another process reclaimed the conversation. Carry the holder into append_message / append_messages_batch and reject the write in the same SQLite transaction when the lease row is missing, expired, or owned by someone else.
Own the surrogate-crash class at three chokepoints instead of leaf sites:
- finalize_turn scrubs final_response once where model text leaves the
conversation loop — covers oneshot stdout (#80366), NIM/any-provider
responses (#19819), and every delivery consumer of the turn result.
- _sanitize_gateway_final_response scrubs at the gateway chat-surface
boundary — Telegram utf16_len (#55309) and Signal formatting (#55143)
can no longer see a lone surrogate; raw-text surfaces keep passthrough.
- run_conversation walks the fully-built api_kwargs with
_sanitize_structure_surrogates so tool descriptions (session_search,
#50959) and every other request-body leaf are JSON-encodable before
any provider sees them.
Regression tests pin all three chokepoints plus helper semantics.
Cherry-picked alongside #79240 (TheophilusChinomona) and #80374
(rainbowgore) whose commits precede this one with authorship preserved.
An enterprise deployment hit sustained SQLite write-lock contention on a
shared multi-gigabyte state.db (gateway + CLI processes writing
concurrently). Turns correctly failed closed with
session_persistence_failed, but the only user-facing wording claimed the
disk was full and the gateway rendered a generic failure.
The fast-fail semantics are deliberate and unchanged. This adds a pure
classifier (locked / disk / unknown) applied where the SQLite error is
still visible, threads the cause through the turn-completion explainer,
and stamps a machine-readable failure_reason
(session_persistence_failed:<cause>) plus a guaranteed non-empty error on
the result for downstream surfaces. The cron scheduler's explainer-text
suppression now matches every cause variant so refined wording cannot
leak into scheduled-job deliveries.
When the CLI streams a response token-by-token, it marks the response as
already displayed and skips re-printing after the tool loop. This means any
content appended by a transform_llm_output plugin fires after streaming — the
appended text is in the final response and stored in history, but never shown
to the user.
Fix by tracking the pre-transform response in finalize_turn() and including it
in the result dict as pre_transform_response. The CLI then checks whether the
response was transformed and, if so, prints only the appended suffix.
Previously the already_streamed branch was a no-op pass. Now it detects
post-stream plugin additions and outputs them without re-printing the streamed
body.
Phase 8 of the Hermes Agent token leak mitigation plan
(ralplan-hermes-token-leaks.md §3.9). Adds a boolean kwarg
`skip_background_review` (default False) to AIAgent.__init__ that
suppresses the end-of-turn _spawn_background_review fork.
Each background review fork instantiates a new AIAgent with its own
~15K input tokens + up to 8 LLM iterations, accumulating ~30K tokens
per event in the worst case. On cron sessions there is no
human-in-the-loop benefit from the review (no skill-creation pressure,
nobody curating MEMORY.md), so the cost is pure waste.
The end-of-turn guard now reads:
if (final_response and not interrupted
and not getattr(self, "skip_background_review", False)
and (_should_review_memory or _should_review_skills)):
skip_memory=True already disables the memory-review trigger; this
flag is the explicit single-switch off for both review paths.
Defaults to False, so behavior is unchanged for gateway/CLI callers
that omit the kwarg.
Tests: 5 new unit tests in tests/agent/test_skip_background_review.py
covering the default value, flag persistence, the gate short-circuit,
the gate fall-through, and a source-text assertion that the cron
scheduler sets the flag to True (separate commit).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
