MiniMax's Anthropic-compatible endpoint rejects an oversized native image
part with "media exceeds size limit: max 10485760 bytes (2013)" — no
occurrence of the word "image", so none of _IMAGE_TOO_LARGE_PATTERNS
matched. The 400 fell through to _REQUEST_VALIDATION_PATTERNS (the body
is type: invalid_request_error) and classified as format_error /
non-retryable.
That skipped the image-shrink recovery in conversation_loop, which is
gated on FailoverReason.image_too_large. Because the oversized part is
already baked into history as a tool_result image block, and the context
compressor rewrites text but not image data, every later turn re-sent the
same bytes and failed identically — the session stayed dead until the
user forked it.
Match on the "media" fragment, mirroring the existing "image exceeds"
entry so reworded vendor variants are caught too. A non-image media
rejection routed here is safe: the shrink pass finds no image parts,
returns False, and the caller surfaces the original error unchanged.
Fixes#76039
The observed wire error — 'Downloaded response does not contain a valid JPG, PNG, WebP, or ICO image.' — has no match in _IMAGE_CORRUPT_PATTERNS, so it falls through to the non-retryable 400 handler and the session replays the same image parts into the same 400 until /new.
Adds the full observed sentence to the pattern list. Deliberately not the shorter prefixes: a bare 'downloaded response does not contain a valid' also matches non-image 400s, and a negative test now pins that a downloaded-response certificate 400 keeps falling through to the existing handler. Covered on both the 400 path and the message-only path.
Reported by ryuhaneul in #69078.
Review (Sol xhigh) on the prior commit found a P1: the generic strip-
and-retry fallback ("any non-retryable 400 with image parts present
strips and retries") was too blunt. It couldn't tell an actual
image-corruption 400 apart from an unrelated one — bad tool schema,
unsupported parameter, billing, content policy — that merely happened
to carry image parts in the request. Any of those would silently erase
vision history and retry the still-invalid request, degrading sessions
that were never bricked in the first place. That's worse than the bug
it was meant to fix.
Revert the generic fallback (agent/conversation_loop.py). Keep only
the classifier-routed path: FailoverReason.image_corrupt +
_IMAGE_CORRUPT_PATTERNS, checked before _IMAGE_TOO_LARGE_PATTERNS
because shrinking corrupt bytes can't repair them. Corrupt-image
wordings still route to strip-and-retry; everything else falls through
to normal (non-retryable) handling as before. Add xAI's second wire
wording for the same corruption class ("base64 string of provided
image cannot be decoded", returned on unaligned truncation vs "Invalid
PNG image." on aligned truncation) and a compound-message test pinning
that image_corrupt wins when a body matches both pattern lists.
Drop TurnRetryState.stripped_images_this_turn. It's unnecessary now
that only one branch is left: the branch already only retries when
_strip_images_from_messages reports it removed something, and that
helper strips every image part from the request in one pass — so a
second corrupt-image hit on the retried (now text-only) request has
nothing left to strip and falls through on its own. No separate
one-shot flag needed.
Add a run_conversation integration test at the sequenced-provider
layer: corrupt 400 on attempt 1, strip, retry succeeds on attempt 2,
with explicit before/after assertions on the outgoing image_url part.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: paultaki <paultaki@users.noreply.github.com>
The permanent-brick class in #69078: xAI returns 'Invalid PNG image'
when a re-serialized image part in replayed history becomes
undecodable. The existing image-error patterns cover only Anthropic
'exceeds max dimension' wordings and 'model does not support images'
strings, so the classifier lands on a generic non-retryable 400 and
neither the shrink path nor the strip path fires. Every subsequent
turn (even bare text) fails identically because the poison stays in
history — the session is permanently wedged until deleted.
Two recovery layers, deliberately separate:
- Semantic split: new FailoverReason.image_corrupt with
_IMAGE_CORRUPT_PATTERNS ('invalid png image' / 'invalid jpeg image'),
checked BEFORE _IMAGE_TOO_LARGE_PATTERNS in both _classify_400 and
_classify_by_message. Corrupt bytes route to strip-and-retry, never
to the shrink path (shrinking corrupt bytes cannot help).
- Generic fallback: any non-retryable 400 whose outgoing messages
still contain image parts gets one strip-and-retry via the existing
_strip_images_from_messages helper, guarded by a new
stripped_images_this_turn one-shot flag on TurnRetryState. This
un-bricks the session for any current or future provider wording
without adding another pattern list to maintain.
Item 3 from the report (multimodal-part integrity across FTS
persistence + compaction handoff) is a separate investigation and
remains follow-up work.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: paultaki <paultaki@users.noreply.github.com>
Consolidates the 429-quota-classifier cluster on top of the merged #93419
Anthropic core. Three independent contributor findings salvaged into one
coherent change to the single 429 branch:
- Broaden the 429 usage-limit check from the narrow 'usage limit' string to
the full _USAGE_LIMIT_PATTERNS ('quota', 'limit exceeded', 'key limit
exceeded') and add _BILLING_PATTERNS detection on 429 ('insufficient
credits' wrapped in a 429 instead of 402), guarded by a _RATE_LIMIT_PATTERNS
exclusion so an explicit 'Rate limit exceeded' never promotes to
non-retryable billing. (credit @Pluviobyte, #39441 — earliest submitter)
- Add 'resets in' to the transient signals: Codex's 'Weekly usage limit
reached. Resets in 6hr 29min.' wrongly read as terminal billing because
main only had 'reset in' (no substring match). (credit @LeonSGP43, #63021)
- Add 'reset after' / 'available in' / 'per minute' / 'per second' transient
signals. (credit @jtstothard, #74785)
Supersedes #65633 (defective branch placement, no tests). The aux-client
path already covers these shapes (_is_payment_error catches weekly/quota
walls; _is_rate_limit_error treats 'resets in' as transient), so no change
there.
Tests: 6 new cases (generic quota wall, insufficient-credits 429, rate-limit
guard, Codex resets-in, extra transient phrases). Guard sabotage-verified.
Co-authored-by: Pluviobyte <Pluviobyte@users.noreply.github.com>
Co-authored-by: LeonSGP43 <LeonSGP43@users.noreply.github.com>
Co-authored-by: jtstothard <jtstothard@users.noreply.github.com>
The Codex OAuth backend (chatgpt.com/backend-api/codex) intermittently
injects prompt_cache_retention into its own upstream call and then rejects
it, returning HTTP 400 invalid_parameter. Hermes never sends that field on
this route (see agent/transports/codex.py::_default_prompt_cache_retention_
for_request, which only sets it for api.meta.ai and bedrock-mantle hosts).
Reproduced live: a minimal 1-message request carrying no cache parameters
at all failed 4/20 (20%) with this error, so the rejection is not
deterministic and retrying the identical request is the correct recovery.
Previously the catch-all in _classify_400 returned format_error/
retryable=False, which tripped the is_client_error abort gate in
conversation_loop and killed the turn on the first attempt - burning an
entire large-context request (~550k tokens) per failure.
Classify these as retryable server_error (should_compress=False - the
request shape was never the problem). The same guard is applied to the
sibling 5xx request-validation branch, where a fronting proxy can surface
the identical rejection.
Deliberately narrow: keyed on parameters we only send on specific routes,
and skipped when the current provider is one that legitimately sends them,
so a genuine client-side bad parameter (max_tokens on GPT-5) still fails
fast as a format_error.
Port from anomalyco/opencode#40707: connection-establishment and DNS
failure messages wrapped in generic exceptions (RuntimeError from local
shims, MCP bridges, SDKs re-raising without chaining) fell through to
FailoverReason.unknown, which misses the retry loop's eager transport
fallback — the full retry budget burned against a dead endpoint before
provider fallback.
New _CONNECTION_MESSAGE_PATTERNS (connect refused, no route, network
unreachable, DNS phrasings across Python/glibc/macOS/Node, fetch failed,
Envoy upstream connect error) classify as retryable timeout via
_classify_by_message, mirroring _TIMEOUT_MESSAGE_PATTERNS. Mid-stream
disconnect strings are deliberately excluded — they keep their
_SERVER_DISCONNECT_PATTERNS routing (large-session compression).
Review follow-up (egilewski): the previous commit only hedged the guidance
text; the exact Anthropic 400 was still classified, persisted, and surfaced
as confirmed billing exhaustion. Carry the ambiguity all the way through:
- agent/error_classifier.py: 'out of extra usage' matches on the 400 and
status-less paths now attach error_context {billing_unverified,
possible_content_filter}. Reason stays FailoverReason.billing (rotation +
fallback remain the right recovery either way); ClassifiedError grows a
billing_unverified property.
- agent/credential_pool.py: new FAILURE_REASON_BILLING_UNVERIFIED. An
unverified billing exhaustion gets the short transient cooldown instead of
the one-hour bench, regardless of pool size: a content-filter rejection
leaves the credential healthy and fails identically on every key, and the
hour-long sole-credential latch is what replayed the stored error and made
real fixes look ineffective. A true 402 keeps the full bench. The marker
persists with the entry so a restart cannot upgrade it back to a bench.
- agent/agent_runtime_helpers.py + run_agent.py: recover_with_credential_pool
threads billing_unverified and hands the pool 'billing_unverified' as the
persisted failure_reason.
- agent/conversation_loop.py: the fallback-switch status, max-retries status,
terminal label, and both structured terminal results hedge when the verdict
is unverified. New _billing_terminal_label + _billing_failure_result build
the returned terminal response in one place; the result dict now carries
billing_unverified and the billing_block gains 'unverified': true. The
confirmed-billing path (a real 402 or an API-key credit depletion) keeps
the original assertive wording, so the caveat no longer dilutes it.
Regression tests: classifier marking (400 + status-less + unambiguous-body
negative), pool cooldown TTLs + persistence round-trip, pool failure_reason
plumbing, and the returned terminal response for both unverified and
confirmed verdicts.
Note: tests/agent/test_credential_pool_routing.py::TestFailureAttribution::
test_unmatched_key_does_not_retry_only_pool_entry fails identically on
current main without this change (pre-existing, unrelated).
Run the expensive-model warning for explicit startup `-m` / `--provider`
overrides before the chat loop starts, and fail closed for non-interactive
invocations that select an expensive or known-confusing model.
Also classify Nous paid-model 404s that say credits are required as billing
exhaustion so they fail fast with billing guidance.
Tested:
- scripts/run_tests.sh tests/hermes_cli/test_cli_startup_model_cost_guard.py tests/hermes_cli/test_model_cost_guard.py tests/agent/test_error_classifier.py -- --tb=short -q
Applies the batch-disposition SALVAGE conditions from #64231: the hook id
moves to the taxonomy transform-family name, and run-all-then-pick-first
dispatch now logs a runtime warning when a valid-but-losing classification
is skipped (the #64714 skipped-transform rule). Chaining semantics are
stated explicitly at the VALID_HOOKS entry, the dispatch helper docstring,
and the hooks.md catalog row and detail section.
Adds a plugin seam at the top of agent/error_classifier.classify_api_error()
(step 0, before the built-in pipeline) so model-provider plugins can classify
their provider's error quirks without patching core:
- New "classify_api_error" entry in VALID_HOOKS. Callbacks receive the parsed
error context (provider, model, status_code, error_type, error_code,
error_message, error_body, error, approx_tokens, context_length,
num_messages), self-scope on `provider`, and return None to pass or a dict
{"reason": "<FailoverReason name>", ...optional recovery-hint overrides}.
- get_plugin_error_classification() helper mirrors
get_pre_tool_call_block_message(): first valid result wins, invalid dicts
and unknown reasons are skipped, callback exceptions are isolated — a
broken plugin can never break classification. Zero behavior change when no
plugin claims the error (all 179 existing classifier tests pass untouched).
- Bundled reference plugin `openrouter-tool-use-404` (opt-in, like all
bundled standalone plugins) re-implements PR #58451: OpenRouter's
"No endpoints found that support tool use" 404 carries no
_MODEL_NOT_FOUND_PATTERNS signal, so it classifies as unknown/retryable
and the retry loop burns 3-5 attempts on a deterministic rejection.
The plugin classifies it as model_not_found (retryable=False,
should_fallback=True) so the fast-fallback path fires immediately —
demonstrating a waiting core PR converted to a publishable plugin.
Motivation: ~10 open PRs are single-provider error-classification patches
(#58451, #58355, #58502, #58474, #58366, ...). This hook turns that whole
class of contribution into plugin territory.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWMcB7RPSYUpsXDfBgwjzM
Follow-up to #84632:
- Guard _llama_cpp_grammar_hit inside status_code == 400 to restore
short-circuit behavior on non-400 errors (minor efficiency)
- Extract _NO_USER_QUERY_SIGNAL constant for the duplicated string
between _INVALID_MESSAGE_BODY_PATTERNS and the llama.cpp exclusion
guard, preventing silent drift if the phrase is ever changed
Local engines wrap Qwen template raise_exception("No user query found…") as
applyPromptTemplate / "Unable to generate parser for this template". That used
to match llama_cpp_grammar_pattern, strip tool schema keywords, and retry while
the real cause was a poisoned/oversized transcript after failed compression.
Classify as format_error so recovery fails fast toward /new instead.
When the Nous Portal returns paid_service_access.allowed=false with
reason=member_spend_cap_exceeded, Hermes was falling through to the
generic 'no active subscription or usable credits' message — even
though the user has ample purchased credits and the real blocker is
an org-level per-member spend cap.
This adds a dedicated branch that surfaces the actual cause: names the
spend cap, shows the cap/spend amounts, and tells the user to ask their
org admin to raise it. Also adds member_spend_cap_exceeded to the
billing error code set so the error classifier and auth error formatter
route it through the Nous entitlement message path.
Selecting an NVIDIA NIM model whose id reached config without the nvidia/
prefix produced a bare "HTTP 404: 404 page not found" — retried three times,
never naming the model. It reads exactly like an outage or an auth failure,
which is where the Discord thread spent its time before the id was spotted.
normalize_model_for_provider() had no branch for nvidia, so a bare id passed
straight through to the API. Repair it from the provider's curated catalogue:
a bare name that matches exactly one entry modulo the prefix gets it back.
That's a lookup, not a guess — build.nvidia.com also fronts local NIM
containers and third-party models, and anything absent from the catalogue is
left alone. Because the repair runs on every runtime setup, an already-broken
config self-heals on the next turn and prints what it changed.
If a bare id still reaches the wire, the 404 now explains itself. The
classifier consults the same catalogue: a prefix-less id the provider only
serves as vendor/model is a deterministic failure, so it classifies as
model_not_found instead of burning three retries on a retryable "unknown",
and the error trace names the id to use.
Fixes#78796
Convert a completed MoA aggregator response into one valid Chat
Completions delta chunk at the MoA facade boundary, normalize completed
message.tool_calls into indexed stream deltas, and classify these local
MoA adapter-shape errors as non-fallback format errors so a local
compatibility bug cannot silently drift the user's MoA route to a
single model (#55933 follow-up).
Salvaged from PR #74903 by @liusencomic-cyber.
Add distinct, greppable log lines at both fix sites so the condition is
observable in the field instead of only inferable from the absence of the old
'Cannot compress further' spiral:
- chat_completion_helpers: warn when an empty partial-stream stub is replaced
with the placeholder (0 chars recovered, no tool call).
- error_classifier: warn when a malformed-body 400 is classified as
format_error rather than context overflow, with num_messages/approx_tokens.
Extend the litellm-shape classifier test to assert the warning is emitted.
A stream that dies after delivering deltas but with 0 recovered chars (and
no captured tool call) produced a content-less assistant stub. That empty
non-final message is invalid for the Anthropic schema, so every later request
400s with 'all messages must have non-empty content' (INVALID_REQUEST_BODY).
On a large session the 400 was misclassified as context overflow, dropping the
loop into a compression spiral that ends in 'Cannot compress further' — a
misleading context-size error on a session nowhere near its limit.
Root cause: substitute a minimal '[response interrupted]' placeholder so the
stub is never empty.
Misclassification: add _INVALID_MESSAGE_BODY_PATTERNS (checked before the
context-overflow heuristic) to classify these as non-retryable format_error,
and teach the body extractors the litellm/Bedrock errorMessage/errorCode/
errorArgs shape so descriptive proxy errors are not mistaken for generic ones.
Adds RED-verified regression tests for the stub path and three classifier tests
(including a guard that real overflows still compress).
Port from anomalyco/opencode#37848 (+ dev-branch twin #37840): expand
context-overflow patterns and guard against rate-limit messages that
mention tokens.
- 'Throttling error: Too many tokens, please wait before trying again.'
(AWS Bedrock / proxy shape) classified as context_overflow and routed a
healthy session into compression on every throttle. Added 'throttling'
to _RATE_LIMIT_PATTERNS, which the message-only path checks BEFORE the
overflow list.
- 'Input length N exceeds the maximum allowed input length of M tokens.'
(Together/Fireworks shape) fell through to unknown — no compression
recovery. Added 'maximum allowed input length' to overflow patterns.
- 'request_too_large' / 'Request exceeds the maximum size' (Anthropic 413
type re-wrapped without a status code by aggregators/proxies) fell
through to unknown. Added to _PAYLOAD_TOO_LARGE_PATTERNS.
All three shapes proven live on main before the fix; 265 classifier +
bedrock tests and 238 sibling rate-guard/compression tests pass.
The api_server adapter returned error code "invalid_api_key" for
API_SERVER_KEY authentication failures, which the Desktop error
classifier misidentified as a provider (OpenRouter/OpenAI) key
problem — showing "OpenRouter API key missing" when the real issue
was gateway auth.
Changes:
- gateway/platforms/api_server.py: return "gateway_auth_failed" code
with descriptive message for API_SERVER_KEY auth failures
- apps/desktop/src/store/notifications.ts: add "gateway_auth_failed"
handler before "invalid_api_key" to show correct error message
- agent/error_classifier.py: add "gateway_auth_failed" to auth patterns
- tests: update test_session_api.py to expect new error code
Fixes#39365
Provider empty-reply text mentions "very low max_tokens", which used to match
the bare overflow pattern and thrash compress until "Cannot compress further".
Keep invalid persisted preset names fail-closed, list the valid configured choices, and classify the local lookup failure as deterministic so it reaches Desktop immediately.
Port from anomalyco/opencode#35671: Z.AI / Zhipu GLM returns
'tokens in request more than max tokens allowed' (error code 1210) on
context overflow. This matched no pattern in _CONTEXT_OVERFLOW_PATTERNS,
so the error classified as unknown/retryable — the agent would retry the
oversized request instead of triggering context compression.
Proven live on main before the fix: classify_api_error() returned
FailoverReason.unknown for the exact Z.AI error shape; now returns
context_overflow.
Keep each xAI OAuth auth-add login as an independent manual device-code pool entry and recognize xAI personal-team spending-limit 403 responses as billing exhaustion. Preserve the structured top-level error message so the failed credential is quarantined and the next healthy account is selected without attempting a pointless token refresh.
Route direct xAI HTTP consumers through the credential pool as well. Proactive and 401-reactive refreshes update the exact issuing manual entry, preserve validated xAI base URL overrides, and serialize single-use refresh-token rotation across concurrent pool instances.
The cross-turn stale-call circuit breaker (_check_stale_giveup in
agent/chat_completion_helpers.py) raises a RuntimeError when the
provider has been unresponsive for N consecutive stale attempts.
This error was classified as FailoverReason.unknown (retryable=True,
should_fallback=False), causing the retry loop to burn all max_retries
against the same dead provider — each retry hitting the circuit breaker
instantly with zero network overhead — before fallback was attempted.
Add a classification rule (section 7b in classify_api_error) that
recognizes the stale-breaker RuntimeError by its signature phrases and
classifies it as FailoverReason.timeout with retryable=False,
should_fallback=True. This makes the retry loop skip retries and go
straight to fallback provider activation on the first hit.
Test: test_stale_breaker_runtime_error_triggers_fallback_not_retry
_classify_by_status() routes every other transient HTTP status to a retryable
reason (500/502 -> server_error, 503/529 -> overloaded, 429 -> rate_limit,
413 -> payload_too_large), but 408 Request Timeout fell through to the generic
`400 <= status < 500` branch and was classified as a non-retryable
format_error -- the same bucket as a 400 Bad Request.
A 408 is a transient timing failure the server itself flags as safe to retry
(RFC 9110 15.5.9), not a malformed request, so the retry loop aborted the turn
when a simple retry would recover. Common trigger: a reverse proxy in front of
a self-hosted backend (llama.cpp / Ollama / vLLM) returns 408 when a long
generation outruns the proxy's request-read window.
Route 408 to the existing FailoverReason.timeout (rebuild client + retry).
Add a regression test plus a boundary test asserting 400 stays non-retryable.
Inspired by Claude Code v2.1.199 (July 2, 2026): SSL certificate errors
(TLS-inspecting proxies, missing CA bundles, expired certs) no longer
burn retries before showing actionable guidance — they fail immediately
with the fix hint.
- agent/error_classifier.py: new FailoverReason.ssl_cert_verification +
_SSL_CERT_VERIFY_PATTERNS, checked BEFORE the transient-SSL patterns
(cert-verify messages also contain '[SSL:' and previously retried
forever as timeout). Non-retryable, no compression, no fallback churn.
- agent/conversation_loop.py: dedicated status line + per-cause fix
hints (corporate proxy CA bundle, certifi refresh, self-signed local
endpoints) on the non-retryable abort path.
- 7 new tests incl. regression guards (transient alerts still retry,
large-session cert failure doesn't trigger compression).
When OpenRouter routes to an endpoint that does not support tool/function
calling, it returns HTTP 404 with the message 'No endpoints found that
support tool use. Try disabling "browser_back".'
The raw error body does not contain 'model not found' or any other
_MODEL_NOT_FOUND_PATTERNS entry, so it falls through to FailoverReason.unknown
with retryable=True. The retry loop wastes 3-5 attempts on the same
deterministic rejection, then surfaces a confusing generic error instead of
automatically failing over to a fallback model or provider.
Adding the OpenRouter phrase to _MODEL_NOT_FOUND_PATTERNS classifies it as
model_not_found (retryable=False, should_fallback=True), which triggers the
client-error fast-fallback path in conversation_loop.py: the agent switches
to a configured fallback model/provider before the user sees the error.
Existing buffered guidance in conversation_loop.py (the 'support tool use'
hint at line ~2967) remains intact and surfaces only if every fallback
exhausts.
Local inference servers (llama.cpp/llama-server, vLLM/Ollama behind a
Cloudflare/Tailscale hop) report context overflow with HTTP 500/502/503/529
instead of 400/413. _classify_by_status returned server_error/overloaded and
retried blindly, then dropped the turn with no compaction. Route explicit
_CONTEXT_OVERFLOW_PATTERNS matches on those 5xx codes to context_overflow
(should_compress=True); plain 500 stays server_error, plain 503 overloaded.
Anthropic returns HTTP 400 with "You're out of extra usage. Add more at
claude.ai/settings/usage and keep going." when the account's extra-usage
allowance is depleted. The existing _BILLING_PATTERNS list did not
include this wording, so classify_api_error fell through to generic
format_error — non-retryable and should_fallback=False — causing the
agent to abort instead of engaging the configured fallback chain.
Add the pattern and a regression test covering the exact Anthropic body.
OpenRouter returns 429 in two shapes: an account-level throttle on the
user's key, and an upstream-provider throttle (DeepSeek/Anthropic/etc.
rate-limiting OpenRouter's aggregate traffic). The classifier treated
both identically and rotated/exhausted OPENROUTER_API_KEY on every 429 —
burning the key for ~24min and silently disabling auxiliary features
(compression, summarization, vision) on an upstream throttle where the
key was healthy.
Add a FailoverReason.upstream_rate_limit classified from OpenRouter's
unambiguous wrapper message "Provider returned error" (the same signal
the metadata-raw parser already trusts). Recovery skips credential
rotation and defers to the fallback chain to switch models instead.
Co-authored-by: Hermes Agent <127238744+teknium1@users.noreply.github.com>
The MiniMax output-layer safety filter surfaces the error verbatim as
`output new_sensitive (1027)` (sometimes with additional provider
wrapping like 'Stream stalled mid tool-call: output new_sensitive (1027)').
When the model emits a large tool-call argument block, the upstream
filter trips and the SSE stream is truncated mid-flight, producing
'stream stalled mid tool-call' errors. Until now this case was
misclassified and retried 3x on the same provider, reproducing the same
refusal and burning paid attempts.
Adding `new_sensitive` to `_CONTENT_POLICY_BLOCKED_PATTERNS` routes
it through the existing is_client_error path: skip 3x retry, activate
configured fallback model immediately, surface a clear provider-safety
message to the user.
Refs #32421
Z.AI / Zhipu reuse HTTP 429 for server-wide overload. The 429 status
path classified these unconditionally as rate_limit with
should_rotate_credential=True, so an overloaded provider exhausted the
credential pool after two errors — fatal for a single-key user, who has
nothing to rotate to.
The credential is valid; the server is just busy. Disambiguate the 429
body against a shared _OVERLOADED_PATTERNS list and route overload
language to FailoverReason.overloaded (retryable, no rotation), matching
the existing 503/529 path and the message-only path (#52890). Genuine
rate limits (no overload language) still rotate.
Extracted the inline overloaded tuple #52890 added into the shared
_OVERLOADED_PATTERNS constant so the status-code and message paths use
one list.
Closes#14038.
Two-part fix:
Part 1 (classifier override at agent/error_classifier.py:720-738):
A transport disconnect on a reasoning model — even on a large session —
now routes to FailoverReason.timeout instead of context_overflow. Without
this, large-session reasoning-model disconnects route to the compression
branch and silently delete conversation history on a phantom
context-length error. The override is strictly targeted: non-reasoning
models (gpt-4o, claude-3-5-sonnet, llama-3.3-70b, etc.) still route to
context_overflow on large sessions — the existing intentional behavior
for chat models whose proxy doesn't idle-kill during prefill/generation.
Part 2 (new agent/thinking_timeout_guidance.py + integration at
agent/conversation_loop.py:3488-3567):
New is_thinking_timeout() and build_thinking_timeout_guidance() helpers.
When a known reasoning model (NVIDIA Nemotron 3 Ultra, OpenAI o1/o3,
Anthropic Opus 4.x thinking, DeepSeek R1, Qwen QwQ, xAI Grok reasoning)
hits a transport-kill on a small session (classifier says timeout
directly) or after Part 1 routes correctly (large session), the user
now sees reasoning-specific guidance with three actionable workarounds
in priority order:
1. Set providers.<provider>.models.<model>.stale_timeout_seconds: 900
in ~/.hermes/config.yaml (Hermes's built-in floor is already 600s
for known reasoning models; raise further if upstream is even
tighter).
2. Lower reasoning_budget or set reasoning_effort: medium on this
model if the provider supports it.
3. Use a smaller / faster reasoning model if the task doesn't
require deep thinking.
The new guidance takes precedence via if/elif over the existing
_is_stream_drop block, so a reasoning-model user with a transport-kill
message sees actionable advice instead of the misleading "try
execute_code with Python's open() for large files" advice (which is
correct for the unrelated large-file-write stream-drop case but
actively wrong for the thinking-timeout case).
Verified:
- 478 tests passing across 9 directly-relevant files (49 new + 429
existing, zero regressions).
- Ruff lint clean on all 4 modified/new files.
- Negative test: 6 parametrized regression guards confirm non-reasoning
models still route to context_overflow on large sessions; 4
parametrized gates confirm non-timeout classifier reasons never
trigger the guidance; 5 parametrized cases confirm non-transport
messages never trigger it.
- Regression guard: new guidance message does NOT contain
"execute_code" or "open()" — the misleading advice is fully
replaced, not appended alongside.
- Cross-vendor dual review via agy -p:
- Gemini 3.5 Flash (Medium) — passed: true, zero blockers, one
SHOULD-FIX (vprint block duplication — fixed by extracting
detection into a helper module).
- GPT-OSS 120B (Medium) — passed: true, zero blockers, two nits
(test placement — adopted at tests/agent/test_thinking_timeout_guidance.py;
primary-model capture — accepted as non-issue per Flash's nit).
Dependency note for maintainers:
This PR includes agent/reasoning_timeouts.py (the reasoning-model
allowlist module from PR #52238) because the Layer 1 override is
load-bearing on get_reasoning_stale_timeout_floor(). After PR #52238
lands on main, this PR's duplicate agent/reasoning_timeouts.py should
be rebased away. Either PR can land first; the other rebase is
mechanical.
Fixes#52271.
Anthropic returns a 400 when the thinking/redacted_thinking blocks in the
latest assistant message are mutated upstream: 'thinking or redacted_thinking
blocks in the latest assistant message cannot be modified. These blocks must
remain as they were in the original response.'
The classifier's thinking_signature branch only matched on the substring
'signature', so this variant fell through to a non-retryable client error
and hard-aborted the turn -- even though the existing strip-reasoning_details
-and-retry recovery would have healed it.
Broaden the 400 match to also catch 'cannot be modified' / 'must remain as
they were' (still gated on 'thinking'), routing it to the same recovery.
Adds a negative-case test so unrelated 'cannot be modified' 400s are not
swept in.
Defense-in-depth, orthogonal to the root-cause work in #35975 / #17861
(which prevent the block mutation in the first place). Only changes a
terminal-failure into a one-shot recovery.
Signed-off-by: Ian Culling <ian@culling.ca>
A GPT-5 model rejecting max_tokens returns a 400 whose message contains the
literal substring 'max_tokens' — one of the _CONTEXT_OVERFLOW_PATTERNS. The 400
path in _classify_400 checked overflow patterns before any request-validation
check (which only existed on the 5xx path), so the parameter error was routed
into the compression loop, re-sent with the same bad param, and ended in
'Cannot compress further' on a tiny context.
Hoist a request-validation guard (unsupported/unknown parameter) above the
context-overflow check in _classify_400. Deliberately excludes the generic
invalid_request_error code, which OpenAI also stamps on real overflow 400s, so
genuine overflows still compress. Pairs with the max_completion_tokens param
fix that stops the bad request at the source.
Also adds AUTHOR_MAP entry for the salvaged PR #13902 commit.
Follow-up to the salvaged #37727. That PR fixed the reactive recovery path
(classifier + post-failure shrinker) but left the PROACTIVE embed-time guard
in vision_tools byte-only — a tall small-byte screenshot (e.g. 1200x12000 at
0.06 MB) still baked into immutable history un-resized, relying on a failed
round-trip to trigger reactive shrink.
- vision_tools: add _image_exceeds_dimension() + _EMBED_MAX_DIMENSION (7900px);
the embed-time cap now fires on bytes OR pixels and passes max_dimension to
the resizer, so tall small-byte images are shrunk before they're embedded.
- vision_tools: best-effort lazy-install of Pillow (tool.vision) in the resize
ImportError fallback so the soft dep self-heals (respects allow_lazy_installs).
- error_classifier: add two more Anthropic dimension-cap wording variants.
- pyproject + lazy_deps: declare Pillow as the [vision] extra / tool.vision
lazy dep (it was undeclared everywhere; without it ALL resize recovery no-ops).
- tests: cover _image_exceeds_dimension (tall/small/edge/no-Pillow/corrupt).
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Anthropic enforces two independent ceilings per image:
1. 5 MB encoded byte size
2. 8000 px longest side
Hermes only guarded #1. A tall screenshot (e.g. 1200x12000 at 0.06 MB)
passes every byte check but fails the pixel check, returning a
non-retryable HTTP 400 that permanently bricks the conversation thread.
Fixes:
- error_classifier: add 'image dimensions exceed' pattern to
_IMAGE_TOO_LARGE_PATTERNS so the 400 is classified as image_too_large
and triggers the shrink/retry path instead of falling through to
non-retryable error.
- conversation_compression: check pixel dimensions (via Pillow) even
when byte size is under the 4 MB target. If max(dims) > 8000, force
shrink.
- vision_tools._resize_image_for_vision: add optional max_dimension param.
When set, images exceeding the pixel cap are downscaled even if they're
under the byte budget. The resize loop now checks both byte AND pixel
limits before accepting a candidate.
Closes#37677
* fix(agent): fallback immediately on provider content-policy blocks
Provider safety-filter refusals (e.g. OpenAI Codex 'flagged for possible
cybersecurity risk', OpenAI moderation 'violates our usage policies',
Anthropic safety-system rejections, Azure content_filter) are
deterministic decisions about a specific prompt. Retrying the same
prompt up to api_max_retries times just reproduces the same refusal and
burns paid attempts before surfacing the generic 'API failed after 3
retries — <provider message>' to Telegram / cron with no indication that
the failure came from the model provider rather than Hermes itself.
Classify these as a new FailoverReason.content_policy_blocked
(non-retryable, should_fallback=True) and route them through the
existing is_client_error path so the loop:
- skips the 3x retry backoff
- activates a configured fallback model immediately
- emits a clear provider-safety message to the user (not the generic
'Non-retryable error (HTTP None)') and surfaces actionable guidance
when no fallback is configured (rephrase, narrow context, or set
fallback_model in hermes config)
- returns a final_response that explicitly tells the user this came
from the model provider, so gateway delivery is unambiguous and
cron last_status reflects the safety block rather than a vague
'agent reported failure'
Patterns are intentionally narrow — verbatim refusal phrasings keyed to
specific provider safety pipelines, not generic words like 'policy' or
'violation' that would collide with billing / format / auth errors.
Regression guards in test_18028_content_policy_blocked.py verify
billing 402s, generic 400s, and OpenRouter account-level
provider_policy_blocked remain distinct classifications.
Salvaged from #18164 onto current main (file restructure: loop logic
moved from run_agent.py to agent/conversation_loop.py, _emit_status →
_buffer_status), broadened patterns beyond the original OpenAI Codex
cybersecurity case to cover OpenAI moderation, Anthropic safety system,
and Azure content_filter; added user-actionable guidance and a clear
final_response so cron/gateway surfaces the policy block instead of a
generic non-retryable error, and added a regression-guard test module
mirroring the is_client_error predicate.
Addresses #18028.
Co-authored-by: Kuan-Chieh Huang <kchuang1015@users.noreply.github.com>
* chore: add kchuang1015 to AUTHOR_MAP
---------
Co-authored-by: Kuan-Chieh Huang <kchuang1015@users.noreply.github.com>
* fix(codex-responses): gracefully recover from invalid_encrypted_content (salvage #10144)
When an OpenAI-compatible Responses API surface accepts an initial
request but later rejects the replayed `codex_reasoning_items`
encrypted blob with HTTP 400 `invalid_encrypted_content`, the
session previously got stuck retrying the same poisoned payload.
Recovery: classify the error as a dedicated FailoverReason, and on the
first hit disable encrypted reasoning replay for the rest of the
session, strip cached items from message history, and retry once.
Changes:
* error_classifier: add FailoverReason.invalid_encrypted_content
branch in _classify_400 (before context_overflow so the messages
that mention 'encrypted content … could not be verified' don't trip
context heuristics), in _classify_by_error_code, and extend
_extract_error_code to peek inside wrapped JSON in error.message and
ignore the bare '400' as a code.
* agent_init: initialize `_codex_reasoning_replay_enabled = True` on
every agent.
* run_agent: add AIAgent._disable_codex_reasoning_replay() helper
that flips the flag and pops cached items.
* codex_responses_adapter: thread a `replay_encrypted_reasoning`
kwarg through _chat_messages_to_responses_input so that when the
flag is False we don't replay codex_reasoning_items.
* transports/codex.py: read `replay_encrypted_reasoning` from params,
thread it into the adapter, and gate the
`include=['reasoning.encrypted_content']` request hint on it.
* chat_completion_helpers: pass the agent's replay flag through to
the transport.
* conversation_loop: in the retry loop, add an
invalid_encrypted_content recovery branch that fires once per
session, only when api_mode == codex_responses, only when replay is
still enabled, and only when at least one assistant message in
history actually carries cached reasoning items (otherwise the 400
has nothing to do with our cache and the normal retry path handles
it).
Tests:
* test_error_classifier: new wrapped-JSON _extract_error_code case;
new TestClassifyApiError cases proving the 400 is retryable with
no fallback, that the broad message match doesn't catch a generic
'parsed' message, and that the error code match is
case-insensitive.
* test_run_agent_codex_responses: end-to-end test of the recovery
branch firing once and disabling replay, plus a sibling test that
proves the branch does *not* fire (and the flag stays True) when
history has no cached reasoning items.
Salvages PR #10144 onto the post-refactor module layout
(error_classifier / codex_responses_adapter / transports/codex /
conversation_loop / agent_init) since the original diff was written
against the pre-refactor monolithic run_agent.py.
* chore(release): map victorGPT in AUTHOR_MAP for #10144 salvage
---------
Co-authored-by: victorGPT <wuxuebin1993@gmail.com>
Standard OpenAI returns request-validation failures (unknown/
unsupported parameter, malformed request) as 4xx. Some
OpenAI-compatible gateways return them as 5xx instead — codex.nekos.me
returns 502 for an unknown parameter.
The generic '5xx -> retryable server_error' rule then misfires: the
error is deterministic (every retry gets the identical rejection), so
the retry loop burns all 3 attempts, the transport-recovery path
resets the counter and burns 3 more, and the result is a request
flood against a request that can never succeed.
Fix: when a 500/502 body carries an unambiguous request-validation
signal — 'unknown parameter' / 'unsupported parameter' /
'invalid_request_error' in the message text, or invalid_request_error
/ unknown_parameter / unsupported_parameter as the structured error
code — classify as a non-retryable format_error so the loop fails
fast and falls back. Genuine 502 Bad Gateway with no such signal
stays retryable as before.
Origin: local-author
Upstream-PR: none
Patch-State: local-only
Some providers (Xiaomi MiMo, some Alibaba endpoints, a long tail of
OpenAI-compatible servers) follow the OpenAI spec strictly and require
tool message `content` to be a string — they reject our list-type
content (text + image_url parts) with HTTP 400 'text is not set' /
'tool message content must be a string'.
Instead of an allowlist of known-good providers (maintenance burden,
guaranteed to miss aggregators like OpenRouter where the underlying
model determines support, not the aggregator name), this lands a
reactive recovery:
1. New `FailoverReason.multimodal_tool_content_unsupported` with a
small pattern list covering the common 400 wordings.
2. `AIAgent._try_strip_image_parts_from_tool_messages` walks the API
message list, downgrades any `role:tool` message whose content is
list-with-image to a plain text summary (preserves text parts) in
place, AND records the active (provider, model) in a session-scoped
`_no_list_tool_content_models` set.
3. `_tool_result_content_for_active_model` short-circuits to a text
summary when (provider, model) is in the cache — so after the first
400 + retry, subsequent screenshots in the same session skip the
round trip entirely.
4. Retry hook in `agent.conversation_loop` mirrors the existing
`image_too_large` recovery: detect the reason, run the helper,
retry once, fall through to the normal error path if no list-type
tool content was actually present.
Cache is transient (per-session) by design — next session retries in
case the provider added support, no persistent state to maintain.
Fixes#27344. Closes#27351 (allowlist approach superseded by reactive
recovery).
