fix(agent): an upstream account ban relayed in a 200 stream fails once instead of retrying as an outage

OpenRouter relays OpenAI's "this user has been blocked for a previous
policy violation" as HTTP 200 + an SSE error event. The SDK raises a
status-less APIError that no classifier rule matched, so it landed in
the retryable `unknown` bucket: every retry was re-sent (3 streamed
requests by default), a configured fallback only engaged after the
ladder, and the user was told the provider "looks temporarily
unavailable. Wait a minute and send /retry".

- error_classifier: match the ban phrase status-agnostically in
  _provider_special_cases -> provider_policy_blocked (non-retryable,
  fallback, no credential rotation: every key on a banned account is
  banned; a 403 variant is not a bad key).
- turn_failure_copy: provider_policy_blocked chat copy now covers an
  account block as well as data/privacy settings; add its cause gloss
  so cron and subagent notices explain it instead of printing raw text.
- cron: provider_policy_blocked action (retrying won't help; pin another
  model).
- docs: FAQ entry for the reply; auto-recovery ladder exclusion list.
This commit is contained in:
teknium1
2026-09-23 08:59:54 -07:00
committed by Teknium
parent 3cf26c82b4
commit 2c948e6aa2
6 changed files with 106 additions and 3 deletions

View File

@@ -342,6 +342,10 @@ _PROVIDER_POLICY_BLOCKED_PATTERNS = (
"no endpoints found matching your data policy",
)
# Upstream account ban relayed by an aggregator, often as HTTP 200 + an SSE error
# event (no status): permanent for this account, so never the transient retry ladder.
_ACCOUNT_POLICY_BLOCK_PATTERNS = ("blocked for a previous policy violation",)
# Per-prompt safety-filter blocks: deterministic for the unchanged request, so
# fallback immediately. Each phrase is verbatim from one provider (Codex cyber
# flags #18028, OpenAI moderation, Anthropic safety, Azure token, MiniMax
@@ -820,6 +824,9 @@ def _provider_special_cases(c: _Ctx) -> Optional[Verdict]:
# to format_error and a status-less block isn't left retryable (#18028).
if any(p in msg for p in _CONTENT_POLICY_BLOCKED_PATTERNS):
return _V_CONTENT_BLOCKED
# Status-agnostic: the stream-relayed ban has no status, and a 403 variant is not a bad key.
if any(p in msg for p in _ACCOUNT_POLICY_BLOCK_PATTERNS):
return _V_POLICY_BLOCKED
# ChatGPT Codex masks a rejected encrypted-reasoning replay behind the same bare
# ``invalid_prompt: Request blocked.`` it uses for real blocks (#92353). Exact envelope
# + provider only. The verdict keeps format_error's abort-and-fallback hints; the one

View File

@@ -183,8 +183,10 @@ _NONRETRYABLE_COPY: Dict[str, str] = {
"with /model."
),
FailoverReason.provider_policy_blocked.value: (
"{label}'s account settings don't allow this model for your request, so it didn't "
"answer. Check the provider's data/privacy settings, or switch models with /model."
"{label} refused this request because of a policy on your account (its data/privacy "
"settings, or a block the model's upstream provider placed on the account), so the model "
"didn't answer and retrying won't help. Check the account with the provider, or switch "
"models with /model."
),
FailoverReason.upstream_blocked.value: (
"A firewall/CDN in front of {label} blocked the request before it reached the model, so "
@@ -228,6 +230,9 @@ FAILURE_CAUSE_GLOSS: Dict[str, str] = {
FailoverReason.upstream_blocked.value: "a firewall/CDN in front of the AI model service blocked the request",
FailoverReason.model_not_found.value: "the model {subject} uses was not found at the AI model service",
FailoverReason.content_policy_blocked.value: "the AI model service's safety filter rejected the request",
FailoverReason.provider_policy_blocked.value: (
"the AI model service refused the request because of a policy on the account"
),
"context_overflow": "{possessive} request grew too large for the model",
"payload_too_large": "{possessive} request grew too large for the model",
}

View File

@@ -78,6 +78,10 @@ _PROVIDER_FAILURE_ACTION["content_policy_blocked"] = (
"Reword the job's prompt with `hermes cron edit {job_id} --prompt <text>`, or pick another "
"model with `hermes cron edit {job_id} --model <name>`."
)
_PROVIDER_FAILURE_ACTION["provider_policy_blocked"] = (
"Retrying won't help: check the account's status and data/privacy settings with the provider, "
"or pin another model with `hermes cron edit {job_id} --model <name>`."
)
_DEFAULT_FAILURE_ACTION = "Run it again with `hermes cron run {job_id}`, or edit it with `hermes cron edit {job_id}`."

View File

@@ -0,0 +1,81 @@
"""An upstream account ban relayed as HTTP 200 + an SSE ``error`` event is permanent.
OpenRouter relays OpenAI's "this user has been blocked for a previous policy violation"
inside a 200 stream; the SDK raises a status-less ``APIError``. Classified ``unknown`` it
was retried ``api_max_retries`` times and the user was told the provider "looks temporarily
unavailable" — advice that can never work for a banned account.
"""
import json
import threading
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import httpx
import openai
import pytest
import run_agent
from agent.error_classifier import FailoverReason, classify_api_error
BAN = (
"Policy Violation: this user has been blocked for a previous policy violation. "
"Learn more: https://platform.openai.com/docs/guides/safety-best-practices"
)
@pytest.mark.parametrize("status", [None, 403])
def test_account_ban_is_a_permanent_policy_block_not_transient_or_auth(status):
request = httpx.Request("POST", "https://openrouter.ai/api/v1/chat/completions")
if status is None: # what openai/_streaming.py raises for a 200 stream carrying {"error": ...}
err = openai.APIError(BAN, request, body={"message": BAN, "code": 403})
else:
err = openai.PermissionDeniedError(
BAN, response=httpx.Response(status, request=request), body={"error": {"message": BAN}}
)
result = classify_api_error(err, provider="openrouter", model="openai/gpt-4.1-nano")
assert result.reason == FailoverReason.provider_policy_blocked
assert result.retryable is False
assert result.should_fallback is True
# Every key on a banned account is banned: rotating the pool only burns credentials.
assert result.should_rotate_credential is False
def test_streamed_account_ban_fails_once_without_transient_advice():
streamed = []
class Handler(BaseHTTPRequestHandler):
def log_message(self, *_a):
pass
def do_POST(self):
body = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}")
if not self.path.endswith("/chat/completions"):
self.send_response(404)
self.end_headers()
return
if body.get("stream"):
streamed.append(body)
self.send_response(200)
self.send_header("content-type", "text/event-stream")
self.end_headers()
self.wfile.write(f"data: {json.dumps({'error': {'code': 403, 'message': BAN}})}\n\n".encode())
self.wfile.write(b"data: [DONE]\n\n")
self.wfile.flush()
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
threading.Thread(target=server.serve_forever, daemon=True).start()
try:
agent = run_agent.AIAgent(
api_key="test-key", base_url=f"http://127.0.0.1:{server.server_address[1]}/v1",
model="m", provider="custom", quiet_mode=True, skip_context_files=True,
skip_memory=True, enabled_toolsets=[], max_iterations=1,
)
result = agent.run_conversation("ping", conversation_history=[], task_id="t")
finally:
server.shutdown()
server.server_close()
assert len(streamed) == 1, f"a permanent ban was re-sent {len(streamed)} times"
assert result["failed"] is True
assert result["failure_retryable"] is False
assert "temporarily unavailable" not in result["final_response"]
assert BAN in result["final_response"]

View File

@@ -230,6 +230,12 @@ To isolate the source:
See [Security](../user-guide/security.md) for Hermes' documented execution controls and [Providers](../integrations/providers.md) for provider configuration.
#### "…refused this request because of a policy on your account"
**Meaning:** the provider rejected the request for an account-level reason that retrying cannot change — an aggregator's data/privacy settings excluded every endpoint for the model, or the model's upstream provider has blocked the account (for example `this user has been blocked for a previous policy violation`, which OpenRouter can relay inside an otherwise successful HTTP 200 stream). Hermes sends the request once, does not retry it or rotate credentials, and moves to your fallback chain if one is configured.
**Solution:** check the account's status and data/privacy settings with the provider named in the reply, or switch to another model or provider with `/model`. `hermes fallback add` routes future blocks to a backup automatically.
#### "Could not open a stream to `<host>` after N attempts (request X KB)"
**Meaning:** every connect attempt to that endpoint failed before a single stream event arrived, so nothing was billed; the normal retry/fallback chain still runs afterwards. The line names the host actually contacted, how many attempts were made, and the serialized request size — the three things that separate an outage from a request-size limit.

View File

@@ -1223,7 +1223,7 @@ agent:
`agent.api_max_retries` controls how many times Hermes retries a provider API call on transient errors (rate limits, connection drops, 5xx) **before** fallback-provider switching engages. The default is `3` — four attempts total. If you have [fallback providers](./features/fallback-providers.md) configured and want to fail over faster, drop this to `0` so the first transient error on your primary immediately hands off to the fallback instead of churning retries against the flaky endpoint.
`agent.auto_recovery_cycles` is the safety net *after* both the retries and the fallback chain are spent. When the failure is a transient outage (HTTP 5xx, an `overloaded`/529 response, a connect or read timeout) and no answer text has reached you yet, Hermes does not end the turn with "API failed after N retries" — it waits and tries again, up to this many cycles (default `5`), with a jittered 15/30/60/60/60 s schedule. A provider `Retry-After` header wins over the schedule (honoured up to 120 s). Every surface shows the same line while it waits — `⏳ Provider temporarily unavailable — retrying automatically in 30s (cycle 2/5); press Esc to stop` on the CLI/TUI/Desktop, a status bubble on messaging platforms (`send /stop to cancel`), a `hermes.status` SSE event on the API server, and a log line for cron jobs. Pressing Esc (or `/stop`) cancels the wait immediately. Fallback still comes first: with a fallback chain configured, exhaustion moves to the next provider as before, and the ladder only engages once the chain has nothing left. Authentication, billing, request-format, entitlement and content-policy errors never enter the ladder. Set `0` to disable it.
`agent.auto_recovery_cycles` is the safety net *after* both the retries and the fallback chain are spent. When the failure is a transient outage (HTTP 5xx, an `overloaded`/529 response, a connect or read timeout) and no answer text has reached you yet, Hermes does not end the turn with "API failed after N retries" — it waits and tries again, up to this many cycles (default `5`), with a jittered 15/30/60/60/60 s schedule. A provider `Retry-After` header wins over the schedule (honoured up to 120 s). Every surface shows the same line while it waits — `⏳ Provider temporarily unavailable — retrying automatically in 30s (cycle 2/5); press Esc to stop` on the CLI/TUI/Desktop, a status bubble on messaging platforms (`send /stop to cancel`), a `hermes.status` SSE event on the API server, and a log line for cron jobs. Pressing Esc (or `/stop`) cancels the wait immediately. Fallback still comes first: with a fallback chain configured, exhaustion moves to the next provider as before, and the ladder only engages once the chain has nothing left. Authentication, billing, request-format, entitlement, content-policy and account-policy errors never enter the ladder. Set `0` to disable it.
## Wall-Clock Run Budget