fix(agent): an upstream account ban relayed in a 200 stream fails once instead of retrying as an outage
OpenRouter relays OpenAI's "this user has been blocked for a previous policy violation" as HTTP 200 + an SSE error event. The SDK raises a status-less APIError that no classifier rule matched, so it landed in the retryable `unknown` bucket: every retry was re-sent (3 streamed requests by default), a configured fallback only engaged after the ladder, and the user was told the provider "looks temporarily unavailable. Wait a minute and send /retry". - error_classifier: match the ban phrase status-agnostically in _provider_special_cases -> provider_policy_blocked (non-retryable, fallback, no credential rotation: every key on a banned account is banned; a 403 variant is not a bad key). - turn_failure_copy: provider_policy_blocked chat copy now covers an account block as well as data/privacy settings; add its cause gloss so cron and subagent notices explain it instead of printing raw text. - cron: provider_policy_blocked action (retrying won't help; pin another model). - docs: FAQ entry for the reply; auto-recovery ladder exclusion list.
This commit is contained in:
@@ -342,6 +342,10 @@ _PROVIDER_POLICY_BLOCKED_PATTERNS = (
|
||||
"no endpoints found matching your data policy",
|
||||
)
|
||||
|
||||
# Upstream account ban relayed by an aggregator, often as HTTP 200 + an SSE error
|
||||
# event (no status): permanent for this account, so never the transient retry ladder.
|
||||
_ACCOUNT_POLICY_BLOCK_PATTERNS = ("blocked for a previous policy violation",)
|
||||
|
||||
# Per-prompt safety-filter blocks: deterministic for the unchanged request, so
|
||||
# fallback immediately. Each phrase is verbatim from one provider (Codex cyber
|
||||
# flags #18028, OpenAI moderation, Anthropic safety, Azure token, MiniMax
|
||||
@@ -820,6 +824,9 @@ def _provider_special_cases(c: _Ctx) -> Optional[Verdict]:
|
||||
# to format_error and a status-less block isn't left retryable (#18028).
|
||||
if any(p in msg for p in _CONTENT_POLICY_BLOCKED_PATTERNS):
|
||||
return _V_CONTENT_BLOCKED
|
||||
# Status-agnostic: the stream-relayed ban has no status, and a 403 variant is not a bad key.
|
||||
if any(p in msg for p in _ACCOUNT_POLICY_BLOCK_PATTERNS):
|
||||
return _V_POLICY_BLOCKED
|
||||
# ChatGPT Codex masks a rejected encrypted-reasoning replay behind the same bare
|
||||
# ``invalid_prompt: Request blocked.`` it uses for real blocks (#92353). Exact envelope
|
||||
# + provider only. The verdict keeps format_error's abort-and-fallback hints; the one
|
||||
|
||||
@@ -183,8 +183,10 @@ _NONRETRYABLE_COPY: Dict[str, str] = {
|
||||
"with /model."
|
||||
),
|
||||
FailoverReason.provider_policy_blocked.value: (
|
||||
"{label}'s account settings don't allow this model for your request, so it didn't "
|
||||
"answer. Check the provider's data/privacy settings, or switch models with /model."
|
||||
"{label} refused this request because of a policy on your account (its data/privacy "
|
||||
"settings, or a block the model's upstream provider placed on the account), so the model "
|
||||
"didn't answer and retrying won't help. Check the account with the provider, or switch "
|
||||
"models with /model."
|
||||
),
|
||||
FailoverReason.upstream_blocked.value: (
|
||||
"A firewall/CDN in front of {label} blocked the request before it reached the model, so "
|
||||
@@ -228,6 +230,9 @@ FAILURE_CAUSE_GLOSS: Dict[str, str] = {
|
||||
FailoverReason.upstream_blocked.value: "a firewall/CDN in front of the AI model service blocked the request",
|
||||
FailoverReason.model_not_found.value: "the model {subject} uses was not found at the AI model service",
|
||||
FailoverReason.content_policy_blocked.value: "the AI model service's safety filter rejected the request",
|
||||
FailoverReason.provider_policy_blocked.value: (
|
||||
"the AI model service refused the request because of a policy on the account"
|
||||
),
|
||||
"context_overflow": "{possessive} request grew too large for the model",
|
||||
"payload_too_large": "{possessive} request grew too large for the model",
|
||||
}
|
||||
|
||||
@@ -78,6 +78,10 @@ _PROVIDER_FAILURE_ACTION["content_policy_blocked"] = (
|
||||
"Reword the job's prompt with `hermes cron edit {job_id} --prompt <text>`, or pick another "
|
||||
"model with `hermes cron edit {job_id} --model <name>`."
|
||||
)
|
||||
_PROVIDER_FAILURE_ACTION["provider_policy_blocked"] = (
|
||||
"Retrying won't help: check the account's status and data/privacy settings with the provider, "
|
||||
"or pin another model with `hermes cron edit {job_id} --model <name>`."
|
||||
)
|
||||
_DEFAULT_FAILURE_ACTION = "Run it again with `hermes cron run {job_id}`, or edit it with `hermes cron edit {job_id}`."
|
||||
|
||||
|
||||
|
||||
81
tests/agent/test_account_policy_block.py
Normal file
81
tests/agent/test_account_policy_block.py
Normal file
@@ -0,0 +1,81 @@
|
||||
"""An upstream account ban relayed as HTTP 200 + an SSE ``error`` event is permanent.
|
||||
|
||||
OpenRouter relays OpenAI's "this user has been blocked for a previous policy violation"
|
||||
inside a 200 stream; the SDK raises a status-less ``APIError``. Classified ``unknown`` it
|
||||
was retried ``api_max_retries`` times and the user was told the provider "looks temporarily
|
||||
unavailable" — advice that can never work for a banned account.
|
||||
"""
|
||||
import json
|
||||
import threading
|
||||
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
|
||||
|
||||
import httpx
|
||||
import openai
|
||||
import pytest
|
||||
|
||||
import run_agent
|
||||
from agent.error_classifier import FailoverReason, classify_api_error
|
||||
|
||||
BAN = (
|
||||
"Policy Violation: this user has been blocked for a previous policy violation. "
|
||||
"Learn more: https://platform.openai.com/docs/guides/safety-best-practices"
|
||||
)
|
||||
|
||||
|
||||
@pytest.mark.parametrize("status", [None, 403])
|
||||
def test_account_ban_is_a_permanent_policy_block_not_transient_or_auth(status):
|
||||
request = httpx.Request("POST", "https://openrouter.ai/api/v1/chat/completions")
|
||||
if status is None: # what openai/_streaming.py raises for a 200 stream carrying {"error": ...}
|
||||
err = openai.APIError(BAN, request, body={"message": BAN, "code": 403})
|
||||
else:
|
||||
err = openai.PermissionDeniedError(
|
||||
BAN, response=httpx.Response(status, request=request), body={"error": {"message": BAN}}
|
||||
)
|
||||
result = classify_api_error(err, provider="openrouter", model="openai/gpt-4.1-nano")
|
||||
assert result.reason == FailoverReason.provider_policy_blocked
|
||||
assert result.retryable is False
|
||||
assert result.should_fallback is True
|
||||
# Every key on a banned account is banned: rotating the pool only burns credentials.
|
||||
assert result.should_rotate_credential is False
|
||||
|
||||
|
||||
def test_streamed_account_ban_fails_once_without_transient_advice():
|
||||
streamed = []
|
||||
|
||||
class Handler(BaseHTTPRequestHandler):
|
||||
def log_message(self, *_a):
|
||||
pass
|
||||
|
||||
def do_POST(self):
|
||||
body = json.loads(self.rfile.read(int(self.headers.get("content-length", 0))) or b"{}")
|
||||
if not self.path.endswith("/chat/completions"):
|
||||
self.send_response(404)
|
||||
self.end_headers()
|
||||
return
|
||||
if body.get("stream"):
|
||||
streamed.append(body)
|
||||
self.send_response(200)
|
||||
self.send_header("content-type", "text/event-stream")
|
||||
self.end_headers()
|
||||
self.wfile.write(f"data: {json.dumps({'error': {'code': 403, 'message': BAN}})}\n\n".encode())
|
||||
self.wfile.write(b"data: [DONE]\n\n")
|
||||
self.wfile.flush()
|
||||
|
||||
server = ThreadingHTTPServer(("127.0.0.1", 0), Handler)
|
||||
threading.Thread(target=server.serve_forever, daemon=True).start()
|
||||
try:
|
||||
agent = run_agent.AIAgent(
|
||||
api_key="test-key", base_url=f"http://127.0.0.1:{server.server_address[1]}/v1",
|
||||
model="m", provider="custom", quiet_mode=True, skip_context_files=True,
|
||||
skip_memory=True, enabled_toolsets=[], max_iterations=1,
|
||||
)
|
||||
result = agent.run_conversation("ping", conversation_history=[], task_id="t")
|
||||
finally:
|
||||
server.shutdown()
|
||||
server.server_close()
|
||||
|
||||
assert len(streamed) == 1, f"a permanent ban was re-sent {len(streamed)} times"
|
||||
assert result["failed"] is True
|
||||
assert result["failure_retryable"] is False
|
||||
assert "temporarily unavailable" not in result["final_response"]
|
||||
assert BAN in result["final_response"]
|
||||
@@ -230,6 +230,12 @@ To isolate the source:
|
||||
|
||||
See [Security](../user-guide/security.md) for Hermes' documented execution controls and [Providers](../integrations/providers.md) for provider configuration.
|
||||
|
||||
#### "…refused this request because of a policy on your account"
|
||||
|
||||
**Meaning:** the provider rejected the request for an account-level reason that retrying cannot change — an aggregator's data/privacy settings excluded every endpoint for the model, or the model's upstream provider has blocked the account (for example `this user has been blocked for a previous policy violation`, which OpenRouter can relay inside an otherwise successful HTTP 200 stream). Hermes sends the request once, does not retry it or rotate credentials, and moves to your fallback chain if one is configured.
|
||||
|
||||
**Solution:** check the account's status and data/privacy settings with the provider named in the reply, or switch to another model or provider with `/model`. `hermes fallback add` routes future blocks to a backup automatically.
|
||||
|
||||
#### "Could not open a stream to `<host>` after N attempts (request X KB)"
|
||||
|
||||
**Meaning:** every connect attempt to that endpoint failed before a single stream event arrived, so nothing was billed; the normal retry/fallback chain still runs afterwards. The line names the host actually contacted, how many attempts were made, and the serialized request size — the three things that separate an outage from a request-size limit.
|
||||
|
||||
@@ -1223,7 +1223,7 @@ agent:
|
||||
|
||||
`agent.api_max_retries` controls how many times Hermes retries a provider API call on transient errors (rate limits, connection drops, 5xx) **before** fallback-provider switching engages. The default is `3` — four attempts total. If you have [fallback providers](./features/fallback-providers.md) configured and want to fail over faster, drop this to `0` so the first transient error on your primary immediately hands off to the fallback instead of churning retries against the flaky endpoint.
|
||||
|
||||
`agent.auto_recovery_cycles` is the safety net *after* both the retries and the fallback chain are spent. When the failure is a transient outage (HTTP 5xx, an `overloaded`/529 response, a connect or read timeout) and no answer text has reached you yet, Hermes does not end the turn with "API failed after N retries" — it waits and tries again, up to this many cycles (default `5`), with a jittered 15/30/60/60/60 s schedule. A provider `Retry-After` header wins over the schedule (honoured up to 120 s). Every surface shows the same line while it waits — `⏳ Provider temporarily unavailable — retrying automatically in 30s (cycle 2/5); press Esc to stop` on the CLI/TUI/Desktop, a status bubble on messaging platforms (`send /stop to cancel`), a `hermes.status` SSE event on the API server, and a log line for cron jobs. Pressing Esc (or `/stop`) cancels the wait immediately. Fallback still comes first: with a fallback chain configured, exhaustion moves to the next provider as before, and the ladder only engages once the chain has nothing left. Authentication, billing, request-format, entitlement and content-policy errors never enter the ladder. Set `0` to disable it.
|
||||
`agent.auto_recovery_cycles` is the safety net *after* both the retries and the fallback chain are spent. When the failure is a transient outage (HTTP 5xx, an `overloaded`/529 response, a connect or read timeout) and no answer text has reached you yet, Hermes does not end the turn with "API failed after N retries" — it waits and tries again, up to this many cycles (default `5`), with a jittered 15/30/60/60/60 s schedule. A provider `Retry-After` header wins over the schedule (honoured up to 120 s). Every surface shows the same line while it waits — `⏳ Provider temporarily unavailable — retrying automatically in 30s (cycle 2/5); press Esc to stop` on the CLI/TUI/Desktop, a status bubble on messaging platforms (`send /stop to cancel`), a `hermes.status` SSE event on the API server, and a log line for cron jobs. Pressing Esc (or `/stop`) cancels the wait immediately. Fallback still comes first: with a fallback chain configured, exhaustion moves to the next provider as before, and the ladder only engages once the chain has nothing left. Authentication, billing, request-format, entitlement, content-policy and account-policy errors never enter the ladder. Set `0` to disable it.
|
||||
|
||||
## Wall-Clock Run Budget
|
||||
|
||||
|
||||
Reference in New Issue
Block a user