fix(classifier): terminal_quota_exhausted 429s classify as billing, not rate_limit

Port from code-yeongyu/oh-my-openagent#6677 (credit: @niStee).

LiteLLM proxies stamp a structured `terminal_quota_exhausted` code on
hard-cap 429s. Hermes' `_status_429` handler always returns a verdict, so
`_by_error_code` (which maps _BILLING_ERROR_CODES to billing) never saw
the code: the exhausted key classified as rate_limit, earned the 429
cooldown, and got retried against a wall that cannot clear until someone
pays. Upstream this respawned duplicate subagent sessions.

- `_status_429` now honors a structured billing code first (decisive
  signal outranks message heuristics).
- `terminal_quota_exhausted` joins _BILLING_ERROR_CODES so every path
  (429, 402, status-less) agrees.
- "hard billing limit" free text joins _BILLING_PATTERNS ("billing hard
  limit" was already there; providers use both orders). "terminal billing
  limit" text is deliberately NOT matched: substring rules cannot negate
  the "non-terminal billing limit" wording — the structured code covers it.
This commit is contained in:
Teknium
2026-09-09 02:38:20 -07:00
parent 833594cda6
commit 92df11f81d
2 changed files with 35 additions and 1 deletions

View File

@@ -92,6 +92,11 @@ _BILLING_PATTERNS = (
"billing hard limit", "exceeded your current quota", "account is deactivated", "plan does not include",
"out of extra usage", "out of funds", "run out of funds", "balance_depleted",
"model_not_supported_on_free_tier", "not available on the free tier",
# LiteLLM proxies word a hard cap as "hard billing limit" (structured twin:
# ``terminal_quota_exhausted`` in _BILLING_ERROR_CODES). "terminal billing
# limit" free text is NOT matched: substring rules can't negate the
# "non-terminal billing limit" wording, and the structured code covers it.
"hard billing limit",
)
# Not proof of exhaustion: Anthropic returns the same "out of extra usage" body
@@ -107,7 +112,7 @@ _XAI_SPENDING_LIMIT_ERROR_CODE = "personal-team-blocked:spending-limit"
_BILLING_ERROR_CODES = frozenset({
"insufficient_quota", "billing_not_active", "payment_required", "insufficient_credits",
"no_usable_credits", "balance_depleted", "model_not_supported_on_free_tier",
"member_spend_cap_exceeded", _XAI_SPENDING_LIMIT_ERROR_CODE,
"member_spend_cap_exceeded", "terminal_quota_exhausted", _XAI_SPENDING_LIMIT_ERROR_CODE,
})
# Transient rate limiting. Bedrock "Throttling error: Too many tokens" also
@@ -725,6 +730,11 @@ def _status_404(c: _Ctx) -> Verdict:
def _status_429(c: _Ctx) -> Verdict:
# A structured billing code is decisive: LiteLLM stamps
# ``terminal_quota_exhausted`` (a hard cap, not throttling) on 429s, and
# this handler always returns, so _by_error_code never sees the code.
if c.code in _BILLING_ERROR_CODES:
return _V_BILLING
# Z.AI/Zhipu reuse 429 for server-wide overload: back off on the same
# key instead of burning the pool (#14038).
if any(p in c.msg for p in _OVERLOADED_PATTERNS):