fix(aux): title calls on reasoning-mandatory routes stop sending a doomed disable
The title lane asks for thinking off (reasoning_config={"enabled": False}). A route
whose model cannot disable reasoning answers 400 "Reasoning is mandatory for this
endpoint and cannot be disabled"; the ladder then retried at the floor effort and
memoised the route, but only in memory. OpenRouter and the Nous Portal already
publish reasoning.mandatory per model in /v1/models, and Hermes mirrors that
catalog to cache/reasoning_caps.json. The aux floor never read it, so an aux-only
OpenRouter route (nothing else warms that catalog) paid the failed round-trip in
every process.
known_reasoning_floor now also floors a thinking-off aux call when the route's
catalog (memory, then disk mirror; never HTTP) flags the model mandatory, and
kicks the background warm on a cold catalog so the mirror answers every later
call and process. Optional-reasoning models keep their disable.
This commit is contained in:
@@ -79,15 +79,45 @@ def remember_reasoning_floor(
|
||||
_FLOORED_ROUTES.add((_route_key(provider, base_url), str(rejected_kwargs.get("model") or "")))
|
||||
|
||||
|
||||
_NOUS_PROVIDERS = {"nous", "nous-portal", "nousresearch"}
|
||||
|
||||
|
||||
def _catalog_marks_mandatory(provider: Optional[str], base_url: Optional[str], model: Optional[str]) -> bool:
|
||||
"""True when the route's ``/v1/models`` catalog (OpenRouter, Nous Portal) flags *model*
|
||||
``reasoning.mandatory``. Cache-only — memory, then the disk mirror — so it never blocks; a cold
|
||||
catalog is warmed in the background, and the mirror it writes answers every later call and process.
|
||||
Without this an aux-only OpenRouter route (nothing else warms that catalog) paid the 400 in every
|
||||
process."""
|
||||
provider_norm = str(provider or "").strip().lower()
|
||||
host = (urlparse(base_url or "").hostname or "").lower()
|
||||
from hermes_cli import models_reasoning_caps as caps_mod
|
||||
if provider_norm == "openrouter" or host == "openrouter.ai" or host.endswith(".openrouter.ai"):
|
||||
lookup, warm = caps_mod.openrouter_model_reasoning_capabilities, caps_mod.warm_openrouter_reasoning_caps_async
|
||||
elif provider_norm in _NOUS_PROVIDERS:
|
||||
lookup, warm = caps_mod.nous_model_reasoning_capabilities, caps_mod.warm_nous_reasoning_caps_async
|
||||
else:
|
||||
return False
|
||||
try:
|
||||
caps = lookup(model)
|
||||
if caps is None:
|
||||
warm()
|
||||
except Exception:
|
||||
return False
|
||||
return bool(caps and caps.get("mandatory"))
|
||||
|
||||
|
||||
def known_reasoning_floor(
|
||||
reasoning_config: Any, provider: Optional[str], base_url: Optional[str], model: Optional[str],
|
||||
task: Optional[str] = None,
|
||||
) -> Any:
|
||||
"""*reasoning_config* lifted to the floor when this route+model is known to refuse a disable; unchanged
|
||||
"""*reasoning_config* lifted to the floor when this route+model is known to refuse a disable — learned
|
||||
from an earlier 400 in this process, or flagged mandatory by the route's model catalog; unchanged
|
||||
otherwise. Runs before the profile projection so every wire shape starts at the floor."""
|
||||
if not _is_disabled(reasoning_config):
|
||||
return reasoning_config
|
||||
if (_route_key(provider, base_url), str(model or "")) not in _FLOORED_ROUTES:
|
||||
if (_route_key(provider, base_url), str(model or "")) not in _FLOORED_ROUTES and not _catalog_marks_mandatory(
|
||||
provider, base_url, model,
|
||||
):
|
||||
return reasoning_config
|
||||
logger.info(
|
||||
"Auxiliary %s: %s (%s) cannot disable reasoning; sending effort=%s up front",
|
||||
|
||||
101
tests/agent/test_auxiliary_reasoning_catalog_floor.py
Normal file
101
tests/agent/test_auxiliary_reasoning_catalog_floor.py
Normal file
@@ -0,0 +1,101 @@
|
||||
"""The route's model catalog decides the title lane's first request (no guaranteed 400 on mandatory routes).
|
||||
|
||||
OpenRouter's (and the Portal's) ``/v1/models`` flags ``reasoning.mandatory`` per model; such a route answers
|
||||
a thinking-off aux call (``reasoning: {enabled: false}``) with 400 "Reasoning is mandatory for this endpoint
|
||||
and cannot be disabled". The floor memo only learned that from the 400 itself, per process, so an aux-only
|
||||
OpenRouter route — nothing else warms that catalog — paid the failed round-trip in every process.
|
||||
"""
|
||||
|
||||
import json
|
||||
import threading
|
||||
from unittest.mock import MagicMock, patch
|
||||
|
||||
import pytest
|
||||
|
||||
import hermes_cli.models as models_mod
|
||||
from agent import auxiliary_reasoning_floor
|
||||
from agent.auxiliary_client import call_llm
|
||||
from hermes_cli import models_reasoning_caps
|
||||
|
||||
_CATALOG = [
|
||||
{"id": "openai/gpt-oss-20b", "supported_parameters": ["reasoning", "tools"],
|
||||
"reasoning": {"mandatory": True, "supported_efforts": ["high", "medium", "low"]}},
|
||||
{"id": "x-ai/grok-4.3", "supported_parameters": ["reasoning", "tools"],
|
||||
"reasoning": {"mandatory": False, "supported_efforts": ["high", "medium", "low", "none"]}},
|
||||
]
|
||||
|
||||
|
||||
@pytest.fixture
|
||||
def fresh_process(monkeypatch):
|
||||
"""Module state of a newly started process (catalog neither in memory nor memo); disk untouched."""
|
||||
def _reset():
|
||||
auxiliary_reasoning_floor._FLOORED_ROUTES.clear()
|
||||
for name in ("_openrouter_reasoning_caps_cache", "_openrouter_reasoning_caps_failed_at"):
|
||||
monkeypatch.setattr(models_mod, name, None)
|
||||
for name in ("_openrouter_caps_disk_checked", "_openrouter_caps_warm_started"):
|
||||
monkeypatch.setattr(models_mod, name, False)
|
||||
_reset()
|
||||
yield _reset
|
||||
auxiliary_reasoning_floor._FLOORED_ROUTES.clear()
|
||||
|
||||
|
||||
def _title_request(model):
|
||||
client = MagicMock()
|
||||
client.base_url = "https://openrouter.ai/api/v1"
|
||||
client.chat.completions.create.return_value = {"ok": True}
|
||||
with (
|
||||
patch("agent.auxiliary_client._resolve_task_provider_model",
|
||||
return_value=("openrouter", model, None, "sk-or-x", None)),
|
||||
patch("agent.auxiliary_client._get_cached_client", return_value=(client, model)),
|
||||
patch("agent.auxiliary_client._validate_llm_response", side_effect=lambda resp, _task, **_kw: resp),
|
||||
patch("agent.auxiliary_client._try_payment_fallback", return_value=None),
|
||||
):
|
||||
call_llm(task="title_generation", messages=[{"role": "user", "content": "hi"}],
|
||||
reasoning_config={"enabled": False})
|
||||
return client.chat.completions.create.call_args_list[0].kwargs.get("extra_body", {}).get("reasoning")
|
||||
|
||||
|
||||
def test_catalog_mandatory_model_gets_the_floor_on_the_first_request(fresh_process):
|
||||
"""A mirror left by an earlier process: the mandatory model's first thinking-off request already
|
||||
carries the floor; an optional model on the same route keeps its disable."""
|
||||
models_reasoning_caps._seed_reasoning_caps(models_reasoning_caps._OPENROUTER_CATALOG_URL, _CATALOG)
|
||||
|
||||
assert _title_request("openai/gpt-oss-20b") == {
|
||||
"enabled": True, "effort": auxiliary_reasoning_floor.REASONING_FLOOR_EFFORT}
|
||||
assert _title_request("x-ai/grok-4.3") == {"enabled": False}
|
||||
|
||||
|
||||
def test_cold_catalog_is_warmed_so_the_next_process_starts_at_the_floor(fresh_process, monkeypatch):
|
||||
"""No mirror yet: the first process can only learn from the 400, but its lookup warms the catalog, so
|
||||
the next process never sends the disable."""
|
||||
monkeypatch.delenv("PYTEST_CURRENT_TEST", raising=False) # the warmer is a no-op under pytest
|
||||
warmed = []
|
||||
real_thread = threading.Thread
|
||||
|
||||
def _joined_thread(*args, **kwargs):
|
||||
thread = real_thread(*args, **kwargs)
|
||||
if kwargs.get("name") == "reasoning-caps-warm":
|
||||
warmed.append(thread)
|
||||
return thread
|
||||
|
||||
class _Resp:
|
||||
def __enter__(self):
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
def read(self):
|
||||
return json.dumps({"data": _CATALOG}).encode()
|
||||
|
||||
monkeypatch.setattr(models_mod, "_urlopen_model_catalog_request", lambda req, timeout: _Resp())
|
||||
monkeypatch.setattr(models_reasoning_caps.threading, "Thread", _joined_thread)
|
||||
|
||||
_title_request("openai/gpt-oss-20b")
|
||||
assert warmed, "a cold catalog lookup must start the background warm"
|
||||
for thread in warmed:
|
||||
thread.join(timeout=10)
|
||||
|
||||
fresh_process()
|
||||
assert _title_request("openai/gpt-oss-20b") == {
|
||||
"enabled": True, "effort": auxiliary_reasoning_floor.REASONING_FLOOR_EFFORT}
|
||||
@@ -1474,6 +1474,8 @@ This is the per-task counterpart of the global `agent.reasoning_effort`: run com
|
||||
|
||||
If the endpoint rejects the reasoning field outright (a chat-only model behind an OpenAI-compatible relay answering `400 Unrecognized request argument supplied: reasoning_effort`, or the reversed wording `400 reasoning_effort 'none' unsupported; use minimal|low|medium|high|xhigh`), the auxiliary call is retried once with every reasoning field omitted, so the task (for example the session title) still completes with the endpoint's default behaviour. The main conversation applies the same recovery: when a route rejects the reasoning-off request Hermes sends for a thinking-only truncated continuation, the disable is dropped for the rest of the session and the request is retried with the route's default.
|
||||
|
||||
Some models cannot turn thinking off at all (`400 Reasoning is mandatory for this endpoint and cannot be disabled`). For those, a thinking-off auxiliary call (title generation, or any task set to `reasoning_effort: none`) goes out at the lowest effort (`low`) instead of the disable. Hermes knows ahead of time when the route's model catalog marks the model mandatory (OpenRouter and Nous Portal `/v1/models`, cached in `cache/reasoning_caps.json`), or when the route already answered an earlier disable that way in the same process. So the rejected request is not sent. A fresh install with no cached catalog can still see that 400 once: the lookup fetches the catalog in the background and later calls use it.
|
||||
|
||||
**Background review is different:** a same-model review fork always inherits the parent's reasoning effort. `auxiliary.background_review.reasoning_effort` is ignored on that path, including when the parent provider/model is explicitly selected. This preserves byte-identical reasoning settings, system prompt, full conversation snapshot, and tool definitions for prompt-cache parity; there is no independent-effort switch for same-model reviews. See [background review reasoning](./features/memory.md#same-model-review-reasoning). When the review is routed to a different provider/model, `reasoning_effort` applies to that routed fork (unset = the routed provider's default). Hermes prints a one-time warning when the key is set but the review runs on the main model.
|
||||
|
||||
**MoA also uses a different configuration:** reasoning depth for Mixture-of-Agents is configured **per slot** in the MoA preset (`moa.presets.<name>.reference_models[].reasoning_effort` / `aggregator.reasoning_effort`), not on the `moa_reference`/`moa_aggregator` auxiliary blocks — see [Mixture of Agents](./features/mixture-of-agents.md).
|
||||
|
||||
Reference in New Issue
Block a user