e7bff4b6d8
* refactor(fallback): share the pinned-owner chain rule
delegate_task's _resolve_child_fallback_chain decides which fallback chain
a child may walk: a pinned child never borrows the parent chain, an explicit
[] disables fallback, a declared list is the child's own. Cron needs the
same rule for pinned jobs (#100437), so the body moves to
hermes_cli.fallback_config.scoped_fallback_chain and the delegation helper
becomes a thin caller. Behaviour is unchanged; the delegation matrix test
still pins every cell.
* fix(cron): a pinned job never falls back to the global chain
A job with its own provider, model or base_url is an explicit operator pin
(since 0469740ab3 unpinned jobs store none of these). It still walked the
global fallback_providers chain in two places, so a pinned job could run
on a different provider and model than the one chosen:
- _resolve_job_runtime walked the chain on an AuthError or transient
network failure while resolving the pinned primary;
- _resolve_cron_agent_setup handed the global chain to every cron agent as
fallback_model, so the conversation loop's provider ladder could swap a
pinned job mid-run.
Both now read _job_fallback_chain(job, cfg), which returns no chain for a
pinned job through the same scoped_fallback_chain rule delegate_task uses
for pinned children. The pre-dispatch key check reads it too: the global
chain used to skip that check for every job, so a pinned job with a
missing key now blocks before the agent is built instead of failing in the
resolver. The transient-failure notice for a pinned job says it does not
fall back and names --unpin, instead of "No backup provider succeeded".
Unpinned jobs (including legacy *_snapshot records) and same-provider
credential-pool rotation are unchanged. The two scheduler tests that
asserted atomic provider+model fallback swaps used pinned jobs; they now
use unpinned jobs and keep the same assertions.
No per-job fallback_providers list: jobs have no generic override field
(create_job/update_job, the cronjob tool schema and the CLI enumerate each
field), so an opt-in chain would be a new surface on all of them. The
escape hatch is to leave the job unpinned and pick its model with
cron.model / cron.model_provider.
Co-authored-by: 686f6c61 <6115107+686f6c61@users.noreply.github.com>
* docs(cron): pinned jobs do not use fallback_providers
cron.md "Provider recovery" and the pre-dispatch key check, the cron rows
and section in fallback-providers.md, and the developer notes in
cron-internals.md / provider-runtime.md said every cron job inherits the
global chain. State the new rule, the compatibility note for users who
relied on a pinned job landing on the chain, and the unpinned + cron.model
alternative.
---------
Co-authored-by: 686f6c61 <6115107+686f6c61@users.noreply.github.com>
230 lines
11 KiB
Markdown
230 lines
11 KiB
Markdown
---
|
|
sidebar_position: 4
|
|
title: "Provider Runtime Resolution"
|
|
description: "How Hermes resolves providers, credentials, API modes, and auxiliary models at runtime"
|
|
---
|
|
|
|
# Provider Runtime Resolution
|
|
|
|
Hermes has a shared provider runtime resolver used across:
|
|
|
|
- CLI
|
|
- gateway
|
|
- cron jobs
|
|
- ACP
|
|
- auxiliary model calls
|
|
|
|
Primary implementation:
|
|
|
|
- `hermes_cli/runtime_provider.py` — credential resolution, custom-endpoint runtime resolution
|
|
- `hermes_cli/auth.py` — provider registry, `resolve_provider()`
|
|
- `hermes_cli/model_switch.py` — shared `/model` switch pipeline (CLI + gateway)
|
|
- `agent/auxiliary_client.py` — auxiliary model routing
|
|
- `providers/` — ABC + registry entry points (`ProviderProfile`, `register_provider`, `get_provider_profile`, `list_providers`)
|
|
- `plugins/model-providers/<name>/` — per-provider plugins (bundled) that declare `api_mode`, `base_url`, `env_vars`, `fallback_models` and register themselves into the registry on first access. User plugins at `$HERMES_HOME/plugins/model-providers/<name>/` override bundled ones of the same name.
|
|
|
|
`get_provider_profile()` in `providers/` returns a `ProviderProfile` for a given provider id. `runtime_provider.py` calls this at resolution time to get the canonical `base_url`, `env_vars` priority list, `api_mode`, and `fallback_models` without needing to duplicate that data in multiple files. Adding a new plugin under `plugins/model-providers/<your-provider>/` (or `$HERMES_HOME/plugins/model-providers/<your-provider>/`) that calls `register_provider()` is enough for `runtime_provider.py` to pick it up — no branch needed in the resolver itself.
|
|
|
|
If you are trying to add a new first-class inference provider, read [Adding Providers](./adding-providers.md) and the [Model Provider Plugin guide](./model-provider-plugin.md) alongside this page.
|
|
|
|
## Chat-completions reasoning shapes
|
|
|
|
OpenAI-compatible relays can return `reasoning` or `reasoning_content` as strings,
|
|
text-part dictionaries, or lists of text parts and string fragments. Hermes flattens
|
|
these fields before string operations in the main stream, Relay recording, synchronous
|
|
and asynchronous auxiliary streams, and completed-response reasoning extraction.
|
|
Fragments retain their explicit whitespace; normalization adds no intra-field separator.
|
|
Main-stream and Relay recording retain the existing paragraph breaks between complete
|
|
bold reasoning headings. Reasoning stays separate from the visible answer.
|
|
|
|
`delta.reasoning_details` is a list of opaque provider records. Both the main
|
|
stream and Relay recording append these records in arrival order without
|
|
flattening, merging, or rewriting their contents. Providers can emit complete
|
|
records on the final delta or across multiple deltas; omit the field on chunks
|
|
without new records. The collected records pass through response normalization
|
|
and assistant-message storage into session replay, including nested signed payloads.
|
|
|
|
## Resolution precedence
|
|
|
|
At a high level, provider resolution uses:
|
|
|
|
1. explicit CLI/runtime request
|
|
2. `config.yaml` model/provider config
|
|
3. environment variables
|
|
4. provider-specific defaults or auto resolution
|
|
|
|
That ordering matters because Hermes treats the saved model/provider choice as the source of truth for normal runs. This prevents a stale shell export from silently overriding the endpoint a user last selected in `hermes model`.
|
|
|
|
## Providers
|
|
|
|
Current provider families include (see `plugins/model-providers/` for the complete bundled set):
|
|
|
|
- AI Gateway (Vercel)
|
|
- OpenRouter
|
|
- Nous Portal
|
|
- OpenAI Codex
|
|
- Copilot / Copilot ACP
|
|
- Anthropic (native)
|
|
- Google / Gemini (`gemini`)
|
|
- Alibaba / DashScope (`alibaba`, `alibaba-coding-plan`)
|
|
- DeepSeek
|
|
- Z.AI
|
|
- Kimi / Moonshot (`kimi-coding`, `kimi-coding-cn`)
|
|
- MiniMax (`minimax`, `minimax-cn`, `minimax-oauth`)
|
|
- Kilo Code
|
|
- Hugging Face
|
|
- OpenCode Zen / OpenCode Go
|
|
- AWS Bedrock
|
|
- Azure Foundry
|
|
- NVIDIA NIM
|
|
- xAI (Grok)
|
|
- Arcee
|
|
- GMI Cloud
|
|
- StepFun
|
|
- Qwen OAuth
|
|
- Xiaomi
|
|
- Ollama Cloud
|
|
- LM Studio
|
|
- Tencent TokenHub
|
|
- Custom (`provider: custom`) — first-class provider for any OpenAI-compatible endpoint
|
|
- Named custom providers (`providers:` dict in config.yaml; the legacy `custom_providers` list is still read for backward compatibility)
|
|
|
|
## Output of runtime resolution
|
|
|
|
The runtime resolver returns data such as:
|
|
|
|
- `provider`
|
|
- `api_mode`
|
|
- `base_url`
|
|
- `api_key`
|
|
- `source`
|
|
- provider-specific metadata like expiry/refresh info
|
|
|
|
## Why this matters
|
|
|
|
This resolver is the main reason Hermes can share auth/runtime logic between:
|
|
|
|
- `hermes chat`
|
|
- gateway message handling
|
|
- cron jobs running in fresh sessions
|
|
- ACP editor sessions
|
|
- auxiliary model tasks
|
|
|
|
## AI Gateway
|
|
|
|
Set `AI_GATEWAY_API_KEY` in `~/.hermes/.env` and run with `--provider ai-gateway`. Hermes fetches available models from the gateway's `/models` endpoint, filtering to language models with tool-use support.
|
|
|
|
## OpenRouter, AI Gateway, and custom OpenAI-compatible base URLs
|
|
|
|
Hermes contains logic to avoid leaking the wrong API key to a custom endpoint when multiple provider keys exist (e.g. `OPENROUTER_API_KEY`, `AI_GATEWAY_API_KEY`, and `OPENAI_API_KEY`).
|
|
|
|
Each provider's API key is scoped to its own base URL:
|
|
|
|
- `OPENROUTER_API_KEY` is only sent to `openrouter.ai` endpoints
|
|
- `AI_GATEWAY_API_KEY` is only sent to `ai-gateway.vercel.sh` endpoints
|
|
- `OPENAI_API_KEY` is used for custom endpoints and as a fallback
|
|
|
|
Hermes also distinguishes between:
|
|
|
|
- a real custom endpoint selected by the user
|
|
- the OpenRouter fallback path used when no custom endpoint is configured
|
|
|
|
That distinction is especially important for:
|
|
|
|
- local model servers
|
|
- non-OpenRouter/non-AI Gateway OpenAI-compatible APIs
|
|
- switching providers without re-running setup
|
|
- config-saved custom endpoints that should keep working even when `OPENAI_BASE_URL` is not exported in the current shell
|
|
|
|
## Native Anthropic path
|
|
|
|
Anthropic is not just "via OpenRouter" anymore.
|
|
|
|
When provider resolution selects `anthropic`, Hermes uses:
|
|
|
|
- `api_mode = anthropic_messages`
|
|
- the native Anthropic Messages API
|
|
- `agent/anthropic_adapter.py` for translation
|
|
|
|
Credential resolution for native Anthropic now prefers refreshable Claude Code credentials over copied env tokens when both are present. In practice that means:
|
|
|
|
- Claude Code credential files are treated as the preferred source when they include refreshable auth
|
|
- manual `ANTHROPIC_TOKEN` / `CLAUDE_CODE_OAUTH_TOKEN` values still work as explicit overrides
|
|
- Hermes preflights Anthropic credential refresh before native Messages API calls
|
|
- Hermes still retries once on a 401 after rebuilding the Anthropic client, as a fallback path
|
|
|
|
## OpenAI Codex path
|
|
|
|
Codex uses a separate Responses API path:
|
|
|
|
- `api_mode = codex_responses`
|
|
- dedicated credential resolution and auth store support
|
|
- a resumed session whose lingering Codex reasoning items (`encrypted_content`) are rejected — as a 400 `invalid_encrypted_content` or as a 401 `token_expired` — self-heals by stripping the cached items and replaying once, before any credential refresh or pool rotation
|
|
|
|
## Auxiliary model routing
|
|
|
|
Auxiliary tasks such as:
|
|
|
|
- vision
|
|
- web extraction summarization
|
|
- context compression summaries
|
|
- skills hub operations
|
|
- MCP helper operations
|
|
- memory flushes
|
|
|
|
can use their own provider/model routing rather than the main conversational model.
|
|
|
|
When an auxiliary task is configured with provider `main`, Hermes resolves that through the same shared runtime path as normal chat. In practice that means:
|
|
|
|
- env-driven custom endpoints still work
|
|
- custom endpoints saved via `hermes model` / `config.yaml` also work
|
|
- auxiliary routing can tell the difference between a real saved custom endpoint and the OpenRouter fallback
|
|
|
|
## Fallback models
|
|
|
|
Hermes supports a configured fallback provider chain — a list of `(provider, model)` entries tried in order when the primary model encounters errors. The legacy single-pair `fallback_model` dict is still accepted for back-compat (and migrated on first write).
|
|
|
|
### How it works internally
|
|
|
|
1. **Storage**: `AIAgent.__init__` stores the `fallback_model` dict and sets `_fallback_activated = False`.
|
|
|
|
2. **Trigger points**: `_try_activate_fallback()` (forwarded to `try_activate_fallback()` in `agent/chat_completion_helpers.py`) is called from three places in the turn phases (`agent/turn_api_error.py`, `agent/turn_response_check.py`, `agent/turn_recovery.py`):
|
|
- After max retries on invalid API responses (None choices, missing content)
|
|
- On non-retryable client errors (HTTP 401, 403, 404)
|
|
- After max retries on transient errors (HTTP 429, 500, 502, 503)
|
|
|
|
3. **Activation flow** (`_try_activate_fallback`):
|
|
- Returns `False` immediately if already activated or not configured
|
|
- Calls `resolve_provider_client()` from `auxiliary_client.py` to build a new client with proper auth
|
|
- Determines `api_mode`: `codex_responses` for openai-codex, `anthropic_messages` for anthropic, `chat_completions` for everything else
|
|
- Swaps in-place: `self.model`, `self.provider`, `self.base_url`, `self.api_mode`, `self.client`, `self._client_kwargs`
|
|
- For anthropic fallback: builds a native Anthropic client instead of OpenAI-compatible
|
|
- Re-evaluates prompt caching (enabled for Claude models on OpenRouter)
|
|
- Sets `_fallback_activated = True` — prevents firing again
|
|
- Resets retry count to 0 and continues the loop
|
|
|
|
4. **Config flow**:
|
|
- CLI: reads the fallback chain via `hermes_cli/fallback_config.get_fallback_chain()` → passes to `AIAgent(fallback_model=...)`
|
|
- Gateway: `gateway/run_config_loaders.py._load_fallback_model()` reads `config.yaml` → passes to `AIAgent`
|
|
- Validation: both `provider` and `model` keys must be non-empty, or fallback is disabled
|
|
|
|
### What does NOT support fallback
|
|
|
|
- **Subagent delegation** (`tools/delegate_tool.py`): subagents inherit the parent's provider but not the fallback config
|
|
- **Auxiliary tasks**: use their own independent provider auto-detection chain (see Auxiliary model routing above)
|
|
|
|
Unpinned cron jobs **do** support fallback: `run_job()` reads `fallback_providers` (or legacy `fallback_model`) from `config.yaml` and passes it to `AIAgent(fallback_model=...)`, matching the gateway's `_load_fallback_model()` pattern. A job with its own `provider` / `model` / `base_url` gets no chain, the same rule as a pinned delegation child. See [Cron Internals](./cron-internals.md).
|
|
|
|
### Test coverage
|
|
|
|
Fallback behavior is exercised across several suites:
|
|
|
|
- `tests/agent/test_fallback_credential_isolation.py` — credential isolation between primary and fallback
|
|
- `tests/hermes_cli/test_fallback_cmd.py` — the `/fallback` CLI command
|
|
|
|
## Related docs
|
|
|
|
- [Agent Loop Internals](./agent-loop.md)
|
|
- [ACP Internals](./acp-internals.md)
|
|
- [Context Compression & Prompt Caching](./context-compression-and-caching.md)
|