* refactor(fallback): share the pinned-owner chain rule
delegate_task's _resolve_child_fallback_chain decides which fallback chain
a child may walk: a pinned child never borrows the parent chain, an explicit
[] disables fallback, a declared list is the child's own. Cron needs the
same rule for pinned jobs (#100437), so the body moves to
hermes_cli.fallback_config.scoped_fallback_chain and the delegation helper
becomes a thin caller. Behaviour is unchanged; the delegation matrix test
still pins every cell.
* fix(cron): a pinned job never falls back to the global chain
A job with its own provider, model or base_url is an explicit operator pin
(since 0469740ab3 unpinned jobs store none of these). It still walked the
global fallback_providers chain in two places, so a pinned job could run
on a different provider and model than the one chosen:
- _resolve_job_runtime walked the chain on an AuthError or transient
network failure while resolving the pinned primary;
- _resolve_cron_agent_setup handed the global chain to every cron agent as
fallback_model, so the conversation loop's provider ladder could swap a
pinned job mid-run.
Both now read _job_fallback_chain(job, cfg), which returns no chain for a
pinned job through the same scoped_fallback_chain rule delegate_task uses
for pinned children. The pre-dispatch key check reads it too: the global
chain used to skip that check for every job, so a pinned job with a
missing key now blocks before the agent is built instead of failing in the
resolver. The transient-failure notice for a pinned job says it does not
fall back and names --unpin, instead of "No backup provider succeeded".
Unpinned jobs (including legacy *_snapshot records) and same-provider
credential-pool rotation are unchanged. The two scheduler tests that
asserted atomic provider+model fallback swaps used pinned jobs; they now
use unpinned jobs and keep the same assertions.
No per-job fallback_providers list: jobs have no generic override field
(create_job/update_job, the cronjob tool schema and the CLI enumerate each
field), so an opt-in chain would be a new surface on all of them. The
escape hatch is to leave the job unpinned and pick its model with
cron.model / cron.model_provider.
Co-authored-by: 686f6c61 <6115107+686f6c61@users.noreply.github.com>
* docs(cron): pinned jobs do not use fallback_providers
cron.md "Provider recovery" and the pre-dispatch key check, the cron rows
and section in fallback-providers.md, and the developer notes in
cron-internals.md / provider-runtime.md said every cron job inherits the
global chain. State the new rule, the compatibility note for users who
relied on a pinned job landing on the chain, and the unpinned + cron.model
alternative.
---------
Co-authored-by: 686f6c61 <6115107+686f6c61@users.noreply.github.com>
Two named custom providers may share one gateway URL with different
credentials. `_resolve_child_credential_pool` matched custom pools by URL
only, so a child inherited whichever pool was registered first for that
endpoint. Pass the child's `requested_provider` (and the parent's) into
`get_custom_provider_pool_key(provider_name=...)`, which already prefers a
name match over URL order.
Second half of the class fixed by #117839; grafted from PR #89021 onto the
split `delegate_tool_config` layout (#45763).
The configured cap was a dispatch-to-death stopwatch: `await_child` waited on a
plain `settled.wait(timeout=child_timeout)`, so any child that outlived the
budget was abandoned even while the provider was actively serving it.
The report's corpus for #116001 (219 tasks / 75 deaths, 0 of them mid-tool) could
not be reproduced here — it needs the reporter's slow OpenAI-compatible endpoint
— but the mechanism it names is exactly this gate: a child waiting on an
in-flight LLM completion, killed with a nearly-finished context. A slow child is
already bounded elsewhere (the per-call stale watchdog, the heartbeat's
staleness verdict), so this cap could only ever kill children the runtime had
judged healthy.
`child_timeout_seconds` now measures time with NO progress: the wait runs in
slices and restarts the window on the same signals the heartbeat's stale verdict
reads (completed call, tool change, activity-clock tick). A frozen child is
still abandoned when the window elapses; a progressing one is never killed for
taking long.
Timeout entries also carry `last_event_age` (how long the child had been silent),
so operators can tell a slow provider from a runaway without transcript
forensics.
Fixes the mechanism reported in #116001. The budget warning and
continuation-respawn items in that issue are separate features and are not part
of this change.
The provider-agnostic half of PR #105863, so a CLI-driven subscription provider can ship as a
standalone `kind: model-provider` plugin instead of a bundled one:
- ProviderProfile: `native_reasoning_details_type`, `model_aliases`, `get_model_context_length`,
`get_usage_cost`, `setup_status`, `discover_models` hooks (all default None / no-op).
- Chat Completions transport: provider-native `reasoning_details` carriers follow only their
declaring profile; standard records still replay on OpenRouter-style routes, strict routes
drop the field wholesale (#70233). Relay/stream accumulate `delta.reasoning_details` verbatim.
- `hermes model`: the generic plugin flow gates an external-process row on the CLI's own login
status (inline `login_command` on a TTY), offers `discover_models()` rows with per-row notes,
and never writes config when the executable is missing.
- `/model` and the pickers: process providers list their live catalog merged with the pinned
one, declared aliases/ids resolve inside the provider, and validation accepts a listed id
without probing `process://`.
- Delegation keeps the selected external-process provider and protocol for the child.
- Model metadata / usage pricing consult the profile's bound and cost hooks first.
- Desktop: `[1m]` renders as a "1M" tag and hyphenated Anthropic versions read "Haiku 4.5".
The bespoke `_model_flow_external_process` and hard-coded `hermes_cli/main.py` paths from the
PR were dropped in favour of main's `_model_flow_plugin_provider`.
Co-authored-by: unsupportedpastels <unsupportedpastels@users.noreply.github.com>
Trim the salvaged fix to the shape main wants:
* Every gate asks ``agent.transports.registered_api_modes()`` directly. The three helper
spellings (``_has_registered_transport``, ``_registry_knows``, ``is_registered_api_mode``)
and the ``sys.modules`` peek are gone: no transport module imports ``providers`` or
``hermes_cli`` at module level, so a plain import cannot re-enter provider discovery.
* ``hermes_cli/auth.py`` late-registration pass dropped — main already re-syncs plugin
profiles into ``PROVIDER_REGISTRY`` on every registry miss
(``auth_plugin_providers.registry_lookup`` / ``sync_plugin_provider_registry``, #102123);
the probe shows a profile registered after the import-time mirror resolves and reaches
the wire on base.
* ``ProviderTransport.normalize_stream_delta`` and the streaming-assembler hook dropped —
legacy ``delta.function_call`` translation is a separate concern from api_mode
propagation and has no in-tree consumer.
* Tests: 15 gate-by-gate unit tests replaced by two invariants that install a REAL plugin
under a temp HERMES_HOME and walk profile → determine_api_mode → resolve_runtime_provider
→ agent ladder → delegation resolver (positive: red on origin/main; negative: an
unregistered mode still degrades to chat_completions).
A provider plugin ships a transport via `register_transport(api_mode, cls)` and
declares that same string as its profile's `api_mode`. Transports are selected by
that string, but every gate that validates an api_mode compared it against a
closed literal, so a plugin's mode was rejected at each one and rewritten to
`chat_completions`. The plugin's transport was then never selected: no error, no
tool call, the turn silently degraded to prose. `register_transport` was a public
seam with no way through.
Accept a mode when the transport registry knows it, via a new
`agent.transports.registered_api_modes()`, at each gate:
* `agent_init._resolve_api_mode` - the agent's mode ladder;
* `runtime_provider._parse_api_mode` - the config gate;
* `delegate_tool_config` - the delegation resolver;
* `providers.get_provider` - the reverse `TRANSPORT_TO_API_MODE` lookup recorded
an unknown mode as `openai_chat`, which made `determine_api_mode` report
`chat_completions` for a provider that has a dialect transport. This one is the
most deceptive: the other gates already pass, and the transport still is not used.
* `providers.determine_api_mode` - the same table lookup at the other end.
The registry read is deliberately lazy (`sys.modules.get("agent.transports")`,
never an import): this code is reached from `determine_api_mode`, which provider
discovery itself calls while the registry is being populated, and importing the
transport package there re-enters discovery.
Also add `ProviderTransport.normalize_stream_delta()`, the response-side twin of
`convert_messages()`: a provider that streams a tool call on the legacy OpenAI
`delta.function_call` pair instead of indexed `delta.tool_calls` had nowhere to
translate it, and the streaming assembler dropped the call. The default returns
the delta unchanged, so existing transports are untouched; the assembler now asks
the transport instead of hardcoding one provider's shape.
Finally, make plugin-provider registration repeatable. `hermes_cli.auth` registered
plugin profiles once, at import, from a list `hermes_cli.config` had already
partially discovered while importing itself. A profile that sorts LAST in discovery
was absent from that snapshot and never reached `PROVIDER_REGISTRY`, so every
consumer reported it unauthenticated and it silently vanished from the model
picker while working fine from the CLI. `ensure_plugin_providers_registered()` is
now called from `get_auth_status()` and `resolve_provider()`, so a late profile is
picked up instead of staying invisible.
Unregistered modes are still rejected everywhere, and the in-tree literal sets are
unchanged - they are simply no longer the only way in.
Without a delegation override the child took base_url from the parent's
live client (_client_kwargs / client.base_url) but api_key from the
surface attribute parent_agent.api_key. After a fallback or any runtime
swap the surface key can lag the live client, so the child was built with
a (base_url, key) pair that was never valid on the parent and died on an
instant, non-retryable 401 (#90009, reporter's log shape).
_inherit_parent_base_url becomes _inherit_parent_endpoint and returns the
(base_url, api_key) pair from one source: the live client kwargs when the
parent has an OpenAI-wire client, the surface attributes otherwise (native
Anthropic/Bedrock runtimes keep client=None). The override branches keep
the salvaged all-or-nothing rule from #111823.
Live probe (real AIAgent, real try_activate_fallback, two fake servers):
before, a parent on fallback B with a lagging surface key sent
`Bearer FAKE-KEY-PRIMARY-A` to server B from the child; after, the child
sends B's key to B. Natural fallback (no lag) is unchanged.
Tests trimmed to three invariants (override never borrows the parent
endpoint, provider without base_url is refused, no-override inherits
live endpoint+key together); all red on origin/main.
Fixes#90009
Salvages #111823
_resolve_child_runtime resolves provider and base_url with independent
per-field fallbacks, so a delegation.provider override whose runtime
comes back without a base_url silently inherits the PARENT's endpoint.
The child then sends e.g. copilot credentials to the parent's Codex URL:
every request 404s, and the fallback chain can't rescue it because its
dedup matches provider+model and skips the entry as a self-loop.
Build the bundle all-or-nothing: with a provider override, base_url
comes from the override only; otherwise everything is inherited from
the parent as before. _runtime_provider_credentials now refuses a
provider that resolves without a base_url, unless it is an ACP
transport (addressed by command) or a native-SDK provider
(bedrock/vertex/google), which legitimately have none.
The delegate fix admits a MIXED same-provider pool (any one entry serves the
child's endpoint) but only guarded the initial lease. A 429/402/401 in the
Azure child later ran recover_with_credential_pool -> mark_exhausted_and_rotate
-> _swap_credential(next_entry) with no endpoint check, and
credential_pool_matches_provider passes a non-custom pool on provider match
alone, so the child was rebound to the api.openai.com entry (#68237 via a
sibling path).
_entry_serves_endpoint moves into agent/credential_pool.py as
credential_pool_entry_serves_endpoint (next to credential_pool_matches_provider;
tools/delegate_tool_config imports it back), and _rotate_and_swap vetoes a
next_entry whose base_url does not serve agent.base_url, returning False exactly
like a rotation that yields nothing: the failed entry stays benched, the
session keeps its credential and the fallback chain takes over.
A delegated child with an explicit base_url (e.g. an Azure OpenAI resource
under provider "openai") shared the parent's "openai" credential pool on bare
provider equality, and _lease_child_credential bound whatever entry
acquire_lease() picked. _swap_credential adopts the entry's base_url as well,
so the child was silently rebound to https://api.openai.com/v1, sent the
Azure key there, got HTTP 401 and only then fell back (#68237).
_resolve_child_credential_pool now requires endpoint coherence on top of
provider identity (credential_pool_matches_provider + at least one entry whose
base_url matches the child's) for both the shared parent pool and the
provider's loaded pool; a mismatched pool is not attached and the child keeps
its fixed credential. _lease_child_credential validates the leased entry
against the child's base_url and, on a mixed pool, releases the wrong-host
lease and leases an endpoint-matching entry by id instead. Entries and
adapters without endpoint metadata cannot rebind and are accepted unchanged.
Salvaged from #68240 (@oferlaor); the acquire_lease/leased_entry filter
parameters and the extra pool-level helper were reduced to the two small
delegate-side predicates.
A finite `hermes chat -q` / `--oneshot` run has no later session in its HERMES_HOME
to learn for, yet it ran the full interactive self-improvement loop. Measured over
21 one-shot benchmark trajectories: 7 skills created and a bundled one patched
mid-task, 37 of ~215 tool calls on skill_view/skill_manage, skill text = 34% of all
tool-result bytes fed back into context, plus reviewer subagents spawned on the
agent's own diff (one task: 5 delegations, 62 subagent API calls, each re-paying a
cold system prompt).
Keyed on the existing HERMES_SINGLE_QUERY_SESSION marker (approval gate, delegation
dispatcher), so interactive and gateway sessions are byte-identical:
* agent/oneshot_footprint.py (new sibling): skill_manage is pruned from the tool
set; the ## Skills block keeps the index + skill_view but drops the record/patch/
offer-to-save coaching and the "load process skills for work you already know"
push (SKILLS_GUIDANCE follows because it is gated on skill_manage).
* delegation.oneshot_max_children (default 2, 0 = unlimited): total children a
one-shot run may spawn; past it delegate_task returns a tool error telling the
model to finish inline.
* requesting-code-review skill: reviewer/fixer subagents (Steps 5 and 7) are
interactive-only; one-shot applies the checklist inline.
Live: `hermes chat -q "list tools starting with skill_"` on the portal — base
"skill_manage, skill_view, skills_list", fix "skill_view, skills_list"; a real
AIAgent under a temp HERMES_HOME shows the interactive prompt unchanged (6643 chars
both) and the one-shot prompt without skill_manage / offer-to-save.
_resolve_child_fallback_chain re-implemented hermes_cli.fallback_config's entry
validation to log per-index warnings, wrapped a pure function in try/except, and
carried two names (routing_cfg / fallback_cfg) for one argument. It now delegates
to get_fallback_chain() and keeps the single warning for "no usable routes";
the facade re-export is dropped (import from the defining module). Tests collapse
to the parametrized decision table plus the pin-derivation, real-config-loader,
review-ownership and activation invariants (22 -> 20 cases, 417 -> ~230 lines).
Three orchestrator failures traced through the Sep 7 gpt-6-astra campaign sessions:
1. delegation.independent_completions (new, default false). #104299 made every
ungrouped task its own completion message, so a 15-task call woke the
orchestrator up to 15 times; one chain received 132 notices and answered
130 of them with "already incorporated". A multi-task call now returns as
ONE consolidated message unless the flag is on; `group` is inert until then.
2. Queued units were killed before they started. Units of one call share a
pool slot but the executor was still sized by slots, so with 15 units live
a new unit queued behind a full pool; the stale monitor's clock ran from
dispatch, interrupted it at 450 s, and the child exited `interrupted 0.02s`
when its thread finally came up (13 such lanes in one session). The
executor now grows to the number of live units and the stall clock arms
when the runner actually starts.
3. The tool text said "do not wait or poll — just continue" without saying
that completions are delivered only BETWEEN turns. A model that never ends
its turn (one 203-minute turn, 717 API calls) never received 40 finished
results. Tool description, dispatch note and completion header now say to
finish independent work, give a one-line status, and end the turn.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
