`key_cmd` (#86891) authenticates a provider with a SHORT-LIVED bearer minted
by a command — SSO/OIDC brokers, cloud IAM, internal auth proxies. The
request path has honoured it since it landed, but the picker resolved probe
credentials from `api_key`/`key_env` ONLY, so a key_cmd provider probed
`/v1/models` with an EMPTY key.
Against an authenticated endpoint the probe 401s, discovery returns nothing,
and the provider falls back to its single configured default model. The
picker shows ONE model, indistinguishable from an endpoint that genuinely
serves one — while inference keeps working, because that path mints
correctly. Reproduced against a LiteLLM gateway behind Entra OIDC: 0 models
discovered with an empty key, 26 with the minted token.
Both picker probe sites already funnel through `_entry_credentials()`, so
the fix lands in one place: it now reports a `cmd:<key_cmd>` identity, and
each site falls back to `resolve_probe_token()` after api_key/key_env. An
explicit static key still wins, so existing configs are unaffected.
The identity is keyed on the COMMAND, never the minted token: the token
rotates on every refresh, so keying on its value would change the group
fingerprint constantly and force a re-probe on every open. Two entries on
one URL with different helpers still get distinct rows.
`resolve_probe_token()` lives in agent.command_token_source, which already
owns key_cmd minting, and shares the CommandTokenSource cache with the
request path — a cache read, not a fresh sign-in. Fail-closed: a helper
needing an interactive sign-in degrades to today's empty-key behaviour
rather than taking down every other provider's row.
`_model_flow_named_custom` (the `hermes model` setup flow) is the sibling
path — it builds its own `Authorization: Bearer` from the same incomplete
resolution — and is fixed the same way, with one ordering constraint: the
value persisted to config.yaml is computed BEFORE the mint, so a short-lived
bearer can never be written back to shadow the key_cmd meant to re-mint it.
Tests drive the real code paths and assert on the credential each probe
receives rather than on function source, so a semantics-preserving refactor
does not fail them. Verified they fail with the fix reverted.
Extend #104477 to the native thinking, vision, metadata, and local header paths identified by #87641. Materialize only at probe boundaries; leave the chat callable and cache ownership untouched. Local-wire A/B: thinking and vision show requests change from 403 to 200, while static credentials and callable chat retain success. Target suites queued.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Follow-ups on the #85006 salvage:
- A key_cmd token with no advertised expiry was cached for the life of the
process. The "refresh on 401" contract it relied on has no implementation
(SDK retries cover 429/5xx only), so an expired no-TTL token would 401
every request until restart. Cache on a bounded 15-minute window instead;
helpers that want a longer cache can advertise their real expiry.
- Test for the no-TTL path updated to pin the bounded-window contract;
the remint test's $RANDOM (bash-only, empty under dash) replaced with
date +%s%N so it exercises remint under any /bin/sh.
- website/docs/integrations/providers.md: document key_cmd in the named
custom providers section (contract, precedence, secrets.command contrast).
Custom providers could only authenticate from a static credential (inline
api_key or a key_env env var). Enterprise gateways -- SSO/OIDC brokers, cloud
IAM, internal auth proxies -- issue short-lived bearers instead, so a value
copied into .env is stale within the hour: long sessions start returning 401s
and the user has to restart or run an external cron that rewrites .env.
The existing `secrets.command` source does not cover this: it runs once per
process at startup (subsequent calls are no-ops by design), so it cannot
re-mint a credential mid-session.
Add providers.<name>.key_cmd: a command that prints a token, wrapped at
resolution in a zero-argument callable. Both wire clients already accept a
callable api_key and invoke it per request (the Entra ID path established
this), so chat_completions, codex_responses and anthropic_messages all work
unchanged and always send a fresh credential. The callable also routes the
Anthropic client through its per-request Authorization hook, which is what
OAuth-gated gateway routes require -- so no per-vendor auth wiring is needed
anywhere in core.
- cached until shortly before the advertised expiry (60s leeway), so the
helper runs about once per token lifetime rather than once per request
- expiry is read from the OAuth 2.0 relative `expires_in` when present, and
otherwise from an absolute ISO 8601 deadline (`expiry`, `expiresOn`), which
is what CLI token helpers commonly print. Reading only `expires_in` treated
those helpers as advertising no TTL at all, cached their token for the life
of the process, and returned 401 on every request once the real deadline
passed. ISO parsing reuses hermes_cli.auth._parse_iso_timestamp rather than
adding another datetime parser.
- no synthetic expiry: when no TTL is advertised, or the advertised one is
unparseable or already past, the token is used and refreshed on 401 instead
of re-minted on an invented schedule
- stdout contract matches OAuth 2.0 token endpoints and existing agent
helpers (bare token or JSON access_token/expires_in); multi-line output is
rejected rather than guessed at, so a misconfigured helper surfaces as a
clear error instead of a corrupt-credential 401
- precedence: explicit --api-key still wins; otherwise key_cmd beats a
static api_key/key_env on the same entry
- failures never include the helper's output (may hold a partial token) or
the command string (may embed a client secret)
Resolution happens on two paths. agent/auxiliary_client.py resolves named
custom providers itself rather than calling _resolve_named_custom_runtime, so
key_cmd is honoured in both: wiring only the runtime resolver leaves the main
agent turn working while every auxiliary call (title generation, compression,
vision, embedding) falls back to the no-key-required placeholder and 401s.
Precedence is identical on both paths, so one config entry cannot yield two
different credentials depending on which resolver the caller reached.
Closes#84162
Signed-off-by: LordMelkor <kray@block.xyz>
