When the announcement carries no call_id and the done item does, the
call_id-exclusive lookup found nothing and left the announcement pending;
response.completed then settled it as a second call. Fall back to the
item/index match only when the matched announcement has no call_id of its
own, so a conflicting call_id still never evicts an unrelated call.
Match pending argument and completion events by output_index when item IDs change. Preserve authoritative arguments and successful output ordering without settling the same call twice.
Verification: independent code review passed; 75 focused tests passed; live Copilot Responses probe passed.
_ensure_codex_session handed the codex app-server session whatever CLI panel
callback the thread had. `hermes chat -q` registers one, so every codex
exec/apply_patch approval request in a -q (kanban worker) turn waited the
full panel timeout for nobody. With no callback the session's existing
"decline" path answers immediately; an approval bypass still auto-approves.
The test previously asserted only wall-clock elapsed < 2s, which passes
whether or not the drain is skipped because the fake stream ends
immediately. Record any read past the terminal frame and assert there
were none, so re-enabling the drain without a shutdown-capable socket
fails deterministically. Also inline the single-use _post_terminal_socket
helper at its call site (keeping the exception guard, since the drain
sits outside the inner try; _socket_from_response already handles a
None response), and drop `x and` / bare-truthiness preambles that the
following set-equality asserts already imply.
_post_terminal_socket duplicated the response -> network_stream/stream ->
connection-candidate walk from _shutdown_stale_attempt_socket. Hoist it into
agent_runtime_helpers._socket_from_response (next to _socket_from_candidate)
and use it from both sites; each caller keeps its own error handling. Drop
the dead event_stream argument: ManagedLlmStream has no .response, only the
raw SDK stream does.
Fold watchdog.start() into the drain try so cancel/join lives once in the
finally (join only when a thread was actually started), replacing the
duplicate start-failure branch and its suppress-wrapped join.
Close guard: relay_llm.stream is ManagedLlmStream, whose close() always
exists and owns the provider stream, so the raw stream only needs closing
when construction failed (event_stream is None). Behaviour-equivalent to the
callable(close) probe.
_close_event_stream's new bool return has no caller that reads it, so
restore the original None-returning shape (and the original comment).
The suppress(Exception) blocks around logger.debug/warning calls in the
post-terminal drain guard against nothing real: logging already swallows
handler errors, and wrapping them hides genuine bugs in the arguments.
Fixes#127390.
Co-authored-by: Eva <239388517+100yenadmin@users.noreply.github.com>
The app-server turn effort was clamped to the Responses vocabulary, so a Hermes
`ultra` pick reached codex as `max` and never ran codex's Ultra harness mode.
Live on codex-cli 0.147 with ChatGPT auth, `turn/start.effort = "ultra"` runs on
gpt-5.6-sol and gpt-5.6-luna (ladders reach `max`) and fails with 400
"'max' is not supported" on gpt-5.5, so `ultra` is now kept wherever the
model's ladder reaches `max` and clamped as before everywhere else. One rule,
route_supported_efforts(..., api_mode="codex_app_server"), drives the turn and
the labels: session.info's reasoning_effort_wire (the Desktop pill) and the CLI
and gateway /reasoning status read "ultra" instead of "ultra (sends max on this
route)" there. A named custom codex provider keeps the clamp.
From #130965 (Ultra passthrough and route-aware labels), narrowed to the
models that accept it.
Co-authored-by: FroRaut <79215537+Froraut@users.noreply.github.com>
The wire model, effort and tier were computed before the try around
run_turn, so an exception there skipped the failed-turn path (log, close
the session, fall back to /codex-runtime auto). They are computed inside it
again, as the wire model was before. The effort clamp reuses
classify_responses_route (also counts a chatgpt.com/backend-api/codex base
URL and xAI) instead of a provider check, and the codex tier names live in
fast_mode.CODEX_TIER_WORDS beside the other tier tables.
The forwarded tier was agent.service_tier mapped by hand, so "auto",
"cold" and "ultrafast" went to codex verbatim, and a null went out on every
normal turn, overriding a tier configured in codex itself.
- The tier now comes from effective_request_overrides(agent), the same
source the Responses path sends: a static /fast tier pinned per
model/route by every surface, or an open auto/cold window. It goes out in
codex's words (OpenAI "priority" is codex "fast"); a tier codex has no
word for (ultrafast) is not sent.
- The session remembers the tier its thread last received and sends
serviceTier only on a change. A new thread starts from codex's configured
default. A resumed thread is treated as unknown, so its first turn sends
Hermes' tier even when that is a clearing null: CLI /fast rebuilds the
agent, and "/fast off" must reach a thread that still carries fast.
The forwarded effort was Hermes' raw reasoning_effort, and Hermes has levels
the backend does not: live on codex-cli 0.147 with ChatGPT auth, a turn
with effort "ultra" fails with 400 "Unsupported value: 'max' is not
supported". Resolve it through the Responses transport's _resolve_reasoning
(nearest weaker supported level, "none" only where the route accepts it),
and only when Hermes has an explicit reasoning setting, so codex's own
configured default is otherwise left alone.
Hermes built turn/start with only threadId and input, so the app-server
kept codex's inherited settings after Hermes changed its live agent state:
reasoning effort and /fast never reached the codex thread. Each turn now
carries the active reasoning effort (none when reasoning is disabled) and
the service tier (Hermes "priority" -> codex "fast"; null clears an
earlier tier), alongside the per-turn model main already sends.
Salvaged from #65599 onto current main; the per-turn model part landed via
#130735, so this keeps the reasoning effort and service tier.
turn/start can change the model but not modelProvider, which is fixed when the thread starts. With the model now forwarded per turn, switching in place between codex's own provider and a named custom provider would send the new provider's model to the old provider's thread. Record the thread's model provider next to its prompt composition and retire the thread when either changes.
The OpenAI API-key rung reaches the app-server as provider=custom with no
codex model_providers id, and its model is not vendor-normalized at agent
init (only openai-codex is), so a configured "openai/gpt-5.5" now went to
codex verbatim where it was previously dropped for codex's default. When
codex's own provider serves the turn, normalize the slug the way
openai-codex does; a named custom provider's model ids pass through.
The provider lookup moves into _codex_model_provider and is passed to
_codex_wire_model, and the session comment now says model is sent for every
provider, modelProvider only for named custom providers.
The CLI/TUI /model switch mutates the running agent, and its codex session
(and thread) is reused, so the model fixed at thread/start stayed in force
for the rest of the session. turn/start accepts a model that codex applies
to that turn and later ones (codex 0.147 schema; live: a thread started on
codex's default ran a turn on gpt-5.5 with modelContextWindow 258400).
Every turn now sends the agent's current wire model.
With the model now forwarded, a selected -900k alias reached codex verbatim
and every turn failed: live on codex 0.147 with ChatGPT auth,
thread/start {model: "gpt-5.6-sol-900k"} -> turn 400 "The
'gpt-5.6-sol-900k' model is not supported when using Codex with a ChatGPT
account". The alias is Hermes-side only (the Responses transport already
strips it); codex applies the catalog's extended window to the base slug on
its own (thread/tokenUsage reports modelContextWindow 828400 for
gpt-5.6-sol). Strip the valid suffix before it goes on the wire.
Codex homes are service-scoped while Hermes model selection is profile and
session scoped. thread/start only carried the model for named custom
providers, so openai/openai-codex sessions on model.openai_runtime:
codex_app_server silently ran codex's own default model, whatever /model or
config.yaml selected. Forward the agent's model on every thread/start;
the custom-provider modelProvider id and the no-credential payload are
unchanged.
Salvaged from #83269 onto current main (the session already accepts model=
since #115905; the forwarding was narrowed to custom providers there).
OpenAI's Ultrafast service tier (service_tier="ultrafast") is broadly
available for GPT-6 Astra in the API and Codex (Pro 500 / Enterprise),
billed at 6x Standard. GPT-6.1 Sol Ultrafast is announced as coming soon.
- One tier table: agent.fast_mode.parse_service_tier / STATIC_TIERS now
back the CLI, gateway and TUI config loaders (three hand-copied parsers).
- resolve_fast_mode_overrides(tier="ultrafast") sends the tier only for
Ultrafast models on api.openai.com / chatgpt.com; anything else gets
nothing rather than a silent swap to another paid tier.
- /fast ultrafast on CLI, gateway (typed + picker when supported) and TUI;
static-tier attach sites pass the tier through.
- Pricing: the response's SERVED service_tier is folded into usage (Codex
stream assembler keeps it); served ultrafast prices from a published
6x row (272K whole-request tier included). A request asking for
Ultrafast but served at default bills at Standard.
- i18n: 4 new keys in all 17 locales; option lists mention ultrafast.
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
Codex app-server webSearch items fire the live tool bubble under
codex_web_search_<itemId>, but the projector had no entry for them, so the
stored transcript kept an opaque "[codex webSearch] {json}" note instead.
Any transcript refresh or reopen then replaced the search card with raw
JSON.
The projector now records webSearch as a web_search tool_call under the
same id and name the live bubble uses. The tool result is
{"provider": "codex", "status": ...}, so a later non-Codex turn can tell
Codex ran the search.
Salvaged from #66995: only the webSearch projection is kept, reworked
onto the _TOOL_PROJECTIONS table and the live web_search name. The MCP
elicitation trust change is not included.
The event bridge listed the text-delta methods twice (display handlers and
the #118410 liveness set) and duplicated _fire_delta's text extraction.
Derive both from _CODEX_TEXT_DELTA_METHODS, share _delta_text(), and hoist
the progress item-type set to a module constant (no per-event frozenset).
The attempt-local first-progress deadline added `progress_timeout` to
`_NonStreamWatchdogs`, but the SimpleNamespace stubs in
test_nonstream_wait_notice.py and test_wait_notice_cadence.py were not
updated. `_emit_wait_notice` reads `wd.progress_timeout` inside its
blanket `except Exception`, so the AttributeError silently produced no
notice and no liveness touch: 11 tests red.
The new "Codex stream opened" debug line also called `time.time()`
eagerly, consuming a tick of the 3-tick stub in
test_codex_first_event_timing.py and shifting the first-event stamp.
The first-event log a few lines below already carries the attempt's
timeline anchor, so the open marker logs without an epoch.
The per-attempt "first parsed event" / "first substantive progress" /
"physical retry" markers are what an operator needs to see where a
lifecycle-only Codex attempt stalled, so they stay at info. The
"stream opened" line fires on every attempt and carries no diagnostic
value on its own, so it goes to debug to keep the default log quiet.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
A codex thread that codex hands back via thread/resume already holds the conversation, but a
thread started fresh did not: a session that ran on another provider before /model switched to
openai-codex, a stored thread codex could not resume, or a thread retired mid-session (prompt
composition change, wedged client) answered the first turn blind. The prior user/assistant text,
tool names and tool-result previews (most recent 32K chars) now ride once on
thread/start.developerInstructions after the prompt composition; thread/resume never carries them,
and the recorded composition stays the bare prompt so the seed cannot make the next turn retire the
thread.
Direction from #26081 (first-turn seeding of the Hermes transcript); redone on the extracted
agent/codex_runtime.py path with the system prompt sent once (#115759) instead of duplicated.
Completes #26035 / #74712 (closed by #115759 for the prompt half; this is the history half).
Co-authored-by: LeonSGP43 <154585401+LeonSGP43@users.noreply.github.com>
Codex app-server thread survives an API-server restart: thread id persisted per session and thread/resume'd, fail-closed to a fresh thread (#100531, salvage #103352)
After a codex app-server turn's projected rows are durable in the session DB,
store the codex thread id as ``codex_thread_id`` in the session row's
model_config (atomic merge via patch_session_model_config; never for a retired
thread). The FIRST CodexAppServerSession an AIAgent builds for that session
passes the stored id as resume_thread_id, so a rebuilt agent — the next
/api/sessions/{id}/chat request, or the first turn after the API server or
gateway restarts — resumes the model-side thread before turn/start instead of
starting an empty one while Hermes' own transcript continues.
Fail closed when codex cannot hand the thread back (rollout gone, CODEX_HOME
changed, previous app-server killed mid-write): drop the stored id, start a
fresh thread on the same client, and say so once —
"Codex thread could not be resumed; starting a new one." — through
_emit_diagnostic_status, the lifecycle status rail every surface renders (CLI
vprint, TUI/Desktop and gateway status_callback). No other lifecycle change:
a retired or prompt-recreated session in the same process keeps today's
fresh-thread behaviour and overwrites the binding once its turn commits.
Why: CodexAppServerSession kept the thread id in memory only, so every
API-server restart (and every per-request agent) silently reset the model's
memory of the conversation (#100531). Supersedes the persistence half of
#100528 (_persist_projected_messages now reports durability) and the
refuse-and-raise policy of #103352 with the maintainer-approved bounded slice.
When every pre-stream connect attempt to an endpoint fails, the user only saw
"Connection error." repeated per outer retry; the host, the attempt count and
the serialized request size lived in agent.log alone. #97548's reporter had an
~829 KB Codex Responses request fail twice before the stream opened while short
chats went through, which is the request-size-limit signature, and nothing on
screen said so.
One buffered diagnostic line now flows through the existing retry-status path
(flushed on terminal failure, dropped on recovery) on both the Codex Responses
runtime and the Chat Completions stream worker: "Could not open a stream to
<host> after N attempts (request X KB); ...". The host comes from the failed
request's URL (the endpoint actually contacted, proxies included) and the size
from the buffered httpx request body. Re-entering the stream call from the
outer retry/fallback loop does not add another copy.
Part of #97548
_on_event is only ever installed as on_event=_fenced(_on_event), and _fenced
already refuses every call once the request is retired, so the inner predicate
could never be false when reached. Comment now points at the fence.
Responses lifecycle events may arrive before answer text, but the streaming runtime never populated the existing post_api_request timing field. Record the first accepted event while preserving request-retirement fencing.\n\nCloses #105311.
On the Relay-managed path ManagedLlmStream.close() cannot reach the provider
response while the loop is still running the drain (RuntimeError, swallowed by
_close_event_stream), so the httpx response and drain thread lingered until the
socket read timeout. Capture the raw stream in _codex_stream_created and close it
too on drain timeout; the unmanaged path is unchanged (idempotent double close).
Replace the hardcoded 2.0s module constant from the salvaged commit with
``agent.stream_drain_timeout`` in DEFAULT_CONFIG (default 2.0, ``0`` skips the
drain), read through ``load_config_readonly`` at drain time. Non-secret knobs
belong in config.yaml, not a new HERMES_* env var. Documents the budget in the
timeouts table and adapts the salvaged test to the new seam.
Why: a relay that never closes the SSE socket after ``response.completed``
wedged the turn until the stale watchdog fired and discarded the already-billed
response (#103864). The drain is a courtesy to Relay's finalizer, so it must be
bounded; the bound needs a user-tunable knob for slow-but-honest relays.
#95429 acceptance criterion 3: retrying a zero-event oversized request must not
resend the same pathological payload unchanged. run_codex_stream's in-place
transport reconnect re-sent dict(api_kwargs) verbatim with no diagnostic.
When an attempt dies before any stream event was received and the serialized
input exceeds the per-turn budget, inline function_call_output strings over the
per-result threshold (results that escaped commit-time persistence) are spilled
through maybe_persist_tool_result -- bounded preview + recoverable
<persisted-output> reference -- and one warning records
serialized_input_bytes=before -> after with the attempt number. When nothing is
prunable the warning says the payload is being resent unchanged. Only the
retried wire payload changes; the caller's kwargs and history are untouched.
No new config or env vars.
Before: zero-event reconnect resent 760K-char tool output byte-identical, no log.
After: retried input < 1/10 the size, spill file holds the full text, one
"Codex zero-event retry (attempt 1/2): ... serialized_input_bytes=N -> M" line.
_record_codex_app_server_usage fed inputTokens — which the app-server protocol
reports INCLUSIVE of cachedInputTokens — into CanonicalUsage.input_tokens while
also recording cachedInputTokens as cache_read. CanonicalUsage.prompt_tokens sums
both buckets, so cached tokens were counted twice: inputTokens=80,
cachedInputTokens=20 produced prompt_tokens=100 instead of 80. The inflated value
reached session accounting, cost estimation and context_compressor.last_prompt_tokens
(premature compaction) on every cache-hit turn (#105412, #63654, #48801).
Subtract cache_read from the reported input to get the uncached remainder,
mirroring normalize_usage's codex_responses branch so both Codex transports yield
identical accounting. The transport exposes no cache-write, so only cache_read is
subtracted. totalTokens stays the provider passthrough. The existing integration
test asserted the wrong totals; it now pins prompt=80, uncached=60, cached=20.
Ported from PR #63654 onto the current adapter shape.
Fixes#105412
codex app-server emits commentary deltas + completed, then final deltas +
completed. The delta buffer accumulated "commentary + final", so the completed
final no longer prefix-matched its own deltas, was delivered with
already_streamed=False and the gateway posted a second copy next to the
streamed message (#74248 boundary 2). Drop the buffer once a completed
agentMessage has been compared against its own deltas.
