The salvage moved generated images/videos to cache/generated/<kind>/ so the
24h sweep stops deleting the only copy (#126445), but two consumers still
only knew the old swept dirs:
- Strict media delivery (HERMES_MEDIA_DELIVERY_STRICT=1) allowlists cache
roots via _MEDIA_DELIVERY_CACHE_SUBDIRS; without "generated" a day-old
generated image was rejected once the recency window lapsed, i.e. the
move broke re-sending artifacts on public gateways.
- iter_cache_files (Modal/remote resync) walked the new never-swept tree in
full, so every sync re-uploaded the whole generation history. Only files
within the sweep window (MEDIA_CACHE_MAX_AGE_HOURS) are synced now.
GENERATED_SUBDIR and MEDIA_CACHE_MAX_AGE_HOURS live in agent/provider_media
as the single source for the image/video writers, the gateway allowlist and
cleanup default, and credential_files. Also trims duplicated comments and
fixes the pet orchestrate comments (nothing prunes cache/generated/images).
Tests use direct function refs instead of importlib strings.
Keep sweep survival (parametrized over image/video writers) and the
cache/generated mount mapping; the outside-dir, per-file sync and SSH
artifact cases restate those two invariants.
Generated image/video output now writes under cache/generated/images|videos
instead of the swept inbound caches (cache/images, cache/videos). The hourly
gateway housekeeping deletes 24h-old files in those dirs; for base64-returning
providers the swept file was the only copy, breaking transcript references,
the Artifacts panel and later edits.
Rebased onto current main, where the writers delegate to agent/provider_media,
so the media kind is now the generated/ subpath. cache/generated is added to
tools/credential_files._CACHE_DIRS: without a mount/sync entry the Docker,
SSH and Modal backends never receive the bytes and the provider result loses
agent_visible_image. All callers consume the returned absolute paths, so no
call-site changes were needed; inbound download paths (get_image_cache_dir /
get_video_cache_dir) and the sweeper are untouched.
RED: tests/agent/test_generated_media_durable.py (9 failed on main) — 25h-backdated
generated files deleted by the real cleanup_image_cache / cleanup_video_cache, and
no mount mapping for cache/generated. tests/tools/test_image_generation_artifacts.py
::test_real_generated_image_is_agent_visible_under_ssh_env covers the sandbox
regression with the real writer. Green: 533 passed, 19 skipped.
Co-authored-by: Froraut <79215537+Froraut@users.noreply.github.com>
(cherry picked from commit fd23181f22232f66aec7076d6bf0868ed45661f2)
RunawayStreamWatch doubled the gap between checks, so a loop that began
after N legitimate characters streamed roughly N more before any check
saw it (100K of prose let ~156K of loop through). Each check already
reads only the 64K tail, so after the 16K -> 32K -> 64K start the watch
now checks every tail window; total work stays linear and the cut lands
within about 100K characters of the loop starting. The provider docs
promised a cut at 16,000 characters, which was never true for a loop
behind a long prefix; they now state the real bound.
The loop-cut test gains a row streaming ~100K distinct characters before
the loop and stalling once 100K loop characters have been sent, so a
watch that waits for the reply to double hangs the turn and fails.
Keep the loop-is-cut test (chat content, reasoning behind changing
details, Anthropic text and thinking) and the legit-repetition-delivered
test. Drop the plain chat-reasoning case (same path as the details case)
and the bounded-tail unit test, which asserted on RunawayStreamWatch
private fields rather than behaviour. Each endless-stream case costs
~13s of wall time.
The watch trimmed its text only when it ran a check, and the gap between
checks doubles, so between two checks it held everything that streamed:
about one tail plus the reply size at the last check. It now trims each
time a tail's worth of new text has arrived, so it holds at most two
tails plus the incoming chunk while the check schedule stays the same.
On the chat wire the watch read the reasoning text chosen for display,
which is the structured details whenever a chunk carries them. The plain
reasoning field is persisted either way, so a loop there went unseen
when details streamed beside it. Each representation has its own watch.
From review on #127242.
(cherry picked from commit 53dcaecc960238d4d0461bd43e80465182cbd098)
A looping model kept a turn streaming for as long as an uncapped endpoint
sent tokens: every repetition check ran after the stream ended, the stale
detector only fires on silence, and no max_tokens reaches these endpoints
by design. The stop path's own criterion (is_runaway_repetition at >= 16k
chars) now runs on the raw content and reasoning channels of the
chat-completions and Anthropic Messages streams, at 16k and every doubling
on a bounded tail. A hit closes the stream on its owner thread and ends the
turn through the repetition stop, without continuation.
Test loops are the reported incidents from #125650, #94224/#94348,
#125351 and #78551.
Fixes#127234
Co-authored-by: auroracapital <info@auroracapital.nl>
Co-authored-by: BrunoBza <189763786+BrunoBza@users.noreply.github.com>
Co-authored-by: gkd2323c <gkd2323c@users.noreply.github.com>
Co-authored-by: Enzo Adami <enzo.eliott.adami@gmail.com>
(cherry picked from commit 5379a417c560e1f669fa2d27e61837b2004e125e)
The replacement no longer depends on any captured group, so drop the
groups from _STRICT_URL_USERINFO_RE and substitute a constant string.
Update the redact_sensitive_text docstring to say the entire userinfo
is masked, and fold the standalone opaque-username test into the
existing parametrize so it also checks redaction stays off by default.
The strict URL redactor now masks the entire userinfo field (***:***@)
instead of keeping the username, so the pre-compress memory-context
tests still asserting the old user:***@ shape failed. Update them to
the new strict shape.
The tail of test_response_between_restarts_resets_the_bound re-ran the
restart_count=0 -> MAX_RETRIES continues -> break sequence that the
neighbouring bound test already asserts (with refunds, exit reason and
steer checks), so it was a copy-paste duplicate. Remove it.
The bound now means "restarts since the last response", not "per turn",
so rename test_restart_refunds_are_bounded_per_turn to
test_back_to_back_restart_refunds_are_bounded to match.
The partial-marker test copied the whole dispatch harness from the test above it,
which already pins the dispatch wiring; the new coverage is only which strings
match, so check them in that test's direct _context_pruned_argument_paths tail.
_respond duplicated _apply's 15-kwarg call; give _apply an optional flag and
response instead. Same assertions, less duplicated setup.
The stack is capped at two invariant tests. The partial-marker guard keeps
its parametrized dispatch test, and the redirect-cap fix keeps the
apply_retry_restarts unit test in test_turn_iteration_prep.py, which already
goes red on the turn-wide counter. Drop the end-to-end run_agent replay,
which covers the same reset through a heavier mocked conversation.
The refunding-restart cap (#106108) counted every redirect for the whole
turn, so a normal interactive turn -- the user answering clarify cards or
sending follow-ups while the model works -- ended as
redirect_restart_limit_exceeded on the (api_max_retries + 1)th mid-turn
message, even though a response completed between each of them.
The cap exists for requests that are cancelled over and over with nothing
completing. Reset restart_count whenever a response reaches processing, so
back-to-back cancelled requests still hit the bound and release the turn
lease, while corrections separated by real progress no longer kill the turn.
Fixes#128000
(cherry picked from commit 3a286a2201ee055b2eb6904523934f60e5e1344c)
The Responses replay kill switch (_codex_reasoning_replay_enabled) was
session state that nothing reset, so a route that tripped it kept replay
off on every route the session moved to afterwards: a /model switch,
fallback activation, or the turn-start restore back to the primary.
The verdict now belongs to the route that earned it: all three route
changes reset it, and the switch rollback snapshot covers it.
Salvage of #61552 (reapplied on current main; the original's
switch_model / restore hunks no longer apply).
An invalid_encrypted_content 400 (or the Codex 401 token_expired twin)
used to strip every cached codex_reasoning_items blob AND turn
replay off for the rest of the agent's life, so a single rotated
sealing key cost all cross-turn reasoning continuity in a long CLI
session or a cached gateway agent.
The first rejection now only strips the stale blobs (persisted messages
and the request copy the retry is rebuilt from); blobs the route mints
afterwards are sealed with its current key and keep replaying. A second
rejection means the route cannot round-trip its own fresh blobs, so
replay is disabled for the session as before: a hostile route costs one
extra 400 compared to main.
The request-copy strip is required: main's global kill switch hid that
the retry was rebuilt from api_messages, which still carried the blob.
Review follow-up. model_preserves_prior_thinking pinned Fable/Mythos to
exactly 5.0/5.1 and returned False for any unparsed Claude id, so the next
minor (claude-fable-5-2, claude-mythos-6, a new family) silently fell back
to the cache-busting strip. Failure directions are asymmetric: replaying a
block the API strips is free, stripping on a model that keeps it rewrites
the cached prefix. Unknown Claude ids now keep; Fable/Mythos keep from 5.0;
Haiku, Claude 3, Opus < 4.5 and Sonnet < 4.6 keep stripping. Non-Claude ids
are unchanged (never keep).
Document in anthropic_thinking_replay why signature-rejection fingerprinting
stays request-wide: the 400's messages.N.content.M path indexes the
converted wire, while recovery only sees pre-conversion api_messages, and
that recovery deliberately covers last-turn-only native models too.
native_anthropic_accounting_projection rebuilt assistant rows from content only, so on
canonical input (preflight, tail walk) a promoted reply with content="" and an
api_content sidecar priced nothing for text that build_api_messages substitutes and
sends. The projection now lets a
non-empty string sidecar win over content, matching the generic wire shadow on main.
Post-build input is unchanged (the sidecar is already in content); ordered turns and
context-selection clones may overcount, never undercount.
The accounting invariant gains an assistant_api_content case that measures replayed
assistant text on the real default assembly: red before (preflight/tail_walk 0 vs
1999), green after.
Collapse 15 single-scenario tests and two parametrized invariants (40 cases)
into two parametrized invariant tests (35 cases):
- estimates charge exactly the thinking the wire replays: one case per route
(16 Anthropic model ids incl. dated snapshots, Nous, OpenRouter) and per
shape (real producer, context-selection clone, usage anchor, carrier dedupe,
opaque signature/redacted bytes, rejected, persistence-disabled,
storage-only reasoning, carrier fallback); wire, preflight, tail walk and
assembled pressure must all move by the same delta.
- rejected thinking never returns: same agent, rebuild, session switch,
compression child, context selection, multi-carrier tool row,
unfingerprintable, and the third-party Kimi route.
Drops projection-internals snapshots; no production change.
The compressor's native tail budget read rejected-signature fingerprints
from the session DB only. With persistence disabled (or a failed persist)
the agent's preflight stripped the rejected blocks while the tail walk
still priced them, so the two estimates diverged after a signature
recovery. The agent now binds its session-keyed rejection set onto its
context_compressor (same set object, so later updates reach both), and
the tail walk prefers that in-memory set for its bound session before
falling back to the persisted fingerprints.
Two parametrized invariants on production entry points (real SessionDB,
_recover_format_errors, build_api_messages, converter, _preflight_request_tokens,
ContextCompressor._walk_tail_budget, _carry_session_state_to_child):
- after a signature rejection every later request replays exactly the never-rejected
signed thinking across rebuild, session switch, compression-child resume and an
unfingerprintable rejection, while a Kimi route keeps its own replay contract;
- on a preserved-thinking native route, preflight and the tail walk move by exactly
the thinking that reaches the wire.
Both fail on origin/main and on the #129620 head (588b746); both pass here.
OPENAI_MODEL_EXECUTION_GUIDANCE told every GPT-family session to use
terminal/execute_code for arithmetic and terminal/read_file/search_files
for hashes, time, system state, files and git, even when the session's
toolset had none of those tools. Lean toolsets then refused plain
arithmetic as "prohibited mental computation".
Drop each guidance line when none of the tools it names are in
valid_tool_names. The toolset is fixed per session, so the system prompt
stays cache-stable.
Fixes#106506
The Codex usage probe caches its verdict per token for five minutes and a
fresh 429 does not clear it. When the endpoint reports quota left while
requests still 429, a fallback-pinned session restored the primary, failed,
and fell back again on every turn of that window, each turn costing a failed
request and two notices.
Lift at most once per probe interval per agent, stamped only when the pool
is actually benched. A genuine reopen is still picked up within one interval,
and benched turns between checks return before touching auth.json again.
A custom endpoint with api_mode codex_responses resolved an opted-in -900k
alias from the static table only (#116329), so it got 900K even when the
gateway's own catalog publishes a lower Codex-shaped max_context_window.
For an opted-in variant the route now asks its own catalog, at its own URL
with its own key, for that cap. The proxy's context_window is still never
trusted, plain slugs make no request, and a gateway without a Codex catalog
is probed once per negative TTL rather than on every resolution.
Reported by andrexibiza on #128288.
The Codex catalog fetch only cached a non-empty result, so a dead endpoint,
a non-200 answer or a body without a Codex catalog was re-probed on every
context resolution (agent init, /status, aux routing, @-reference turns),
each miss costing up to two GETs with a (5, 10)s timeout. A miss is now
cached for _CODEX_OAUTH_CONTEXT_NEGATIVE_TTL (300s): the entry is stamped
as if cached TTL - NEGATIVE_TTL ago, so the existing freshness check expires
it and the cache tuple keeps its shape. A fresh catalog stored by a
concurrent probe is never overwritten, and the max_context_window map is
left untouched so an observed cap still survives the outage (#130686). A
new token or endpoint is a new fingerprint and probes immediately. An empty
200 now reports fresh=False like the other misses (nothing to confirm).
turn/start can change the model but not modelProvider, which is fixed when the thread starts. With the model now forwarded per turn, switching in place between codex's own provider and a named custom provider would send the new provider's model to the old provider's thread. Record the thread's model provider next to its prompt composition and retire the thread when either changes.
The OpenAI API-key rung reaches the app-server as provider=custom with no
codex model_providers id, and its model is not vendor-normalized at agent
init (only openai-codex is), so a configured "openai/gpt-5.5" now went to
codex verbatim where it was previously dropped for codex's default. When
codex's own provider serves the turn, normalize the slug the way
openai-codex does; a named custom provider's model ids pass through.
The provider lookup moves into _codex_model_provider and is passed to
_codex_wire_model, and the session comment now says model is sent for every
provider, modelProvider only for named custom providers.
The CLI/TUI /model switch mutates the running agent, and its codex session
(and thread) is reused, so the model fixed at thread/start stayed in force
for the rest of the session. turn/start accepts a model that codex applies
to that turn and later ones (codex 0.147 schema; live: a thread started on
codex's default ran a turn on gpt-5.5 with modelContextWindow 258400).
Every turn now sends the agent's current wire model.
With the model now forwarded, a selected -900k alias reached codex verbatim
and every turn failed: live on codex 0.147 with ChatGPT auth,
thread/start {model: "gpt-5.6-sol-900k"} -> turn 400 "The
'gpt-5.6-sol-900k' model is not supported when using Codex with a ChatGPT
account". The alias is Hermes-side only (the Responses transport already
strips it); codex applies the catalog's extended window to the base slug on
its own (thread/tokenUsage reports modelContextWindow 828400 for
gpt-5.6-sol). Strip the valid suffix before it goes on the wire.
Codex homes are service-scoped while Hermes model selection is profile and
session scoped. thread/start only carried the model for named custom
providers, so openai/openai-codex sessions on model.openai_runtime:
codex_app_server silently ran codex's own default model, whatever /model or
config.yaml selected. Forward the agent's model on every thread/start;
the custom-provider modelProvider id and the no-credential payload are
unchanged.
Salvaged from #83269 onto current main (the session already accepts model=
since #115905; the forwarding was narrowed to custom providers there).
The salvaged phase fix added _role_message_with_phase beside the
_role_message_item constructor that every role item already goes through
(type=message for llama.cpp). Two builders of the same wire shape drift,
so the phase rule moves into _role_message_item and the second helper
goes away.
That path is the first one where caller-supplied phase text reaches the
wire (exact replay only forwards server-issued values), and the Responses
API accepts only "commentary" and "final_answer" on input messages. The
phase is now trimmed and lowercased and kept only when it is one of
those; anything else is dropped instead of forwarded into a 400.
The regression file's five tests become two invariants in
test_codex_responses_adapter.py: phase survives conversion with preflight
a no-op on converter output (pins "phase is not a replay marker" on the
codex_backend typed-list path), and only accepted assistant phases are
kept. Each guard is mutation-checked: main's adapter, dropping the
allow-list, treating phase as a replay marker, or copying phase onto user
items each turn a test red.
After the 1h context TTL, a failed catalog refresh sent resolution to the
static fallback, which applied the -900k bump without the max_context_window
the same token and endpoint had already published. A known 600K cap became
900K with no newer evidence. The fallback branch now passes the cached cap.
Introduced with the cap itself in #115872.
Reported by andrexibiza on #128288.
A reader that saw a fresh _codex_oauth_context_cache entry could find no
matching max_context_window yet (the max map was written second) and resolve
an opted-in -900k alias to the uncapped 900K instead of the catalog's ceiling.
Writing the max first means any reader that sees the context entry also sees
its cap. Reachable only with true thread interleaving (free-threaded builds);
the store order has been this way since #115872.
Reported by andrexibiza on #128288.
