The FTS fail-open detach now waits up to the caller's write budget (20 s /
60 s) for the write lock, so the one-time quarantine check before the loop
left a long window: a sibling that quarantined the file meanwhile still got
its triggers dropped and the stale breadcrumb committed on the quarantined
handle. Re-check the handle flag and the process-wide storage latch at the
top of every attempt, via the same _raise_if_db_corrupt(storage=True) that
_execute_write runs per attempt.
Classify the retryable lock error with is_sqlite_lock_error (result code
first) instead of a locked/busy substring match, matching #120488.
When a canonical write trips a corrupt FTS index, SessionDB detaches the derived
indexes (breadcrumb + trigger drop) and retries the write. The detach ran one
BEGIN IMMEDIATE on the writer connection, whose busy timeout is only 1 s, and gave
up on "database is locked" — so the canonical write escaped as "database disk
image is malformed". The usual lock holder is a sibling writer (gateway + TUI)
detaching the same index, so under load the second writer's turn was lost.
The detach now waits out lock contention on the caller's write budget with the
same jittered retry as _execute_write (default _WRITE_PATIENCE_S for the search
fail-open callers).
Repro: a second process takes BEGIN IMMEDIATE the instant the corruption error
surfaces and holds it 2.5 s. Base: append raises after 1.02 s (3/3). Fixed: the
row lands after the holder releases, FTS detached (3/3). Found by the E2E sqlite
torture chamber (fts_corruption_fail_open) at load ~200.
Re-review follow-up (N1, N2) for the group room failed-turn fix.
- Stranded harvest: the retained error only counts as the stranded turn's
own when no user row follows the turn's prompt. When a later turn in the
same session (a gateway/CLI message, a later room turn) failed, its error
is not the stranded turn's, and the real late reply now posts instead of
being dropped and reported as a failure.
- Live poll: the pre-submit comparison keys the retained error on
`turn_started_at` plus the `inflight` snapshot. The snapshot alone carries
no start time, so a failure identical to the previous turn's read as the
leftover and the pre-tool text was posted as the reply.
Independent-review follow-up for the group room failed-turn fix.
- Group room poll: a retained error newer than the pre-submit snapshot
(turn start replaces it with a fresh started_at) is this turn's failure
and wins over transcript text. The core closer writes no failed-turn row
behind a tool row, so "said X, called a tool, provider 401" left X as the
newest assistant row and the room posted it, dropped the 401 and re-drove
the member to the round cap (live repro on the PR head).
- Both pickers end the turn at a failed_turn row instead of scanning past
it; with the retained error gone (backend restart) the row's notice is
reported through the failed path instead of a silent pass / dropped
stranded marker. The stranded harvest treats a retained error as the
stranded turn's own the same way.
- REST cold-load/paging (/api/sessions/{id}/messages, /messages/around)
type legacy untyped notice rows like session.resume does, via one
read-side helper in agent/turn_failure_copy.py.
- Gateway closer: the fresh-session closure test now asserts the row's
display_kind through a real SessionDB round-trip (red without the
run_turn.py stamp).
- E2E: mock trigger that says text + calls a tool, then 401s; spec asserts
the room reports the error and never posts that text.
The main chat painted the failed-turn boundary row as an assistant message:
"Your request was not processed. Send it again..." sat above the provider
error card in the model's own voice, live and after a reload. Rows typed
display_kind="failed_turn" now hydrate as a system row, like the other
Hermes timeline markers.
Live: a 1:1 chat with the mock provider answering 401 rendered the notice in
aui_assistant-message-root before, aui_system-message-root after (both live
and after a cold reload).
Both were deleted by #120071 as failing on main. Triage: group-member-backend-failure
was a real regression (fixed in the previous commit); group-approval-click-submits
only failed under a leaked HERMES_YOLO_MODE (harness fixed two commits up) and passes
on a clean env. Restored verbatim from d9ca819cc4.
A spec run from inside an agent terminal passed HERMES_YOLO_MODE,
HERMES_INTERACTIVE, HERMES_SESSION_ID... straight into the sandboxed backend.
HERMES_YOLO_MODE alone made group-approval-click-submits fail (the gated
rm -rf ran without a prompt), which read as an approval regression; CI never
has these vars. buildAppEnv now drops inherited HERMES_* except the
harness's own HERMES_DESKTOP_* / HERMES_E2E_* knobs.
Since 8f0322da5b the core loop closes a failed turn with a Hermes-authored
assistant row ("Your request was not processed..."). The group room's turn
poll took the newest assistant row as the member's answer, so a provider 401
posted that line as the bot speaking, recorded the member as replied (no
failure row, no roster badge, no error reason from #117366), re-drove it, and
the drive ended at the round cap.
pickGroupTurnReply / pickStrandedGroupTurnReply, and the external-write
mirror, now skip rows typed display_kind="failed_turn" (stamped by the
previous commit), so the retained `inflight` error surfaces through the
existing failed-turn path again. No copy of the English text on this side.
Live: the deleted group-member-backend-failure.spec.ts is red on origin/main
(3/3, "turn stopped at the round/message cap"), green with this change
("Programmer hit an error — HTTP 401: ..."). Backend-only A/B with the main
renderer: green at 8f0322da5b8^, red at 8f0322da5b.
8f0322da5b closes a failed turn with a Hermes-authored assistant row
(FAILED_TURN_NOTICE / PARTIAL_FAILED_TURN_NOTICE). It carried no marker, so
the only way a client could tell it from a real answer was matching the
English copy.
Both writers (agent/conversation_loop.py::_close_durable_failed_turn and
gateway/run_turn.py::_hmwa_close_failed_turn) now stamp
display_kind="failed_turn" (agent/turn_failure_copy.py::FAILED_TURN_DISPLAY_KIND).
display_kind is a DB/display column already stripped from every provider
request (agent/turn_context.py), so the wire bytes and prompt cache are
unchanged; the ACP loopback test asserts the replayed row is exactly
{"role": "assistant", "content": FAILED_TURN_NOTICE}.
session.resume (tui_gateway/session_history.py::_legacy_display_kind) also
types untyped rows already on disk from the last five days, matching the
Python constants in-process, so clients key on the type alone.
Under persistent Docker, profile B's session-bound work keys profile:B but its
session-less (cron) work keyed home:<path>, so the same profile ran two
long-lived containers. The routed no-session branch now returns the same
profile key as branch 3 when Docker is persistent (profile-scoped); other
backends keep the per-home key.
Invariant for the import-time config bridge: a multiplexed backend's first import of
gateway.run can happen inside a routed profile's session (hermes serve imports it lazily
from an agent build), and the bridge must still write the launch home's agent.max_turns and
terminal.cwd into the process env. Red on the previous gateway.run (HERMES_MAX_ITERATIONS
came from the routed profile), green with get_process_hermes_home().
gateway.run bridges config.yaml into os.environ at import, keyed on
get_hermes_home(). The Desktop backend (hermes serve) first imports it lazily
from a session's agent build (tui_gateway.agent_callbacks._wire_callbacks),
under that session's routed profile override, so whichever secondary profile
built first latched its terminal.* (TERMINAL_CWD, backend...) and bridged
settings into the launch process env for every later launch-profile turn and
cron job.
Found by tests/e2e/core/tenancy/test_two_tenant_desktop_backend.py (C7 canary):
the default profile's cron env snapshot carried alpha's/beta's TERMINAL_CWD.
Use get_process_hermes_home(); identical for a standalone gateway.
A multiplexed host runs every profile's cron jobs without a session key, and
_resolve_container_task_id collapsed all of them onto the shared "default"
environment. Profile B's cron tool calls therefore reused the LocalEnvironment
the launch profile's job created: B's terminal subprocess saw the launch
profile's .env residue and bridged TERMINAL_* (TERMINAL_CWD, backend policy).
Found by tests/e2e/core/tenancy/test_two_tenant_gateway.py (C7 canary): alpha's
cron env snapshot carried default's TENANT_MARKER and TERMINAL_CWD. Session-less
work under a routed home override now keys home:<realpath>; the launch profile
and single-profile processes keep "default". The unit test that pinned the
shared key for a routed profile is updated.
The Desktop serve backend polls /api/fs/default-cwd, whose git-branch probe
captured git output with text=True and no encoding. subprocess then decodes
with locale.getencoding() - cp936 on zh-CN Windows - so a UTF-8 branch name or
git's localized stderr raised UnicodeDecodeError inside communicate()'s
_readerthread on every poll ([gateway-crash] ... 'gbk' codec can't decode).
Pin encoding='utf-8', errors='replace' on the remaining in-process captures
whose children emit UTF-8: the git branch probe, the self-repo guard's git
alias read, the Bot Mode DM transport (a Hermes CLI child, stdio forced to
UTF-8; builds on the salvaged errors='replace'), the agent-browser npx probe,
the lightpanda help probe, and the cua-driver stderr drain thread.
Fixes#83851
Same class as the cron script runner: text=True decodes with the locale codec under strict error handling, so one undecodable byte in a child's output raises UnicodeDecodeError inside subprocess.run. That is a ValueError - neither except (subprocess.SubprocessError, OSError) nor except TimeoutExpired catches it - so it escaped into user-visible paths instead of the intended error handling.
tools/bot_mode_dm._run_local_turn: a transport that exits 0 but prints a byte the locale cannot decode crashed the delivery instead of re-emitting the transport's streams (stdout is the reply text the completion notification carries back).
agent.command_token_source._mint: a key_cmd printing a non-UTF-8 byte killed token minting with a traceback instead of the documented CommandTokenError path, so provider auth failed in a way the caller could not report.
Both decode with errors=replace now: the exit code and the surrounding checks still decide what happens, and damaged bytes appear as U+FFFD in the text we carry.
* refactor(update): one predicate for serve rows outside the gateway matrix
The inventory branch of `_marker_only_restart_obsolete` inlined the rule for which
serve/dashboard rows the gateway matrix neither covers nor needs to (supervisor-owned,
or a manual serve handed to its own reminder). The inventory-less branch needs the same
rule for #118742, so it moves to `update_cmd_fleet_gatewayless.runtime_outside_gateway_evidence`
and both branches will read one definition. No behaviour change.
* fix(update): a Desktop-only host settles an inventory-less restart obligation
A host that runs no gateway (the Desktop app alone) can be left with an inventory-less
fleet-restart obligation: an updater that died before recording its inventory, or the
pre-inventory writer. With no owed set, the live gateway matrix is its only evidence, and
on that host the matrix is empty forever, so `_marker_only_restart_obsolete` never settled
and every later `hermes update` exited 1 with "gateways are still off the checkout code"
(#118742).
An empty fleet alone cannot tell that host from one whose gateway the dying update stopped,
so the inventory-less branch now asks the live host, never a historical receipt:
`host_owes_no_gateway_restart` settles only when no profile's `gateway_state.json` claims a
state other than stopped/startup_failed (a gateway that went away without a clean stop keeps
the obligation) and every live runtime sits outside the gateway matrix
(`runtime_outside_gateway_evidence`, shared with the inventory branch). HEAD must still
contain the pulled SHA (`checkout_contains`, same rule as #119367). Probe failures keep it.
Tests: the scoped-reconciliation matrix now holds its host at "the update stopped a gateway"
so it keeps pinning receipt independence; a new host-evidence matrix covers Desktop-only,
clean stop, carried commit, stopped gateway in a named profile, unclassified and
unidentified serves, a gateway row without fleet identity, and a diverged checkout. Two
manual-serve tests that assumed an empty fleet always stays pending now stub the live host
and assert the manual reminder survives the gateway obligation settling.
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
---------
Co-authored-by: KoNit-K <124019182+KoNit-K@users.noreply.github.com>
Adapt #93931 to the current packaging lifecycle with real packager paths.
Retain the non-blocking restore from 0fbb1bc337539902d1132af4a02412b52e73d6ed,
exclude Chromium gender packs, and leave Windows stamping in afterExtract.
(cherry picked from commit c37a15c2fe5665081e7c9e6bfbe93a4c2f3aadfe)
Co-authored-by: brooklyn! <brooklyn.bb.nicholson@gmail.com>
Builds on the salvaged voice.stop_phrases matcher:
- /api/config merges DEFAULT_CONFIG, so an untouched install reports
stop_phrases: ["stop"] instead of omitting the key. Compare against
/api/config/defaults so that case keeps the built-in English list
("goodbye", "never mind", ...) and only a list the user changed replaces
it. Parse malformed values (mapping, null) as the backend does: default.
- Normalise transcripts and phrases with NFKC and Unicode punctuation
(\p{P}), so NFD Cyrillic from STT, «guillemets» and CJK full stops match.
- Wire the existing voice.barge_in_threshold_multiplier into the desktop
barge-in monitor: it scales the quiet-floor multiplier and the playback
clamp (PLAYBACK_MIN_TRIGGER_LEVEL) by its ratio to the backend default
(3.0), so stock behaviour is unchanged and a quiet BT HFP headset can
interrupt with a lower value. No new setting or env var.
The desktop voice loop advertised configured stop phrases in the notice but matched only a hardcoded English list, so non-English STT phrases (e.g. отбой) were submitted as prompts. Wire the matcher to the same voice.stop_phrases config the backend uses.
Read Aloud removed Markdown tables with no spoken trace (#86602). Speak the
header row in their place ("Model, Price.") and keep skipping the body rows,
so the listener hears that a table is on screen and what it compares
without cell-by-cell data. The header is the reply's own text, so it is
always in the voice's language; a localized "table omitted" notice would
follow the UI locale instead and reintroduce the mismatch when the two
differ. A table whose header cells are all empty stays silent.
Also keep the sentence punctuation after an inline MEDIA: token on both
the desktop and backend paths ("see MEDIA:/x.py. Then" no longer loses
its full stop), per review on #89367.
- MEDIA:/path tokens (which render as 'Open <hyphen-slug>.xlsx') are stripped
in both the backend normalizer and desktop sanitizer; a voice cannot say
them and ElevenLabs-style models loop on the hyphenated slug ('eeeeee').
- Desktop no longer speaks the 'code block omitted' / 'link' placeholders;
unspeakable tokens are silence, not English words (#86602).
- Line-final colons close to periods before newlines flatten, on both sides,
so 'here is the list:' followed by a code block is spoken as two sane
sentences instead of hanging the voice on an open colon-pause.
Regression tests on both sides pin each behavior, including the exact
colon-plus-code-fence reproduction.
Salvaged from #89367 (79db15c1a8). The symbol/dash/euro expansion and the
bare-path "the path" placeholder were not carried: on the desktop they put
English words into non-English voices, which is the #86602 bug class.
On Windows (fr-FR, es-AR, de-DE reports) the zone name arrives in the ANSI
code page but is decoded under a UTF-8 LC_CTYPE (UTF-8 mode, or Piper/espeak
flipping the process locale mid-run) with surrogateescape. datetime.strftime
splices tzname() in as UTF-8, so "%Z" raised UnicodeEncodeError while the
system prompt was being built and every new/compressed conversation died.
Rework safe_strftime into a small repair: output is untouched for valid text
(the system prompt stays byte-identical), surrogateescape'd bytes decode back
through the ANSI code page ("heure d'été"), anything else degrades to U+FFFD,
and a raising "%Z" is rendered from the repaired tzname(). hermes_time no
longer imports agent.* at module load.
Route the remaining locale-name sites through it: cron quota-hold notice,
auxiliary cooldown notice, cron session titles, session_search dates,
insights, learning graph and billing renew dates. Tests use real datetimes
with a surrogate zone name instead of stubbed strftime.
Fixes#102910
Co-authored-by: Aniruddha Adak <127435065+aniruddhaadak80@users.noreply.github.com>
The bounded startup locale retry (#96177's second half) already landed on
main; this pins its cleanup path, which had no coverage: once the provider
unmounts, the scheduled retry is cleared instead of polling /api/config for
a tree nobody renders. Adapted from the unmount case in #96213.
Co-authored-by: 686f6c61 <github@00b.tech>
Follow-up to the progress-aware probe (#96177):
- Wait while the spawned child is alive instead of "alive and wrote output
in the last 30s". The stall this fixes is silent: the backend answers
HTTP, then holds the GIL importing gateway platform modules and prints
nothing until the loop recovers (the web_server heartbeat only logs
"event loop stalled" afterwards). Output recency counted down during
exactly that window, so a long stall still failed. A live loopback child
whose listener accepted the TCP connect is busy, not refusing; refusals
and auth rejections are immediate error/close events, not timeouts.
Drops the now-unused BackendOutputTail.lastActivityAt/backendMakingProgress.
- One policy home: spawnedBackendProbeOptions() in gateway-ws-probe.ts
(base 10s, cap 90s = the port-announcement cold-start budget); both the
primary and pool spawn paths use it. Remote gateways, "Test remote" and
host-backend attach keep the fixed budget.
- Without keepWaitingWhile the probe is again a single one-shot timer (no
1s polling during the base budget). Timeout reasons name what ended the
wait: base budget, backend stopped after the budget, or the cap.
- Tests drive every deadline with fake timers (no wall-clock races) and
cover the production policy: a 30s stall succeeds while alive, a child
that exits mid-wait fails at the next check, a live backend that never
answers fails at the cap.
Windows cold start can hold the backend's GIL for 12-28s while gateway
platform modules import (feishu/lark, weixin, telegram C extensions),
leaving the /api/ws upgrade unanswered well past the probe's fixed 10s
connect budget. The probe then fails, the desktop declares the backend
unhealthy, spawns a second backend, and boot lands in the
'Ignoring stale Hermes backend exit (1)' cascade (#96177).
The probe now supports backend-progress-aware waiting: once the quiet
connect budget passes, a keepWaitingWhile callback is polled and the
deadline only fires when the backend stops reporting progress (hard
capped by maxConnectWaitMs so a hung backend still fails the boot).
- gateway-ws-probe.ts: keepWaitingWhile / progressCheckIntervalMs /
maxConnectWaitMs options; behavior is unchanged when the callback is
omitted (remote-gateway probes keep the fixed timeout).
- main.ts: both local-backend boot paths (primary window + pool) pass
keepWaitingWhile = backendMakingProgress(child, outputTail) with a
90s cap aligned with the existing port-announcement cold-start budget.
- backend-claim.ts: BackendOutputTail tracks lastActivityAt; new
backendMakingProgress() signal: alive child + (no output yet, i.e.
inside the block-buffered import window, or recent output within 30s).
- Regression tests: probe waits past the base budget while progress is
reported, fails once progress stops, respects the hard cap, fails
closed on a throwing callback; tail activity + progress-signal units.
Benchmark (simulated 13s import stall, production budgets):
before: FAIL at 10.0s -> boot cascade after: OK at 13.8s
warm start: OK 0.96s after: OK 0.96s (no delta)
dead backend: FAIL 10.0s after: FAIL 10.1s (same)
hung backend (alive, never ready): after: FAIL 90.8s (hard cap)
Closes#96177
(cherry picked from commit e8dbe9e0a2bfb6853401d55a2bbda02393560de0)
The anthropic.com-only refresh guard stopped token rotation on hosts the
resolver itself accepts as native Anthropic (*.claude.com, /anthropic
proxies), which already receive the Anthropic token at startup. Long
sessions there hit 401 and the recovery refresh was refused too.
Official hosts (anthropic.com, claude.com) rotate as before; any other
host rotates only when the current key is already an Anthropic
credential (sk-ant- or OAuth), so the #17829 case (custom key swapped
for ANTHROPIC_API_KEY / the OAuth token) stays blocked. Azure keeps its
static-key exclusion. Also corrects the _resolve_openrouter_runtime
docstring, which still claimed OPENAI_BASE_URL is never consulted.
The OpenRouter key ladder falls back to OPENAI_API_KEY (legacy home of an
OpenRouter key). When OPENAI_BASE_URL binds that key to another host, a
bare `provider: custom` with no endpoint fell through to openrouter.ai and
shipped the OpenAI-proxy key there instead of failing fast. Use it for
OpenRouter only when OPENAI_BASE_URL is unset or names the same host.
Hosts compare through utils.base_url_hostname, so a scheme-less
OPENAI_BASE_URL (proxy.corp:8080/v1) still counts as binding the key.
Found by the C11 routing truth table (tests/e2e/core/tenancy).
_try_refresh_anthropic_client_credentials runs before every request and on
401 for provider 'anthropic'. A URL-bearing model alias under the built-in
'anthropic' label (#28660) points that provider at a foreign host with no
key; the pre-request refresh then swapped in ANTHROPIC_API_KEY / the OAuth
token and sent it to that host.
The third-party guard above matched "anthropic.com" as a substring of the
URL, so a host like 127.0.0.1:8080/anthropic.com still passed as native and
got the refresh. Refresh only when the endpoint's hostname is Anthropic's own
(or no endpoint is set).
Found by the C11 routing truth table (tests/e2e/core/tenancy): a /model
switch to such an alias carried the vendor key to the alias host.
With provider 'anthropic' pointed at a third-party Anthropic-compatible
endpoint, _try_refresh_anthropic_client_credentials only skipped Azure and
otherwise re-resolved native Anthropic credentials (ANTHROPIC_API_KEY, the
stored OAuth token) and rebuilt the client with them, so the endpoint got
a credential it was never given. Skip the refresh for any endpoint
build_anthropic_client already classifies as third-party.
Ported from run_agent.py onto agent/client_lifecycle.py, where the method
lives now; the separate Azure test is dropped (Azure is one such endpoint).
Salvaged from #17829.
ACP resolves toolsets like the gateway for the same platform config,
which includes plugin toolsets. An empty platform_toolsets.acp list still
adds enabled plugin toolsets, so the docs point at agent.disabled_toolsets
for removing them.
A fresh ACP agent appended mcp-<server> for every enabled config MCP server
unconditionally, so a platform_toolsets.acp allowlist of server names and the
no_mcp sentinel were ignored on ACP while the gateway honoured both.
The MCP half now comes from the same _get_platform_tools(config, "acp") call
as the base toolsets: its server names (default every enabled server, a listed
allowlist, or none for no_mcp) are keyed as mcp-<server>. Editor-provided
session/new servers are unchanged.
Docs: `hermes tools` has no ACP platform entry, so drop the claim that it
configures platform_toolsets.acp; document the MCP rules with a config example.
A fresh ACP agent hardcoded enabled_toolsets=["hermes-acp"], so
platform_toolsets.acp never narrowed the editor tool surface, unlike the
gateway, cron and api_server which all resolve via
hermes_cli.tools_config._get_platform_tools. Resolve the ACP base the same
way (ACP keeps appending its own mcp-<server> entries), and treat only None,
not an explicit empty list, as "use the hermes-acp default" in the /tools
and MCP-refresh rebuilds so a deny-all list cannot re-widen mid-session.
With the unconfigured default the resolved tool definitions are
byte-identical to the hermes-acp composite, so existing sessions keep the
same tool list and prompt cache.
The fresh-session assertion in test_make_agent_prefers_passed_toolsets_over_config_servers
now checks membership of the config MCP entry: the resolver returns the
expanded toolset keys rather than the bare composite name.
Refs #74582, #79516. Credit: #64045 (@israellot), #80309 (@thatssoheil),
#106834 (@nicolasramos) proposed the resolver routing.
test_acp_real_agent_gets_session_db_for_recall replaced the whole
hermes_cli.config module with a one-attribute stub, so any real import
from it inside _make_agent (now hermes_cli.tools_config, for the shared
toolset resolver) died with ImportError. Patch the one function instead;
the assertions are unchanged.
Keep the behavioural /tools check (real get_tool_definitions, red on main);
drop the kwargs-forwarding and memory-provider cases, which the fresh-agent
tool-surface test in the next commit covers end to end.
Review follow-up: _cmd_tools rebuilt its listing without
state.agent.disabled_toolsets, so a config-disabled toolset was filtered
from execution but still advertised by /tools. Pass it through, matching
the session tool-surface rebuild.
New tests exercise the real get_tool_definitions (no patching) and assert
a disabled toolset is absent from both the /tools listing and the rebuilt
valid_tool_names — with a baseline assertion that the toolset is present
when nothing is disabled, so the check cannot pass vacuously.
The CLI (cli.py: CLI_CONFIG['agent'].get('disabled_toolsets')) and the
gateway (gateway/run.py) both read agent.disabled_toolsets from config
and pass it to AIAgent. The ACP adapter's _make_agent never did, so
state.agent.disabled_toolsets was always None and the ACP tool-registry
rebuild (acp_adapter/server.py -> get_tool_definitions) included every
tool in the enabled toolsets — a toolset the user disabled in config
(todo, browser, ...) remained fully executable in editor/ACP sessions.
Observed live: a 'todo' tool call executed from a profile whose config
lists todo in agent.disabled_toolsets.
Read agent.disabled_toolsets in _make_agent and pass it through,
mirroring the CLI and gateway paths.
Claude-Session: https://claude.ai/code/session_01YNvCUipheR7yx4VorUL2jW
After an overflow split the heads are sealed and the message id is cleared,
so the tail goes out as a first send in the tick that carries the tool
boundary. When that send failed, _end_segment skipped the unseen-tail flush
(it required a truthy _message_id) and _reset_segment_state cleared
_accumulated: ~1.1k chars vanished. The same loss existed for a plain failed
first send at a segment break.
The flush now runs whenever the boundary update did not land, except for the
__no_edit__ sentinel (its continuation still goes out once via the fallback
final). With a real message id the condition is unchanged, so no new send
there. The flush also honours the egress guard like the other new-message
fallbacks, since it can now run on the path where a draft was declined.
Tests: the oversized-leftover file now pins one invariant, parametrized over
none / commentary / segment_break / tail_send_fails: every tail line reaches
the channel, never twice as a new message, and post-boundary text is not
glued onto the pre-boundary preview. tail_send_fails is red on f361b39
(24 lines lost), green here.
Review finding on the seal re-split: `continue` after `_split_first_send`
skipped the rest of the tick, so a commentary drained with the oversized
burst was dropped and a tool boundary was lost (post-tool text then
extended the pre-tool preview). The initial-overflow gate had the same
`continue`.
Seal first (a no-op without a message id), split once, then push the
tail through `_push_update` in the same tick so commentary, segment end
and the flush barrier run normally after it. `continue` is kept only when
a head send failed and the full text must stay for the fallback final.
- test_stream_consumer_oversized_leftover.py: keep the user-visible invariant
(no line of the answer reaches the channel twice as a new message) and drop
its lane-size twin, which pins the same re-split.
- test_stream_consumer.py: a failed first send leaves no uneditable partial
preview; the complete reply is the only message that reaches the chat.
Both are red on origin/main and green with the fixes.
When the stream consumer's first send failed (e.g. a Telegram timeout that
never reached the platform), _first_send disabled edits but left the
message id unset, so the next tick sent another first send: a partial
preview that could never be edited. The final reply then went out as a new
message and the truncated preview stayed on screen next to it (on every
first-send timeout, not only under load). With edits disabled, skip
non-final first sends; the final is delivered once, complete.
Found by the C12 exactly-once suite (fault_matrix[stream_timeout_first_send-
fk_tg|fk_dc]).
Two follow-ups to the re-split after _seal_overflow_heads (previous commit),
found by the exactly-once E2E suite driving the real GatewayRunner on
platforms whose limit a streamed reply overflows (Discord 2000):
1. _split_first_send kept the adapter's " (n/n)" chunk indicator on the
tail chunk it reuses as the live preview, so later deltas were appended
after it: the user saw "...wo (3/3)rd0575..." embedded mid-reply and the
visible text no longer matched the transcript. Strip the indicator from
the kept tail.
2. The re-split after a seal now `continue`s like the gate above it when the
turn is not finished: the tail is still unsent, and falling through into
the segment-break reset would clear it.
tests/gateway/test_stream_consumer.py::...fence_aware_split asserted the old
behaviour (the tail starts with the full indicator-suffixed chunk); it now
asserts the indicator is dropped from the tail and never appears in an edit.
_seal_overflow_heads clears the edit target mid-iteration, so the overflow gate
evaluated at the top of run() is already stale when the leftover is pushed. The
seal loop exits after ONE successful seal (its condition includes
_message_id is not None), so a buffer far past the limit leaves most of itself
behind. That leftover then reaches _push_update -> _first_send with no message to
edit, on a NON-final tick, and adapter.send() chunks and caps it itself, numbering
the pieces (i/n). The turn-final lane later publishes the same text again with its
own denominator, because the two lanes use different budgets and different payload
shapes.
Observed in production on Discord: one inbound message, one API call, no tool
turns, a 36642-char answer, and two interleaved sequences on screen - (i/10) and
(i/9) - whose first chunks were byte-identical once the indicator was stripped and
whose concatenations shared a prefix. The gateway's normal final send was
correctly suppressed (streamed=True, content_delivered=True), which places the
duplicate inside the consumer rather than in the gateway's delivery ledger.
Fix: re-check the overflow gate AFTER the seal and hand a still-oversized leftover
to _split_first_send, which is the consumer's own cap-aware, ledger-aware splitter
and already owns sealing. This removes the same-iteration invalidation rather than
compensating for it downstream, and it does not add a fourth delivery flag: the
existing flags behaved correctly here.
Negative control on this branch: with the change reverted, both new tests fail,
including the one asserting that no line is published twice.
ev_long_run used to flag a missing claim refresh as stalled_heartbeat
and carry on, so neutralising heartbeat_fire_claim in _heartbeat_loop
left all five scenarios green. Every virtual 30 s of the 15-minute hold
now (a) waits for the run's heartbeat to restamp the claim and fails if
it never does, and (b) has a contender call the real claim_job_for_fire
and asserts it loses, also once the first stamp is older than the TTL.
With the heartbeat replaced by True all 5 scenarios are red on the
refresh assertion; on main they are green.
The #119970 cell no longer carries a whole-scenario strict xfail. While
its behavioural probe reproduces, only the oracle's fired-set and
next_run_at checks may deviate: each deviation is recorded, the model
follows the stored slot, and the soak runs every virtual day with all
other checks live, XFAILing at the end only if a deviation was seen
(and failing if the probe says open but none was). The #120314 entry
is gone (merged). A scenario that fails mid-hold now releases its held
run before teardown so it cannot bleed into the next scenario.
