The model-picker fix added a second copy of the choice picker's auth
check, and each copy rebuilt the callback ctx that _handle_callback_query
had already computed as `cb`. The chat-id picker loop is the only route
into both handlers. Gate there once with the existing cb, so a picker
added to that loop later is covered without anyone having to copy the
check again. Fail-closed behaviour is unchanged: _callback_authorized
returns True only for an authorized tapper and answers the denial text
otherwise. The back-button test calls the handler directly and no longer
needs its auth stub.
Co-authored-by: Dusk1e <yusufalweshdemir@gmail.com>
Every other ModelPickerView callback (provider/model select, expensive
confirm, back) runs through the shared _HermesView._gate auth check, but
_on_cancel did not. A non-allowlisted member could tap Cancel on the
owner's picker, marking it resolved and clearing the view. No model
switch was possible, so impact is low, but the gate should be uniform.
Reuse the same _gate call the sibling handlers use.
Model picker taps (mp:/mpg:/mpv:/mm:/mc:/mb/mx/mg:) never went through the
callback allowlist, so a group member who is not allowlisted could tap the
owner's /model picker and switch the owner's session or config model. Gate
the handler on _callback_authorized like the choice-picker, approval and
clarify buttons, before any picker state is read.
Ported from #33859 (gateway/platforms/telegram.py) to the plugin adapter.
(cherry picked from commit b002be8feacd09fde0ee9752de7b396d0aac69ba)
- salvage summary cap and _bound_oversized_record still composed bare
truncation idioms in model-visible text; route them through elide /
elide_middle so a copied marker is guard-visible.
- the active-task line repr()'d the elided text, escaping the marker's
apostrophe when the user text held both quote kinds and hiding it from
the guard; elide after repr instead (text within the cap stays whole).
- a leftover budget smaller than the marker produced a content-free,
over-budget marker line in _build_verbatim_user_section and the Slack
nested-attachment path; skip the item instead.
- _build_verbatim_user_section elided twice, reporting the wrong total;
one elide at min(cap, remaining).
- drop redundant len() pre-checks before elide() and name verification
stop's repeated 1200.
Co-authored-by: ahisblessed <ahisblessed@users.noreply.github.com>
Co-authored-by: salch-cred <salch-cred@users.noreply.github.com>
The Slack Block Kit payload dump and the nested-attachment text budget
are both fed to the agent, and trajectory_compressor's summarizer input
becomes training data; all three still used the imitable bare
"... [truncated]" idiom. Route them through elide()/elide_middle() with
module-level imports and extend the no-idiom invariant to scan
plugins/platforms/slack and trajectory_compressor.py.
Co-authored-by: salch-cred <salch-cred@users.noreply.github.com>
With reply_in_thread: false the whole channel is one session and the bot
answers top-level, so an unmentioned top-level message there is a
follow-up in a conversation the bot is part of, like a thread reply.
reply_expected is now False for a free-channel message only when it starts
its own session (a new top-level thread), else None. The bot-id set is
built inside _slack_reply_expected, as _channel_gate_allows does.
The Slack rule marked every admitted message that was not a DM, a mention
or a command as not addressed, so a plain "done?" in a thread the bot is part
of, or a reaction trigger, could end on a bare silence marker and vanish,
the case #111624 fixed (#110952).
reply_expected is now False only for a message that opens by @mentioning
someone else, or a top-level message a free-response channel admitted
without a mention. Reaction triggers and pipe-form self mentions count as
addressed; other thread replies are None (visible fallback). The
free-channel predicate moves into _slack_is_free_channel so the gate and
the rule read the same one. The test drives the real _handle_slack_message.
Docs describe the rule in its own note instead of the
ignore_other_user_mentions tip, and the messaging index documents the
human-turn fallback.
Since 5ea8fb2b78 (#111624, for #110952) the gateway rejects a bare silence
marker on any human turn and delivers "The model returned only a silence
marker for a message that needed a reply" instead. That protects a human
who asked this bot something and got nothing back. It also fires on every
human message the adapter admitted without the bot being addressed at all:
a free-response channel, a thread follow-up under
`thread_require_mention: false`, or a message @-mentioning another person
or bot with `ignore_other_user_mentions: false`. A bot whose SOUL declines
peer-addressed turns with a deliberate marker now posts that notice on
every such message. A fleet running several bots in shared Slack threads
reported it as spam on v2026.9.21. #37940 established that intentional
silence must not be re-inflated. Both contracts hold once the turn knows
whether a reply was expected.
`MessageEvent.reply_expected` (True, False, None) is set by the adapter
where the message is admitted. Slack (`slack_reply_expected`): a 1:1 DM,
an @mention of this bot or a command is True, anything else it admits is
False. Other adapters leave None, which keeps today's behaviour, so nothing
changes for them until they are ported. `response_filters.silence_allowed`
holds the one rule (machinery turn, or reply not expected) and both call
sites use it: the live turn in `run_turn._hmwa_shape_agent_response` and
the crash-recovery redelivery from #120377 (1136f135dd), which reads the
flag back from the persisted turn metadata. The suppressed case logs one
DEBUG line naming platform and chat.
Operator workaround until this lands: `platforms.slack.extra.
ignore_other_user_mentions: true` drops peer-addressed messages before a
turn exists.
(cherry picked from commit 094439776ab898cccde303a1c2c911c8ab5bfb75)
Four surfaces build a throwaway AIAgent and never call close() — the
owner boundary that releases memory-provider sessions, tool
subprocesses and httpx clients. In long-lived processes each run leaked
all of them until exit:
- batch_runner._process_single_prompt: one agent per prompt, N prompts
per batch process.
- feishu_comment._run_comment_agent: one agent per comment run in the
gateway process.
- tui_gateway prompt.background: one side agent per background turn.
- cli /bg: one agent per background task in the CLI process.
Wrap each run in try/finally with a suppressed close(), mirroring
gateway/run.py's owner pattern. preview.restart stays deliberately
unclosed (its task exists to leave a detached server running), and the
prompt.background side agent is safe to close: its session_id is the bg
task id, so close() reaps only its own task resources.
Fixes#50197
- A mid-turn notify reply (/status, /approve, clarify answer) shares the
stream's thread key; it no longer seals and overwrites the half-streamed
answer. In-place replacement now requires the final to match the stream
after normalizing mrkdwn markers and whitespace; anything else posts fresh
and leaves the stream open.
- One _commit_stream helper for both seal-then-commit paths, so the rewrite
path also falls back to chat.update when stopStream fails.
- A stream reopened after a server-side seal is seeded with only the text
past the sealed message (tracked as 'base'), not the whole segment.
- Streams older than 15 min are sealed and dropped on the next start.
- _stream_key reuses _workspace_thread_key/scope_id_for_chat; the stream
dict no longer duplicates chat/team ids.
5648f81431 fixed this exact server-side seal (Slack closes a native stream
after a few minutes of a long turn, live-observed at ~5m20s; the lifetime
is not documented) for the native task-card stream: on
message_not_in_streaming_state from appendStream, drop the dead ts and
start a fresh stream, seeded with the full current content so nothing is
lost.
send_draft — the plain-text native streaming path used when task cards
are not enabled — hits the identical seal but never got the fix: its
generic except block only recognizes the feature-gate markers
(not_allowed, missing_scope, ...) and otherwise just logs debug and
returns failure. gateway/stream_consumer_transport.py's
_send_draft_frame() docstring is explicit that "any failure permanently
disables drafts for this run" — so a long turn streaming as plain text
degrades to the edit-based fallback for its remainder exactly the way
the task-card bug did before 5648f81431.
Mirror the task-card fix: on message_not_in_streaming_state from
chat.appendStream, drop the dead ts and _start_stream() a fresh one
seeded with the full accumulated text (not just the delta), so the next
frame's delta still resumes correctly. One reopen per frame; a second
rejection propagates as a real failure, matching the twin's behavior.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
(cherry picked from commit 2b4ff4e23bdf684d2fde1a9512f11dcd7ef99c42)
A mrkdwn-rewritten turn-final (e.g. *Done:* -> _Done:_) no longer continues the
streamed text, so it was classified unrelated: the stale stream was sealed and
send() posted a second message (#95430 cause B). Seal, then chat.update the
sealed ts with the final; post fresh only if the in-place update fails.
Co-authored-by: liguoyu <guoyu.li@lcfuturecenter.com>
Problem: with native streaming (chat.startStream/appendStream/stopStream)
the same answer could land twice in a thread — once as the streamed
message, once as a fresh chat.postMessage — while the streamed message
kept its live-typing indicator.
Mechanism: `_try_finalize_stream` matched the turn-final against the
streamed text with a raw `startswith`. The agent strips `final_response`
and joins footers with `rstrip()`, so any surrounding-whitespace
difference made the finalize fall through to a plain post although the
open stream already showed the whole answer (and was never sealed). A
`chat.stopStream` failure took the same fresh-post path even when the
streamed text equalled the final. Streams were also keyed per `chat_id`
only, so two concurrent turns in two threads of one channel sealed or
overwrote each other's stream.
Fix:
- Key native streams per `(team_id, chat_id, thread_ts)`; the stream
consumer stamps the same `thread_id` on every draft frame and on the
turn-final `send()`, so both resolve to the same key.
- Honor the streaming contract (gateway/AGENTS.md): sends carrying
`_interim_send` or `expect_edits` never seal a stream.
- Classify the final against the streamed text as equal / extends /
unrelated with edge-whitespace tolerance (`_stream_relation`). The
stopStream delta is sliced from the RAW final, so nothing inside the
answer (blank lines, fences, tables) is dropped or repeated.
- Commit rule: one `chat.stopStream`, one retry only when no tail is
appended (`markdown_text` APPENDS, so an ambiguous failure must not
repeat it), then `edit_message(finalize=True)` on the stream ts as the
idempotent in-place commit — it already owns format/truncate/Block Kit
and the block-rejection retry. Only when both fail does `send()` post
a fresh message (a duplicate beats a lost answer).
- Oversized tails and rewritten finals (`notify=True`) seal the stale
stream on what is visible before falling back, so no stream is left
with a live-typing indicator.
- `_seal_stream` takes the exact unsent delta instead of recomputing it
from `final_text`; `disconnect()` and the stream API calls route
through the stream's own team client.
Tests: tests/gateway/test_slack_native_streaming.py covers the
whitespace-only difference, the stopStream-failure commit path, the
bounded retry, the uncommittable fallback, interim/preview sends,
per-thread keying, oversized tails, rewritten finals and the
GatewayStreamConsumer end-to-end path.
(cherry picked from commit a64d10071ce7816b124467e29407d7d47bfdde8d)
dingtalk-stream 0.24.3 start() retries forever inside the SDK, logging a
malformed logger.exception() every 3s, so the adapter breaker never saw the
error. Install a dedup filter on the SDK logger that can't raise on bad
format args, detect the websockets incompatibility (bare or chained
TypeError), log one ERROR with a pin-consistent hint, and hand off via
_set_fatal_error(retryable=False) + _notify_fatal_error(): only a
reinstall of the pinned versions and a restart fixes it. Breaker stays
tripped until the error type changes; constants moved to module level.
The reconnect storm in #24851 is driven by a dingtalk-stream/websockets
incompatibility that raises TypeError on every start(); backoff never
recovers it. Log the first occurrence per error run at ERROR with an
upgrade hint instead of a generic WARNING.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
DingTalk stream-mode reconnection storms the gateway: when start() raises
the same error every cycle, _run_stream logs a WARNING and reconnects at the
60s cap forever, generating hundreds of MB of identical log lines and hanging
the gateway.
Add a per-error-type circuit breaker: after 5 consecutive identical errors,
suppress the repeated WARNING (one ERROR summarises) and pause 300s instead of
spinning at 60s. Reset backoff + counters after a clean start() so a recovered
connection is treated fresh.
Salvage of #24881 re-implemented on current main: the original fix targeted
gateway/platforms/dingtalk.py, which has since been refactored to
plugins/platforms/dingtalk/adapter.py. Same logic, new path; tests import the
new module.
Closes#24851.
(cherry picked from commit 84201c58255ae3d3c9b269ac777ea0ff444d9229)
Surface code/msg at debug when the recall API returns non-success
(matches the exception path) and trim the docstring. Drop the trivial
disconnected-client test.
Feishu had no delete_message, so a failed finalize-edit plus fallback
send left the truncated edit bubble next to the full final. Implement
the SDK delete and thread the fallback send to the originating message.
(cherry picked from commit c61add84ad40b1bc39288405a0a05b4f621e00fc)
a74e0155b6 made attachments[].blocks[] reach the agent through
_append_link_unfurls, but rendered each attachment's blocks with no ceiling.
Slack allows 20 attachments per message, so one alert could project 20x what
a single attachment does (measured: 3,247 chars for 1 -> 64,855 for 20 with
8x400-char rich_text sections each), while the top-level blocks path caps
once at 6000.
Share one budget (_SLACK_UNFURL_BLOCKS_MAX_CHARS, the same 6000 the top-level
path uses) across the array: the first attachment keeps its body, later ones
are truncated against the remainder, and a spent budget still leaves every
header visible. After: 3,247 -> 6,659 chars at 20 attachments.
(cherry picked from commit 3efc1e4c532c38a220d4c6ea53745d59d334aa3e)
Telegram's per-(chat_id, status_key) status-message cache grew without
bound; give it the same _STATUS_MESSAGE_IDS_MAX=2000 FIFO half-trim the
Slack adapter already has. In both adapters, guard the post-await write-back
after a successful edit with a compare-before-write (only re-store the id if
the cached entry is still the one we edited) so an eviction or replacement
that happened during the await is not undone.
Partial salvage of #87480: kept the Telegram bound and both compare-before-write
guards (re-applied by hand, 17586 behind), defined the max as a class attr
like Slack instead of an instance attr, dropped the 4 new tests.
(cherry picked from commit 43ae95e98e)
_handle_events called _save_cursors inline whenever a batch moved the
cursor. It ends in atomic_json_write (mkstemp + fsync + os.replace), and
on the WebSocket transport _handle_events runs once per inbound EVENT
frame, so every message paid an fsync on the gateway's event loop,
stalling every other adapter and in-flight turn for its duration.
Split the snapshot from the write: the payload is still built on the
loop (_channel_state is loop-owned), and the write goes through
asyncio.to_thread. The snapshot is taken under an asyncio.Lock so a
slower, older write can never land after a newer one and regress the
durable cursor. connect() keeps the synchronous _save_cursors.
(cherry picked from commit 299929741429262f497b86a67940ffb37faa1694)
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
EmailAdapter._sender_accepted runs before any MessageEvent exists and
read only EMAIL_ALLOWED_USERS. Unset, it dropped every sender unless
allow-all was on; set, it dropped everyone not listed. The gateway's
own handling therefore never ran for email:
platforms.email.unauthorized_dm_behavior "pair" (the setup wizard's
"Use DM pairing") and "decline" sent nothing, and a sender admitted by
GATEWAY_ALLOWED_USERS or an approved pairing was dropped. bb304b4914
turned the empty-allowlist branch into drop-all after #50568 had made
"pair" email's explicit opt-in.
The gate now keeps a sender listed by address in EMAIL_ALLOWED_USERS
or GATEWAY_ALLOWED_USERS, a sender the registered gateway
authorization check admits (that is the only reader of the pairing
store), and, under an explicit pair or decline, an unknown sender the
gateway will answer. The default "ignore" still drops unknown senders
before a MessageEvent exists, so the mail-loop guard from fd9c32c0f2
holds.
Three guards keep the wider gate from widening access, and close two
forged-From: paths main already had:
- A sender admitted only so the gateway can answer it (pair or
decline) must authenticate its From:, open access or not: the
pairing code or refusal is mailed back to that address. A granted
sender still needs it short of open access, since a pairing grant
keys on From: just as the allowlist does. Open access follows the
gateway's own order: EMAIL_ALLOW_ALL_USERS wins over a list, while
GATEWAY_ALLOW_ALL_USERS beside a list admits nobody extra, so it no
longer exempts a listed address from From: authentication either
(on main a forged From: of a listed address got through there).
- Open access comes from the gateway's own verdict when a check is
registered. GATEWAY_ALLOW_ALL_USERS beside a GATEWAY_ALLOWED_USERS
list grants a stranger nothing there, so the env flag alone no
longer exempts one from From: authentication (that path mailed a
pairing code to a forged From: on main too).
- A sender whose local part alone matches an allowlist entry is
dropped. The gateway's check also matches an address by its bare
local part (#119446), so without this, GATEWAY_ALLOWED_USERS=alice
(a chat username) would admit or pair alice@<any domain>. The lists
are parsed as the gateway parses them, JSON list literals included,
or '["alice"]' would slip past this guard.
_allowlist_in_effect only served the old condition and is removed.
The scope tests now assert the same scoped reads through
_sender_accepted, with GATEWAY_ALLOWED_USERS covered as well.
Measured end to end with the real GatewayRunner callback wired
(adapter -> gateway ingress):
- pair, decline, GATEWAY_ALLOWED_USERS and an approved pairing each
went from 0 events reaching the gateway to 1. pair mails a pairing
code, decline mails one refusal.
- An unauthenticated From: in pair mode, for a paired address or for a
GATEWAY_ALLOWED_USERS address still reaches nothing.
- A bare GATEWAY_ALLOWED_USERS=stranger entry lets nothing from
stranger@<domain> through, under ignore or pair. Without the
local-part guard that mail reached the gateway in both.
- The same holds for a JSON-literal list, and a pair-mode stranger
with a forged From: under allow-all beside an EMAIL_ or
GATEWAY_ALLOWED_USERS list reaches nothing.
- The default still drops.
Completed update IDs lived only in the adapter's memory. The gateway
reconnect watcher builds a new TelegramAdapter and connects it with
is_reconnect=True, which keeps Telegram's pending queue, and a new PTB
Updater polls from offset 0. Telegram then resends every update whose
acknowledgement (the next getUpdates offset, or the cleanup call in
Updater.stop) never landed, and the fresh adapter admitted them again.
Write completed IDs to telegram_update_receipts_<bot_id>.json in the
adapter's Hermes home and seed admission from it once per bot. Receipts
older than 24h are dropped: the Bot API keeps unconfirmed updates no
longer than that, and it keeps the lookup clear of the random ID restart
Telegram may do after a week without updates. Writes are coalesced and
run off the loop; disconnect waits for the last one.
Refs #68502
Co-authored-by: Joe Githler <5716896+NoTimeforInfinity@users.noreply.github.com>
Follow-up on the two salvaged commits:
- One _sidecar_payload_text() helper decides the /send text for both the adapter
path and _standalone_send (cron / send_message), which had the same raw-markdown
leak on URL-bearing messages and never stripped with PHOTON_MARKDOWN=false.
- BlueBubbles (same iMessage surface) keeps [label](url) targets as bare URLs.
- Tests trimmed to two invariants.
The gateway's tool-progress path emits terminal commands as fenced code
blocks on any adapter whose supports_code_blocks is True. The Photon
adapter set that flag from PHOTON_MARKDOWN, but the sidecar's /send
router (send-format.mjs) silently routes every URL-bearing message
through the plain-text builder — where fences survive as literal backtick
characters — and even the markdown path renders a fence as inline
monospace text, not a block. Net effect: raw fenced terminal commands
(and raw markdown around them) surfaced in iMessage bubbles no matter
what the user's prompt-level style rules said.
Fix the whole class: the adapter never claims code-block support, so the
gateway emits its compact one-line tool preview instead. Prose markdown
passthrough (bold/italic/headings) is unchanged, and no config change is
required — tool_progress can stay at its user's preferred mode.
Tests: new capability + E2E tests pin that the gateway cannot emit a
fence for Photon with real adapter + real display resolution; the old
supports_code_blocks-mirrors-env expectation (which pinned the buggy
behavior) now asserts the flag is always False.
chooseSendFormat() routes markdown containing a raw http(s) URL through
spectrum-ts' text() builder, because the markdown builder's iMessage data
detection 500s on those messages (#73615). text() ships the payload
verbatim, and nothing strips the markers on the way down, so any reply
that mentions a link arrives in iMessage as literal markdown source:
**Release 1.2.0** is out
- **EUR 5** off this month
https://example.com/releases/1.2.0
Every ** is visible in the bubble. Remove the URL from that same reply
and it renders correctly, which is what makes the URL the trigger rather
than the content.
spectrum-ts documents markdown() as degrading to readable plain text on
platforms without native support "instead of surfacing raw ** markers".
Selecting text() ourselves opts out of that guarantee, so the adapter has
to honour it instead.
Strip in _sidecar_send() when the payload is markdown and the sidecar will
downgrade it. The format key is deliberately preserved: the sidecar owns
the builder choice (test_rich_links.py pins that contract), and an older
sidecar without chooseSendFormat must keep rendering natively.
_send_plain_fallback() selects the text builder explicitly via
markdown=False and had the same leak, so it strips too.
Reuses the shared strip_markdown() helper rather than adding a second
implementation, so the PHOTON_MARKDOWN=false path and this path produce
identical output and both inherit any future fix to that helper.
Diagnosis previously reported in #85733, which was closed unmerged.
Stripping must not take the URL with it, though. The shared helper
collapsed [label](url) to label alone, which on this path is worse than
raw markdown: iMessage auto-links bare URLs and nothing else, so the
reply arrives with a description and no way to reach the link.
Open the itinerary on Google Flights <- URL gone entirely
So strip_markdown() takes keep_link_targets, which rewrites
[label](https://url) as "label\nurl" (own line, because these URLs are
often long) and leaves non-http targets such as mailto: or relative
paths label-only, since iMessage won't linkify those either. The default
is unchanged, so the SMS, IRC, Feishu and QQ callers keep dropping the
target as before.
plugins/platforms/line/adapter.py already carries a private
strip_markdown_preserving_urls() for exactly this reason ("LINE
auto-links bare URLs only"). This moves the behaviour behind the shared
helper instead, so Photon's three plain-text paths -- the downgrade,
_send_plain_fallback(), and PHOTON_MARKDOWN=false -- all agree.
gateway/platforms/bluebubbles.py is the same iMessage surface with the
same loss; left alone here to keep this change to one platform.
The Photon hint told the model "Markdown is rendered (bold, italics, lists,
code)", so replies came back with headers, code fences and backticks. That
is wrong on the real delivery path: the sidecar sends any message containing
a URL as raw text (every *, #, ``` and | shows literally), and even on the
markdown path spectrum-ts flattens headings to bold, tables to "a | b"
rows, and turns code into Unicode math-monospace glyphs that break when
copied. The hint also claimed attachments are metadata-only, which stopped
being true when native media send landed.
Both iMessage hints (Photon plugin, BlueBubbles built-in) now ask for a
texting register: short, answer first, no headers/tables/fences/backticks,
commands on their own plain line so they copy, bare URLs. BlueBubbles also
notes that strip_markdown drops the URL of [text](url) links.
A plugin that finished loading after an adapter connected never got its platform
handlers (slash commands, button callbacks, inbound transforms) registered until a
gateway restart, silently. Three pieces, one seam shared by every surface:
1. Discovery listener: PluginManager.on_plugin_loaded(cb) fires from INSIDE
discover_and_load for the plugins a sweep newly loaded (diff of the loaded set),
with a per-plugin activation summary (hermes_cli/plugins_activation.py):
activated_now {gateway_commands, gateway_transforms, hooks, callbacks} vs
deferred {tools, prompt, mcp_servers}. Every mid-run load path now performs a real
discover_plugins(force=True): CLI install/enable (via the gateway), Desktop/TUI
plugins.manage install/toggle/update, dashboard REST install, tool-triggered
force re-discovery, the new `reload-plugins` control-socket verb. A non-forced
discover_plugins() short-circuits on _discovered, which is why reload.mcp after
a mid-run install used to reload the OLD server set.
2. Idempotent re-wire: BasePlatformAdapter.rewire_plugin_handlers() runs only
factories not yet wired on the live native client (keyed (plugin, qualname);
a force reload hands back new function objects). Telegram hoists late handlers
ahead of core's catch-all filters.COMMAND / CallbackQueryHandler (PTB dispatches
the first match per group) and re-wires on the transient-init rebuild; Slack
dedupes register_slack_action_handler per AsyncApp. The gateway runner
subscribes per served profile and re-wires on the loop.
3. Scope limit + honest messaging: handlers only. Tools/prompt stay deferred to
the next session (prompt-cache invariant), MCP servers to mcp.reload; the CLI
hint and plugins.manage results (activation, gateway_reloaded,
restart_required only when no gateway answered) say exactly that.
_auto_create_thread's dedup pre-seed (is_duplicate(str(thread.id))) stops
the echo MESSAGE_CREATE Discord fires for the starter (id == thread.id)
from re-running the request. This PR turned the tracker persist into
`await self._threads.mark_async(thread_id)` and placed it BEFORE the
pre-seed; to_thread always suspends, so the echo's handler could reach
_discord_message_admission -> is_duplicate during the os.replace window,
claim the id first, and rerun the starter. On main both statements were
sync, so no window existed.
Move the pre-seed (with its comment) directly after
`thread_id = str(thread.id)`; the await now follows it.
Other mark_async call sites checked (discord :4717 slash create-thread,
discord :6140 pre-dispatch, matrix :1438 create_thread, matrix :2083
inbound): nothing after those awaits relies on state a concurrent event
could claim first, so no further reorder.
PROOF: tests/gateway/test_discord_double_dispatch.py::
test_thread_starter_duplicate_dropped now installs a recording mark_async
that asserts thread.id is already in _dedup._seen when it is awaited.
Red with the pre-swap order (1 failed), green after; ruff clean,
check-windows-footguns clean, real import of the adapter from the
worktree ok, scripts/run_tests.sh on the PR's test files green.
`record`/`record_media` do a read-modify-write of the JSON index ending in
`atomic_json_write`/`os.replace`, and every coroutine on the WhatsApp Cloud,
WhatsApp bridge and Telegram send/inbound paths called them inline on the
loop thread, stalling the whole gateway for a filesystem write nothing
awaits. Add `record_async`/`record_media_async` as thin
`asyncio.to_thread` wrappers (same precedent as
gateway/channel_directory.py) and await them from the seven coroutine call
sites; the sync functions stay for sync callers. The write completes before
the await returns, so lookup-after-send behaviour is unchanged.
Salvaged from #118883 by @Kyzcreig; re-shaped onto asyncio.to_thread (no
writer thread, no in-memory shadow, no queue), plus the missed
plugins/platforms/whatsapp/adapter.py record_media site.
`ThreadParticipationTracker._save` (gateway/platforms/helpers.py) ends in
`atomic_json_write` -> `os.replace`, whose duration is unbounded under
filesystem pressure. All five call sites are coroutines on the inbound-message
or slash-command path:
plugins/platforms/matrix/adapter.py _resolve_message_context
create_handoff_thread
plugins/platforms/discord/adapter.py _handle_message (x2)
_handle_thread_create_slash
So that rename was paid inline on the running loop, stalling every other
adapter's polling, every in-flight turn and every heartbeat for as long as it
took.
The fix is at the CHOKE POINT rather than at five call sites:
- `mark_async` does the in-memory insert synchronously and offloads only the
persist via `asyncio.to_thread`. The insert must stay synchronous because
both adapters gate on `thread_id in self._threads` immediately after
marking; deferring it would make mention-gating depend on executor
availability.
- all five coroutine call sites now await it.
- `mark` keeps its exact synchronous contract for the non-loop callers.
- an RLock is added in the SAME commit that introduces the concurrency: the
event loop used to serialize every caller by accident, and
`atomic_json_write` makes each write atomic without making
check/insert/trim/write atomic. Without it two concurrent marks lose one.
Enforcement is an AST class sweep, not an inventory: no `async def` under
gateway/ or plugins/ may call `<x>._threads.mark(...)`, and every
`mark_async(...)` must be awaited -- an un-awaited one never runs at all, so
the thread is neither persisted nor recorded in memory and mention-gating
re-prompts forever in a thread the bot already joined. A new adapter fails the
gate without anyone remembering a list.
tests/gateway/test_discord_thread_slash_expired_defer.py stubbed the tracker
with `SimpleNamespace(mark=...)`; it now uses the real tracker against a
tmp_path, so the test cannot rot silently the next time this surface moves, and
it additionally asserts the thread really was recorded.
Verified on this exact head, PYTHONPATH pinned to the worktree:
tests/gateway/test_thread_tracker_mark_off_loop.py 7 passed
expired-defer + admission-exemption + off-loop 10 passed
-k "discord or matrix or thread" over tests/gateway 1136 passed, 13 failed
The 13 failures are INHERITED: a clean worktree at upstream/main 75e9567ca7
with none of these changes fails the identical 13.
Every guard is gate-proven -- reverting the RLock, the to_thread, one call
site, one `await`, the sync insert, or the dedupe short-circuit each fails its
own test and only its own.
(cherry picked from commit 6fc8037d292ec051fe76ad213903b2ee5534080a)
`gateway.sticker_cache._save_cache` ends in `atomic_json_write` -> `os.replace`,
whose duration is unbounded under filesystem pressure. Its only production
caller is Telegram's `_handle_sticker` -- an inbound-message coroutine -- so
every sticker that missed the cache paid that rename inline on the event loop,
stalling every other adapter and every in-flight turn in the process for its
duration.
Adds `cache_sticker_description_async`, which dispatches the existing sync form
via `asyncio.to_thread`; the Telegram call site awaits it. The sync form keeps
its exact contract and is what the wrapper dispatches to, so it is unchanged for
non-loop callers.
`cache_sticker_description` is a read-modify-write (`_load_cache` -> merge ->
`_save_cache`). `atomic_json_write` makes each WRITE atomic, not the TRIPLE.
While the call was inline the event loop serialized every caller and the race
could not be observed; moving the write to a worker thread introduces real
concurrency, so `_CACHE_LOCK` is added in this same commit rather than deferred.
Tests use no wall-clock thresholds. The liveness witness is ORDERING: the rename
is held open on a barrier released only by a background timer, and the sibling
task must have ticked BEFORE that release. Verified RED first:
- revert `asyncio.to_thread` to an inline call -> 2 failed
("the cache write ran on the event-loop thread")
- revert `_CACHE_LOCK` -> the concurrency test fails on the DURABLE FILE:
"lost an entry ... keys = ['uid_a']"
`test_gate_proof_the_sync_form_does_block_the_loop` drives the identical barrier
through the sync form and asserts the loop DOES starve, so the liveness
assertion cannot pass vacuously.
GREEN: sticker off-loop + existing sticker cache tests -> 10 passed;
`tests/gateway -k "sticker or telegram"` -> 937 passed, 4 skipped; Ruff clean.
(cherry picked from commit 9752a9b1aba7536181d081cad9c1c32917a89680)
Declare `_TERMINAL_HEALTH_REASONS = frozenset({"socket_closed",
"client_closed"})` beside `_read_websocket_health` (the producer of those
literals at adapter.py:1705/:1707/:1715) and use it at the escalation
site in `_liveness_loop` (was `reason in ("socket_closed",
"client_closed")` at :1791).
WHY: the first-strike classification was a magic-string match against
literals produced 80 lines away with no shared definition. A rename in
the producer, or a new hard-closed reason, would silently demote the
escalation back to the threshold path with nothing failing. One named
set next to the producer makes the coupling visible. The
`tuple[bool, str]` return shape is unchanged (existing tests monkeypatch
the sampler with 2-tuple lambdas).
Proof: mutating the set to {"socket_closed"} fails
test_closed_transport_first_strike_forces_reconnect[client_closed];
restored, the liveness file is green.
Collapse the INFO/WARNING level switch on the probe-exit log
(plugins/platforms/discord/adapter.py:1755-1766) to one unconditional
logger.info and delete `expected_exit`. Drop
test_unexpected_probe_exit_logs_warning, which was the only caller that
could reach the WARNING half.
WHY: the WARNING branch is unreachable in production. It needs
`_running=True and not _disconnecting and _client is None` at the guard,
and `_client` has exactly two production setters to None:
- adapter.py:1269 inside connect(): synchronously followed by
`self._client = commands.Bot(...)` at :1271 with no await between, so
the probe coroutine can never observe the None.
- adapter.py:1941 inside disconnect(): runs after `_disconnecting = True`
(:1915) and after `await self._cancel_liveness_task()` (:1917), so the
probe is already cancelled (exits via CancelledError at :1752) and the
flag would read as an expected exit anyway.
No other `_client = None` in plugins/platforms/discord/, gateway/platforms/
base.py or gateway/run.py. `_running=False` and `_disconnecting=True` come
only from teardown paths, all "expected". The deleted test pinned this by
poking `adapter._client = None` directly — a state the code never produces.
This drops the level switch borrowed from #118504 but keeps its intent:
the probe exit is always logged with its full state (#118487, the probe
must never disappear silently).
The six-line comment above the terminal-reason check
(plugins/platforms/discord/adapter.py:1789-1794) restated the resume-swap
mechanism already documented in the test docstring and the issue. Keep the
one sentence that carries the intent (closed transport = confirmed death,
soft signals keep the threshold) and the #118487 pointer. No code change.
_read_websocket_health reports client_closed when Bot.is_closed() is
true. That is the same transport-dead state as socket_closed, yet it
stayed on the two-strike confirmation path and could show the same
1/2-then-silent-reset pattern if the bot task's done callback were ever
suppressed. Escalate both terminal reasons on the first strike.
The guard exit in _liveness_loop fires for two very different reasons:
ordinary teardown (adapter stopped or disconnecting), and a still-running
adapter whose client vanished. The first is noise at INFO; the second
means the gateway keeps running with no watchdog (#118487) and must stand
out in an incident log. Pick the level from the exit cause instead of
logging both at INFO.
Co-authored-by: Konstantin Khlopkov <47825603+kokhlo@users.noreply.github.com>
