The r3 fold scoped only the dmarc verdict to its clause. SPF and DKIM
still came from a whole-string, last-match-wins regex scan, so the same
quoted/comment smuggle closed for dmarc still authenticated a spoofed
From (GHSA-rxqh-5572-8m77), e.g.
spf=fail smtp.mailfrom="x spf=pass smtp.mailfrom=example.com "@evil.test
spf=fail (spf=pass) smtp.mailfrom=a@example.com
spf=fail smtp.mailfrom=a.spf=pass@example.com
dkim=pass header.d=evil.test header.i="x header.d=example.com y"@evil.test
Every verdict now comes from the leading method=result token of its
own clause (from _ar_clauses, comments dropped), and its domains only
from that clause. Properties are read by a token scanner that consumes
quoted-strings and other key=value tokens whole, so quoted contents are
never read as properties while a quoted value still is
(header.from="example.com"). SPF fails closed on more than one spf
clause; DKIM accepts any single dkim=pass clause whose own header.d
aligns (multi-signature mail is normal), never mixing clauses. The
whole-string methods/props (methods["dmarc"] was dead) are gone.
Also: a stray ')' at depth 0 is now unbalanced (it split header.from
out of the dmarc clause); a From with more than 64 '(' takes the silent
empty-sender drop instead of a parseaddr RecursionError logged as an
error; the cap test asserts the cap directly instead of wall-clock
timing; the empty-From drop assertion moves next to the other
_extract_email_address rejects; and the untested >1-dmarc, unbalanced
and backslash-escape rules get reject strings.
Round-3 review of the #124322 salvage:
- stdlib parseaddr is pure Python and superlinear on hostile input: a
100KB `From: <a@a@...` held the GIL ~1s per message (main ~0), and any
remote sender reaches it before auth. Refuse values over _MAX_FROM_LEN
(2048; RFC 5322 lines cap at 998) right after unfolding so they take the
existing empty-sender drop. The timing assertion now times
_extract_email_address itself instead of a private regex.
- The dmarc clause split ignored quoted-strings and stripped comments one
level only, so an attacker-controlled SPF-passing envelope sender like
smtp.mailfrom="x;dmarc=pass header.from=example.com x"@evil.test planted
a fake dmarc=pass ahead of the real dmarc=fail and authenticated
admin@example.com. Split clauses with a quote- and nested-comment-aware
scanner (comments dropped, quoted values kept so header.from="x" is still
read and unquoted), and fail closed on an unbalanced value or when more
than one clause starts with dmarc=. Tests pin the smuggled clause,
reason="a;b", the nested comment, a quoted aligned header.from, and that
every header.from in the dmarc clause must align (all->any goes red).
- `Doe, John (CEO) <j@x>` was dropped because the fallback refused any
paren. Strip (comments) from the display part first; `attacker@evil.test
(c) <victim>` stays rejected by the '@' rule.
- Results without '@' (`John` -> `john`, `a\"b <v@x>` -> `a\`) were
dispatched as sender ids; return "" so they take the drop path.
The r1 malformed-From fallback regex ([^<>\s]+@[^<>\s]+) backtracked
catastrophically on `From: <a@a@a@...`: 32KB held the GIL ~23s, and any
remote sender reaches it before auth. Capture <([^<>\s]+)> instead and
check the '@' in Python (same accepted set, 100KB in ~3ms).
The fallback also mapped multi-mailbox / group / comment-prefixed From
values (`attacker@evil.test, <victim@x>`, `Grp: a@evil; <victim@x>`,
`attacker@evil.test (c) <victim@x>`) to the bracketed victim, which a
dmarc=pass without header.from then authenticated. Only fall back when the
display part has no ; : ( ) and no '@' unless it is exactly the bracketed
address; `Doe, John <j@x>` and `j@x <j@x>` still resolve. The rule now lives
only in the docstring (the old inline comment was wrong).
dmarc: strip (comments) before splitting clauses, and take the verdict and
header.from from the same first clause that starts with dmarc=, so
`dmarc=pass (p=none; sp=none) header.from=evil.test`, a later dmarc=pass
clause after dmarc=fail, and duplicate misaligned header.from are rejected,
while `arc=pass (dmarc=fail ...); dmarc=pass header.from=<ours>` passes.
Property clean-up is shared via _auth_props.
The empty-sender drop now runs right after address parsing and gets an
assertion (it was untested). Tests extend existing ones (no new tests).
Strict parseaddr returns '' for common RFC-invalid From values that the
old regex resolved: an unquoted address as display name
(john@example.com <john@example.com>), an unquoted comma (Doe, John
<j@example.com>), or a@x.test <b@y.test>. Allowlisted senders using such
clients were silently dropped, and under open access every one of them
shared an empty chat_id. Fall back to the bracketed address only for the
unambiguous shape (no quotes, exactly one <...> pair), so the quoted
display-name spoof still resolves to the attacker. _parse_fetched_message
now drops messages whose From yields no address instead of dispatching
an empty identity.
The dmarc header.from alignment check read header.from from props merged
across all clauses, so a later dkim clause's header.from could override
the dmarc clause's value. Read it from the dmarc clause only, and pin the
misaligned dmarc=pass rejection in the existing dmarc test (previously
no test covered it). Also drop a redundant _domain_of ternary, trim the
_extract_email_address docstring and a duplicate test assertion.
_verify_sender_authentication accepted any dmarc=pass verdict, even one
issued for a different domain than the From we parsed. That is what let
the quoted-display-name spoof ride on the attacker's own truthful DMARC
pass. When the trusted Authentication-Results names header.from, require
it to align with the From domain; otherwise fall through to the aligned
SPF/DKIM checks. Defense in depth for the #124322 parser fix.
_extract_email_address took the FIRST <...> pair, so
From: "Victim <victim@example.com>" <attacker@evil.test> resolved to the
victim. The attacker's own domain passes DMARC truthfully, so the
allowlist (EMAIL_ALLOWED_USERS), pairing and session identity were all
evaluated against an address the sender does not control.
Use email.utils.parseaddr, which keeps the quoted text as the display
name and returns the real addr-spec. RFC 5322 folding is unfolded first
so a folded quoted display name is not mistaken for the mailbox.
Salvages #124322.
Runtime identity resolved through hermes_cli.__version__ (a static 0.0.0
on source installs, rewritten by release stamping) leaked v0.0.0 into
About, /api/health, User-Agents, and plugin compat, and source updates
showed "couldn't reach update server" because identity and channel
authority disagreed with the checkout.
Now: get_version_info() resolves install stamp -> live git -> unknown,
never pyproject metadata, never a package constant. Source checkouts
derive identity from their reachable release tag; the completion tail of
every successful install/update/historical takeover atomically rewrites
install-stamp.json with that identity; a stale source stamp whose commit
no longer matches HEAD defers to live git. ACP/TUI use derived_version
for display and base_version for protocol fields; all ~44 runtime
__version__ consumers migrated; hermes_cli.__version__ and generated
_version.py are gone; release stamping only touches the native manifests
external builders consume (nix/tauri/cargo) and passes release identity
straight into write_install_stamp.py; pyproject.toml stays inert 0.0.0.
Desktop no longer synthesizes a competing install-stamp.json: the
checkout owns its stamp, and desktop-bootstrap classification keys on
the bootstrap-complete marker. verify-bootstrap-version-stamp.py now
cross-checks the checkout's stamp (baseVersion + commit == HEAD).
Validation: 31-file focused suite green (version identity, stamping,
adoption, providers, gateway, acp/tui runtime identity, api server via
extras env, release graph); desktop tsc + 25 vitest green; real-repo
probe: base=unknown derived=git.0635606.dirty source=git on this
checkout; clean-env imports resolve entirely from this tree; windows
footgun + compat-pointer scans clean.
EmailAdapter._sender_accepted runs before any MessageEvent exists and
read only EMAIL_ALLOWED_USERS. Unset, it dropped every sender unless
allow-all was on; set, it dropped everyone not listed. The gateway's
own handling therefore never ran for email:
platforms.email.unauthorized_dm_behavior "pair" (the setup wizard's
"Use DM pairing") and "decline" sent nothing, and a sender admitted by
GATEWAY_ALLOWED_USERS or an approved pairing was dropped. bb304b4914
turned the empty-allowlist branch into drop-all after #50568 had made
"pair" email's explicit opt-in.
The gate now keeps a sender listed by address in EMAIL_ALLOWED_USERS
or GATEWAY_ALLOWED_USERS, a sender the registered gateway
authorization check admits (that is the only reader of the pairing
store), and, under an explicit pair or decline, an unknown sender the
gateway will answer. The default "ignore" still drops unknown senders
before a MessageEvent exists, so the mail-loop guard from fd9c32c0f2
holds.
Three guards keep the wider gate from widening access, and close two
forged-From: paths main already had:
- A sender admitted only so the gateway can answer it (pair or
decline) must authenticate its From:, open access or not: the
pairing code or refusal is mailed back to that address. A granted
sender still needs it short of open access, since a pairing grant
keys on From: just as the allowlist does. Open access follows the
gateway's own order: EMAIL_ALLOW_ALL_USERS wins over a list, while
GATEWAY_ALLOW_ALL_USERS beside a list admits nobody extra, so it no
longer exempts a listed address from From: authentication either
(on main a forged From: of a listed address got through there).
- Open access comes from the gateway's own verdict when a check is
registered. GATEWAY_ALLOW_ALL_USERS beside a GATEWAY_ALLOWED_USERS
list grants a stranger nothing there, so the env flag alone no
longer exempts one from From: authentication (that path mailed a
pairing code to a forged From: on main too).
- A sender whose local part alone matches an allowlist entry is
dropped. The gateway's check also matches an address by its bare
local part (#119446), so without this, GATEWAY_ALLOWED_USERS=alice
(a chat username) would admit or pair alice@<any domain>. The lists
are parsed as the gateway parses them, JSON list literals included,
or '["alice"]' would slip past this guard.
_allowlist_in_effect only served the old condition and is removed.
The scope tests now assert the same scoped reads through
_sender_accepted, with GATEWAY_ALLOWED_USERS covered as well.
Measured end to end with the real GatewayRunner callback wired
(adapter -> gateway ingress):
- pair, decline, GATEWAY_ALLOWED_USERS and an approved pairing each
went from 0 events reaching the gateway to 1. pair mails a pairing
code, decline mails one refusal.
- An unauthenticated From: in pair mode, for a paired address or for a
GATEWAY_ALLOWED_USERS address still reaches nothing.
- A bare GATEWAY_ALLOWED_USERS=stranger entry lets nothing from
stranger@<domain> through, under ignore or pair. Without the
local-part guard that mail reached the gateway in both.
- The same holds for a JSON-literal list, and a pair-mode stranger
with a forged From: under allow-all beside an EMAIL_ or
GATEWAY_ALLOWED_USERS list reaches nothing.
- The default still drops.
Scoped secrets — `gateway.platforms._shared.get_scoped_secret` is the single implementation of
the "scope authoritative, unscoped default-profile falls back to os.environ" read:
- plugins/platforms/buzz/adapter.py::_get_scoped_secret (113 LOC, ~100 of which were one
docstring paragraph pasted 16x) -> 3-line forwarder over the canonical with
`external_fallback=True`. Its one genuine extra rung (one-shot profile-scope build so a
Bitwarden-managed key is visible to the startup gate, #95216) moves into `_shared` as that
keyword plus `_unscoped_profile_secrets`.
- weixin::_wx_secret, matrix::_startup_env_secret, the inline try/except copies in slack
(SLACK_APP_TOKEN) and telegram (TELEGRAM_WEBHOOK_SECRET/_URL) -> canonical.
- The "extra-first, then scoped env" reader written 11x under 6 names (weixin._extra_or_env,
bluebubbles/ntfy/photon/wecom `_setting`, dingtalk `_extra_get`, mattermost `_extra_or_env`,
slack `_extra_or_env_flag/_channel_set`, feishu closures) -> `_shared.extra_or_secret`.
- `authz_mixin._platform_gate_env` -> `_shared.platform_gate_env`; discord/telegram drop their
`_scoped_gate_env` twins; run.py / run_config_loaders.py / slack import it directly.
Boilerplate — three table-driven helpers in `_shared` replace the pasted docs template:
- `seed_extra_from_env(spec, home_env=)` replaces 8 `_env_enablement` bodies (buzz, google_chat,
irc, line, ntfy, photon, simplex, teams; raft is a one-liner and untouched).
- `apply_yaml_bridge(cfg, spec)` replaces 7 `_apply_yaml_config` bodies (buzz, dingtalk, feishu,
matrix, mattermost, slack, whatsapp); discord/telegram keep bespoke bridges (alias keys,
nested `platforms.*.extra`, generic-key exclusions). buzz and mattermost previously bypassed
`yaml_env_setter` with hand-rolled `os.environ` writes.
- `env_is_connected(*vars)` replaces 5 identical `_is_connected` (discord, homeassistant,
mattermost, slack, sms).
- 8 identity `_build_adapter` wrappers deleted; `adapter_factory=<Class>`.
Behavior change:
- buzz `_apply_yaml_config` returned None, so under multiplex a secondary Buzz profile got
neither env (correctly skipped) nor `extra` for relay_url/channels/allow_all_users/...; it now
seeds `extra` like every other hook. It also wrote reply_in_thread/reply_to_mode to the process
env even inside a secondary profile's scope (first-writer-wins leak, #80099 class); it no longer
does. BUZZ_POLL_INTERVAL is bridged through the same table.
- `home_channel.name` default when `<X>_HOME_CHANNEL_NAME` is unset is now the literal "Home" for
all plugins (irc/ntfy/buzz used the chat id; simplex/teams/photon/google_chat already used
"Home", as do the built-in platforms in gateway/config_env.py).
- weixin's non-secret tunables (send_chunk_*, rate_limit_circuit_*) now read through the scoped
reader instead of raw os.getenv — a secondary profile no longer inherits the default's values.
- `extra_or_secret` treats a blank string in extra as unset (falls to env) and an explicit False
as a real value, the strictest of the merged copies.
- slack `reaction_trigger_target` bridges via str(); `reaction_triggers` comma-joins any list-ish
value (was list/tuple/set only) — same env text for every real YAML shape.
Docs: website/docs/developer-guide/adding-platform-adapters.md (the template the copies were
pasted from) and gateway/platforms/ADDING_A_PLATFORM.md now show the helpers and the scoped
reader; gateway/AGENTS.md points at the one implementation.
Tests: tests/gateway/test_shared_platform_boilerplate.py — every plugin `_env_enablement`
reads only through the scoped getter (parametrized over the 8 plugins, spy on the seam, raw
`os.getenv`/`get_env_value` asserted untouched); buzz bridge seeds `extra` for a secondary
profile and still bridges env for the default; one home-name rule; extra_or_secret contract;
external_fallback rung. Existing tests repointed: tests/agent/test_secret_scope_tier1_migration.py,
tests/plugins/platforms/buzz/test_buzz_unscoped_requirement_gate.py.
Six adapters defined their own `_cancel_task` and nine more inlined the same
cancel + suppress(CancelledError) + await block; five kept a hand-rolled TTL-dict
`_is_duplicate` next to the existing `helpers.MessageDeduplicator`; three carried a
`_bounded_put`. Each copy fixed the same bugs on its own schedule (self-cancel deadlock,
done-task re-await, eviction under load).
- `helpers.cancel_task`: None/done no-op, never awaits the current task, swallows the
task's own exception at teardown. Replaces qqbot/signal/yuanbao/buzz/photon/simplex
definitions and the inline copies in weixin, discord, email, irc, line, mattermost,
whatsapp and telegram.
- `helpers.MessageDeduplicator` replaces `_is_duplicate` in qqbot, ntfy, photon,
wecom_callback and LINE's `_MessageDeduplicator`; every site keeps its own
max_size/TTL (qqbot and ntfy 1000/300s, photon 4000/48h, wecom_callback 2000/300s,
LINE 1000/no TTL).
- `helpers.bounded_put` replaces photon/wecom/whatsapp_cloud copies; a re-put now
refreshes the key to the newest slot at every site.
- telegram gmail-triage scripts resolve under `get_hermes_home()` instead of a hard
`~/.hermes`, so profiles with HERMES_HOME set find them.
Not changed: `get_chat_info` stays `@abstractmethod` because
tests/gateway/test_relay_capability_surface.py locks the abstract set to exactly
{connect, disconnect, send, get_chat_info} as a cross-repo contract, so the ~17 no-op
overrides remain.
Behavior change: whatsapp_cloud `_bounded_put` was a pure FIFO (no refresh on re-put);
it now refreshes like the other two sites. Task cancellation at the migrated sites
swallows a task's terminal exception where a few copies previously only suppressed
CancelledError (all are shutdown/disconnect paths).
20 `plugins/platforms/*/adapter.py::_standalone_send` paths (the out-of-process cron /
send_message delivery) built `{"error": f"... {e}"}` by hand — 83 literals. The exception text
of an httpx/aiohttp failure can carry the Authorization header, a signed URL or a response body
with the token in it, and that string became the tool result the model reads. Only sms went
through the redacting `tools.send_message_senders._error`; discord kept a private regex that
only knew `Authorization: Bot`.
`gateway.platforms._shared.send_error(message)` wraps that helper (agent.redact +
URL-secret scrub) and every standalone literal now goes through it, including the three
envelopes that carry extra keys (discord warnings, photon error_class/retryable, whatsapp's
`(None, err)` tuple). The sms and discord local wrappers are deleted. Telegram already
delegated to the core sender and is untouched.
Behavior change (security): vendor exception text in standalone-send failures is redacted
before reaching the model.
send_document() accepted reply_to but never passed it down, so attachments
always threaded from the cached per-address context (or not at all) even
when the caller named the message to reply to. The plain-text path
(_send_email) already honored it; the attachment path now does too.
The metadata half of #10131 (send_image rejecting metadata=) was already
fixed on main by the adapter parity pass. Diagnosis from #10131 and the
explicit-reply_to-wins shape from PR #10321 (which targeted the
pre-plugin path).
Fixes#10131
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Same bug class as the Slack/Feishu picks: buzz, dingtalk, email, google_chat,
line, ntfy, photon, sms, teams, wecom and whatsapp already had the platform
message id on the MessageEvent but built the SessionSource without it, so
source.message_id consumers (reply anchor in run.py, /sethome synthetic-thread
check, relay _event_ids fallback, shutdown notice anchor) saw None. Only sites
where the id variable was already in scope are widened.
Under gateway.multiplex_profiles, os.environ holds the DEFAULT profile's .env. Several
adapter-owned authorization gates still read GATEWAY_ALLOW_ALL_USERS, GATEWAY_ALLOWED_USERS
or their platform allowlist/allow-all raw from os.environ, so the default profile opting
into open access opened every secondary email/QQ/WhatsApp/Matrix/Teams/Slack/LINE/DingTalk
bot to any sender (email additionally skipped From: authentication), the default's Matrix
allowlist decided who may approve tool calls on a secondary bot, and a secondary that
opted in only in its own .env was silently deny-all.
Every such read now goes through the adapter's existing module-local scoped reader
(gateway.platforms._shared.get_scoped_secret / matrix _startup_env_secret): profile
scope first, scoped miss = default, never os.environ; the unscoped default-profile and
single-profile paths keep the environ read, where it IS the profile's own value.
Sites: email _allow_all_senders/_allowlist_in_effect; qqbot _open_dm_opted_in;
whatsapp_common _open_dm_opted_in/_live_dm_allow_from; teams _card_action_denied;
matrix _is_authorized_user, MATRIX_ALLOWED_USERS, MATRIX_IGNORE_USER_PATTERNS,
_extra_csv_set (allowed/free-response rooms); slack _slack_allow_bots/_slack_api_human_users;
line _truthy_env/allowlist (allow-all, user/group/room allowlists); dingtalk _extra_get
(allowed_users/chats, free-response chats, require_mention).
Live repro (temp HERMES_HOME, multiplex on, default env GATEWAY_ALLOW_ALL_USERS=true,
secondary scope without opt-in): EmailAdapter._allow_all_senders() True -> False,
QQAdapter._open_dm_opted_in() True -> False, Matrix _is_authorized_user('@stranger')
True -> False, Teams card action allowed -> denied.
Co-authored-by: Drexuxux <drexux0@gmail.com>
Co-authored-by: MoonsvnLyn <FirmamentalSpring@users.noreply.github.com>
Co-authored-by: svector-anu <anuoluwakolapo94@gmail.com>
Co-authored-by: babatorik <durgun.ismail@gmail.com>
Co-authored-by: salch-cred <salch-cred@users.noreply.github.com>
#106167 widened the base contract to SendResult but left six native-batch
overrides (Discord, Email, Matrix, Mattermost, Slack, Telegram) returning
None, which (a) meant a media-only reply on those platforms still reported
FAILURE because _record_delivery(None) records nothing, and (b) produced new
`ty` invalid-method-override diagnostics against the widened base (#106192).
Make the contract honest instead of annotating it Optional: each override
now rolls its batches (and any per-image fallback) into one SendResult, so
the turn-outcome accounting works on every platform, not just Signal and
the base loop. The "legacy overrides return None" comment in
_send_image_batch goes away with the legacy.
ty on the 8 touched files: origin/main 241 diagnostics / 20 override,
this branch 241 / 20 — byte-identical diagnostic set; the intermediate
`-> SendResult` head without this commit had 246 / 25.
Refs #106192
Breaks the two import cycles that forced Protocol stand-ins in the F821 sweep, so the two
sites now name the real types.
gateway/platforms/event.py (new leaf): MessageType, ProcessingOutcome, MessageEvent moved
out of base.py verbatim. Their only dependency is gateway.session.SessionSource; base.py
imported helpers.py at module level, so helpers could not name MessageEvent. Now
TextBatchAggregator is typed by the real MessageEvent. 249 importers repointed
(`from gateway.platforms.base import` -> `.event`, preserving each import's layout);
gateway.platforms.__init__ re-exports from .event. The three revert-scheduled PLUGIN-COMPAT
pointers that named these symbols (gateway.slash_commands → MessageType, dingtalk → MessageType,
photon → ProcessingOutcome) and their COMPAT_MANIFEST rows now target gateway.platforms.event.
Docs updated: ADDING_A_PLATFORM.md, adding-platform-adapters.md (en + zh-Hans).
tools/mcp_tool_sampling.py: ElicitationHandler no longer holds a back-reference to its
MCPServerTask (mcp_tool imports sampling, so the task type cannot be named there). It only
ever read owner._pending_call_context, so it takes `call_context: Callable[[], Context | None]`
and MCPServerTask passes `lambda: self._pending_call_context`. The consent call is one
`functools.partial`, run directly or inside the captured Context.
ty on the 11 touched production files vs origin/main: 0 new diagnostics, 14 resolved.
(The one `source: SessionSource = None` diagnostic moves with the class; typing it Optional
exposes ~60 unguarded call sites — separate follow-up.)
Tests: tests/gateway + tests/plugins + tests/tools + touched files, 18,235 passed; the 31
failures reproduce identically on origin/main (macOS /private/tmp, systemd socket,
long-path fixtures, live-service tests).
Two shape fixes from the review of #92979, no behavior change:
- Read imap.capabilities directly instead of the getattr(...) or ()
fallback — a real imaplib.IMAP4 always sets the attribute in
_connect() (raising if the server sends no CAPABILITY), so the
default branch only existed for a MagicMock(spec=["xatom"]) that no
production path produces. Drop the test that pinned it.
- Trim tests to the invariant bar: the four mock tests duplicated what
test_email_imap_id_protocol.py proves on the wire (real imaplib, real
socket, command order), and the phase parametrize (startup vs poll)
exercised the same single _send_imap_id call site twice. Keep the
three id_mode cases through the poll path.
42 tests pass; mutation check: guard removed -> the absent-ID case
fails with the #39856 SELECT error, restored -> green.
_send_imap_id() sent RFC 2971 ID unconditionally after login (a
163/NetEase requirement). Servers without the extension can react badly:
Purelymail answers with an untagged '* BYE Unknown command.' and closes
the connection, which imaplib cannot surface at the call site — the
failure appears one command later as 'IMAP connection failed: command:
SELECT => Unknown command.' and the adapter retries forever.
Gate the ID send on the server's advertised capabilities
(imap.capabilities, populated by imaplib at connect), keeping the
existing exception handler for servers that advertise ID but reject it.
Ports PR #39861 to the current plugin layout, as requested by the
hermes-sweeper review there.
Fixes#39856
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
dingtalk: drop DINGTALK_TYPE_MAPPING/EXT_MAP re-exports. google_chat: card_spec_to_cards_v2 test -> .cards.
matrix: drop module-level MAX_MESSAGE_LENGTH alias (no importers). teams: drop TeamsSummaryWriter re-export
(teams_pipeline/runtime + tests -> summary_writer). wecom: drop WeComStreamExpiredError/STREAM_EXPIRED_ERRCODE/
MAX_INTERMEDIATE_FRAMES re-exports (tests -> .streaming). parallel: drop _get_parallel_client/_get_async_parallel_client
aliases (tests -> _get_sync_client). email: drop stale 'alias' comment (_esecret_int is the only name).
photon: re-remove credential_summary() (shim-only, cb9b7c36f3); its no-leak test now drives print_credential_summary.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
Follow-up to the #99641 salvage:
- One module-level _normalize_security() (ssl/tls/implicit -> tls, starttls,
plain/none -> plain; unknown -> WARNING + secure default) replaces the three
copies of the alias set; _connect_imap/_connect_smtp/_standalone_send all
compare against the canonical value. Unknown modes no longer raise.
- _tls_context(verify, host) is module-level and shared by all sites; when
verification is disabled for a non-loopback host it logs a WARNING.
- _esecret_bool: an unset/empty env var now yields the caller's default
(previously is_truthy_value('') returned False, silently disabling TLS
verification whenever EMAIL_*_TLS_VERIFY was unset).
- Documented surface is platforms.email.extra.{imap,smtp}_security and
{imap,smtp}_tls_verify in config.yaml; env vars remain an internal bridge
and are NOT added to plugin.yaml (optional_env feeds hermes setup prompts).
- Docs: Proton Mail Bridge / local relays recipe in user-guide/messaging/email.md.
- Tests: starttls builds IMAP4 then .starttls(); unknown mode falls back to
tls/starttls with verification still on.
Adds EMAIL_IMAP_SECURITY / EMAIL_SMTP_SECURITY and EMAIL_IMAP_TLS_VERIFY /
EMAIL_SMTP_TLS_VERIFY (env or platforms.email.extra.*) so the adapter can
talk to local relays such as Proton Mail Bridge (IMAP 1143 / SMTP 1025 with
STARTTLS and a self-signed certificate) instead of hardcoding IMAP4_SSL and
SMTP+STARTTLS with a verified default context.
Salvaged from #99641 (adapter.py only).
ctx.register_platform_handler(platform, factory) — the generic surface for
plugins to wire native handlers into any platform adapter at connect()
time. Factories receive (native, adapter): the platform's client/app
object (PTB Application, discord.py Bot, slack_bolt AsyncApp, Teams App,
DingTalkStreamClient, aiohttp web.Application) or None for adapters with
no separate native object.
- BasePlatformAdapter._wire_plugin_handlers(native): shared, isolated
invocation helper — a raising plugin cannot block a platform connect.
- All 27 connectable adapters call it: telegram/slack/teams/line/
api_server/msgraph_webhook wire before their dispatch tables freeze;
the rest hook at connect success.
- register_telegram_handler and get_telegram_handler_factories retained
as thin back-compat aliases over the telegram bucket.
- Source-invariant test guarantees every adapter with connect() keeps
calling the hook.
On macOS (256 soft fd limit), routing the weixin/email pollers through a
local HTTP proxy leaked one TCP socket per failed poll/connect cycle
until the gateway hit `[Errno 24] Too many open files` and crashed
(launchd respawn loop). Live capture showed 216 of 256 fds pinned on
connections to the proxy, ~214 of them abandoned.
Code-side gaps fixed:
- email adapter, `connect()`: no try/finally around the IMAP test
connection — a failure in login/ID/select/search abandoned the
connected socket with no owner. Every reconnect-watcher retry builds
a fresh adapter, so each retry against an unreachable/proxied host
leaked another fd. Teardown now runs in `finally`.
- email adapter, IMAP teardown: `imaplib.IMAP4.logout()` only swallows
`OSError` internally; on a broken connection `LOGOUT` raises
`IMAP4.abort` before the internal `shutdown()`, leaving the socket
open. New `_close_imap()` helper chases a failed `logout()` with an
unconditional `shutdown()`; used in `connect()` and
`_fetch_new_messages()`.
- weixin adapter: repeated poll failures through a proxy strand
sockets in the aiohttp connector where the tight keepalive reaper
never sees them. The poll loop now recycles its ClientSession
(swap-then-close, safe for concurrent `_process_message` tasks)
after each MAX_CONSECUTIVE_FAILURES streak, tearing down the
connector and every socket it holds.
Targeted tests: tests/gateway/test_poller_fd_lifecycle.py (9 tests).
Reported by @EthanHunter1229 with measured fd captures.
Follow-ups to the salvaged #80032 fatal-error escalation, closing the
gaps its review thread identified plus a sibling of the same class:
1. Partial-batch loss: _check_inbox now dispatches whatever the fetch
returned BEFORE escalating a failure — the early-return dropped
already-fetched messages whose UIDs were marked seen.
2. Seen-after-fetch: UIDs enter _seen_uids only after their fetch
returns a response, so a mid-batch connection failure leaves the
remaining UIDs eligible for the next poll. Per-message processing
moved to _parse_fetched_message behind a poison guard: a message
that fails parsing/auth-verification is marked seen, logged with
its UID, and skipped once — never an eternal crash loop.
3. Reconnect mail loss: connect(is_reconnect=True) restores the
account's seen-UID baseline from a class-level snapshot instead of
re-marking the entire mailbox seen — mail that arrived during an
outage is now processed after the reconnect the escalation triggers.
7 new regression tests.
_fetch_new_messages() wrapped the whole IMAP connect/login/select/search/
fetch sequence in a bare except that logged and returned an empty list —
indistinguishable from a genuinely empty inbox. The adapter never invoked
its fatal-error handler, so the gateway's reconnect/backoff/status
machinery never learned the mailbox was unreachable; outages lasted until
a manual restart.
Track fetch failure on the adapter and, when the poll loop observes it,
set a retryable fatal error (email_imap_fetch_failed) and notify the
gateway handler so the platform enters the reconnect queue just like a
startup connection failure.
Fleet triage after the 2026-08-11 storm resolution found agents whose sole
platform had been silently 'retrying' for weeks: revoked Telegram tokens,
Discord privileged-intent rejections, and Photon sidecars that can never
start were all funnelled into the indefinite reconnect queue with no owner
signal (OOF-151/152/153, epic OOF-156).
Two-part fix:
1. Per-adapter classification — by exception TYPE only, never message text:
- telegram: InvalidToken/Forbidden -> telegram_auth_error, retryable=False
(new _looks_like_auth_error, mirrors _looks_like_network_error)
- discord: LoginFailure -> discord_auth_error, PrivilegedIntentsRequired
-> discord_intents_required (both retryable=False); every other path now
sets an explicit code (previously the generic branch set NO fatal info,
which the gateway read as 'probably transient')
- photon: new typed PhotonSidecarStartupError; deps-install failure ->
SIDECAR_DEPS_MISSING and missing node binary -> SIDECAR_NODE_MISSING
(retryable=False); ambiguous startup crashes stay retryable
- email: IMAP/SMTP failures now always set a fatal code;
SMTPAuthenticationError -> email_auth_error, retryable=False (IMAP4.error
is type-ambiguous between bad creds and transient NOs, so IMAP stays
retryable)
2. Gateway escalation — platforms continuously in the reconnect queue past
HERMES_RECONNECT_ATTENTION_AFTER_SECONDS (default 2h, 0 disables) get
needs_attention=true + retrying_since stamped into runtime status, once
per episode, cleared on successful reconnect.
Deliberately NOT a circuit breaker: retries never stop. The auto-pause
mechanism was removed for good reason (transient DNS outages left bots
silently dead); this preserves that and only adds visibility. No new
platform_state enum values — NAS's status schema is strict — only additive
fields.
Unknown exception types always stay retryable: a false terminal recreates
the silently-dead-bot problem, and the escalation path covers
misclassified permanent failures.
Unknown charset labels (QQ Mail's RFC 1428 'unknown-8bit' placeholder,
misspelled names, garbage encoded-word charsets) raised LookupError from
bytes.decode — errors='replace' only guards decode errors, not a missing
codec — aborting the whole fetch batch. UIDs are marked seen before the
fetch, so the crash permanently dropped every message in the batch.
- _safe_decode(): alias table (unknown-8bit→utf-8, gb2312/gbk→gb18030,
ks_c_5601-1987→cp949, ...) then utf-8, then latin-1 last resort.
- _decode_header_value(): wraps decode_header() so a malformed RFC 2047
header degrades to the raw string instead of crashing.
- _extract_text_body(): all three decode sites now use _safe_decode.
Fixes#35901, fixes#55381, fixes#55383.
Follow-up to the salvaged #59076 commit:
- Replace the bare get_secret import with a module-level Slack-pattern
helper (_get_esecret): try get_secret, on UnscopedSecretError fall back
to os.getenv. The DEFAULT profile's email adapter constructs UNSCOPED
under multiplexing, where a bare get_secret raises and would crash the
email path on startup — the exact WhatsApp defect fixed in 5438e9c629
(whatsapp_common._get_wsecret).
- Extend scope coverage to the remaining scope-blind reads:
EMAIL_IMAP_PORT / EMAIL_SMTP_PORT / EMAIL_POLL_INTERVAL (_esecret_int
replacing utils.env_int) and EMAIL_TRUST_FROM_HEADER (_esecret_bool
replacing utils.env_bool).
- Add tests: default-profile unscoped-under-multiplex construction, and
scoped ports/trust-flag no-environ-inheritance.
The email adapter (plugins/platforms/email/adapter.py) read
EMAIL_ADDRESS, EMAIL_PASSWORD, EMAIL_IMAP_HOST, EMAIL_SMTP_HOST,
EMAIL_ALLOWED_USERS, and EMAIL_ALLOW_ALL_USERS via os.getenv()
directly. In a multiplexed gateway, os.environ holds the default
profile's .env values, so every secondary profile inherited the
default profile's email credentials instead of its own.
This was a sibling of the api_server env-leak bug (#52307/#50051):
the same os.getenv→get_secret migration that PR #50094 applies to
gateway/config.py, but for the email adapter itself, which neither
PR #50094 nor #51374 covers.
Changes:
- plugins/platforms/email/adapter.py: replace os.getenv with
agent.secret_scope.get_secret for all EMAIL_* credential reads
(adapter __init__, check_email_requirements, _allowlist_in_effect,
_dispatch_message allowlist gate, _send_email SMTP helper).
- gateway/config.py: add _getenv/_getenv_str/_getenv_int helpers
(from PR #50094) and replace os.getenv with _getenv for the email
block in _apply_env_overrides, so config.platforms[EMAIL].extra
is populated from the scoped value.
- tests/gateway/test_email_secret_scope.py: 5 new tests covering
scoped credential reads, environ fallback without scope, missing-
key-no-leak, allowlist scoping, and check_email_requirements scoping.
Related: #50051, #52307, PR #50094, PR #51374
Salvage of #2794 by @CharmingGroot, ported to the relocated
plugins/platforms/email/adapter.py:
- Guard raw_email = msg_data[0][1] against IndexError/TypeError and
non-bytes payloads. UIDs are added to _seen_uids before fetch, so an
exception mid-batch permanently skipped every remaining message in
the batch — now the bad message is logged and skipped instead.
- Message-ID domain generation falls back to 'localhost' when
EMAIL_ADDRESS lacks '@' (now via a shared _message_id_domain() helper
covering all 3 send paths; the PR fixed 2 of 3).
Aligns runtime behaviour with SECURITY.md 2.6: externally reachable
messaging adapters must fail closed unless access is explicitly
configured. Closes the confirmed multiplex authorization bypass a
secondary profile's open dm/group policy no longer inherits the default
profile's allowlist trust.
- Own-policy adapters (WhatsApp, WeCom, Weixin, QQBot, Yuanbao) default
dm_policy/group_policy to pairing/allowlist instead of open; open now
requires an explicit GATEWAY_ALLOW_ALL_USERS or per-platform allow-all.
- Startup guard (_own_policy_open_startup_violation) refuses to boot when
an enabled adapter is open without the allow-all opt-in; the guard now
runs for every secondary profile in multiplex mode too.
- Profile-aware own-policy authorization: _authorization_adapter /
_adapter_for_source resolve the live adapter via SessionSource.profile,
so _is_user_authorized and the ingress/pairing/busy/queue paths read the
originating profile's adapter policy, not the default profile's.
- Fail-closed intake for Email, Feishu P2P, and Discord (blank-principal
denial, empty-allowlist deny, missing-interaction.user deny).
Salvaged from #44073 (external-surface hardening), split into a focused
gateway-authz PR per maintainer request. Follow-up fix by Hermes Agent:
the Discord slash-auth channel bypass now matches DISCORD_ALLOWED_CHANNELS
by the same name-inclusive keys (id + name + #name + parent) the on_message
scope gate uses, so a name-form channel allowlist authorizes slash
interactions consistently (was id-only, breaking #name matching).
Co-authored-by: Hermes Agent <agent@nousresearch.com>
After a prolonged outage the in-process network-error ladder escalates to
fatal and GatewayRunner._platform_reconnect_watcher rebuilds a fresh adapter
that reconnects through the bootstrap path. That path called
start_polling(drop_pending_updates=True), discarding every update Telegram
queued during the outage — all messages sent while the bot was down were
silently lost. The in-process ladder and 409-conflict handler already passed
drop_pending_updates=False; only bootstrap did not distinguish a cold first
boot from a reconnect.
Thread an is_reconnect signal from the watcher through
_connect_adapter_with_timeout into adapter.connect(). The base
BasePlatformAdapter.connect() gains a keyword-only is_reconnect=False so every
adapter inherits a tolerant signature (no per-platform breakage when the
runner forwards the kwarg). Telegram translates is_reconnect into
drop_pending_updates=not is_reconnect on both the polling and webhook bootstrap
calls. Cold boot still drops the stale queue; a watcher reconnect preserves it.
Fixes#46621.
Co-authored-by: annguyenNous <annguyen@nousresearch.com>
Co-authored-by: kyssta-exe <kyssta-exe@users.noreply.github.com>
Co-authored-by: Kewe63 <Kewe63@users.noreply.github.com>
The email adapter authorized senders entirely off the From: header, which is
attacker-controlled and unauthenticated by IMAP. An attacker could forge
From: an-allowlisted-address and pass both the adapter's EMAIL_ALLOWED_USERS
pre-filter and the gateway's allowlist authz (both key on the same spoofable
sender_addr), getting unauthorized commands executed by the agent.
Verify the From: domain against the trusted Authentication-Results header the
receiving mail server stamps (SPF/DKIM/DMARC) before trusting it for
authorization. Enforced only when an allowlist is in effect and allow-all is
off — fail-closed. Operators whose server does not stamp the header can opt
out via platforms.email.require_authenticated_sender: false (or
EMAIL_TRUST_FROM_HEADER=true).
Fold in the #40715 blank-env OOM fix on top of the host-resolution change:
- connect() now sets a non-retryable fatal error when required settings are
missing, so the gateway stops reconnecting against an empty host instead of
looping forever and leaking memory until the host OOM-kills.
- check_email_requirements() treats blank/whitespace-only EMAIL_* values as
missing, so an abandoned setup with empty keys no longer enables the platform.
Credits the parallel fixes by zerone0x (#40745) and liuhao1024 (#40829).
The email adapter read address/host purely from env vars and never stripped
them, so a missing or whitespace-padded EMAIL_IMAP_HOST reached
imaplib.IMAP4_SSL("") and surfaced as the misleading
"[Errno 8] nodename nor servname provided, or not known" — sending users down a
DNS rabbit hole when the real problem was an empty/dirty host string. A
config.yaml-only setup also left the host empty because __init__ ignored
PlatformConfig.extra, even though the "connected" check, the send helper, and
`hermes config show` already read address/imap_host/smtp_host from it.
Resolve address/imap_host/smtp_host from the env var first, then fall back to
config.extra, and strip surrounding whitespace — matching the send helper's
existing pattern. Validate the required settings at the start of connect() and
return False with an actionable message instead of attempting a connection with
an empty host.
Adds regression tests for whitespace stripping, config.extra fallback, and the
no-IMAP-attempt-on-missing-host path.
Salvage of PR #41284 onto current main. Relocates the last 9 inline messaging
adapters (+ satellites: telegram_network, feishu_comment/_rules/meeting_invite,
wecom_crypto, wecom_callback) from gateway/platforms/ into self-contained
bundled plugins under plugins/platforms/<x>/, discovered via the platform
registry. Strips the per-platform core touchpoints from gateway/run.py,
gateway/config.py, hermes_cli/gateway.py, hermes_cli/setup.py, and
tools/send_message_tool.py.
Carries forward the migration fixes (explicit enabled:false honored,
get_connected_platforms forces discovery, plugin is_connected via
gateway.get_env_value, logs --component gateway matches plugins.platforms.*,
matrix hidden on Windows).
Additionally ports config keys main added since the PR base: the matrix
plugin's _apply_yaml_config now also covers allowed_users,
ignore_user_patterns, process_notices, and session_scope (the inline
gateway/config.py matrix block gained these in the 1340 commits the PR sat
open; they would otherwise have been silently dropped on deletion).
