Follow-up fixes from /hermes-pr-review + /simplify-code on PR #83437:
1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True)
— the plugin's 11-pattern list was a strict subset of the 50+ patterns in
agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens,
HuggingFace tokens, DB connection strings, and Telegram bot tokens would
all leak through the plugin's list but are caught by the existing redactor.
Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py.
2. Remove dead 'not isinstance(client, object)' check in on_session_finalize —
always False for any Python value.
3. Fix MoAClient.last_reference_metrics() to call the public
self.chat.completions.last_reference_metrics() instead of reaching into
the private _last_reference_metrics attribute via getattr.
4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass
pre_coerced=input_messages to _messages_for_langfuse_input to avoid
double-coercion + double _capture_content serialization per API request.
5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py
for consistency with the other HERMES_LANGFUSE_* env vars.
6. Fix test_sanitized_mode_redacts_secrets test data — the old samples
('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too
short to match the regex thresholds and never actually tested redaction.
Updated to realistic-length secrets and changed assertions to check that
the output differs from input (redact_sensitive_text masks rather than
inserting the literal string 'REDACTED').
Two review items raised on #65449 (thanks @hansai-art):
1. Explicit test that post-module-load registration REBUILDS the
_PREFIX_SUBSTRINGS pre-screen tuple — plugin patterns flow through
the same fast path as built-ins, never around it. This was covered
implicitly by the masking tests; now it is asserted directly.
2. Plugin patterns are now stored keyed by registration source, giving
the #64229 lifecycle/ownership-ledger work a clean seam to drop one
plugin's patterns on unload. No public removal API is added —
additive-only stands; unload remains a host-owned lifecycle concern.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnMCvi2vXqfs996AjVeF2F
Nested unbounded quantifiers ((a+)+, (?:x*)*, (a{2,})+) backtrack
catastrophically, and registered patterns run against every log line
and tool output, so a pathological pattern from a buggy plugin would
stall the host process. Registration now rejects the structural
nesting shape with a logged warning, same fail-soft contract as the
other validators.
Detection is a hand-rolled scanner matching the top-level-alternation
check's idiom: escapes and character classes skipped, group stack
tracks whether each group body contains an unbounded repeat, reject
when such a group closes into an unbounded quantifier. Overlapping
alternation ambiguity ((a|aa)+) is documented as out of scope.
Also refreshes the test module docstring left stale by the demo-plugin
unbundling.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnMCvi2vXqfs996AjVeF2F
'ab|.*' compiled and carried the accepted 'ab' literal prefix while its
'.*' branch stayed unprefixed, escaping the no-redact-everything
guarantee (_extract_literal_prefix stops at '|'). Registration now
rejects top-level alternation with a regression test for exactly that
shape; grouped alternation after the prefix, escaped pipes, and
character-class pipes remain accepted.
The bundled nvapi-redaction reference plugin is removed per repo policy
(vendor integrations ship as standalone plugin repos); the end-to-end
register() coverage now uses a synthetic plugin written at test time.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PnMCvi2vXqfs996AjVeF2F
Every new vendor token format has required a core PR appending to
_PREFIX_PATTERNS in agent/redact.py (fw_, retaindb_, hsk-, mem0_, brv_
all landed that way; #58466/#58501 are the latest of the class). This
adds an additive-only registry so provider plugins own their format:
- agent/redact.py: register_redaction_patterns(patterns, source) —
validates each pattern (must compile; must start with >=2 literal
characters so the pre-screen substring gate keeps working and
redact-everything patterns like `.*` are structurally impossible),
dedupes against built-ins and prior registrations, then atomically
rebuilds _PREFIX_RE and _PREFIX_SUBSTRINGS. Registered patterns get
identical treatment to built-ins everywhere: same head/tail masking,
same non-reusable «redacted:label…» sentinel on file_read, same
security.redact_secrets operator opt-out. Additive-only by design —
a plugin can extend masking, never weaken it. Includes a
test/teardown reset helper.
- hermes_cli/plugins.py: PluginContext.register_redaction_patterns()
delegating with per-plugin attribution; warns and returns 0 on any
failure so a broken plugin can never break startup.
- Bundled reference plugin `nvapi-redaction` (opt-in): masks NVIDIA
API keys (nvapi-, used by NIM / build.nvidia.com) — a real format
missing from core, shipped as the one-liner plugin that previously
would have been a one-line core PR.
13 new tests: baseline gap, masking + built-ins unaffected, invalid
regex / no-literal-prefix / dedupe / non-string rejection, file_read
sentinel labeling, reset semantics, PluginContext wiring incl.
exception isolation, and a no-mocks end-to-end through the demo
plugin. Existing redaction suites (tests/agent/test_redact.py,
tests/tools/test_kanban_redaction.py) pass untouched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FWMcB7RPSYUpsXDfBgwjzM
Review follow-ups on the guard: state the accepted \n-residual in the
comment, reuse the span local in the next condition instead of
re-slicing, and short-circuit the substring checks before the regex.
Post-merge review of aecb9ca89 found the join guard over-broad: skipping
the join whenever ANY fragment self-matches _PREFIX_RE reopened a leak
for non-newline splits — sk-<15 chars>ESC<25 chars> masked only the
self-matching head and left the 25-char tail in cleartext (fully masked
before the guard; main never masked this shape at all, so the merged
state was still >= main, but the salvage's own coverage regressed).
Skip the join only when the span crosses a line boundary (\n / \r) —
that is the shape where adjacent legitimate text gets swallowed
(ghp_<token>-then-'button [ref=e3]' annotation bug). ESC/zero-width
controls never legitimately separate a token from prose, so joining
there is safe and restores full-tail masking.
Both legs mutation-checked: reverting to the unconditional skip fails
the new tail-mask test; removing the guard fails the annotation test.
CI slice 1/12 caught a regression in _mask_control_split_tokens: a
COMPLETE prefix token at end-of-line followed by ordinary text (browser
accessibility annotations: 'ghp_<tok>\nbutton [ref=e3]: Copy') was
joined across the newline into one stripped-copy match, and the mask
swallowed the adjacent line ('button' disappeared).
Join only when no fragment inside the span matches _PREFIX_RE on its
own — a self-matching fragment is already handled by the ordinary
prefix pass, so joining can only cause damage. All smuggling shapes
(ESC/ZWSP/newline splits with under-length fragments) still mask;
regression test added and mutation-checked (fails without the guard).
A masked secret's visible head/tail could carry control bytes (newline,
NUL, DEL, C1 0x80-0x9F, zero-width) into config/status/dump output.
Strip every control incl. \n/\t (display differs from redact_sensitive_text,
which preserves \n/\t as line structure) before slicing; all-control values
return the configured empty fallback.
Consolidates the previously-closed #58079 approach (strip controls before
masking) - supersedes it.
- Import file_safety._BLOCKED_PROJECT_ENV_BASENAMES instead of copying
it (comment-enforced parallel lists drift); lookup is now
case-insensitive to match file_safety's .lower() semantics (cat .ENV
on macOS/Windows case-insensitive filesystems reads the same secrets).
- Strip shell quotes plain split() leaves attached (cat ".env").
- Drop the dead _ENV_FILE_EXCLUDE_SUFFIXES logic (exact-basename
membership already excludes templates) and the stray blank-line noise.
- Document the defense-in-depth limits (sudo/full-path/substitution
readers) mirroring is_env_dump_command's precedent, and correct the
docstring overclaim about name-independence.
- Annotate command as str | None (tests pass None).
Terminal output from file-read commands (cat, head, tail, ...) uses
code_file=True, which skips the generic ENV-assignment redaction pass.
Reading a .env file through the terminal therefore leaked any key whose
value has no recognized vendor prefix (Mistral, Gemini AQ.*, tvly-dev-,
bu_, Spotify client secrets).
Detect file-read commands targeting .env-style basenames (mirroring
agent/file_safety's blocked list) and route them to code_file=False so
the existing ENV pass masks opaque values. Templates (.env.example,
.env.sample, ...) are excluded.
Salvaged from #61352 (145 commits of drift; conflict with the test-prune
wave resolved by NOT resurrecting pruned tests). Authored by @ShaoRou459.
Closes#61352
_CFG_DOTTED_RE's nested quantifier (?:[A-Za-z0-9_\-]+\.)+ backtracks
exponentially on long non-matching dotted runs (doubles every ~4
segments). Flatten it and use possessive quantifiers (py3.11+) in
_CFG_DOTTED_RE and _YAML_ASSIGN_RE wherever the successor is disjoint.
Zero behavior change: equivalence fuzz-verified over 120k structured
and random inputs comparing full sub() output including groups. Adds a
ReDoS regression test.
Port from openclaw/openclaw#112954. The redactor knew GitHub, Slack,
Google, Stripe, AWS access-key-ID and ~25 other vendor prefixes but had
zero GitLab coverage — glpat-/gloas-/gldt-/glrt-/glrtr-/glcbt-/glptt-/
glft-/glimt-/glagent-/glsoat-/glffct-/glwt- tokens and legacy GR1348941
runner registration tokens passed through display and log surfaces
verbatim. Follow-up explicitly invited when #4541 was closed.
Each pattern keeps a full literal prefix so the _PREFIX_SUBSTRINGS
pre-screen (derived at module load) stays false-negative-free; routable
runner tokens allow dotted segments. Sibling site: skills_guard's
credential-exposure scan gains a gitlab_token_leaked pattern.
- Remove tests/-shadowing sys.path.insert(dirname/'..') from 11 test files:
it prepended the tests/ dir itself to sys.path, so 'import agent' /
'import hermes_cli' resolved to the test packages and collection died
with ModuleNotFoundError depending on import order (2 files failed in
every full-suite run; 9 more were latent).
- Patch call_llm in 5 context-compressor tests that called compress()
unmocked: each burned ~50s attempting live LLM traffic through the
relay before falling back (572s file — the slowest in the suite, and
flaky under the 300s per-file timeout). File now runs in ~5s.
- agent/redact.py: fix two catastrophically-backtracking regexes hit by
the compressor's redaction pass on large payloads —
_STRICT_URL_USERINFO_RE anchors on the mandatory '//' (optional-scheme
prefix backtracked O(n^2): ~55s on a 320KB payload, now sub-ms;
output-equivalence fuzz-verified on 20k random strings), and the
_CFG_DOTTED_RE/_CFG_ANCHORED_RE subs gain an exact linear keyword
pre-gate so secret-free text skips the quadratic pattern entirely.
- tests/gateway/test_feishu.py: version-guard the extra_ua_tags SDK
signature check; the repo pins lark-oapi==1.6.8 but stale local
installs (1.5.3) fail the assertion — skip below the pin.
- tests/tools/test_managed_browserbase_and_modal.py: stub
agent.redact + agent.credential_persistence in the fake agent package
(empty __path__ blocks all real agent.* imports added since the fake
was written).
- tests/gateway/test_startup_restart_race.py: raise wait_for timeouts
2s -> 30s; 2s wall-clock on a loaded 40-worker box flaked in the
baseline run (passes instantly when the box is quiet).
Port from nearai/ironclaw#6129: their sensitive-marker scrubber matched
markers as bare substrings, so tool results containing 'Secretary of the
Treasury' were scrubbed as 'secret' on replay, evicting legitimate content
and forcing the model into a re-fetch loop. Hermes' lowercase/dotted/YAML
config-key redaction patterns (_CFG_DOTTED_RE, _CFG_ANCHORED_RE,
_YAML_ASSIGN_RE) had the same false-positive class: their key classes allow
arbitrary alphanumeric affixes around the keyword, so ordinary document
text like 'Secretary: J.Smith', 'tokenizer: cl100k_base' (HF model cards),
and BibTeX 'author=Smith' got value-masked on the surfaces that run these
passes (browser snapshots, log lines, kanban summaries, CLI-echoed output).
Fix: post-match word-boundary validation of the keyword occurrence inside
the matched key. Boundaries: key edges, non-letters (_ - . digits),
camelCase transitions (clientSecret, secretKey, APIToken), plural 's'
(secrets:, tokens:). Concatenated real-world compounds keep matching via
explicit alternatives (authtoken, authkey, secretkey, accesstoken). ALL-CAPS
keys keep legacy embedded matching (MYTOKEN=...) — all-caps is almost never
prose, same rationale as _ENV_ASSIGN_RE. Same discipline the file already
applies to exact-match body/query keys (ported from ironclaw#2529) and the
deliberate 'auth' exclusion that keeps 'author:' from matching.
_redact_env already skips redaction when a KEY=value assignment's value is
a programmatic env lookup (os.getenv(...), os.environ[...], process.env.X)
per issue #2852 — masking it would corrupt a code snippet, not redact a
secret. _redact_json (JSON "key": "value" syntax) and _redact_yaml
(unquoted key: value syntax) are separate closures in the same function
and never got the same check, so the identical code-snippet-in-config-
syntax case still gets mangled:
{"apiKey": "os.getenv('OPENAI_API_KEY')"} -> {"apiKey": "os.get...EY')"}
api_key: os.getenv("OPENAI_API_KEY") -> api_key: os.get...EY")
Fix: apply the same _ENV_LOOKUP_VALUE_RE.match(value) check in both
closures before masking, mirroring _redact_env exactly. Real secret
values in JSON/YAML syntax are still redacted (verified live and via new
tests) — this only skips the case where the "value" already look like a
code snippet.
'KEY=os.getenv(...)' / 'os.environ[...]' / 'process.env.X' values are
variable-name references in code snippets, not leaked secrets. Masking
them corrupted pasted code in prose/log contexts (issue #2852):
ha_token=os.getenv('HOMEASSISTANT_TOKEN') -> ha_token=os.get...EN').
Skip these values inside _redact_env, which covers all three passes that
share the closure (_ENV_ASSIGN_RE, _CFG_DOTTED_RE, _CFG_ANCHORED_RE).
Real secret values are still masked.
Salvage of PR #2852-fix #2854 — the PR's own placement (an unconditional
pass before the code_file gate) would have reintroduced the code-file
false-positive class; the skip is applied inside the existing gated pass
instead. Tests adapted from the PR.
Co-authored-by: crazywriter1 <sampiyonyus@gmail.com>
The xapp-<num>-<hash> format used by Slack App-Level / Socket Mode
tokens was missing from both agent/redact.py prefix patterns and
gateway/run.py gateway secret patterns, so SLACK_APP_TOKEN values could
leak through to chat users even with security.redact_secrets enabled.
Adds an anchored xapp-\d+- pattern to both redaction paths.
Review of the salvage found the timeout-message redaction left the more
common failure mode unguarded: when the first websockets.connect(cdp_url)
fails (bad URI / refused / TLS), the raw websockets exception -- which
embeds the full cdp_url incl. ?token= and user:pass@ -- is stashed as
_start_error and re-raised verbatim by start(), and two reconnect
logger.warning sites log the same raw exception.
Add a module-level _redact_cdp_error_text() chokepoint (delegating to
agent.redact.redact_cdp_url) and route all four supervisor egress points
through it:
- start() TimeoutError message (already covered; kept)
- start() _start_error re-raise -> now raises a redacted RuntimeError with
'from None' so no secret leaks via message OR traceback cause chain
- connect-failed and session-dropped reconnect warnings
Guard tests assert the re-raised message is redacted for both token and
userinfo, the raw cause is suppressed, and the helper preserves non-secret
context (host/reason). Verified with a mutation check: reverting to the raw
'raise err' fails the new tests. Correct the redact_cdp_url docstring to
scope its guarantee to direct-URL redaction and point exception callers at
the supervisor helper.
The session-log fix (browser_tool._sanitize_url_for_logs) and the
supervisor attach-timeout fix (CDPSupervisor.start) both composed the
same three redactors (redact_sensitive_text -> _redact_url_query_params
-> _redact_url_userinfo) to mask CDP endpoint credentials. Two copies of
one policy drift: tune one site (e.g. add fragment masking) and the other
silently re-leaks.
Promote that composition to a single public helper redact_cdp_url() in
agent/redact.py -- the one place the CDP-URL redaction policy lives -- and
route both call sites through it (_sanitize_url_for_logs becomes a thin
wrapper; the supervisor imports the helper instead of re-composing the
private redactors). Add direct unit tests for the seam covering query
tokens, multiple credentials, userinfo passwords, plain-URL passthrough,
non-string/exception coercion, and None.
No behavior change at the call sites; both leak paths remain closed.
Fireworks AI is a first-class provider in hermes-agent — FIREWORKS_API_KEY
is listed in tools/environments/local.py and the provider is selectable via
the model picker (api.fireworks.ai in model_metadata, hermes_cli/models.py).
Fireworks API keys follow the format fw_<40 alphanumeric chars> and were
absent from _PREFIX_PATTERNS in agent/redact.py. The ENV-assignment and
Bearer header patterns catch FIREWORKS_API_KEY=fw_... in config output,
but a raw key in a stack trace, debug print, or tool error passed through
completely unmasked.
Four unit tests added to TestFireworksToken covering bare token masking,
env assignment, short-prefix false positive, and visible prefix in output.
git remote set-url with an embedded password (https://PASSWORD@github.com)
leaked the credential into agent output — the redaction engine only masked
user:pass@ DB connection strings, never the colon-less bare-token userinfo
form a git remote uses.
Add _URL_BARE_TOKEN_RE: scheme://TOKEN@host for web/transport schemes
(http/https/wss/git/ssh/ftp), 8+ char floor to skip short usernames, token
class forbidding /:@ so an @ in a path/query is never treated as userinfo.
Deliberately scoped to the bare-token form only. The user:pass@ colon form
and query-string tokens stay passing through (#34029, 'pass web URLs through
unchanged') so magic-link / OAuth round-trip skills keep working — a bare
credential in userinfo is never a workflow token (those live in the query
string), so masking it can't break a skill.
When security.redact_secrets is on (default), read_file/search_files/cat
applied redact_sensitive_text(code_file=True) to file content, which still
ran prefix masking. An API key in config.yaml (ghp_..., sk-..., xai-..., etc.)
came back as a head/tail mask like `ghp_S1...Pn2T` — a plausible-looking
truncated key. When an agent read that and wrote it back to config, the masked
value replaced the real credential, silently breaking auth (401). Production
evidence: a config.yaml found containing the exact 13-char masked GitHub PAT.
The two community PRs (#35529, #35534) fixed the corruption by NOT redacting
prefixes for config reads — but that exposes the user's real keys to the agent
context, model, and logs (a security regression). This takes the safer route:
keep redacting, but for file content emit a NON-REUSABLE sentinel.
- New `_mask_token_nonreusable`: prefix secrets -> `«redacted:ghp_…»` (vendor
label preserved for debuggability; zero secret bytes; angle-bracket/ellipsis
wrapper is syntactically invalid as a token so it can't be mistaken for or
written back as a usable key).
- New `redact_sensitive_text(file_read=True)` routes prefix matches through it
(implies code_file=True). Default/log/display mode is UNCHANGED — `_mask_token`
still keeps head/tail (fine for logs, never written back).
- Wired the 3 file_tools.py call sites (read_file / search_files / cat) to
file_read=True.
Fixes both the corruption AND avoids the secret-exposure of the un-redact
approach. 6 new tests (sentinel shape, no-leak, not-a-plausible-key, default
mode unchanged, file_read implies code_file, sk- prefix); 88 redact tests pass;
mutation-verified (reverting to the old mask fails the sentinel/leak tests).
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
Co-authored-by: adammatski1972 <289282750+adammatski1972@users.noreply.github.com>
Closes#35519. Supersedes #35529, #35534.
* fix(security): redact secrets in background process + foreground env-dump output
Terminal-output redaction was incomplete (#43025):
- Gap 1: process(action=poll/log/wait) returned background stdout verbatim —
no redaction at all. A background printenv/server/test emitting a key leaked
raw to the model, session.db, and CLI display. Same for the gateway
background-process watcher's completion/progress notifications.
- Gap 2: the foreground terminal path hardcoded code_file=True, which skips the
ENV-assignment pass, so an opaque token (no vendor prefix) from env/printenv
leaked even there.
Adds agent.redact.redact_terminal_output(output, command) as the single policy
for ALL terminal-output surfaces: env-dump commands (env/printenv/set/export/
declare) get the ENV-assignment pass (code_file=False) to mask opaque tokens;
other commands stay on code_file=True to avoid false positives on source dumps.
Wired into terminal_tool, process_registry (_handle_process boundary), and the
gateway watcher. Respects security.redact_secrets (no force) — opt-out preserved.
* docs: add infographic for #43025 terminal-output redaction fix
Two related redaction bugs from #43083:
1. build_assistant_message redacted tool-call arguments in-memory. That dict
feeds both the replayed conversation history and state.db (which is itself
replayed verbatim on session resume), so the model read back its own
PGPASSWORD='***' psql call and copied the placeholder, breaking every
credential-dependent command on the second turn. The masking gave no real
protection either — the same secret still leaks through tool OUTPUT. Remove
it. Keeping secrets out of the replayable store is a separate
tokenization/vault concern (security.redact_secrets still governs
storage-time redaction elsewhere).
2. _AUTH_HEADER_RE's greedy \S+ credential class ate a closing quote when the
token sat flush against it (Authorization: Bearer sk-.."), turning value
corruption into syntax corruption (unterminated quote -> shell EOF /
SyntaxError). Exclude " and ' from the token class; real credentials never
contain them.
Closes#43083.
Secret redaction is display/output-scoped on main — write_file writes
content verbatim, terminal/execute_code redact only output not the
command/source. The real bug is in displayed tool OUTPUT (read_file,
terminal, execute_code):
_DB_CONNSTR_RE's password group [^@]+ was greedy across newlines, so on a
multi-line block it scanned past the DSN line to the next stray '@' (a
Python @decorator), replacing every intervening character — including line
breaks — with ***. That dropped lines and concatenated the next line onto
the f-string line, making read_file output look corrupted (the file on disk
was always correct). Reported in #33801.
Fix:
- Forbid whitespace in the userinfo/password groups ([^:\s]+ / [^@\s]+) so
the match can never span a line break. A real DSN password never contains
whitespace. This alone kills the catastrophic line-dropping.
- Under code_file=True, preserve a password group that is a pure {...} brace
expression — f"postgresql://{user}:{pass}@{host}" is an f-string template,
not a live credential. Literal passwords are still masked.
- Pass code_file=True at the terminal and execute_code output redaction call
sites (file_tools already did) so code-execution output isn't corrupted by
ENV/JSON/template false positives. Real prefixes, auth headers, JWTs, and
private keys are still redacted.
Verified E2E against the reporter's exact pydantic-settings module: file
written verbatim, read_file shows the DSN f-string + @model_validator intact
with zero *** corruption, while a literal postgresql://admin:pw@host DSN and
a real sk- key are still masked.
Reported-by: koishi70
Reported-by: pfrenssen
The secret redactor only matched uppercase env-style keys ([A-Z0-9_]),
so config-file assignments like spring.datasource.password=secret,
app.api.key=xyz, and YAML password: secret leaked verbatim when the
agent ran cat/grep on application.properties or .env files (issue #16413).
Adds three case-insensitive config-key matchers that run only in a
config-file context, preserving the existing #4367 (lowercase code/prose)
and web-URL-passthrough carve-outs:
- _CFG_DOTTED_RE: namespaced keys (contain a dot) — unambiguously config
- _CFG_ANCHORED_RE: bare secret-word keys at line start (incl. export)
- _YAML_ASSIGN_RE: unquoted colon config (password: value)
Value capture stops at whitespace and '&' so form bodies stay pair-wise;
the '://' guard keeps intentional web-URL query-param passthrough intact.
Reported-by: Murtaza1211
Secret redaction only matched `Authorization: Bearer <token>`. Other auth
headers passed through verbatim into logs, tool output, and transcripts:
- `Authorization: Basic <base64>` — leaks base64(user:password)
- `Authorization: token <pat>` / any non-Bearer scheme
- `Proxy-Authorization: ...`
- `x-api-key: <key>` (Anthropic and many providers) and `api-key`,
`x-goog-api-key`, `x-auth-token`, `x-access-token`, ... — opaque values with
no known vendor prefix were caught by nothing
A logged request or an echoed `curl -H "x-api-key: ..."` command therefore
leaked live credentials.
Generalize the Authorization rule to mask the credential for any scheme (and
Proxy-Authorization) while preserving the header name and scheme word for
debuggability, and add an api-key header rule for the single-opaque-value
headers. Bearer behavior is unchanged; plain prose containing the word
"authorization" (no colon-delimited value) is left untouched.
Adds regression tests for Basic/token/Proxy auth and the x-api-key/api-key
headers, including inside a curl command.
dump_api_request_debug() masks the provider Authorization header but writes
the request `body` (system prompt, tool defs, context-embedded values) and the
error message raw via atomic_json_write. This path also fires unconditionally
on API errors (not only under HERMES_DUMP_REQUESTS), so any secret surfaced
into context (e.g. an integration token) lands in cleartext at
request_dump_*.json on every failed call.
Run the serialized dump through the existing redact_sensitive_text() scrubber
(already used for logs/tool output) before persisting and before the
HERMES_DUMP_REQUEST_STDOUT print; preserve atomicity via temp-file +
Path.replace. Also add the Notion internal-integration prefix (ntn_) to
_PREFIX_PATTERNS so bare values are caught.
Per SECURITY.md §3.2 this is a redaction (in-process heuristic) hardening, not
a §3.1 vulnerability. Refs #46583.
* fix(redact): pass web URLs through unchanged
Magic-link checkout URLs, OAuth callbacks the agent is meant to follow,
and pre-signed share URLs were getting `?token=***` / `?code=***` /
`?signature=***` blanket-redacted by parameter NAME, which breaks any
skill that has to round-trip a URL through history (the model's tool
call arguments get sanitized before persistence — the live call fires
with the real URL, but the next turn sees `***`).
Joe Rinaldi Johnson hit this with a checkout-acceleration skill that
uses magic links in URLs.
Drops three call sites from `redact_sensitive_text`:
- `_redact_url_query_params` (was redacting `access_token`, `token`,
`api_key`, `code`, `signature`, `key`, `auth`, etc.)
- `_redact_url_userinfo` (was redacting `https://user:pass@host`)
- `_redact_http_request_target_query_params` (was redacting access-log
request targets like `"POST /hook?password=... HTTP/1.1"`)
The helpers themselves are kept in the module — still importable by
anything that wants to opt in explicitly.
Still redacted (unchanged):
- Vendor-prefix credential shapes (sk-, ghp_, AKIA, gAAAA, etc.)
anywhere they appear, including inside URLs — see the
`test_known_prefix_inside_url_still_redacted` case.
- JWTs (`eyJ...`)
- DB connection-string passwords (`postgres://admin:pw@host`) —
these are connection strings, not web URLs the agent navigates to.
- Authorization headers, ENV assignments, JSON `apiKey`/`token` fields,
Telegram bot tokens, private key blocks, Discord mentions, E.164
phone numbers, and form-urlencoded bodies (request bodies, not URLs).
Tests: replaces `TestUrlQueryParamRedaction` + `TestUrlUserinfoRedaction`
with `TestWebUrlsNotRedacted`, asserting representative URLs (OAuth
callback, magic link, S3 pre-signed, websocket, userinfo, access log)
pass through unchanged. Adds positive cases proving the prefix and DB
connstr nets still fire. 74 redact tests + 10 browser-exfil + 16 PII
redaction tests all pass.
* test(codex_app_server): drop URL-query assertion from stderr-tail redaction test
The test bundled (a) sk-live-* credential-prefix redaction with (b)
URL query-param redaction. (a) is still in effect via _PREFIX_RE;
(b) was the contract we just removed in the parent commit so the
'querysecret12345' assertion stopped holding. Keep the credential-shape
assertion, drop the URL-query one.
Send-message tool's local _URL_SECRET_QUERY_RE in tools/send_message_tool.py
is independent of agent/redact.py and unchanged — its tests
(test_top_level_send_failure_redacts_query_token,
test_http_error_redacts_access_token_in_exception_text) still pass.
* perf(config): add load_config_readonly() fast path for hot agent loop
`load_config()` is called from the agent loop's per-API-call hot path via
`get_provider_request_timeout()` and `get_provider_stale_timeout()` —
both invoked once per turn from `_resolved_api_call_timeout()` in
run_agent.py.
Profiling a synthetic 20-tool-call agent run revealed:
- 21 invocations of `load_config()` cumulating 56ms (~17% of agent loop)
- 34,398 deepcopy calls totaling 37ms (config defensive deepcopy + chain)
- 8,652 `_expand_env_vars` invocations (~412 per turn)
Microbench (cache-hit, real config.yaml present):
load_config() 265us/call (125us deepcopy + 140us infra)
load_config_readonly() 138us/call (~48% faster)
`load_config_readonly()` returns the cached dict directly without the
defensive deepcopy. Documented contract: caller must not mutate. Returns
plain dict (not MappingProxyType) so downstream `isinstance(x, dict)`
guards keep working — caught during initial implementation when
MappingProxyType broke get_provider_request_timeout's guard logic.
Wired into hermes_cli/timeouts.py (the two functions called per agent
turn). load_config() is unchanged for the 263 other call sites that
mutate the result before save_config(), are not in the hot path, or
where the safety guarantee matters more than the perf.
Profile A/B (cached config, 21-turn agent loop):
BEFORE AFTER delta
get_provider_request_timeout 55ms 16ms -71%
total function calls 399k 160k -60%
deepcopy calls (in hotspots) 34,398 ~0 ~elim
Verified:
- isinstance(load_config_readonly(), dict) is True
- timeout/stale resolutions correct
- load_config() still returns isolated mutable deepcopies
- tests/hermes_cli/test_config*.py / test_timeouts.py: 102/102 pass
- tests/cli/ + tests/agent/test_auxiliary_client.py: 883/883 pass
* perf(redact): substring pre-screens skip non-matching regex chains
Every log record passes through `RedactingFormatter.format` which calls
`redact_sensitive_text`, which historically ran ALL 13 secret-pattern
regexes against every line — including DB connection strings, JWTs,
Discord mentions, Signal phone numbers, etc. — even for typical clean
log records like 'INFO run_agent: API call completed'.
Add cheap substring pre-checks before each regex pass. False positives
still run the regex (which then matches nothing); false negatives are
impossible because every pattern requires the gated substring to match
its leading anchor:
- `_PREFIX_RE` gated on any of 33 known credential prefix substrings
- `_ENV_ASSIGN_RE` gated on `=` in text
- `_JSON_FIELD_RE` gated on `:` and `"` in text
- `_AUTH_HEADER_RE` gated on `uthorization`/`UTHORIZATION` in text
- `_TELEGRAM_RE` gated on `:` in text
- `_PRIVATE_KEY_RE` gated on `BEGIN` and `-----`
- `_DB_CONNSTR_RE` gated on `://` in text
- `_JWT_RE` gated on `eyJ` in text
- URL userinfo/query gated on `://`
- `_redact_form_body` gated on `&` and `=`
- `_DISCORD_MENTION_RE` gated on `<@`
- `_SIGNAL_PHONE_RE` gated on `+`
Microbench (5 typical log records, 20k iterations each):
BEFORE AFTER delta
redact_sensitive_text per call 5.63us 1.79us -68%
Real-world impact: ~244 log records emitted in a 30-turn agent loop, so
the chain saves ~1ms of CPU per conversation. Bigger win is the
reduction in regex execution and GC pressure during heavy logging
sessions (verbose logging, gateway message processing).
Security regression test: 30 secret-containing inputs (sk-/ghp_/JWT/DB
connstr/Auth-Bearer/private key/URL userinfo/Discord/Signal/etc.)
verified to produce identical redacted output before/after. All 75
existing tests/agent/test_redact.py cases pass.
The `?access_token=foo&code=bar` (bare query string, no scheme) case
that 'leaks' is pre-existing behavior — the URL query redaction
requires a well-formed URL with scheme+host. Not a regression.
* perf(run_agent): cache _needs_thinking_reasoning_pad result per (provider, model, base_url)
Profile of a 31-turn synthetic agent run shows `_needs_thinking_reasoning_pad`
fires 495 times (~16 per turn) and each call ran 3 helper methods, each
hitting `base_url_host_matches` 1-4 times via `urlparse`. Total cost:
3,342 base_url_host_matches calls + 3,373 urlparse calls accounting for
~36ms of agent-loop overhead (~7% of the entire post-network work).
Provider / model / base_url don't change during a conversation except via
`switch_model` and fallback activation — both of which already overwrite
those attributes atomically. Cache the result on a tuple key; since the
key is derived from the very fields that would change, the cache
auto-invalidates on the next read after a switch. No manual invalidation
needed in switch_model / _try_activate_fallback.
Profile A/B (31-turn cached-config agent run):
BEFORE AFTER delta
_needs_thinking_reasoning_pad cum 18ms 1ms -94%
_copy_reasoning_content_for_api cum 17ms 1ms -94%
base_url_host_matches calls 3,342 372 -89%
urlparse calls 3,373 403 -88%
total function calls 296k 223k -25%
Verified:
- tests/run_agent/test_deepseek_reasoning_content_echo.py: 36/36 pass
- tests/run_agent/ (full): 1383/1383 pass + 3 skipped
xAI is a first-class provider in hermes-agent with its own credential
pool entry (XAI_API_KEY / xai-oauth). API keys follow the format
xai-<60+ alphanumeric chars> and were absent from _PREFIX_PATTERNS in
agent/redact.py.
When a key appears raw in log output, tool results, or error messages,
it passed through completely unmasked. The ENV-assignment and Bearer
header patterns catch the most common cases, but a raw token in a
stack trace or debug print had no protection.
Verified before fix:
redact_sensitive_text("using key xai-ABCD...rstu to call xAI", force=True)
# "using key xai-ABCD...rstu to call xAI" <- exposed
After fix:
# "using key xai-AB...rstu to call xAI" <- masked
Five unit tests added to TestXaiToken covering bare token masking,
env assignment, short-prefix false positive, company name false
positive, and visible prefix in masked output.
Replace with for all literal-tuple
membership tests. Set lookup is O(1) vs O(n) for tuple — consistent
micro-optimization across the codebase.
608 instances fixed via `ruff --fix --unsafe-fixes`, 0 remaining.
133 files, +626/-626 (net zero).
Flip the default for HERMES_REDACT_SECRETS from off to on so the redactor
already wired into send_message_tool, logs, and tool output actually runs
on a fresh install.
- agent/redact.py: env-var default "" → "true"
- hermes_cli/config.py: DEFAULT_CONFIG security.redact_secrets True;
two config-template comments rewritten
- gateway/run.py + cli.py: startup log / banner warning when the user
has explicitly opted out, so the downgrade is visible in agent.log
and at CLI banner time
- docs/reference/environment-variables.md: description reconciled
- tests: flipped the default-pin, restructured the force=True
regression test to explicit-false instead of unset
Users who need raw credential values (redactor development) can still
opt out via security.redact_secrets: false in config.yaml or
HERMES_REDACT_SECRETS=false in .env.
Closes#17691.
Addresses #20785 (short-term output-pipeline recommendation).
ENV-assignment and JSON-field regex patterns in redact_sensitive_text()
cause false positives when reading source code files:
- MAX_TOKENS=*** triggers the ENV assignment pattern
- "apiKey": "test" in test fixtures triggers the JSON field pattern
Add code_file=False parameter. When code_file=True, skip only the
ENV-assignment and JSON-field regex passes; all other patterns (prefixes,
auth headers, private keys, DB connstrings, JWTs, URL secrets) are
still applied.
Update file_tools.py (read_file and search_files) to pass code_file=True
so agent code analysis is not polluted by false-positive redactions.
Closes#15934
Three modules independently implemented the same "preserve head+tail of
a secret, mask the middle" logic with slightly different behaviors that
had started to drift:
hermes_cli/config.py redact_key — 12-char floor, 4+4, DIM '(not set)'
hermes_cli/status.py redact_key — 12-char floor, 4+4, plain '(not set)' ← drift
hermes_cli/dump.py _redact — 12-char floor, 4+4, empty string
The visible bug: 'hermes status' displayed the '(not set)' placeholder
in plain text while 'hermes config' showed it in dim text. Same concept,
inconsistent UI.
Introduces mask_secret() in agent/redact.py as the canonical helper,
with head/tail/floor/placeholder/empty kwargs. The three call sites
become one-line wrappers that differ only in the 'empty' handling:
config.redact_key → mask_secret(k, empty=color('(not set)', Colors.DIM))
status.redact_key → mask_secret(k, empty=color('(not set)', Colors.DIM))
dump._redact → mask_secret(v) # empty → ''
agent.redact._mask_token (log redactor, different policy: 18-char floor,
6+4 visible, '***' on empty) also ports to mask_secret but retains its
own empty-case handling to preserve the historical '***' return.
Net: the three display-time redactors now agree on formatting, the
canonical helper lives in one place, and future tweaks (e.g. adding
bullet-point masking, changing the head/tail widths) happen once.
Verified:
- 3/3 tests/hermes_cli/test_web_server.py::TestRedactKey pass
- 89/89 agent/tests/test_redact.py + tests/tools/test_browser_secret_exfil.py
+ tests/hermes_cli/test_redact_config_bridge.py pass
- Live 'hermes status', 'hermes config', 'hermes dump' all render the
same way they did before (verified against actual env with real
keys: OpenRouter, Firecrawl, Browserbase, FAL, Tinker all show
'prefix...suffix'; Kimi shows '***' at <12 chars; unset shows
'(not set)' uniformly).
Co-authored-by: teknium1 <teknium@users.noreply.github.com>
Flips security.redact_secrets from true to false in DEFAULT_CONFIG, and
the HERMES_REDACT_SECRETS env-var fallback in agent/redact.py now
requires explicit opt-in ("1"/"true"/"yes"/"on") to enable.
New installs and users without a security.redact_secrets key get pass-
through tool output. Existing users whose config.yaml explicitly sets
redact_secrets: true keep redaction on — the config-yaml -> env-var
bridges in hermes_cli/main.py and gateway/run.py still honor their
setting.
Also updates the inline config comments, website docs, and the
hermes-agent skill so /hermes config set security.redact_secrets true
is now the documented way to turn it on.
* feat(security): URL query param + userinfo + form body redaction
Port from nearai/ironclaw#2529.
Hermes already has broad value-shape coverage in agent/redact.py
(30+ vendor prefixes, JWTs, DB connstrs, etc.) but missed three
key-name-based patterns that catch opaque tokens without recognizable
prefixes:
1. URL query params - OAuth callback codes (?code=...),
access_token, refresh_token, signature, etc. These are opaque and
won't match any prefix regex. Now redacted by parameter NAME.
2. URL userinfo (https://user:pass@host) - for non-DB schemes. DB
schemes were already handled by _DB_CONNSTR_RE.
3. Form-urlencoded body (k=v pairs joined by ampersands) -
conservative, only triggers on clean pure-form inputs with no
other text.
Sensitive key allowlist matches ironclaw's (exact case-insensitive,
NOT substring - so token_count and session_id pass through).
Tests: +20 new test cases across 3 test classes. All 75 redact tests
pass; gateway/test_pii_redaction and tools/test_browser_secret_exfil
also green.
Known pre-existing limitation: _ENV_ASSIGN_RE greedy match swallows
whole all-caps ENV-style names + trailing text when followed by
another assignment. Left untouched here (out of scope); URL query
redaction handles the lowercase case.
* feat: replace kimi-k2.5 with kimi-k2.6 on OpenRouter and Nous Portal
Update model catalogs for OpenRouter (fallback snapshot), Nous Portal,
and NVIDIA NIM to reference moonshotai/kimi-k2.6. Add kimi-k2.6 to
the fixed-temperature frozenset in auxiliary_client.py so the 0.6
contract is enforced on aggregator routings.
Native Moonshot provider lists (kimi-coding, kimi-coding-cn, moonshot,
opencode-zen, opencode-go) are unchanged — those use Moonshot's own
model IDs which are unaffected.
Found via trace data audit: JWT tokens (eyJ...) and Discord snowflake
mentions (<@ID>) were passing through unredacted.
JWT pattern: matches 1/2/3-part tokens starting with eyJ (base64 for '{').
Zero false-positive risk — no normal text matches eyJ + 10+ base64url chars.
Discord pattern: matches <@digits> and <@!digits> with 17-20 digit snowflake
IDs. Syntactically unique to Discord's mention format.
Both patterns follow the same structural-uniqueness standard as existing
prefix patterns (sk-, ghp_, AKIA, etc.).
Consolidated salvage from PRs #5301 (qaqcvc), #5339 (lance0),
#5058 and #5098 (maymuneth).
Mem0 API v2 compatibility (#5301):
- All reads use filters={user_id: ...} instead of bare user_id= kwarg
- All writes use filters with user_id + agent_id for attribution
- Response unwrapping for v2 dict format {results: [...]}
- Split _read_filters() vs _write_filters() — reads are user-scoped
only for cross-session recall, writes include agent_id
- Preserved 'hermes-user' default (no breaking change for existing users)
- Omitted run_id scoping from #5301 — cross-session memory is Mem0's
core value, session-scoping reads would defeat that purpose
Memory prefetch context fencing (#5339):
- Wraps prefetched memory in <memory-context> fenced blocks with system
note marking content as recalled context, NOT user input
- Sanitizes provider output to strip fence-escape sequences, preventing
injection where memory content breaks out of the fence
- API-call-time only — never persisted to session history
Secret redaction (#5058, #5098):
- Added prefix patterns for Groq (gsk_), Matrix (syt_), RetainDB
(retaindb_), Hindsight (hsk-), Mem0 (mem0_), ByteRover (brv_)
