Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
#116713 parsed HERMES_LANGFUSE_MAX_DEPTH inside `_safe_value`, i.e. once per
captured prompt, response, tool input and tool output. With an invalid value
(`abc`) every captured field logged the same "Invalid ... Falling back to 4"
WARNING for the life of the process — one multi-tool turn fills agent.log.
Resolve the depth in `_resolve_max_depth`, an lru_cache keyed on the raw env
string: the warning fires once per distinct bad value, a changed env var is
still picked up by a long-lived process (mirrors `_capture_mode`, which also
reads per call and warns once), and valid values skip the int() parse after
the first call.
Follow-up to #116713 (independent review finding).
Conflicts resolved toward the PM model: main's lazy_deps/update_cmd_deps/npm
stamp machinery stays deleted (PM + scripts/build/node-deps.mjs own it), the
systemd ExecStop stop-mark rides the installation launcher, legacy
linux_only/macos_only/windows_only markers are rewritten to platforms(), and
finalize_update_receipt carries pending manual-serve obligations forward
again (lost when the ContextVar receipt rewrite crossed c0aa3ce354).
Test harness: the real-home I/O guard exempts /proc/<pid>/fd metadata reads
(deleted-WAL holder scans) and run_tests.sh drops ~/.hermes PATH entries so
shutil.which() cannot trip the tripwire.
_finalize_all_traces called _get_langfuse(), which lazily BUILDS a client when the
launch-profile slot is empty. Under a multiplex gateway every turn runs scoped, so
only the per-home slots are populated; at exit (no scope) the credential read
raised UnscopedSecretError and the flush loop was skipped for every profile,
losing all pending traces. The finalizer now iterates the already-settled
clients (launch slot + per-home map) and never initializes. Test drives two
profiles under real home-override + secret scopes, then finalizes unscoped.
Three shims caught UnscopedSecretError and degraded to "" (mem0._scoped_env)
or to os.environ (langfuse._secret, azure_identity_adapter._scoped_env). Under
multiplex os.environ holds the DEFAULT profile's .env, so the langfuse/azure
fallback could ship another profile's keys, and the mem0 fallback silently
routed a mis-spawned turn's memories into the default profile's account. The
exception exists to surface exactly that spawn-site bug (agent/AGENTS.md:
never add environ fallthrough, never swallow it). All three now call
agent.secret_scope.get_secret directly: with a scope installed a miss returns
the default; single-profile deployments (multiplex off) still read the process
env inside get_secret; a scope-less multiplex caller raises.
Behavior change: a mis-spawned child under gateway.multiplex_profiles now
fails loud with UnscopedSecretError instead of running silently unauthenticated
/ on the default profile's identity. The #99121 contract (OSS mode needs no
MEM0_API_KEY in scope) is unchanged and its test now installs an empty profile
scope, which is the situation the issue described; a genuinely scope-less
caller is asserted to raise in a new test.
Tests: tests/plugins/test_scoped_secret_readers_fail_closed.py (scope wins over
environ; scope-less multiplex raises) for langfuse + azure, sabotage red when
the langfuse fallthrough is restored; tests/plugins/memory/test_mem0_v3.py::
test_load_config_fails_closed_without_scope_even_for_identity_settings,
sabotage red with a swallowing wrapper reinstated.
Competing installers and checkout-local venv assumptions bypassed PM
selection, install consent, and generation lifetimes. Route consumers
through PM and installation-bound launchers. Refresh source launchers
before obsolete Python entries can be collected.
Remove Node, browser, and CUA acquisition engines, obsolete venv-holder
handling, detached sync, and unused PM APIs. Keep historical updater
exports inert and preserve external tool ownership and native integration.
Share product freshness and prepared inputs across builders. Align plugin
admission, Docker provisioning, setup instructions, and behavioral tests.
Verified targeted Python and JavaScript tests, desktop and web typechecks,
scoped lint, real product builds, and the Docker frontend smoke test.
The missed post-setup test cleanup is included and verified.
Native Windows/macOS execution, full Rust compilation, and the complete
repository suite remain unverified. Historical compatibility requirements
were preserved and extended, not fully rescanned.
Ramp Router efforts cache + warm/disk flags, xAI and OpenRouter image catalogs,
Hindsight append-capability verdict, memory-provider skill registry, OpenViking
atexit provider, Honcho loopback flow status, Langfuse client (os.environ-only
credentials) and disk-cleanup's protected cron paths held one profile's
credential- or home-derived value process-wide; YuanbaoAdapter._active_instance
was last-connected-wins across profiles.
Keyed by home key / credential fingerprint under an override, credentials read
through the secret scope, warm threads run under copy_context(); unscoped module
slots stay for the single-profile path and the existing monkeypatch tests.
The plugin's _Runtime.run_in_session wrapper serves every mark/event it
emits (turn start/end, approvals, subagent marks) and runs synchronously
on the agent's conversation thread. It passed no timeout, so the host's
run_in_session default (timeout=None) made each mark an UNBOUNDED native
call. With a wedged native Relay pipeline the agent blocked between API
calls with zero activity ticks — observed live 2026-08-15: two cron jobs
died at the 600s inactivity kill and a gateway chat session at 1800s,
all with last_activity="API call #N completed".
The core's scope push/pop/flush/close sites were bounded with
_SCOPE_OP_TIMEOUT after the 2026-08-10 delegation stall; the plugin's
event marks were the missed sibling class.
Changes:
- plugins/observability/nemo_relay: the wrapper always passes
timeout=relay_runtime._SCOPE_OP_TIMEOUT (10s) to the host. A breach
costs one telemetry span, never the agent; it also sets scope_errored
(so close_session skips the ATIF export for the wedged session) and
warns once so the sick pipeline is visible.
- tests/plugins/test_nemo_relay_bounded_marks.py: proves the budget
reaches the host (fails on the pre-fix code — sabotage-verified),
a TimeoutError flags the session and disables its export, and the
generic error path keeps its scope_errored contract.
Session-finalize hooks ran synchronously on the gateway event loop from
three call sites (shutdown drain, session-expiry watcher, /new reset).
A plugin hook doing heavy blocking work froze the whole loop: adapter
heartbeats stopped, the drain machinery could not run, and systemd
eventually SIGKILLed the process mid-export. Observed live on a
multi-day 4.7G session where the nemo_relay observability plugin
serialized a full-session ATIF trace inside on_session_finalize.
Changes:
- gateway/run.py: new GatewayRunner._finalize_session_off_loop()
dispatches hermes_cli.lifecycle.finalize_session via the gateway
executor under asyncio.wait_for (10s budget), mirroring
_cleanup_agent_resources_off_loop (#53175). Shutdown finalize and
the session-expiry watcher now use it.
- gateway/slash_commands.py: /new reset path uses the same helper.
- plugins/observability/nemo_relay: ATIF export is now bounded
(HERMES_NEMO_RELAY_ATIF_EXPORT_TIMEOUT_S, default 30s) and skipped
entirely for sessions whose Relay scope operations already errored
(their exporter state is unreliable and the export can be
pathologically slow).
- tests/gateway/test_finalize_session_off_loop.py: regression tests
proving the loop stays live under a wedged hook and the budget is
enforced.
Adds an observability/nemo_relay section to the built-in plugins page
(the plugin had no section despite appearing in the shipped table) with
the gateway.telemetry.session_segments keys, defaults-off contract, and
segment metadata; mirrors a summary in the plugin README.
Follow-up fixes from /hermes-pr-review + /simplify-code on PR #83437:
1. Replace _redact_secrets with agent.redact.redact_sensitive_text(force=True)
— the plugin's 11-pattern list was a strict subset of the 50+ patterns in
agent/redact.py. Secrets like Stripe keys, Google API keys, GitLab tokens,
HuggingFace tokens, DB connection strings, and Telegram bot tokens would
all leak through the plugin's list but are caught by the existing redactor.
Added pk-lf- (Langfuse public key) to _PREFIX_PATTERNS in agent/redact.py.
2. Remove dead 'not isinstance(client, object)' check in on_session_finalize —
always False for any Python value.
3. Fix MoAClient.last_reference_metrics() to call the public
self.chat.completions.last_reference_metrics() instead of reaching into
the private _last_reference_metrics attribute via getattr.
4. Deduplicate _coerce_request_messages call in on_pre_llm_request — pass
pre_coerced=input_messages to _messages_for_langfuse_input to avoid
double-coercion + double _capture_content serialization per API request.
5. Add HERMES_LANGFUSE_CAPTURE to OPTIONAL_ENV_VARS in hermes_cli/config.py
for consistency with the other HERMES_LANGFUSE_* env vars.
6. Fix test_sanitized_mode_redacts_secrets test data — the old samples
('sk-abc...1234', 'sk-ant...1234', 'Authorization: Bearer ***') were too
short to match the regex thresholds and never actually tested redaction.
Updated to realistic-length secrets and changed assertions to check that
the output differs from input (redact_sensitive_text masks rather than
inserting the literal string 'REDACTED').
Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054),
@aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress
(#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797),
and @liuhao1024 (#43130).
Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two
attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes
8 prior community PRs with interaction-fix follow-ups.
Model attribution: on_pre_llm_request and on_post_llm_call now prefer the
wire value (request body model, response model) over the agent attribute,
which goes stale after /model switch or provider fallback.
Cost total: both cost paths now send a summed total alongside the per-type
breakdown, since Langfuse does not derive calculatedTotalCost from
cost_details keys. Subscription-included routes send no cost keys at all.
New coverage: api_request_error closes failed generations with ERROR level;
on_session_finalize/on_session_end close dangling traces for tool-only and
interrupted turns; subagent_start/subagent_stop trace delegated children as
spans; MoA advisor fan-out emits one generation per advisor priced at the
advisor's own model.
Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default
sanitized). Sanitized mode redacts secret patterns before truncation.
Adopted lifecycle fixes: shutdown client at session finalize when
reason=shutdown (not on session rotation); atexit finalizer ends open root
spans for short-lived processes; root context manager exited to prevent
interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock;
reasoning_content surfaced in traces; system prompt included in generation
input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io.
Closes#29482, #43129, #72661.
Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130.
Partially addresses #67544 (capture modes + secret redaction; user_id remains open).
Scope events export when their OWNING scope closes. Turn scopes close
every turn; session scopes close only at session end. Marks were attached
to the session handle, so a long-lived conversation — a Slack thread open
all day, the normal enterprise case — emitted no approval or turn marks
for hours, and none at all if the process died first. Audit dashboards
showed an empty approval table while approvals were demonstrably firing;
the operator had to end the session to see anything.
Attach marks to the live turn handle when one exists for the mark's
session (active_turn already validates live/same-profile/same-session/
unreleased), falling back to the session handle otherwise — correct for
session-level events like session.end and for marks emitted outside a
turn. Parentage semantics are unchanged: the turn is a child of the
session, so the session tree is identical, only export cadence changes
from per-session to per-turn.
