Files

kshitij e665300d6b feat(langfuse): widen tracing to errors, sessions, subagents, and MoA fan-out

Salvaged from PR #83437 by @erosika, with adopted fixes from @bgodlin (#81054),
@aldoeliacim (#82332), @nftpoetrist (#42326), @rodboev (#39653), @FnExpress
(#64292, supersedes #32175 by @db-aeon), @Per0-1 (#61166), @NaMinhyeok (#64797),
and @liuhao1024 (#43130).

Widens the bundled Langfuse plugin from 6 to 11 hooks and fixes two
attribution bugs. Also adopts shutdown/atexit lifecycle fixes and composes
8 prior community PRs with interaction-fix follow-ups.

Model attribution: on_pre_llm_request and on_post_llm_call now prefer the
wire value (request body model, response model) over the agent attribute,
which goes stale after /model switch or provider fallback.

Cost total: both cost paths now send a summed total alongside the per-type
breakdown, since Langfuse does not derive calculatedTotalCost from
cost_details keys. Subscription-included routes send no cost keys at all.

New coverage: api_request_error closes failed generations with ERROR level;
on_session_finalize/on_session_end close dangling traces for tool-only and
interrupted turns; subagent_start/subagent_stop trace delegated children as
spans; MoA advisor fan-out emits one generation per advisor priced at the
advisor's own model.

Capture modes: HERMES_LANGFUSE_CAPTURE=metadata|sanitized|full (default
sanitized). Sanitized mode redacts secret patterns before truncation.

Adopted lifecycle fixes: shutdown client at session finalize when
reason=shutdown (not on session rotation); atexit finalizer ends open root
spans for short-lived processes; root context manager exited to prevent
interpreter-teardown TypeError; TOCTOU on _get_langfuse() fixed with lock;
reasoning_content surfaced in traces; system prompt included in generation
input for Anthropic/Codex/Bedrock; SDK v3 update_trace replaces set_trace_io.

Closes #29482, #43129, #72661.
Supersedes #81054, #82332, #42326, #39653, #64292, #32175, #61166, #64797, #43130.
Partially addresses #67544 (capture modes + secret redaction; user_id remains open).

2026-08-13 23:10:16 +05:30

2.9 KiB

Raw Blame History

Langfuse Observability Plugin

This plugin ships bundled with Hermes but is opt-in — it only loads when you explicitly enable it.

Enable

Pick one:

# Interactive: walks you through credentials + SDK install + enable
hermes tools  # → Langfuse Observability

# Manual
pip install langfuse
hermes plugins enable observability/langfuse

Required credentials

Set these in ~/.hermes/.env (or via hermes tools):

HERMES_LANGFUSE_PUBLIC_KEY=pk-lf-...
HERMES_LANGFUSE_SECRET_KEY=sk-lf-...
HERMES_LANGFUSE_BASE_URL=https://cloud.langfuse.com   # or your self-hosted URL

Without the SDK or credentials the hooks no-op silently — the plugin fails open.

Verify

hermes plugins list                 # observability/langfuse should show "enabled"
hermes chat -q "hello"              # then check Langfuse for a "Hermes turn" trace

Generation observations include the Hermes system prompt when the provider uses a separate system param (Anthropic Messages API). Open an LLM call child span to inspect role: system (truncated via HERMES_LANGFUSE_MAX_CHARS).

Optional tuning

HERMES_LANGFUSE_ENV=production       # environment tag
HERMES_LANGFUSE_RELEASE=v1.0.0       # release tag
HERMES_LANGFUSE_SAMPLE_RATE=0.5      # sample 50% of traces
HERMES_LANGFUSE_MAX_CHARS=12000      # max chars per field (default: 12000)
HERMES_LANGFUSE_CAPTURE=sanitized    # content capture mode (see below)
HERMES_LANGFUSE_DEBUG=true           # verbose plugin logging

Capture modes

HERMES_LANGFUSE_CAPTURE controls how much content (prompts, responses, tool arguments/results) is exported. Structural metadata — IDs, roles, tool names, token usage, cost, timing — is always captured in every mode.

mode	behavior
`metadata`	No content. Each content field is replaced by a shape/size stub (`{"omitted": true, "type": "text", "chars": N}`).
`sanitized`	(default) Content is exported after secret-pattern redaction (API keys, tokens, JWTs, private keys, `password=`-style assignments) and truncation. Redaction runs before truncation.
`full`	Raw content, truncated only. Explicit opt-in — traces will contain whatever passed through the conversation, including injected memory and file contents.

The active mode is recorded on every trace as metadata.capture_mode.

Note: sanitized is pattern-based defense in depth, not a DLP guarantee. For personal sessions or shared Langfuse projects, prefer metadata.

Error + shutdown coverage

Failed model requests (api_request_error hook) close their generation with level=ERROR, status code, retry counters, and a capture-mode-scrubbed error message. Non-retryable failures also finish the turn trace.
Session end/finalize closes any still-open traces for that session and flushes queued events, so interrupted or tool-only turns don't dangle.

Disable

hermes plugins disable observability/langfuse

2.9 KiB Raw Blame History