Before the type-ahead fix, an Enter pressed while `hermes chat` started
could save "No thanks" before the offer was on screen, and every surface
then treated the profile as answered. Every answer now also writes
`telemetry.shared_metrics.offer_version`. An "off" answer without it reads
as `reask`: undecided, so the CLI, Desktop strip and dashboard banner offer
it once more with a one-line reason. Any response settles it, including
Esc in the terminal and the banner close, which keep "No thanks", so
nobody is asked a third time. Opt-in answers were always deliberate picks
and are never re-asked.
Review follow-ups for the failure_class split: finish_compression_attempt now
decides skipped/failed from the normalized class it records (a padded or
mixed-case skip name no longer reads failed with a skip reason, and a
non-string class no longer raises); the pre-chat consent offer names the new
reason like the setup copy; the setup copy keeps 80 columns; the doc says a
refused batch's op rows carry the class of the op that stopped it.
Fleet telemetry shows built-in memory writes failing 34-47% and automatic
compression failing ~15%, and several merged fixes each target one cause
(#133038 for memory, #133054/#133040/#133051/#133032 for compression). Both
metrics only carried outcome, so the fleet could not tell which cause a fix
moved.
- hermes.memory.op.count and hermes.compression.count gain one closed-enum
dimension, failure_class; `none` on success.
- Memory: every refusal/failure return in tools/memory_tool.py and
tools/memory_tool_store.py names its class (no_match, ambiguous,
over_budget, would_empty, drift, read_failed, retry_cap, scan_blocked,
missing_content, missing_old_text, invalid_args, stale_entry, disabled,
staged, gate_refused) through a context variable; the result the model
sees is unchanged. Plugin providers report provider_error / exception.
- Compression: the attempt log's own failure_class, collapsed onto the
compressor's closed names (exception:<Type> / rollback:<Type> keep only the
prefix, anything else reads other, a classless failure unknown). Skipped
rows carry the skip reason, and the skip set is now defined once in the
contract.
- Package schema v4 / event schema v4; v3 field sets of both metrics stay
valid so pending rows drain. Doc table rows and the consent copy (CLI
setup and every Desktop locale's collectedActivity) name the new reason.
hermes.compression.count recorded every non-committed attempt as outcome=failed
except lock contention. Structural no-ops (insufficient_messages,
no_compressible_window, empty_post_handoff_window: the transcript is all
protected tail, no model call is made), a user stop (explicit_interrupt) and
superseded/stale attempts all landed in `failed`, so the fleet dashboard read a
session that simply had nothing summarizable as a broken compactor.
The outcome now comes from the resolved failure class (the previous commit keeps
the compressor's class instead of the caller's generic `no_progress`), and those
classes count `skipped`. Real failures (summary errors, would_grow, no_progress
after a real summary, stalls, commit errors) stay `failed`.
The pre-chat shared-metrics offer is drawn a couple of seconds into
`hermes chat`. curses reads any keys already queued on stdin, so an Enter
pressed while Hermes booted selected the preselected "No thanks" and saved
it before the question was ever visible. The profile then read as decided
and no surface asked again. Flush pending input before drawing the offer.
Review fixes for the once-per-episode connect_failed dedupe:
- The episode was latched before the collection-consent gate, so a failure
seen while shared metrics were off suppressed the first consented day's
connect_failed. _record_connect now returns without latching when
collection is off: only a recorded failure opens an episode.
- Local episode state was keyed by the privacy-projected platform name, so
two custom adapters (both emitted as "plugin") shared one episode and one
adapter's recovery closed the other's outage. The key is now the adapter's
real platform value; projection happens only at emission.
- Docs: the hermes.platform.health row describes the episode semantics.
hermes.platform.health recorded connect_failed for every attempt through the
runner's connect seam, and the reconnect watcher retries a failed platform
forever (30s -> 300s backoff). One platform that stayed down for a day
therefore emitted ~300 connect_failed rows against a single connect_ok,
so the fleet dashboard read 61-96% "connect failure" for Telegram, WhatsApp,
email, Signal and BlueBubbles, rates set by a few looping installs rather
than by how many users could not connect.
record_platform_connect now counts a failed connect EPISODE once per
profile home, platform and UTC day: the first failure opens the episode,
watcher retries inside it are silent, and the next connect_ok/reconnect
closes it, so the following outage counts again. A platform that is still
failing the next day counts once more, so daily aggregates keep showing
installs that are still down. A non-retryable failure closes its episode
(nothing retries it). The state is a dict touched only on the
single-threaded metrics worker.
Review fixes for the abandoned-sign-in change:
- A device-code poll that ran out while the token endpoint kept answering
408/429/5xx/WAF chains a ConnectionError cause, so it records failed/network
instead of abandoned; a code left on authorization_pending is still abandoned.
- xAI keeps the wire OAuth error code: access_denied and expired_token are
abandoned, invalid_client stays failed/auth.
- Codex 401/403 at user-code fetch or token exchange raise AuthError (auth), and
the start route's HTTPException carries the worker's recorded class.
hermes.provider_setup.count recorded every user cancel as event=failed
failure_class=cancelled, on every surface: Esc/Ctrl-C/Back in the CLI pickers,
Cancel on a Desktop/dashboard OAuth sign-in (DELETE /sessions), a cancel that
landed before the start response, and consent declined on the provider page.
A device code left to run out was also failed/other on the Nous, xAI and
Codex dashboard pollers (only the free-tier `timeout` reason and Codex
`expired` status were abandoned). Every provider's failure rate was inflated by
walk-aways (openai-codex desktop: 64 of 75 failed rows were `cancelled`).
- provider_setup_fields: failed + cancelled/expired is recorded abandoned, so
every emitter (CLI, web routers, pollers) gets the same answer.
- setup_failure_class: a lapsed device code (DeviceCodeExpired on the Nous poll,
AuthError code device_code_timeout/timeout, RFC 8628 expired_token) is
`expired`; access_denied is `cancelled`; an error whose __cause__ is a network
error is `network`.
- Dashboard/Desktop pollers keep the exception's closed class on the session
(note_oauth_failure) so `status=error` no longer collapses to `other`;
free-tier anon_unreachable is network, a start-route 504 is network.
- DELETE /sessions settles the metric immediately: a Nous/xAI poll blocks for
the code's lifetime, so the cancel row used to land minutes later or never.
Live (real dashboard server, Desktop-owned, isolated HOME, real providers):
base: nous/openai-codex cancel -> failed/cancelled (xai cancel never
settled); nous code left to lapse -> failed/other
fix: nous/xai-oauth/openai-codex cancel -> abandoned/none, each at cancel
Live (real `hermes model` in tmux, Ctrl-C at the OpenRouter key prompt):
base failed/cancelled, fix abandoned/none.
The Home Assistant gateway platform (plugins/platforms/homeassistant) and the
homeassistant toolset (tools/homeassistant_tool.py) now ship as the official
catalog plugin `homeassistant` (NousResearch/hermes-homeassistant), with the
same platform name, toolset/tool names, config keys and env vars.
Migration (hermes_cli/left_core_migration.py, a generic "left core" table
reusing the memory-provider migration's helpers): every profile home that
used Home Assistant (HASS_TOKEN in its .env, platforms.homeassistant enabled
or holding a token, or the homeassistant toolset selected) gets the plugin
installed from the catalog at its reviewed pin by `hermes update`, and lazily
once per process at agent start / gateway start (honours
security.allow_lazy_installs). Every outcome reaches terminal, Desktop and
chat; a gateway-start outcome is handed to that home's first agent.
Core HA special cases become generic PlatformEntry seams the plugin uses:
trusted_inbound (allowlist bypass for event-bus platforms), display_tier
(built-in display defaults) and shared_env_prefixes (profile-clone ownership
rule for env shared with the plugin's tools). Cron delivery uses the existing
cron_deliver_env_var seam; HASS_TOKEN stays stripped from subprocesses via the
installed plugin's manifest secrets.
Every failed auxiliary model_route row read error_class=unknown because the
aux logical scope only carried outcome/model/provider and the projection
hard-coded unknown for any non-success. Every exception escaping call_llm,
including Hermes' own aborts (Ctrl+C / /stop mid-compression, interrupted
aux streams, task cancellation), was recorded as failed.
- classify the error that ended an aux call (and the last one a successful
call recovered from) with agent.error_classifier; HTTP-200 bodies that
carry a provider error object are classified from that object
- Hermes aborts (non-Exception BaseExceptions, InterruptedError) close the
call as cancelled/none, matching the contract
- MoA aggregator streams report the same way
- aux calls that finish under the turn's own live scopes (title generation
runs beside the turn) were never counted: the scope could not be popped
and the turn-close drain closed it as an orphan without a route. The
drain now closes it with the stored real result.
Fleet telemetry labelled every `hermes -z` / `chat -q` / `-Q` run as
entrypoint=interactive because the cli surface mapped straight to
interactive and no caller declared anything else, so 95% of CLI sessions
read as one-turn humans and scripts could not be told apart.
- New closed entrypoint `one_shot` (contract + v3 schema enum). The task
scope start declares it for cli-surface runs carrying the
HERMES_SINGLE_QUERY_SESSION marker the one-shot paths already set (the
signal session source and cache_ttl: auto read). Children Hermes
dispatchers spawn (kanban worker source=kanban, A2A forward source=a2a)
read the existing `background`. Delegated children, other surfaces and
the REPL are unchanged.
- Engagement keeps treating one_shot like interactive (its prior label).
- `hermes -z` exits via os._exit, skipping the atexit hook that closes the
metrics session: its pre-exit cleanup now runs the runtime shutdown, so
task_run.finished and session.count are actually written (before, only
task_run.started survived and the store kept a hot journal).
- Smoke expectations move to one_shot (it drives `chat --query --quiet`).
- relay-shared-metrics.md documents the entrypoint vocabulary.
Follow-up to the off-turn flush worker, found while running the runtime tests:
- The worker was a bare threading.Thread, so the export it runs lost the
contextvars of the turn that scheduled it. _send_exported_packages reads the
profile's send/consent config through them, so under a multi-profile serve the
export resolved the launch profile's config instead of the finishing turn's.
Start it through spawn_context_thread.
- A trigger arriving after the worker saw "nothing pending" but before the
thread finished dying found is_alive() true, set the pending flag and started
no worker, so that flush waited for the next trigger. The worker now clears
its slot under the lock schedulers check, and the slot is set only after
start() succeeds.
MEMORY_PROVIDERS was the plugins/memory/ directory listing, so a provider that
left core for the plugin catalog (hindsight, now honcho) collapsed into
"plugin". Catalog names are public, like the bundled ones.
One answer per profile, asked once, with the Desktop strip's three equal
choices (Send to Nous / Local only / No thanks) now offered by:
- `hermes setup`: at the end of every flow (Quick, Full, Blank Slate,
Portal, --quick), via the setup-completed record they all funnel into.
- `hermes` / `hermes --tui`: once before an interactive chat on a profile
that never answered. Skipped for -q, piped/JSON output, spawned actions
(HERMES_NONINTERACTIVE), Desktop-hosted panes and managed installs.
- Web dashboard: a banner for the profile being managed, backed by
GET/PUT /api/shared-metrics/consent.
"No thanks" is the terminal default so Enter never opts anyone in; Esc and
the banner's X leave the question open.
Also fixes the answer that could not stick: a CLI "no" equals the shipped
defaults, so save_config stripped it and every surface (Desktop strip
included) kept asking. The answer is now written to config.yaml directly,
and `decided` has one definition shared by the RPC, the CLI and the router.
Main removed the intro cinematic (#126139): the gate no longer has a
`cinematic` phase and store/intro-reveal is gone. The funnel stops recording
an `intro` step (dropped from the contract and v3 schema, never shipped) and
the consent strip no longer waits on the intro film.
A volume simulation through the real store showed two costs worth cutting.
Rows: hermes.task_run.finished carried duration, retry, model-call and
tool-call buckets on top of outcome/end reason/surface, so almost every task
became its own row; hermes.tool_call.count did the same with latency and
retry buckets. The terminal rows now keep only the fields they are filtered
by, and the same end events feed two small split counters:
hermes.task_run.duration (surface, outcome, duration, retries) and
hermes.tool_call.latency (tool category, latency, retries). Per-task call
counts already ride on hermes.task_cost.count. v3 still accepts the v2 field
sets of both counters so rows recorded before an upgrade drain.
Local store: every package lived three times for 30 days (counter rows, the
payload column, a pretty-printed outbox file). Outbox files are now compact
JSON, and once the ingest has accepted or refused a package the database
drops its copy of the body; the file stays as the local history the docs
promise, and pending packages keep their body for resends.
Measured (simulated day, skewed draws, before -> after):
heavy 2,788 -> 2,347 rows, 21.0 -> 17.7 KB on the wire, 102 -> 44 MB local over 30 days
extreme 6,561 -> 5,047 rows, 46.0 -> 33.6 KB, 272 -> 102 MB
typical 663 -> 650 rows, 7.1 -> 6.8 KB, 24 -> 13 MB
Azure calls a model by a deployment name its owner chose, which can name
a team, customer or environment. On azure* providers the model id now
passes only when it is in Hermes' static catalogs or the local models.dev
cache (never a network call); anything else reads custom.
The M13 fix stopped counting any key a tool panel also asks for, since a
bare PUT /api/env cannot tell connecting a provider from configuring a
tool. Desktop onboarding and model settings now send provider_setup=true
and the server counts those saves; the Keys page and tool panels keep
the exclusion.
Only the Desktop consent RPC purged desktop_onboarding/; opting out with
hermes config set or the setup wizard left the latches until the Desktop
switch was next flipped. An opted-out process start now removes them
with the other pending markers.
In a multiplexed gateway the final send runs after the routed profile
scope is reset, so delivery rows went to the launch profile while the
same turn's reply_latency went to the owner. start_reply_clock now also
remembers the owning home per (platform, chat) in a bounded map that
outlives the clock, and records_delivery submits through it.
model_route('lm-studio', id) kept the user's model id because only the
canonical lmstudio was in user_named_model_providers while the alias
passed provider_metric_name as a shipped name. Bedrock ARNs carry the
AWS account id. Both now read custom.
hermes.execution_backend.count still counted terminal/browser/code calls made
under the background_review write origin (the review and curator forks),
while hermes.terminal.outcome and file_edit already skip them. Owner ruling:
match terminal.outcome. record_execution_backend now returns early when
is_background_review() is set (one contextvar read, before any work).
Doc: the execution_backend row lists the forks as not counted.
Probe probe_internal.py (terminal_tool inside a background-review origin):
before: [execution_backend {backend: local, kind: terminal, outcome: success} x1]
after: []
test_background_review_fork_terminal_calls_are_not_backend_usage is RED on base
(2 rows != 1), green here.
m22: migrations ran save_config -> record_config_saved inside surfaced
processes (`hermes config migrate`, console migrate, dashboard/TUI profile
create), so e.g. v45 appending `connections` to platform_toolsets read as a
user re-enabling a toolset. _persist_migration (the single migration write
path) now wraps save_config in hermes_applied_write(), a ContextVar the hook
honours.
m23: the old side is the raw file (`memory_enabled: ${MEM_ON}`) and the new
side the env-expanded config, so any unrelated save reported `disabled` once a
day forever. A setting whose value on either side is an unexpanded `${...}`
string is now skipped (its value is unknown to the diff).
m24: dashboard handlers call save_config inside config_write_scope
(_CONFIG_MUTATION_LOCK), and the diff (tools_config._get_platform_tools per
platform) + record ran there. record_config_saved now runs only the cheap
gate inline (surface, enabled, surface/home resolution) and hands deep copies
to a non-daemon thread bound to the owning profile (non-daemon so a CLI
command exiting right after its write still records).
Test contract change: test_record_config_saved_needs_a_surface_and_collection
and test_real_config_writes_report_the_move_away_from_default now join the
recording thread before asserting (the record is asynchronous by design).
Probes (review/signals/mine):
p2_migration.py config|dashboard
before: re_enabled toolset connections row; after: <no db>
p4_envref_configset.py
before: memory.memory_enabled disabled after an unrelated save;
after: no row (config set curator.enabled false still records)
p3_lock_v2.py (times config_transitions, try-acquires the web lock)
before (p3_lock.py): record_config_saved 0.047s/0.004s, lock held=True
after: diff 0.003-0.008s, lock held=False
Tests test_migrations_and_env_templates_are_not_user_disables and
test_diff_and_record_run_off_the_callers_lock_in_the_owning_profile are RED on
base.
- `curator` latched on the default-on scheduled pass (outcome=success, all
buckets 0), so it meant "idled through one curator interval". It now needs
trigger=manual (the user ran it).
- `skills_created` and the v4 milestone `first_skill_created` latched when
the background-review fork created a skill (provenance=agent_created).
Both now require provenance != agent_created (foreground creates are
stamped "learn" -> not agent_created, see tools/skill_usage.record_created).
- days_since_install_bucket caught only sqlite3.Error; a non-numeric
sessions.started_at raised ValueError into the subscriber on every event,
re-opened state.db each time and never latched. It now also catches
OSError/TypeError/ValueError -> `unknown`.
Probes:
probe_internal_adoption.py
before: feature_adoption curator + skills_created, milestone first_skill_created
after: no feature_adoption rows, no milestone
probe_statedb_errors.py textstamp
before: ValueError warnings, 0 rows, settled 0 of 1
after: 1 state.db read, feature_adoption {unknown, projects}, settled 1 of 1
Tests test_hermes_internal_work_never_latches_adoption and
test_unreadable_first_session_reads_unknown are RED on base.
tool_unavailable_fields compared against agent.valid_tool_names, the
model-visible array after tool-search assembly. With the default
tools.tool_search.enabled=auto, 14 enabled built-ins (session_search,
todo_list, process_manage, ...) live behind the tool_call bridge, so a direct
call to one was reported as "disabled in this session". Cron agents had
clarify stripped on purpose by the scheduler and still counted.
- Names in the session's tool-search scoped set
(agent.tool_executor._tool_search_scoped_names: the deferrable names its
enabled/disabled toolsets allow, cached on the agent) are enabled, not
unavailable. A session that turned the toolset off still reports it.
- platform == "cron" returns None, like delegated children and background
review.
Probes:
probe_deferred_tools.py (real AIAgent, default config)
before: rows for session_search and todo_list; after: no rows
probe_cron_narrowed.py
before: tool_name=clarify row; after: no rows
Test test_deferred_builtins_and_cron_narrowing_are_not_unavailable is RED on
base.
begin_process's collection-off branch purged only parked update receipts.
A provider_setup marker (pid, start_time, surface, provider) or a process
exit marker written while collection was on stayed on disk through the
opt-out and was reported (`abandoned` / `killed`) once collection came back
on, in a period the user had not consented to.
The same branch now removes process_markers/ and provider_setup_markers/.
Probe p2b_off_then_on.sh:
before: marker survives two OFF starts; the ON start emits
`abandoned openrouter`
after: markers {} after the first OFF start; the ON start emits only its
own started/completed rows
Test test_opted_out_start_purges_markers_left_while_opted_in is RED on base.
Every PUT /api/env write of a provider-shaped variable recorded
started+completed: an empty value (a clear), a same-value re-save,
GITHUB_TOKEN/GH_TOKEN (-> copilot), HF_TOKEN, and keys the TTS/STT/image
panels also save (GEMINI_API_KEY, XAI_API_KEY, DEEPINFRA_API_KEY). TUI
model.save_key counted re-saves; editing a custom endpoint counted as a new
setup.
- record_api_key_saved takes the value and the stored value read before the
save: only a non-empty, changed value counts.
- provider_for_api_key_env returns None for ecosystem tokens (GITHUB_TOKEN,
GH_TOKEN, HF_TOKEN) and any key a TOOL_CATEGORIES panel asks for: a bare
key save cannot tell a provider connection from a tool setting (the PUT
body has no provider flag; adding one is Desktop work, see NOT_COVERED).
Provider pickers (CLI, TUI model.save_key with its explicit slug) still
count those providers.
- model.save_key skips a same-key re-save.
- upsert_custom_endpoint records only when the endpoint did not exist.
Probe p4_web_double.py (real route functions):
before: openrouter started x3/completed x3 (save, re-save, clear),
copilot x1 from GITHUB_TOKEN, custom x3 (create + 2 edits)
after: openrouter x1, no copilot row, custom x1
Tests test_web_forms_count_only_a_new_provider_key_or_endpoint and the
extended test_api_key_env_maps_to_provider_only are RED on base.
cli_provider_setup treated the setup menus' BaseException control flow as
errors: Esc (_SetupCancelled) recorded failed/other, and Back (_SetupGoBack)
recorded failed/other and then a second `started` when the wizard replayed
the provider picker.
- setup_failure_class maps _SetupCancelled to `cancelled`.
- Back leaves the flow open on the thread-local; the replayed picker
re-entering the same surface+provider continues it (no new `started`).
Picking another provider, or leaving the entry point
(provider_setup_surface exit other than a Back), ends it failed/cancelled.
Probe p5_cli_nav.py (real cmd_model):
before: back_then_pick started x2, failed/other x1, completed x1;
escape started x1, failed/other x1
after: back_then_pick started x1, completed x1;
escape started x1, failed/cancelled x1;
back_then_esc_menu started x1, failed/cancelled x1;
back_then_other openrouter started+cancelled, anthropic started+completed
Test test_cli_setup_navigation_esc_cancels_and_back_resumes_one_flow is RED
on base (failed/other, extra started).
m13: the renderer stored and sent raw plugin palette/keybinding ids, rage-click
targets and config paths below user-named containers (providers.<name>.api_key);
the backend collapsed them, but only after truncating the action list to 200.
- recordAction collapses anything outside the built-in ids (KEYBIND_ACTIONS +
KEYBIND_READONLY + the named buttons, minus numbered slots — exactly the
backend's DESKTOP_ACTION_IDS) to `other` before it is counted or used as a
rage-click target.
- recordSettingsSaved sends only keys the config schema publishes (the Settings
page passes its schema: DEFAULT_CONFIG leaves), so nothing below a user-keyed
container leaves or is kept.
- Backend DAILY_ACTION_ROWS_MAX = |DESKTOP_ACTION_IDS| x |vias| (296): every
distinct collapsed (action, via) fits, so none is cut before collapsing.
m14: first-run steps happen before the consent question and collection defaults
off, so every step but consent/first_message was dropped. Chose the in-memory
buffer (smaller than removing the steps from contract, schema and TS): while the
answer is unknown/undecided, recordOnboarding/closeOnboardingStep calls are held
in memory (max 100, never persisted or sent) and replayed on the switch turning
on; a decided "off", a profile switch or quitting discards them.
setDesktopMetricsGate takes `decided` (hook passes consent.decided).
Tests (desktop-metrics.test.ts): recordSettingsSaved callers pass a schema
(contract change). New invariants, RED on base: plugin action id / rage target
stored and sent as `other`; a providers.<name> key is not sent; pre-consent
onboarding held, sent on yes (with closes applied), dropped after a decided no.
Probe (review/desktop/jsrun/.../desktop-metrics.probe.test.ts, raw ids):
before: localStorage actions {"acme-plugin:/home/alice/clients/bigco|palette":1, ...|click:3};
wire rage_click target "acme-plugin:/home/alice/clients/bigco", setting
"providers.acme-corp-internal.api_key"
after: localStorage actions {"other|palette":1,"other|click":3}; wire
[rage_click target "other"] only
m16. shared_metrics.set(enabled=false) purged the renderer's onboarding latch
but left telemetry/shared_metrics/desktop_onboarding/<step>.<event> in the
profile, so a content-free per-profile record outlived the opt-out and the two
sides disagreed on re-opt-in. The Desktop's consent RPC now removes that
directory when it turns collection off (shared_metrics_desktop.
purge_onboarding_latches). Opting out from the CLI (config set / setup wizard)
leaves them until the Desktop's switch is next turned off — see NOT_COVERED.
Test: test_turning_collection_off_drops_the_onboarding_latches — RED on base
(latch file survives), green now.
Probe (review/desktop/probe_optout_set.py):
before: surviving desktop latches: [.../desktop_onboarding, .../intro.reached, ...]
after: surviving desktop latches: [] (only metrics.sqlite3 / outbox remain)
M10 + m17 (+ the backend half of M9). The backend's once-per-day latches for
hermes.desktop.{action_use,mode_use} (the finished-day report) and
hermes.desktop.feature_use lived in process memory, so a second window's report,
a backend restart or a pooled backend re-counted the same day/area. The daily
rows also went through the relay's current-day marks, so a day flushed a week
late was counted in the flush day's period.
- record_desktop_daily now writes through SharedMetricsStore.update_rollup_state:
one write transaction checks a `desktop_daily_reported` state row (the usage
days already reported, bounded to the last 8 days) and records the rows with
period_start = the usage day. Days outside [today-8, today] settle unrecorded.
- record_desktop_feature_use uses store.record_counter_once_per_day (the same
durable per-UTC-day-per-dimension latch feature_disabled uses).
- With collection off, desktop_daily answers recorded:false instead of true, so a
renderer that has not learnt the switch yet never treats that as "settled" and
drops the day (the client's own gate purges it when it reads off).
- Neither metric has feature/milestone side effects in the subscriber, so
writing to the store directly is equivalent apart from the latch and period.
Tests: the fixture mocked record_process_marks_saved; the daily/feature-use
assertions now read the profile's store (their contract changed: rows no longer
go through the relay). The disabled-profile test now expects recorded:false.
The store-refusal test raises from update_rollup_state instead of mocking the
relay's saved count. New invariant test (two windows / a restarted backend, all
in-process state reset between reports): RED on base, green now.
Probe (review/desktop/probe_latch.py, run twice on one home = backend restart):
before: run 2 → action_use composer.send 2, feature_use terminal_pane 2,
mode_use 2, all in period 2026-09-28 for usage day 2026-09-20
after: run 2 → action_use 1, feature_use 1, mode_use 1; action_use/mode_use in
period 2026-09-20. Opted-out home: daily -> {'recorded': False}.
What:
- m1: record_known_cache_break emits only when it is the first announcement for the pending cold
read (the first cause names the break); several causes before one cold read were one row each.
- m5: enabled() while OFF returns before taking the process-wide _RUNTIME_LOCK when no runtime
exists for the profile (nothing to tear down).
Tests: compression x2 + model_switch before one cold read -> one compression row (RED on the
previous commit: compression 2 + model_switch 1). Existing cache_break test unchanged and green.
Measured (uncontended, fakehome): enabled() OFF 17.72 -> 17.24 us/call. The lock was the
cross-thread serialisation point, not the bulk of the cost: _raw_config() is 13.0 us of it (see
NOT_COVERED.md).
What:
- M2: engagement.record reads the clock inside the rollup write transaction, so writers apply in
clock order and no process closes a day another already closed. apply() never moves the stored
day backwards: an interaction that lands behind the stored day folds into it (the only close path
is a later clock), instead of being dropped or re-opening a day.
- m7: SharedMetricsStore counter and rollup writes that hit the 250 ms busy timeout are kept in a
bounded in-memory queue (4096) and land with the next write, the next package export, or an exit
drain (atexit, 5 s busy timeout, registered only after a deferral). The caller still waits at most
250 ms; a non-busy error drops only the failing write.
Why: the Relay subscriber thread saw "database is locked" and lost the row (engagement under
multi-process contention; model-call counters under host load).
Tests (RED on the previous commit): a write while another connection holds BEGIN IMMEDIATE ->
no raise, lands with the next write (value 2); a late interaction folds into the newer day and the
closed day reports once.
Stress: 20 parallel copies of the existing
test_cross_process_model_call_updates_are_transactional under -n 12 (load avg 6-15, a second
-n 12 suite running): fixed store 10 rounds -> 200/200 passed; previous store 3 rounds -> 55/60
(5 failed: child exit 1, database is locked).
What:
- M5: `relay` is a gateway surface alias, so an inbound the connector could not stamp is
entrypoint gateway_message / surface gateway (it counts as engagement) instead of other/none;
`relay` joins the task/session gateway platform set (it was collapsing to `plugin`).
- m9: RelayAdapter._metrics_platform labels a chat by its inbound platform, and falls back to the
connector's primary platform only when the socket fronts exactly one identity; _source_platform
uses it (egress routing keeps _chat_platform unchanged).
Why: the same turn was `other/none` in task/session while delivery/latency said `discord`; an
unknown chat on a multi-platform connector was misattributed to the primary platform.
Tests: the existing unresolvable-chat stub now implements _metrics_platform (renamed hook). New:
task_start_fields({"platform": "relay"}) -> gateway_message/gateway/relay; unstamped chat on a
two-identity connector -> delivery platform `relay`. Both RED on the previous commit
(other/other/none; discord).
Probe (review/engagement/relay/probe_d_coverage.py), before: source.platform='relay' ->
{'entrypoint': 'other', 'execution_surface': 'other', 'platform': 'none'}, engagement surface None.
After: the contract test above.
Not done: resolving the turn-start task platform to the connector's platform (needs the adapter at
agent construction) and M6 (delivery row profile under multiplex); see NOT_COVERED.md.
What:
- M1: the engagement turn mark (primary model) needs a user turn (pre_llm_call) on an interactive
surface or a gateway message; API-server, python embedding and curator turns no longer create
engagement.day rows or outvote the person's model. close_day emits nothing for a day with no
engaged surface (the root profile's host row with an active-profile count stays).
- M3 (owner ruling): unattended cron runs are dropped from engagement entirely. surface_day always
carries an active-minutes bucket, so it cannot express cron presence without minutes; hermes.cron.run
already counts runs. The `cron` enum value stays in the contract for already-stored rows.
- M7: background review forks (no pre_llm_call) no longer advance model_switch_after's turn run.
- M8: the run observes the route each turn was sent on (its first request), so a provider failover
neither restarts the run nor names the fallback as the model left.
- m6: documented that the root host row (surfaces_used_count 0) is excluded from days-active counts.
Why: Hermes-internal and unattended activity is not user engagement; switch_after measures how long
a person stayed on the model they chose.
Tests (RED on the previous commit): API/cron-only day -> no day row, mixed day -> primary = person's
model; failover + review fork -> switch_after names the selected model with 2_to_3.
Probe (review/engagement/probe_internal.py, switch/probe_switch.py -k "p3c or p7"), before:
API-only / curator-only day -> engagement.day {"primary_model": "openai/gpt-5", "surfaces_used_count": "0"}
cron all day -> day {"active_minutes_bucket": "gte_6h", "active_profile_count_bucket": "1"} + surface_day cron
2 CLI + 5 API turns -> primary openai/gpt-5
P3c (6 turns on A, 7th failed over to B) -> {"model": "gpt-5", "turns_before_switch_bucket": "1"}
P7 review fork -> 4_to_10 (routed=False) / gpt-5-mini "1" (routed=True)
after: the equivalent invariant tests pass (no day row, primary = PUBLIC, selected model 2_to_3).
What: the metrics lineage no longer keys off the ambient Portal conversation id (it walks every
parent_session_id: gateway reset/idle expiry, /new, /branch). Compression calls
relay_shared_metrics.rotate_segment(old, new) from the committed rotation
(_notify_context_engine_compression_complete) and from a stale agent adopting the live tip
(_adopt_live_compression_child). The hand-off registers the new id in the lineage, moves the
route run and the spent-turn history into it, and closes the old segment now (or when its in-flight
compression turn ends). A rotated-to id closed before serving a turn, or still open at shutdown,
closes too; a 512-lineage backstop flushes conversations a surface never closed.
- m2: /undo or /retry on the new id reaches the turns earlier segments spent (tokens_bucket known).
- m11: /model right after a rotation finds the conversation's turn run.
- m8: background review forks (same session id, no pre_llm_call) add no turns/calls/messages.
Why: one conversation = one session / context_peak / tool_overhead / tool_enabled_unused row, on
time; N gateway conversations collapsed into one row emitted only at process exit, and rotated
segments stayed in memory until exit.
Tests: two existing tests simulated rotation by sharing a conversation context; their contract
changes to the explicit hand-off (test_a_compressed_conversation..., test_context_peak_is_one_row...).
New: reset/branch chain after a compression + tip-first close order, the compression seam, switch/undo
right after a rotation, review fork volume. RED on base (rotate_segment missing / switch_after [] /
turn bucket '2' for one user turn).
Probe (review/engagement/lineage probe_lineage.py -k c2, probe_branch.py; fix-lineage/probe_handoff.py):
before: c2 3 conversations -> session.count 0 until shutdown, then 1; branch -> 1; 50 compressed
conversations -> 0 rows, 50 lineages/sessions live.
after: c2 -> 3 rows before shutdown (turns 2,1,1), lineages=0; branch -> 2; 50 compressed -> 50 rows,
0 lineages, no sessions; tip-first close with the old turn running -> 0 then 1; shutdown with an
open tip -> 1 per conversation.
What:
- shared_metrics_catalog.model_metric_name also collapses a model id that is
a network address: a loopback/IPv4 host or localhost at the start, or a
leading host:port with a 2-5 digit port (so Bedrock's "...-v1:0" and
OpenRouter's ":free" / Ollama's ":120b" tags stay readable). The existing
URL/path/weight-file rules move into the same compiled pattern.
- shared_metrics_contract._bounded_dimensions (every mark -> counter
projection) and model_token_counters re-run provider_metric_name /
model_metric_name on the mark's provider/model identifier fields and drop
the row when either would rewrite it ("none" = unset provider passes).
Record-time only: counter_dimensions_are_valid (also used at packaging) is
unchanged, so catalog drift can never raise out of _package_metric and
block a package of rows already stored.
Why: `127.0.0.1:8080/x` on a public provider passed verbatim through every
route metric, and the subscriber validated identifier shape only, so a mark
that skipped the producer's catalog pass stored custom:acme, ollama/<model>,
unknown vendors and even http:// URLs.
Probe (fix-catalog/probe_b1_m12.py):
before: route(openrouter, 127.0.0.1:8080/x) -> model '127.0.0.1:8080/x'
(also localhost/qwen, 10.0.0.5/x, gpu-box.lan:8000/qwen verbatim)
subscriber injected switch_after {127.0.0.1:8080/x, openrouter} -> stored
subscriber injected switch_after {acme-secret, acmecorp-internal} -> stored
after: all of the above -> model 'custom'; both injected marks -> None;
bedrock anthropic.claude-3-5-sonnet-20241022-v2:0, openai/gpt-4o:free,
gpt-oss:120b and the legit anthropic mark unchanged.
Test: test_network_address_model_ids_and_raw_injected_marks_never_reach_counters
(RED with the catalog reverted: model_route kept 127.0.0.1:8080/x; RED with the
contract reverted: the injected host:port mark was stored; GREEN with both).
What: shared_metrics_catalog.provider_names() no longer reads the live
PROVIDER_REGISTRY / models._KNOWN_PROVIDER_NAMES. It is now the built-in
auth rows (BUILTIN_PROVIDER_IDS), HERMES_OVERLAYS, the three static alias
tables, openrouter/custom, and the names+aliases of profiles registered
from the in-tree plugins/model-providers dir (providers._SOURCES ==
"bundled", process-wide layer, independent of the bound profile home).
Why: auth_plugin_providers mirrors every $HERMES_HOME (and pip) provider
plugin, plus its aliases, into PROVIDER_REGISTRY and the picker labels, so a
user-chosen provider name was treated as shipped and its model id was not
collapsed either. Every consumer goes through provider_metric_name, so the
one fix covers model_route and every per-model v4/v5 metric (tokens,
context_peak, friction, tool_quality, loop_guard, tool_recovery,
model_reply_issue, task_cost, wasted_tokens, cache_break, switch_after,
tool_unavailable, engagement.day), provider_setup (all surfaces, incl. the
PUT /api/env key path), the on-disk pending marker, setup.completed,
model_switch/fallback and install.snapshot main_provider. Pip-installed
provider plugins now read custom too (third-party, not shipped); doc says so.
Probe (fix-catalog/probe_b1_m12.py, user plugin acmecorp-internal alias
acme-llm):
before: provider_metric_name('acmecorp-internal') -> 'acmecorp-internal'
route: {'model': 'acme-secret-model-v2', 'provider': 'acmecorp-internal'}
marker on disk: {..., "provider": "acmecorp-internal"}
after: provider_metric_name('acmecorp-internal') -> 'custom' ('acme-llm' too)
route: {'model': 'custom', 'provider': 'custom'}; marker "provider": "custom"
install main_provider custom; switch/fallback/setup custom
shipped plugin deepinfra: {'model': 'meta-llama/...', 'provider': 'deepinfra'}
Test: test_user_provider_plugin_name_and_model_never_leave (RED on base:
marker carried "acme-llm"; GREEN after).
- hermes.tool_unavailable.count: model called a shipped built-in (BUILTIN_TOOL_NAMES) not enabled in
the session; tool name + catalog provider/model. Unknown names stay in v4 unknown_tool quality.
- hermes.provider_setup.count: started/completed/abandoned/failed per provider and surface
(cli_setup, cli_model, tui, desktop, dashboard) with a closed failure_class. Pending marker per flow;
a dead or stale marker is reported abandoned at the next start (v4 process-marker pattern).
- hermes.feature_adoption.count: once per feature per install at first real use, derived from existing
counters in the subscriber (metric -> feature table) plus Bot Mode / Projects hooks; bucketed by the
owning profile's install age.
- hermes.feature_disabled.count: turning off a default-on toolset/skill/plugin/platform/setting (and the
re-enable back to default), diffed at the config write chokepoints; once per (kind,name,event)/day.
Surface comes from the user entry point; setup/migrations record nothing.
Desktop love/hate telemetry needs a backend half: six counters
(feature_use, action_use, mode_use, friction, dislike, onboarding) with
closed dimension sets, the v3 JSON schema entries, the public doc rows,
and profile-scoped gateway RPCs the app reports through.
Every value the app sends is collapsed server-side onto the closed set
(unknown -> other); feature use latches per area per UTC day, friction
and dislike are capped per day, the daily report (action counts + mode
time + bot count) latches per day and answers recorded=false on a store
failure so the app keeps the day for a retry. For setting changes the
app sends only the config key: the backend reads the saved value and
compares it to DEFAULT_CONFIG itself, so values never cross the wire;
keys outside DEFAULT_CONFIG read other. All helpers check enabled()
before doing any work.
Engagement (local daily rollup, emitted once the UTC day closes, exactly once per profile):
- hermes.engagement.surface_day.count {surface, active_minutes_bucket} per surface used
(cli/tui/desktop/gateway/acp/cron); active time = inter-interaction gaps capped at 5 min.
- hermes.engagement.day.count {active_minutes_bucket, surfaces_used_count, primary_provider,
primary_model, active_profile_count_bucket}; primary model = most attended turns that day.
Active profiles are folded into one host accumulator in the root profile's store (opaque local
hashes); only the root profile's row reports the count, other profiles report 0.
- Days-active-per-week / return-by-model are derived server-side; no weekly state, no new id.
Model satisfaction:
- hermes.model_switch_after.count {provider, model, turns_before_switch_bucket}: turns the model
being left served in the conversation, from every /model site (CLI, TUI, ACP, gateway).
Decisions:
- hermes.session.count counts once per conversation across compression rotation (merged over the
v4 conversation lineage), and gains message/model-call/tool-call count buckets with a longer
tail (251_to_1000, gte_1001); rows recorded before the upgrade still package.
- Relay-delivered hermes.gateway.reply_latency and hermes.platform.delivery report the originating
platform instead of `relay`.
Six opt-in, bucketed shared-metrics counters (schema + contract + docs in `v5 efficiency` blocks):
- hermes.task_cost.count {provider, model, tokens_bucket, tool_calls_bucket, api_calls_bucket,
outcome}: one row per user turn the user saw end (pre_llm_call-started, attended; forks, delegated
children and session-close aborts excluded). Tokens = prompt+completion over the turn's primary
calls. Tool/API call buckets reach gte_101 (TURN_ACTIVITY_BUCKETS; COUNT_BUCKETS untouched).
- hermes.wasted_tokens.count {provider, model, reason in undo/retry/interrupt, tokens_bucket}:
emitted from the v4 friction sites (no re-detection). /undo N counts N turns; an interrupted turn
later undone counts once; turns this process never saw read unknown.
- hermes.tool_output_truncation.count {tool, truncated, original_size_bucket}: one row per tool
result, judged after the per-turn budget; covers the per-result spill, the turn budget, and the
shared head/tail notice tools write when they cut their own output (original size reported).
- hermes.tool_overhead.count {enabled_tool_count_bucket, tool_schema_tokens_bucket,
execution_surface} + hermes.tool_enabled_unused.count {toolset (shipped TOOLSETS else custom),
used}: once per closed interactive conversation (merged across compression lineage).
- hermes.cache_break.count {provider, model, cause}: compression (committed), model_switch /
system_prompt_rebuild (continuing conversation rebuilt its prompt), toolset_change (tool array
changed mid-conversation, Bot Chat capability rebuild), provider_reported_miss (cold read after a
warm read on the same route with no Hermes-known cause), cache_expired (same after >=5 min idle).
A Hermes-known break suppresses the miss it causes. No prompt hashing.
Live-proven against a fake OpenAI-compatible server (chat -q, --resume -m, TUI gateway JSON-RPC
undo/retry/interrupt); disabled => zero telemetry files.
Five opt-in counters that tune the agent loop itself, recorded through
record_process_mark (disabled config => zero rows, zero files):
- hermes.file_edit.count: tool, mode, outcome, match_strategy. The fuzzy
matcher reports the strategy that landed (or no_match / ambiguous) into a
context-local probe opened only around the patch/write_file handlers, so we
learn which strategies earn their keep without counting internal callers.
- hermes.loop_guard.count: provider, model, signal, detector. Hooked where
the guardrail already warns/blocks/halts and at turn end for the iteration
budget; latched once per turn per (signal, detector).
- hermes.tool_recovery.count: provider, model, tool, next_tool,
next_outcome. One row per failed tool call, resolved against the model's
next round (same tool first) or the turn end (no_tool_call / gave_up).
- hermes.terminal.outcome.count: backend, command_kind, outcome. The kind is
a table lookup on the first program word, never the text. Hermes' own
deadline/interrupt now set hermes_timed_out / hermes_interrupted on the
backend result, so a command's own `exit 124` reads nonzero, not timeout.
Disjoint dims from hermes.execution_backend.count (served vs not).
- hermes.model_reply_issue.count: provider, model, issue. Refusal and
truncation only from the structured finish reason; empty/reasoning_only
from the normalized message; issue=none per response as the denominator
(model_route counts attempts and files empty replies as failures).
Background review / curator loops and _host_local / unmetered terminal
calls never count. Helpers live in shared_metrics_harness.py; call sites
are one or two lines each.
- m9: shared_metrics.startup_latency had no server-side latch, so any repeat
call added a row. The TUI and Desktop now send an opaque per-launch id
(TUI: one per process; Desktop: minted when the once-per-launch main-process
claim succeeds) and the backend claims (surface, launch_id) once per
process: a reconnect re-sending the same launch counts once, a new Desktop
launch against a long-lived backend still counts. Legacy clients without an
id count once per backend process and surface. The id is only a local latch
key (bounded set, length-capped), never recorded. Only a usable measurement
spends the claim. Contract + apps/shared regenerated.
- m10: relaunch() execs in place, keeping the PID, so `sessions browse` ->
resume counted the picker time as CLI startup. relaunch() stamps its PID in
HERMES_RELAUNCHED_PID right before exec; process-start surfaces skip when it
matches, while children (other PIDs) inheriting the env still count.
- m8 (psutil half): record_process_ready checks the opt-in gate before reading
the process start time (psutil) or starting a thread. The claim is taken
first so a later opt-in never records a stale "startup" mid-session.
Test changes to existing assertions: the TUI/Desktop param assertions now
expect launch_id (a new wire param); the RPC surface test sends distinct
launch ids so its three calls stay three launches under the new latch.
Two false negatives in the install snapshot's local-server flag (review m11):
- an auxiliary slot under provider auto / no provider with base_url + api_key
was skipped as "inherits main", but agent.auxiliary_client routes exactly
that shape to the endpoint as custom; the filter now mirrors the runtime;
- a providers:/custom_providers: entry referenced by bare name (provider:
home-llama) was only resolved for the custom:<name> spelling. The lookup
now goes through _get_named_custom_provider for any non-inheriting id, so
built-ins still shadow entries exactly as at runtime.
Review findings M7, M8, m6, m7 and the browser half of m8 on the v4 loop metrics.
- M7: TUI/Desktop @-path completion on a non-local backend lists the directory through
terminal_tool, so every keystroke added a hermes.execution_backend.count row as if the user
ran a command. Hermes-owned calls now run inside shared_metrics_loop.unmetered_backend_calls()
(a contextvar checked in record_execution_backend) instead of a flag threaded through
terminal_tool. It was the only non-_host_local internal terminal_tool caller (bot DM runners
already use _host_local; the prompt-builder probe calls env.execute directly).
- M8: a scheduled curator tick that finds the run claim held recorded scheduled/skipped every
tick (5 ticks -> 5 rows for one pass). The holder's own pass is the one counted; the held
branch now records nothing. Real dry runs still report skipped.
- m6: a foreground timeout returns exit 124 with partial output and no error field, so it was
recorded as success. exit 124 now records outcome=failed, error_class=timeout.
- m7: vercel_sandbox (and managed_modal) are built-in backends in
tools/terminal_tool_config._BUILTIN_BACKENDS but collapsed to other. Added to
TERMINAL_BACKENDS and both schema enums; a test pins every built-in backend to its own bucket.
- m8 (browser): record_browser_call resolved the browser backend on every call even with
collection off. The resolver is now passed lazily and only runs inside the gated builder.
The existing test asserting a held claim records scheduled/skipped encoded M8 itself and is
replaced by one asserting held ticks record nothing.
