teknium1
8d8836ddb1
feat(telemetry): harness-accuracy shared metrics for the agent loop
...
Five opt-in counters that tune the agent loop itself, recorded through
record_process_mark (disabled config => zero rows, zero files):
- hermes.file_edit.count: tool, mode, outcome, match_strategy. The fuzzy
matcher reports the strategy that landed (or no_match / ambiguous) into a
context-local probe opened only around the patch/write_file handlers, so we
learn which strategies earn their keep without counting internal callers.
- hermes.loop_guard.count: provider, model, signal, detector. Hooked where
the guardrail already warns/blocks/halts and at turn end for the iteration
budget; latched once per turn per (signal, detector).
- hermes.tool_recovery.count: provider, model, tool, next_tool,
next_outcome. One row per failed tool call, resolved against the model's
next round (same tool first) or the turn end (no_tool_call / gave_up).
- hermes.terminal.outcome.count: backend, command_kind, outcome. The kind is
a table lookup on the first program word, never the text. Hermes' own
deadline/interrupt now set hermes_timed_out / hermes_interrupted on the
backend result, so a command's own `exit 124` reads nonzero, not timeout.
Disjoint dims from hermes.execution_backend.count (served vs not).
- hermes.model_reply_issue.count: provider, model, issue. Refusal and
truncation only from the structured finish reason; empty/reasoning_only
from the normalized message; issue=none per response as the denominator
(model_route counts attempts and files empty replies as failures).
Background review / curator loops and _host_local / unmetered terminal
calls never count. Helpers live in shared_metrics_harness.py; call sites
are one or two lines each.
2026-09-28 12:43:03 -07:00
..
2026-09-28 10:21:41 -07:00
2026-09-28 12:36:05 -05:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 12:43:03 -07:00
2026-09-24 14:47:06 -04:00
2026-09-28 06:42:00 -07:00
2026-09-24 06:03:29 -04:00
2026-09-23 12:02:05 -04:00
2026-09-25 17:57:38 -05:00
2026-09-23 11:41:01 -04:00
2026-09-27 03:37:09 -07:00
2026-09-27 02:55:39 -07:00
2026-09-28 10:21:41 -07:00
2026-09-22 09:55:47 -04:00
2026-09-24 02:03:59 -04:00
2026-09-28 03:37:09 -07:00
2026-09-23 11:41:01 -04:00
2026-09-24 18:06:48 +05:30
2026-09-25 21:27:06 +05:30
2026-09-27 18:37:01 +05:30
2026-09-28 10:21:41 -07:00
2026-09-26 21:38:28 +05:30
2026-09-28 10:21:41 -07:00
2026-09-24 02:03:59 -04:00
2026-09-25 14:22:12 -04:00
2026-09-28 10:21:41 -07:00
2026-09-24 01:40:31 -04:00
2026-09-28 10:21:41 -07:00
2026-09-25 12:07:32 -05:00
2026-09-28 10:21:41 -07:00
2026-09-27 18:19:17 +05:30
2026-09-28 03:37:09 -07:00
2026-09-24 01:55:51 -04:00
2026-09-24 22:32:57 +05:30
2026-09-25 13:49:31 -05:00
2026-09-28 12:33:01 -07:00
2026-09-28 10:21:41 -07:00
2026-09-27 00:56:11 +05:30
2026-09-28 12:43:03 -07:00
2026-09-22 13:57:51 +05:30
2026-09-28 12:43:03 -07:00
2026-09-26 22:19:53 -05:00
2026-09-24 13:13:35 -04:00
2026-09-23 10:05:40 -07:00
2026-09-23 22:22:32 +05:30
2026-09-23 17:09:54 -07:00
2026-09-28 12:43:03 -07:00
2026-09-26 17:15:30 -05:00
2026-09-24 13:10:14 -04:00
2026-09-23 22:05:49 +00:00
2026-09-24 00:17:13 -07:00
2026-09-23 22:05:49 +00:00
2026-09-28 03:34:07 -07:00
2026-09-25 21:27:06 +05:30
2026-09-28 10:21:41 -07:00
2026-09-28 12:33:01 -07:00
2026-09-23 19:51:34 -04:00
2026-09-28 02:00:41 -07:00
2026-09-28 12:33:01 -07:00
2026-09-23 15:05:44 -07:00
2026-09-28 12:33:01 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 04:04:23 -07:00
2026-09-22 09:55:47 -04:00
2026-09-28 02:00:41 -07:00
2026-09-28 12:43:03 -07:00
2026-09-27 02:33:57 +05:30
2026-09-28 10:21:41 -07:00
2026-09-28 04:44:29 -07:00
2026-09-26 21:43:07 -05:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 06:42:00 -07:00
2026-09-24 01:55:51 -04:00
2026-09-27 22:04:26 -07:00
2026-09-28 10:21:41 -07:00
2026-09-23 11:41:01 -04:00
2026-09-28 12:23:25 -07:00
2026-09-23 10:42:18 -04:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-27 01:04:39 +05:30
2026-09-28 04:04:23 -07:00
2026-09-23 10:58:15 -07:00
2026-09-23 10:58:15 -07:00
2026-09-28 04:04:23 -07:00
2026-09-23 07:48:32 -07:00
2026-09-27 02:57:28 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-23 18:29:35 -05:00
2026-09-27 15:27:43 -05:00
2026-09-28 10:21:41 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:56:47 -05:00
2026-09-28 10:21:41 -07:00
2026-09-28 03:37:09 -07:00
2026-09-27 18:29:18 +05:30
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-27 18:29:18 +05:30
2026-09-22 13:41:11 +05:30
2026-09-28 03:37:09 -07:00
2026-09-28 03:37:09 -07:00
2026-09-23 08:09:47 -07:00
2026-09-23 11:41:01 -04:00
2026-09-28 10:21:41 -07:00
2026-09-21 23:47:51 -07:00
2026-09-28 12:24:52 -05:00
2026-09-28 19:02:20 +05:30
2026-09-25 14:22:12 -04:00
2026-09-25 14:22:12 -04:00
2026-09-28 04:44:29 -07:00
2026-09-25 16:28:14 -05:00
2026-09-27 11:44:31 -07:00
2026-09-28 06:42:00 -07:00
2026-09-24 01:55:51 -04:00
2026-09-28 12:53:29 -05:00
2026-09-28 12:43:03 -07:00
2026-09-23 15:26:34 -04:00
2026-09-28 12:43:03 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-26 17:04:25 -05:00
2026-09-24 11:50:30 -04:00
2026-09-23 07:49:35 -07:00
2026-09-23 10:52:20 -04:00
2026-09-24 14:08:21 -04:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 05:37:55 -07:00
2026-09-28 10:21:41 -07:00
2026-09-27 18:31:33 -05:00
2026-09-28 10:21:41 -07:00
2026-09-26 07:14:26 +05:30
2026-09-24 09:40:41 -04:00
2026-09-28 10:21:41 -07:00
2026-09-22 13:57:51 +05:30
2026-09-23 06:46:02 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-22 22:00:24 +00:00
2026-09-28 10:21:41 -07:00
2026-09-23 09:26:07 +05:30
2026-09-27 17:06:55 -07:00
2026-09-28 10:21:41 -07:00
2026-09-22 05:16:50 -04:00
2026-09-24 02:03:03 -07:00
2026-09-28 12:43:03 -07:00
2026-09-24 05:10:12 -04:00
2026-09-25 14:22:12 -04:00
2026-09-23 12:45:32 +05:30
2026-09-23 07:09:45 -07:00
2026-09-24 14:08:21 -04:00
2026-09-24 11:50:29 -04:00
2026-09-27 06:26:52 -05:00
2026-09-24 11:50:29 -04:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-24 11:50:29 -04:00
2026-09-25 21:19:04 +05:30
2026-09-27 01:42:52 +05:30
2026-09-28 10:21:41 -07:00
2026-09-28 04:04:23 -07:00
2026-09-22 01:19:12 -07:00
2026-09-22 17:16:00 +05:30
2026-09-28 10:21:41 -07:00
2026-09-23 11:41:01 -04:00
2026-09-25 14:22:12 -04:00
2026-09-22 05:16:50 -04:00
2026-09-22 05:16:50 -04:00
2026-09-27 01:42:52 +05:30
2026-09-25 14:22:12 -04:00
2026-09-28 10:21:41 -07:00
2026-09-24 16:17:16 -04:00
2026-09-27 06:26:52 -05:00
2026-09-27 21:29:39 -07:00
2026-09-22 01:07:59 -07:00
2026-09-23 05:37:40 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-23 05:37:40 -07:00
2026-09-28 04:04:23 -07:00
2026-09-28 10:21:41 -07:00
2026-09-25 14:25:16 -05:00
2026-09-23 07:36:31 -04:00
2026-09-28 04:59:38 -07:00
2026-09-28 12:43:03 -07:00
2026-09-27 03:41:42 -07:00
2026-09-27 00:56:11 +05:30
2026-09-28 10:21:41 -07:00
2026-09-28 04:44:29 -07:00
2026-09-28 03:34:07 -07:00
2026-09-28 10:21:41 -07:00
2026-09-24 11:50:30 -04:00
2026-09-24 12:30:00 -04:00
2026-09-23 22:22:32 +05:30
2026-09-26 20:40:40 -05:00
2026-09-23 15:43:51 -07:00
2026-09-25 10:46:54 -05:00
2026-09-28 15:54:01 +05:30
2026-09-26 21:12:19 -04:00
2026-09-27 20:49:04 +05:30
2026-09-24 18:08:24 +05:30
2026-09-27 20:49:04 +05:30
2026-09-23 10:43:45 -04:00
2026-09-28 08:33:43 +00:00
2026-09-28 12:43:03 -07:00
2026-09-28 12:33:01 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 12:43:03 -07:00
2026-09-25 20:00:57 -05:00
2026-09-28 10:21:41 -07:00
2026-09-28 06:42:00 -07:00
2026-09-27 01:36:34 +05:30
2026-09-27 02:54:03 -07:00
2026-09-24 14:23:48 -04:00
2026-09-23 22:05:57 -04:00
2026-09-25 14:22:12 -04:00
2026-09-28 10:21:41 -07:00
2026-09-23 18:29:35 -05:00
2026-09-27 07:43:23 -05:00
2026-09-28 12:33:01 -07:00
2026-09-28 19:02:20 +05:30
2026-09-24 11:12:27 -04:00
2026-09-28 10:21:41 -07:00
2026-09-24 02:03:59 -04:00
2026-09-26 17:09:16 -05:00
2026-09-22 22:24:31 +00:00
2026-09-27 23:53:40 -07:00
2026-09-24 16:17:16 -04:00
2026-09-28 10:21:41 -07:00
2026-09-26 21:00:12 -04:00
2026-09-27 14:09:15 -05:00
2026-09-21 22:02:46 -04:00
2026-09-28 03:08:56 -07:00
2026-09-28 04:44:29 -07:00
2026-09-26 20:07:47 +05:30
2026-09-25 14:22:12 -04:00
2026-09-23 18:48:33 -05:00
2026-09-28 04:44:29 -07:00
2026-09-28 03:08:56 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 05:33:25 -07:00
2026-09-28 04:44:29 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 12:43:03 -07:00
2026-09-24 14:08:22 -04:00
2026-09-23 11:41:01 -04:00
2026-09-27 02:57:28 -07:00
2026-09-23 06:51:47 -07:00
2026-09-28 04:44:29 -07:00
2026-09-28 10:21:41 -07:00
2026-09-25 14:22:12 -04:00
2026-09-28 05:33:25 -07:00
2026-09-28 12:43:03 -07:00
2026-09-27 14:09:15 -05:00
2026-09-22 15:55:23 +05:30
2026-09-27 03:38:52 -07:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 16:23:22 +05:30
2026-09-23 03:25:06 -07:00
2026-09-28 12:33:01 -07:00
2026-09-23 15:26:34 -04:00
2026-09-28 05:37:55 -07:00
2026-09-28 12:18:58 -07:00
2026-09-28 12:43:03 -07:00
2026-09-23 06:46:02 -07:00
2026-09-23 01:16:41 -07:00
2026-09-23 11:41:01 -04:00
2026-09-28 12:18:58 -07:00
2026-09-28 12:18:58 -07:00
2026-09-25 12:09:37 -05:00
2026-09-28 12:43:03 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 10:21:41 -07:00
2026-09-28 16:23:22 +05:30
2026-09-24 15:38:22 +05:30
2026-09-23 19:51:34 -04:00