hermes-agent/evals/compaction

Compaction Eval Harness

Measures what context compaction actually costs in recall, not just tokens.

What it does

1. Takes a real long transcript (JSON: {"messages": [...]}, chat format).
2. Generates a bank of factual recall questions from the region that compaction will summarize away (cached per transcript for reproducibility).
3. Runs the transcript through ContextCompressor.compress() under each policy in the matrix (current default, aggressive tail, codex-style, ...).
4. For each policy, asks a fresh LLM the recall questions with ONLY the post-compaction context, and judges answers against gold.
5. Emits a scorecard: recall accuracy vs tokens retained, per policy.

Usage

# from repo root, venv active
python evals/compaction/runner.py \
    --transcript /path/to/lineage.json \
    --policies current+recovery,lean+recovery \
    --questions 15 \
    --out evals/compaction/results/run1
python evals/compaction/report.py evals/compaction/results/run1

Transcripts are NOT committed (they contain real session data). Point --transcript at a local file. See fixtures.py for the expected shape and a synthetic-transcript generator used by CI smoke tests.

Building transcripts from real sessions (scripts/)

Compaction rotations mean a single active session rarely exceeds ~300K tokens, but the lineage (parent→children chain) carries the full uncompacted history. The scripts reconstruct those into eval transcripts:

# 1. ALWAYS copy the DB first — never point at the live state.db
cp ~/.hermes/state.db /tmp/state_copy.db

# 2. Find big lineages (sessions with parent_session_id form chains), then:
python evals/compaction/scripts/reconstruct_lineage.py \
    /tmp/state_copy.db <root_session_id> /tmp/lineage.json

# 3. (optional) Replay a 500K prefix through one checkout's compressor and
#    dump before/after for the HTML viewer:
python evals/compaction/scripts/replay_lineage.py <checkout> /tmp/lineage.json out.json 500000
python evals/compaction/scripts/build_html_report.py <runs_dir> report.html

reconstruct_lineage.py walks the whole descendant tree chronologically, dedupes rotation-copied rows by content hash, strips synthetic compaction artifacts (summaries, todo snapshots), and resolves the system prompt through the system_prompts dedup table (sessions only carry a hash). The HTML report renders before/after transcripts side by side with compaction artifacts color-coded.

Region-scoping tripwire

test_region_scoping.py plants sentinels in head/middle/tail and asserts the summarizer's serialized-turns input carries ONLY the middle (compacted) region in both legacy and lean modes. Run it directly or via pytest.

Policies

Defined in policies.py. Each policy maps to ContextCompressor constructor kwargs plus optional attribute overrides applied post-construction (e.g. tail_token_budget). Add new policies there — the runner picks them up by name.

A policy with "engine": "jev" bypasses ContextCompressor and runs jev_arm.py, a Python port of fast-jev-compaction: no summary at all — TypeSafe's Jev decision model scores every tool call/result (noul keep probabilities over the whole history) and stale ones are dropped or truncated while user/assistant text stays verbatim. Transport is OpenRouter's Decisions API (~typesafe/jev-latest, needs OPENROUTER_API_KEY); "jev": {...} overrides JevOptions (threshold, pinned tail, state/request ceilings). When the fitted state cannot get under the 25K-token ceiling the arm records jev_fallback (the plugin throws and Claude Code falls back to its built-in summary) instead of scoring.

Every arm's result carries its compaction spend: compaction_calls, compaction_input_tokens / compaction_output_tokens, compaction_model and compaction_cost_usd (Jev reports cost directly; summary calls are priced at the OpenRouter list price of the model that answered). The run also writes eval_usage.json — the harness's own question/answer/judge token bill.

Repeated-compaction simulation (scripts/jev_cycles.py)

A one-shot recall score misses the failure mode of "decide, don't summarise" compaction: it never removes user/assistant text, so each cycle frees only threshold − text_floor and the floor grows monotonically. jev_cycles.py feeds a lineage chronologically and compacts with the Jev arm every time the estimate crosses the threshold, recording per cycle: tokens before/after, percent freed, text floor, candidate/dropped calls, fitting stage, state tokens, requests and Jev cost. It stops at end of transcript, when a cycle frees nothing (stuck), or when the state cannot fit Jev's 25K ceiling (fallback — the plugin throws there).

# lineage from a state.db COPY (see above), then, with OPENROUTER_API_KEY set:
python evals/compaction/scripts/jev_cycles.py /path/lineage.json 500000 40 > cycles-500k.json
python evals/compaction/scripts/jev_cycles.py /path/lineage.json 160000 60 > cycles-160k.json
python evals/compaction/scripts/jev_cycles_report.py cycles-*.json      # markdown table

Threshold 500000 ≈ Hermes' 1M-window posture; 160000 ≈ a 200K-window host. Each cycle costs 1–8 Jev requests (< 1¢); a 40-cycle run is ~$0.20. The 2026-09-19 runs are committed under results/jev-cycles-2026-09-19/ (counts only, no transcript content) and summarised in SCORECARD-2026-09-19-jev.md: freed-per-cycle decayed 63% → 8% / 76% → 20% / 89% → 55% over 32–40 cycles, one 200K run was stuck after 0.42M tokens of work, one transcript never fit.

Notes

• Question generation and judging use agent.auxiliary_client.call_llm (same transport the compressor uses), so the harness needs a configured provider. Costs real tokens: ~(policies x questions) answer calls plus one generation and one judge pass.
• Accuracy is judged 2/1/0 (correct / partial / wrong); the scorecard reports normalized percent. The judge sees gold answers, the answerer does not.
• --also-uncompacted adds a control arm that answers from the full original transcript — the recall ceiling.
• Default arm is current+recovery: the production path. Compaction in Hermes is the summary plus the session_search pointer it carries, so the answerer gets one search round-trip over the archived region (same FTS5+BM25 engine as production). A bare policy name (current) is closed-book — the summary with its recovery pointer unused — and scores 30+ pts lower on needle questions. Use it only when you specifically want that floor.