00919d1e8a
jev_cycles_report.py renders the cycles/freed/floor/end-state table from jev_cycles.py outputs; README carries the exact commands, thresholds and cost so the diminishing-returns and stuck/fallback findings can be reproduced. Fallback records now count the real tool calls.
134 lines
6.5 KiB
Markdown
134 lines
6.5 KiB
Markdown
# Compaction Eval Harness
|
||
|
||
Measures what context compaction actually costs in *recall*, not just tokens.
|
||
|
||
## What it does
|
||
|
||
1. Takes a real long transcript (JSON: `{"messages": [...]}`, chat format).
|
||
2. Generates a bank of factual recall questions from the region that
|
||
compaction will summarize away (cached per transcript for reproducibility).
|
||
3. Runs the transcript through `ContextCompressor.compress()` under each
|
||
policy in the matrix (current default, aggressive tail, codex-style, ...).
|
||
4. For each policy, asks a fresh LLM the recall questions with ONLY the
|
||
post-compaction context, and judges answers against gold.
|
||
5. Emits a scorecard: recall accuracy vs tokens retained, per policy.
|
||
|
||
## Usage
|
||
|
||
```bash
|
||
# from repo root, venv active
|
||
python evals/compaction/runner.py \
|
||
--transcript /path/to/lineage.json \
|
||
--policies current+recovery,lean+recovery \
|
||
--questions 15 \
|
||
--out evals/compaction/results/run1
|
||
python evals/compaction/report.py evals/compaction/results/run1
|
||
```
|
||
|
||
Transcripts are NOT committed (they contain real session data). Point
|
||
`--transcript` at a local file. See `fixtures.py` for the expected shape and
|
||
a synthetic-transcript generator used by CI smoke tests.
|
||
|
||
## Building transcripts from real sessions (`scripts/`)
|
||
|
||
Compaction rotations mean a single active session rarely exceeds ~300K
|
||
tokens, but the *lineage* (parent→children chain) carries the full
|
||
uncompacted history. The scripts reconstruct those into eval transcripts:
|
||
|
||
```bash
|
||
# 1. ALWAYS copy the DB first — never point at the live state.db
|
||
cp ~/.hermes/state.db /tmp/state_copy.db
|
||
|
||
# 2. Find big lineages (sessions with parent_session_id form chains), then:
|
||
python evals/compaction/scripts/reconstruct_lineage.py \
|
||
/tmp/state_copy.db <root_session_id> /tmp/lineage.json
|
||
|
||
# 3. (optional) Replay a 500K prefix through one checkout's compressor and
|
||
# dump before/after for the HTML viewer:
|
||
python evals/compaction/scripts/replay_lineage.py <checkout> /tmp/lineage.json out.json 500000
|
||
python evals/compaction/scripts/build_html_report.py <runs_dir> report.html
|
||
```
|
||
|
||
`reconstruct_lineage.py` walks the whole descendant tree chronologically,
|
||
dedupes rotation-copied rows by content hash, strips synthetic compaction
|
||
artifacts (summaries, todo snapshots), and resolves the system prompt through
|
||
the `system_prompts` dedup table (sessions only carry a hash). The HTML
|
||
report renders before/after transcripts side by side with compaction
|
||
artifacts color-coded.
|
||
|
||
## Region-scoping tripwire
|
||
|
||
`test_region_scoping.py` plants sentinels in head/middle/tail and asserts the
|
||
summarizer's serialized-turns input carries ONLY the middle (compacted)
|
||
region in both legacy and lean modes. Run it directly or via pytest.
|
||
|
||
## Policies
|
||
|
||
Defined in `policies.py`. Each policy maps to `ContextCompressor` constructor
|
||
kwargs plus optional attribute overrides applied post-construction (e.g.
|
||
`tail_token_budget`). Add new policies there — the runner picks them up by
|
||
name.
|
||
|
||
A policy with `"engine": "jev"` bypasses `ContextCompressor` and runs
|
||
`jev_arm.py`, a Python port of
|
||
[fast-jev-compaction](https://github.com/tamaratran/fast-jev-compaction): no
|
||
summary at all — TypeSafe's Jev decision model scores every tool call/result
|
||
(`noul` keep probabilities over the whole history) and stale ones are dropped
|
||
or truncated while user/assistant text stays verbatim. Transport is
|
||
OpenRouter's Decisions API (`~typesafe/jev-latest`, needs
|
||
`OPENROUTER_API_KEY`); `"jev": {...}` overrides `JevOptions` (threshold,
|
||
pinned tail, state/request ceilings). When the fitted state cannot get under
|
||
the 25K-token ceiling the arm records `jev_fallback` (the plugin throws and
|
||
Claude Code falls back to its built-in summary) instead of scoring.
|
||
|
||
Every arm's result carries its compaction spend: `compaction_calls`,
|
||
`compaction_input_tokens` / `compaction_output_tokens`, `compaction_model`
|
||
and `compaction_cost_usd` (Jev reports cost directly; summary calls are
|
||
priced at the OpenRouter list price of the model that answered). The run
|
||
also writes `eval_usage.json` — the harness's own question/answer/judge
|
||
token bill.
|
||
|
||
## Repeated-compaction simulation (`scripts/jev_cycles.py`)
|
||
|
||
A one-shot recall score misses the failure mode of "decide, don't summarise"
|
||
compaction: it never removes user/assistant text, so each cycle frees only
|
||
`threshold − text_floor` and the floor grows monotonically. `jev_cycles.py`
|
||
feeds a lineage chronologically and compacts with the Jev arm every time the
|
||
estimate crosses the threshold, recording per cycle: tokens before/after,
|
||
percent freed, text floor, candidate/dropped calls, fitting stage, state
|
||
tokens, requests and Jev cost. It stops at end of transcript, when a cycle
|
||
frees nothing (`stuck`), or when the state cannot fit Jev's 25K ceiling
|
||
(`fallback` — the plugin throws there).
|
||
|
||
```bash
|
||
# lineage from a state.db COPY (see above), then, with OPENROUTER_API_KEY set:
|
||
python evals/compaction/scripts/jev_cycles.py /path/lineage.json 500000 40 > cycles-500k.json
|
||
python evals/compaction/scripts/jev_cycles.py /path/lineage.json 160000 60 > cycles-160k.json
|
||
python evals/compaction/scripts/jev_cycles_report.py cycles-*.json # markdown table
|
||
```
|
||
|
||
Threshold 500000 ≈ Hermes' 1M-window posture; 160000 ≈ a 200K-window host.
|
||
Each cycle costs 1–8 Jev requests (< 1¢); a 40-cycle run is ~$0.20. The
|
||
2026-09-19 runs are committed under `results/jev-cycles-2026-09-19/` (counts
|
||
only, no transcript content) and summarised in `SCORECARD-2026-09-19-jev.md`:
|
||
freed-per-cycle decayed 63% → 8% / 76% → 20% / 89% → 55% over 32–40 cycles,
|
||
one 200K run was stuck after 0.42M tokens of work, one transcript never fit.
|
||
|
||
## Notes
|
||
|
||
- Question generation and judging use `agent.auxiliary_client.call_llm`
|
||
(same transport the compressor uses), so the harness needs a configured
|
||
provider. Costs real tokens: ~(policies x questions) answer calls plus
|
||
one generation and one judge pass.
|
||
- Accuracy is judged 2/1/0 (correct / partial / wrong); the scorecard
|
||
reports normalized percent. The judge sees gold answers, the answerer
|
||
does not.
|
||
- `--also-uncompacted` adds a control arm that answers from the full
|
||
original transcript — the recall ceiling.
|
||
- **Default arm is `current+recovery`: the production path.** Compaction in
|
||
Hermes is the summary *plus* the session_search pointer it carries, so the
|
||
answerer gets one search round-trip over the archived region (same FTS5+BM25
|
||
engine as production). A bare policy name (`current`) is closed-book — the
|
||
summary with its recovery pointer unused — and scores 30+ pts lower on
|
||
needle questions. Use it only when you specifically want that floor.
|