hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity

evals(compaction): Jev repeated-compaction simulator, cycle data and verdict

Browse Source 
jev_cycles.py compacts a lineage with Jev every time the estimate crosses a
threshold and records freed tokens, text floor and fitting stage per cycle.
Six runs show the floor (never-removed user/assistant text) is the binding
limit: freed-per-cycle decays to 0-8% and a 200K-window host is stuck after
0.42M tokens; >~500 tool calls between compactions cannot fit the 32K state.
Scorecard gains a decisive verdict section.
This commit is contained in:
teknium1
2026-09-19 10:46:43 -07:00
committed by Teknium
parent 770c47779f
commit d659843dbf
8 changed files with 2752 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
evals/compaction/results/SCORECARD-2026-09-19-jev.md
View File

@@ -1,5 +1,28 @@
# fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19)
## Verdict
**Do not adopt Jev as Hermes' compaction. Do adopt the retention rule it demonstrates.**
- The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool
output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a
matched token budget, ranked no better than plain recency (77.8 vs 77.8).
- Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never
reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40
cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens
of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be
a pre-pass.
- The 32K request window forces the whole history into 25K tokens: in every measured cycle
the ceiling was binding and Jev decided on tool name + 60-char input + result size with
message texts replaced by `[… N chars omitted …]`. >~500 tool calls between compactions
cannot fit at all (1 of 4 transcripts never got a first compaction).
- What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention
of everything not a tool result.
Recommended follow-up (not this PR): change the compressor's retention posture to prune
tool results first and summarise only the remainder, keeping assistant text verbatim within
the tail budget. Zero external dependency, captures most of the measured gain.
Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the
compaction summary with Jev decisions: score every tool call/result, drop or truncate
stale ones, keep everything else verbatim") beat our compressor on remaining tokens,
@@ -75,6 +98,45 @@ posture would capture most of Jev's gain at zero extra cost. If a scoring model
for the prune ranking, Jev is fast and cheap enough (1.5 s, < 1¢) but this data shows no
signal over recency at equal budget; re-test before wiring it in.
## Repeated compaction: how many cycles does Jev-only compaction survive?
`scripts/jev_cycles.py` feeds a lineage chronologically and compacts with Jev (plugin
defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K
≈ a 200K-window host. Raw JSON per run in `results/jev-cycles-2026-09-19/`. Jev spend for all
six runs: $0.60.
| run | cycles | raw session consumed | freed per cycle | text floor | end state |
|---|---|---|---|---|---|
| prreview @500K | 32 | 11.4M (whole lineage) | 89% → 55% | 43K → 139K | still working |
| sysprompt @500K | 40 | 8.6M of 23.3M | 76% → 20% | 91K → 240K | degrading |
| sigsegv @500K | 40 | 5.2M of 8.6M | 63% → 8% | 175K → 359K | 40K freed/cycle; wall ≈ 8M |
| prreview @160K | 60 | 3.9M of 11.4M | 87% → 22% | 12K → 87K | compacting every ~14 rows |
| sigsegv @160K | 11 | 0.42M of 8.6M | 52% → 0% | 85K → 151K | **STUCK** (floor ≥ threshold) |
| afff57 @500K | 0 | — | — | — | fallback on cycle 1 (541 calls) |
Reading: every cycle has candidates (new tool calls arrive between compactions and Jev
drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to
remove" never happens. What runs out is headroom: each cycle frees at most
`threshold − floor`, and the floor (all user/assistant rows) only grows. Well before the
hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts
every ~40K tokens of new work with every turn billed at ~460K input.
## The 32K window in practice
- `state_tok` was 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always
binds. Fitting stages reached at 500K: "old messages collapsed" (texts → `[… N chars
omitted …]`), "old calls compacted" (one line per call, input cut to 60 chars), "old
messages left out" (text-only old rows removed from the state), "old calls merged". Jev
never sees tool result contents (by design) and, at these stages, barely sees message
text either — it decides on tool name, input stub, result size and the last 3 user prompts.
- Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls);
the plugin throws and the host falls back to its summary. The text floor does not affect
fit (old text rows are left out of the state); the call count does.
- Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per
cycle at 500K, 1–2 s, $0.005–0.008.
- Untouchable content: anything in assistant text (pasted logs, long analyses) is never
compacted, which is exactly what makes the floor grow.
## Method notes
- Transcripts reconstructed with `scripts/reconstruct_lineage.py` from a state.db copy;

18
evals/compaction/results/jev-cycles-2026-09-19/cycles-afff57-500k.json Normal file
View File

@@ -0,0 +1,18 @@
{
"transcript": "afff57",
"threshold": 500000,
"raw_tokens_consumed": 500301,
"raw_total": 7357845,
"msgs_consumed": 1108,
"cycles": [
{
"cycle": 1,
"at_msg": 1108,
"before": 500301,
"fallback": "history too large for Jev (~25972 tokens after truncation, limit 25000)",
"floor": 22650,
"calls": 0
}
],
"jev_usd_total": 0.0
}

850
evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-160k.json Normal file
View File

@@ -0,0 +1,850 @@
{
"transcript": "prreview",
"threshold": 160000,
"raw_tokens_consumed": 3908081,
"raw_total": 11361152,
"msgs_consumed": 3778,
"cycles": [
{
"cycle": 1,
"at_msg": 84,
"before": 161753,
"after": 20516,
"freed_pct": 87.3,
"floor": 11651,
"candidates": 43,
"dropped": 38,
"stage": "full",
"state_tok": 24216,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 2,
"at_msg": 204,
"before": 160345,
"after": 25821,
"freed_pct": 83.9,
"floor": 20754,
"candidates": 69,
"dropped": 67,
"stage": "texts abridged",
"state_tok": 18878,
"requests": 1,
"usd": 0.001
},
{
"cycle": 3,
"at_msg": 417,
"before": 160545,
"after": 43910,
"freed_pct": 72.6,
"floor": 38723,
"candidates": 97,
"dropped": 95,
"stage": "texts abridged",
"state_tok": 16544,
"requests": 1,
"usd": 0.001
},
{
"cycle": 4,
"at_msg": 603,
"before": 161273,
"after": 62761,
"freed_pct": 61.1,
"floor": 50469,
"candidates": 82,
"dropped": 78,
"stage": "texts abridged",
"state_tok": 23411,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 5,
"at_msg": 770,
"before": 161973,
"after": 77531,
"freed_pct": 52.1,
"floor": 61000,
"candidates": 76,
"dropped": 66,
"stage": "texts abridged",
"state_tok": 23272,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 6,
"at_msg": 864,
"before": 161232,
"after": 104685,
"freed_pct": 35.1,
"floor": 65705,
"candidates": 48,
"dropped": 46,
"stage": "texts abridged",
"state_tok": 23088,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 7,
"at_msg": 905,
"before": 169559,
"after": 81925,
"freed_pct": 51.7,
"floor": 66307,
"candidates": 30,
"dropped": 29,
"stage": "texts abridged",
"state_tok": 21283,
"requests": 1,
"usd": 0.001
},
{
"cycle": 8,
"at_msg": 948,
"before": 164874,
"after": 90444,
"freed_pct": 45.1,
"floor": 66783,
"candidates": 24,
"dropped": 23,
"stage": "texts abridged",
"state_tok": 21077,
"requests": 1,
"usd": 0.0009
},
{
"cycle": 9,
"at_msg": 961,
"before": 165153,
"after": 112109,
"freed_pct": 32.1,
"floor": 66783,
"candidates": 9,
"dropped": 8,
"stage": "texts abridged",
"state_tok": 19750,
"requests": 1,
"usd": 0.0008
},
{
"cycle": 10,
"at_msg": 995,
"before": 166475,
"after": 90196,
"freed_pct": 45.8,
"floor": 67678,
"candidates": 25,
"dropped": 25,
"stage": "texts abridged",
"state_tok": 21316,
"requests": 1,
"usd": 0.001
},
{
"cycle": 11,
"at_msg": 1035,
"before": 161835,
"after": 90088,
"freed_pct": 44.3,
"floor": 68354,
"candidates": 20,
"dropped": 20,
"stage": "texts abridged",
"state_tok": 21796,
"requests": 1,
"usd": 0.001
},
{
"cycle": 12,
"at_msg": 1101,
"before": 160756,
"after": 76512,
"freed_pct": 52.4,
"floor": 68354,
"candidates": 42,
"dropped": 42,
"stage": "texts abridged",
"state_tok": 23610,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 13,
"at_msg": 1175,
"before": 162050,
"after": 81482,
"freed_pct": 49.7,
"floor": 69504,
"candidates": 43,
"dropped": 41,
"stage": "texts abridged",
"state_tok": 24232,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 14,
"at_msg": 1269,
"before": 160807,
"after": 83741,
"freed_pct": 47.9,
"floor": 69679,
"candidates": 62,
"dropped": 61,
"stage": "old messages collapsed",
"state_tok": 24863,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 15,
"at_msg": 1346,
"before": 162548,
"after": 78873,
"freed_pct": 51.5,
"floor": 70824,
"candidates": 47,
"dropped": 47,
"stage": "texts abridged",
"state_tok": 24955,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 16,
"at_msg": 1433,
"before": 160045,
"after": 80210,
"freed_pct": 49.9,
"floor": 70824,
"candidates": 41,
"dropped": 41,
"stage": "texts abridged",
"state_tok": 24696,
"requests": 1,
"usd": 0.0012
},
{
"cycle": 17,
"at_msg": 1481,
"before": 164436,
"after": 102906,
"freed_pct": 37.4,
"floor": 71719,
"candidates": 23,
"dropped": 23,
"stage": "texts abridged",
"state_tok": 23269,
"requests": 1,
"usd": 0.001
},
{
"cycle": 18,
"at_msg": 1514,
"before": 169351,
"after": 98544,
"freed_pct": 41.8,
"floor": 71719,
"candidates": 21,
"dropped": 21,
"stage": "texts abridged",
"state_tok": 22944,
"requests": 1,
"usd": 0.001
},
{
"cycle": 19,
"at_msg": 1606,
"before": 169524,
"after": 99953,
"freed_pct": 41.0,
"floor": 72852,
"candidates": 59,
"dropped": 59,
"stage": "old messages collapsed",
"state_tok": 24973,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 20,
"at_msg": 1627,
"before": 161132,
"after": 92569,
"freed_pct": 42.6,
"floor": 72852,
"candidates": 14,
"dropped": 14,
"stage": "texts abridged",
"state_tok": 22610,
"requests": 1,
"usd": 0.001
},
{
"cycle": 21,
"at_msg": 1690,
"before": 160119,
"after": 80377,
"freed_pct": 49.8,
"floor": 72852,
"candidates": 40,
"dropped": 40,
"stage": "texts abridged",
"state_tok": 24904,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 22,
"at_msg": 1754,
"before": 160964,
"after": 88796,
"freed_pct": 44.8,
"floor": 73786,
"candidates": 38,
"dropped": 38,
"stage": "old messages collapsed",
"state_tok": 24978,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 23,
"at_msg": 1843,
"before": 160168,
"after": 87896,
"freed_pct": 45.1,
"floor": 73786,
"candidates": 53,
"dropped": 53,
"stage": "old messages collapsed",
"state_tok": 24870,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 24,
"at_msg": 1871,
"before": 165096,
"after": 95841,
"freed_pct": 41.9,
"floor": 74776,
"candidates": 15,
"dropped": 15,
"stage": "texts abridged",
"state_tok": 23510,
"requests": 1,
"usd": 0.001
},
{
"cycle": 25,
"at_msg": 1923,
"before": 160843,
"after": 110837,
"freed_pct": 31.1,
"floor": 75759,
"candidates": 34,
"dropped": 34,
"stage": "old messages collapsed",
"state_tok": 24877,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 26,
"at_msg": 1953,
"before": 160363,
"after": 88775,
"freed_pct": 44.6,
"floor": 75759,
"candidates": 16,
"dropped": 14,
"stage": "texts abridged",
"state_tok": 24027,
"requests": 1,
"usd": 0.001
},
{
"cycle": 27,
"at_msg": 2047,
"before": 171545,
"after": 115659,
"freed_pct": 32.6,
"floor": 76397,
"candidates": 45,
"dropped": 44,
"stage": "old messages collapsed",
"state_tok": 24930,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 28,
"at_msg": 2093,
"before": 162639,
"after": 91133,
"freed_pct": 44.0,
"floor": 76397,
"candidates": 24,
"dropped": 23,
"stage": "old messages collapsed",
"state_tok": 24991,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 29,
"at_msg": 2239,
"before": 160686,
"after": 92844,
"freed_pct": 42.2,
"floor": 76989,
"candidates": 70,
"dropped": 70,
"stage": "old messages collapsed",
"state_tok": 24842,
"requests": 2,
"usd": 0.0023
},
{
"cycle": 30,
"at_msg": 2273,
"before": 160421,
"after": 94811,
"freed_pct": 40.9,
"floor": 76989,
"candidates": 15,
"dropped": 14,
"stage": "texts abridged",
"state_tok": 24605,
"requests": 1,
"usd": 0.001
},
{
"cycle": 31,
"at_msg": 2357,
"before": 171760,
"after": 128557,
"freed_pct": 25.2,
"floor": 77549,
"candidates": 38,
"dropped": 38,
"stage": "old messages collapsed",
"state_tok": 24981,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 32,
"at_msg": 2369,
"before": 160215,
"after": 114061,
"freed_pct": 28.8,
"floor": 77549,
"candidates": 7,
"dropped": 6,
"stage": "texts abridged",
"state_tok": 24087,
"requests": 1,
"usd": 0.001
},
{
"cycle": 33,
"at_msg": 2415,
"before": 167023,
"after": 121299,
"freed_pct": 27.4,
"floor": 78211,
"candidates": 22,
"dropped": 21,
"stage": "old messages collapsed",
"state_tok": 24971,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 34,
"at_msg": 2442,
"before": 160572,
"after": 96497,
"freed_pct": 39.9,
"floor": 78211,
"candidates": 16,
"dropped": 15,
"stage": "old messages collapsed",
"state_tok": 24963,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 35,
"at_msg": 2498,
"before": 161248,
"after": 132997,
"freed_pct": 17.5,
"floor": 78779,
"candidates": 26,
"dropped": 26,
"stage": "old messages collapsed",
"state_tok": 24924,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 36,
"at_msg": 2523,
"before": 160787,
"after": 103239,
"freed_pct": 35.8,
"floor": 78779,
"candidates": 16,
"dropped": 15,
"stage": "old messages collapsed",
"state_tok": 24941,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 37,
"at_msg": 2625,
"before": 160431,
"after": 96660,
"freed_pct": 39.7,
"floor": 78779,
"candidates": 57,
"dropped": 57,
"stage": "old messages collapsed",
"state_tok": 24891,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 38,
"at_msg": 2665,
"before": 161961,
"after": 123182,
"freed_pct": 23.9,
"floor": 79463,
"candidates": 17,
"dropped": 16,
"stage": "old messages collapsed",
"state_tok": 24942,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 39,
"at_msg": 2692,
"before": 160484,
"after": 101551,
"freed_pct": 36.7,
"floor": 80148,
"candidates": 14,
"dropped": 14,
"stage": "old messages collapsed",
"state_tok": 24926,
"requests": 1,
"usd": 0.001
},
{
"cycle": 40,
"at_msg": 2711,
"before": 169093,
"after": 118561,
"freed_pct": 29.9,
"floor": 80148,
"candidates": 9,
"dropped": 8,
"stage": "old messages collapsed",
"state_tok": 24919,
"requests": 1,
"usd": 0.001
},
{
"cycle": 41,
"at_msg": 2770,
"before": 160083,
"after": 99987,
"freed_pct": 37.5,
"floor": 80148,
"candidates": 30,
"dropped": 29,
"stage": "old messages collapsed",
"state_tok": 24997,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 42,
"at_msg": 2805,
"before": 162263,
"after": 124566,
"freed_pct": 23.2,
"floor": 80797,
"candidates": 19,
"dropped": 17,
"stage": "old messages collapsed",
"state_tok": 24898,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 43,
"at_msg": 2849,
"before": 160207,
"after": 101501,
"freed_pct": 36.6,
"floor": 80797,
"candidates": 24,
"dropped": 24,
"stage": "old messages collapsed",
"state_tok": 24900,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 44,
"at_msg": 2986,
"before": 160607,
"after": 101684,
"freed_pct": 36.7,
"floor": 80797,
"candidates": 70,
"dropped": 69,
"stage": "old messages collapsed",
"state_tok": 24947,
"requests": 2,
"usd": 0.0023
},
{
"cycle": 45,
"at_msg": 3022,
"before": 162446,
"after": 105814,
"freed_pct": 34.9,
"floor": 81538,
"candidates": 17,
"dropped": 17,
"stage": "old messages collapsed",
"state_tok": 24930,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 46,
"at_msg": 3111,
"before": 160129,
"after": 115971,
"freed_pct": 27.6,
"floor": 82171,
"candidates": 42,
"dropped": 42,
"stage": "old messages collapsed",
"state_tok": 24841,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 47,
"at_msg": 3130,
"before": 163879,
"after": 113877,
"freed_pct": 30.5,
"floor": 82171,
"candidates": 9,
"dropped": 9,
"stage": "old messages collapsed",
"state_tok": 24981,
"requests": 1,
"usd": 0.001
},
{
"cycle": 48,
"at_msg": 3235,
"before": 161073,
"after": 105216,
"freed_pct": 34.7,
"floor": 82732,
"candidates": 52,
"dropped": 52,
"stage": "old messages collapsed",
"state_tok": 24982,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 49,
"at_msg": 3262,
"before": 165994,
"after": 113169,
"freed_pct": 31.8,
"floor": 84296,
"candidates": 16,
"dropped": 16,
"stage": "old messages collapsed",
"state_tok": 24878,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 50,
"at_msg": 3295,
"before": 162010,
"after": 110995,
"freed_pct": 31.5,
"floor": 84296,
"candidates": 18,
"dropped": 18,
"stage": "old messages collapsed",
"state_tok": 24936,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 51,
"at_msg": 3383,
"before": 160170,
"after": 107831,
"freed_pct": 32.7,
"floor": 84296,
"candidates": 42,
"dropped": 42,
"stage": "old messages collapsed",
"state_tok": 24977,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 52,
"at_msg": 3495,
"before": 161271,
"after": 111319,
"freed_pct": 31.0,
"floor": 84897,
"candidates": 64,
"dropped": 64,
"stage": "old messages collapsed",
"state_tok": 24975,
"requests": 2,
"usd": 0.0022
},
{
"cycle": 53,
"at_msg": 3525,
"before": 160973,
"after": 116263,
"freed_pct": 27.8,
"floor": 85505,
"candidates": 19,
"dropped": 19,
"stage": "old messages collapsed",
"state_tok": 24905,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 54,
"at_msg": 3567,
"before": 165268,
"after": 118425,
"freed_pct": 28.3,
"floor": 85782,
"candidates": 23,
"dropped": 23,
"stage": "old messages collapsed",
"state_tok": 24996,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 55,
"at_msg": 3616,
"before": 162401,
"after": 123737,
"freed_pct": 23.8,
"floor": 86298,
"candidates": 20,
"dropped": 20,
"stage": "old messages collapsed",
"state_tok": 24838,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 56,
"at_msg": 3632,
"before": 160426,
"after": 123165,
"freed_pct": 23.2,
"floor": 86298,
"candidates": 7,
"dropped": 6,
"stage": "old messages collapsed",
"state_tok": 24944,
"requests": 1,
"usd": 0.001
},
{
"cycle": 57,
"at_msg": 3690,
"before": 161487,
"after": 118670,
"freed_pct": 26.5,
"floor": 86298,
"candidates": 29,
"dropped": 29,
"stage": "old messages collapsed",
"state_tok": 24911,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 58,
"at_msg": 3755,
"before": 165439,
"after": 123947,
"freed_pct": 25.1,
"floor": 86924,
"candidates": 31,
"dropped": 31,
"stage": "old messages collapsed",
"state_tok": 24879,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 59,
"at_msg": 3764,
"before": 160849,
"after": 139191,
"freed_pct": 13.5,
"floor": 86924,
"candidates": 3,
"dropped": 3,
"stage": "old messages collapsed",
"state_tok": 24852,
"requests": 1,
"usd": 0.001
},
{
"cycle": 60,
"at_msg": 3778,
"before": 162699,
"after": 126592,
"freed_pct": 22.2,
"floor": 86924,
"candidates": 11,
"dropped": 11,
"stage": "old messages collapsed",
"state_tok": 24871,
"requests": 1,
"usd": 0.001
}
],
"jev_usd_total": 0.0792
}

458
evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-500k.json Normal file
View File

@@ -0,0 +1,458 @@
{
"transcript": "prreview",
"threshold": 500000,
"raw_tokens_consumed": 11361152,
"raw_total": 11361152,
"msgs_consumed": 10074,
"cycles": [
{
"cycle": 1,
"at_msg": 531,
"before": 500120,
"after": 52660,
"freed_pct": 89.5,
"floor": 43449,
"candidates": 252,
"dropped": 252,
"stage": "old calls compacted",
"state_tok": 24975,
"requests": 7,
"usd": 0.0083
},
{
"cycle": 2,
"at_msg": 961,
"before": 511654,
"after": 111265,
"freed_pct": 78.3,
"floor": 66783,
"candidates": 199,
"dropped": 199,
"stage": "old messages collapsed",
"state_tok": 24916,
"requests": 6,
"usd": 0.0071
},
{
"cycle": 3,
"at_msg": 1313,
"before": 502998,
"after": 88652,
"freed_pct": 82.4,
"floor": 70824,
"candidates": 216,
"dropped": 216,
"stage": "old messages collapsed",
"state_tok": 24882,
"requests": 6,
"usd": 0.0071
},
{
"cycle": 4,
"at_msg": 1627,
"before": 501264,
"after": 92569,
"freed_pct": 81.5,
"floor": 72852,
"candidates": 177,
"dropped": 177,
"stage": "old messages collapsed",
"state_tok": 24921,
"requests": 5,
"usd": 0.0059
},
{
"cycle": 5,
"at_msg": 1947,
"before": 500375,
"after": 94749,
"freed_pct": 81.1,
"floor": 75759,
"candidates": 193,
"dropped": 191,
"stage": "old messages collapsed",
"state_tok": 24852,
"requests": 5,
"usd": 0.0059
},
{
"cycle": 6,
"at_msg": 2411,
"before": 519129,
"after": 114932,
"freed_pct": 77.9,
"floor": 78211,
"candidates": 219,
"dropped": 218,
"stage": "old calls compacted",
"state_tok": 24999,
"requests": 6,
"usd": 0.0072
},
{
"cycle": 7,
"at_msg": 2741,
"before": 502448,
"after": 104732,
"freed_pct": 79.2,
"floor": 80148,
"candidates": 168,
"dropped": 167,
"stage": "old messages collapsed",
"state_tok": 24954,
"requests": 5,
"usd": 0.0059
},
{
"cycle": 8,
"at_msg": 3236,
"before": 504872,
"after": 117119,
"freed_pct": 76.8,
"floor": 82732,
"candidates": 245,
"dropped": 245,
"stage": "old calls compacted",
"state_tok": 24979,
"requests": 7,
"usd": 0.0084
},
{
"cycle": 9,
"at_msg": 3636,
"before": 500075,
"after": 119512,
"freed_pct": 76.1,
"floor": 86298,
"candidates": 210,
"dropped": 209,
"stage": "old calls compacted",
"state_tok": 24973,
"requests": 6,
"usd": 0.0072
},
{
"cycle": 10,
"at_msg": 4000,
"before": 502270,
"after": 128473,
"freed_pct": 74.4,
"floor": 88838,
"candidates": 181,
"dropped": 181,
"stage": "old messages collapsed",
"state_tok": 24956,
"requests": 5,
"usd": 0.006
},
{
"cycle": 11,
"at_msg": 4354,
"before": 501400,
"after": 151425,
"freed_pct": 69.8,
"floor": 91026,
"candidates": 192,
"dropped": 192,
"stage": "old messages collapsed",
"state_tok": 24842,
"requests": 5,
"usd": 0.006
},
{
"cycle": 12,
"at_msg": 4592,
"before": 511090,
"after": 158483,
"freed_pct": 69.0,
"floor": 93179,
"candidates": 128,
"dropped": 127,
"stage": "old messages collapsed",
"state_tok": 24897,
"requests": 4,
"usd": 0.0046
},
{
"cycle": 13,
"at_msg": 4806,
"before": 501974,
"after": 142234,
"freed_pct": 71.7,
"floor": 95299,
"candidates": 110,
"dropped": 110,
"stage": "old messages collapsed",
"state_tok": 24984,
"requests": 3,
"usd": 0.0035
},
{
"cycle": 14,
"at_msg": 5124,
"before": 501941,
"after": 156309,
"freed_pct": 68.9,
"floor": 98017,
"candidates": 161,
"dropped": 160,
"stage": "old messages collapsed",
"state_tok": 24877,
"requests": 5,
"usd": 0.0059
},
{
"cycle": 15,
"at_msg": 5463,
"before": 500937,
"after": 154241,
"freed_pct": 69.2,
"floor": 101588,
"candidates": 176,
"dropped": 176,
"stage": "old messages collapsed",
"state_tok": 24813,
"requests": 5,
"usd": 0.006
},
{
"cycle": 16,
"at_msg": 5819,
"before": 503417,
"after": 162498,
"freed_pct": 67.7,
"floor": 102640,
"candidates": 233,
"dropped": 233,
"stage": "old calls compacted",
"state_tok": 24996,
"requests": 7,
"usd": 0.0083
},
{
"cycle": 17,
"at_msg": 6033,
"before": 507005,
"after": 188509,
"freed_pct": 62.8,
"floor": 104655,
"candidates": 110,
"dropped": 110,
"stage": "old messages collapsed",
"state_tok": 24876,
"requests": 3,
"usd": 0.0035
},
{
"cycle": 18,
"at_msg": 6314,
"before": 501223,
"after": 163564,
"freed_pct": 67.4,
"floor": 105926,
"candidates": 143,
"dropped": 143,
"stage": "old messages collapsed",
"state_tok": 24865,
"requests": 4,
"usd": 0.0047
},
{
"cycle": 19,
"at_msg": 6646,
"before": 501654,
"after": 178869,
"freed_pct": 64.3,
"floor": 109116,
"candidates": 170,
"dropped": 170,
"stage": "old calls compacted",
"state_tok": 24992,
"requests": 5,
"usd": 0.006
},
{
"cycle": 20,
"at_msg": 6883,
"before": 501207,
"after": 172627,
"freed_pct": 65.6,
"floor": 111444,
"candidates": 126,
"dropped": 126,
"stage": "old messages collapsed",
"state_tok": 24957,
"requests": 4,
"usd": 0.0047
},
{
"cycle": 21,
"at_msg": 7016,
"before": 506054,
"after": 218132,
"freed_pct": 56.9,
"floor": 114654,
"candidates": 73,
"dropped": 73,
"stage": "old messages collapsed",
"state_tok": 24816,
"requests": 2,
"usd": 0.0023
},
{
"cycle": 22,
"at_msg": 7283,
"before": 503596,
"after": 204002,
"freed_pct": 59.5,
"floor": 116369,
"candidates": 129,
"dropped": 125,
"stage": "old messages collapsed",
"state_tok": 24970,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 23,
"at_msg": 7503,
"before": 504496,
"after": 191152,
"freed_pct": 62.1,
"floor": 117845,
"candidates": 109,
"dropped": 109,
"stage": "old messages collapsed",
"state_tok": 24917,
"requests": 3,
"usd": 0.0036
},
{
"cycle": 24,
"at_msg": 7749,
"before": 501038,
"after": 205540,
"freed_pct": 59.0,
"floor": 119715,
"candidates": 120,
"dropped": 119,
"stage": "old messages collapsed",
"state_tok": 24988,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 25,
"at_msg": 8050,
"before": 515776,
"after": 211866,
"freed_pct": 58.9,
"floor": 122519,
"candidates": 155,
"dropped": 155,
"stage": "old calls compacted",
"state_tok": 24996,
"requests": 5,
"usd": 0.006
},
{
"cycle": 26,
"at_msg": 8314,
"before": 504042,
"after": 216111,
"freed_pct": 57.1,
"floor": 125001,
"candidates": 132,
"dropped": 132,
"stage": "old messages collapsed",
"state_tok": 24980,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 27,
"at_msg": 8533,
"before": 500651,
"after": 205303,
"freed_pct": 59.0,
"floor": 126706,
"candidates": 110,
"dropped": 105,
"stage": "old messages collapsed",
"state_tok": 25000,
"requests": 3,
"usd": 0.0036
},
{
"cycle": 28,
"at_msg": 8819,
"before": 502475,
"after": 251171,
"freed_pct": 50.0,
"floor": 129085,
"candidates": 143,
"dropped": 138,
"stage": "old messages collapsed",
"state_tok": 24993,
"requests": 4,
"usd": 0.0049
},
{
"cycle": 29,
"at_msg": 9083,
"before": 507185,
"after": 215329,
"freed_pct": 57.5,
"floor": 131182,
"candidates": 137,
"dropped": 137,
"stage": "old messages collapsed",
"state_tok": 24856,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 30,
"at_msg": 9432,
"before": 500044,
"after": 213241,
"freed_pct": 57.4,
"floor": 133547,
"candidates": 178,
"dropped": 173,
"stage": "old calls compacted",
"state_tok": 24983,
"requests": 5,
"usd": 0.0063
},
{
"cycle": 31,
"at_msg": 9654,
"before": 501024,
"after": 222477,
"freed_pct": 55.6,
"floor": 136282,
"candidates": 112,
"dropped": 112,
"stage": "old messages collapsed",
"state_tok": 24876,
"requests": 3,
"usd": 0.0036
},
{
"cycle": 32,
"at_msg": 9856,
"before": 505769,
"after": 228548,
"freed_pct": 54.8,
"floor": 138773,
"candidates": 98,
"dropped": 96,
"stage": "old messages collapsed",
"state_tok": 24861,
"requests": 3,
"usd": 0.0036
}
],
"jev_usd_total": 0.1755
}

165
evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-160k.json Normal file
View File

@@ -0,0 +1,165 @@
{
"transcript": "sigsegv",
"threshold": 160000,
"raw_tokens_consumed": 417257,
"raw_total": 8553913,
"msgs_consumed": 1115,
"cycles": [
{
"cycle": 1,
"at_msg": 449,
"before": 181861,
"after": 88112,
"freed_pct": 51.5,
"floor": 84762,
"candidates": 140,
"dropped": 140,
"stage": "old messages collapsed",
"state_tok": 24961,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 2,
"at_msg": 581,
"before": 160149,
"after": 107770,
"freed_pct": 32.7,
"floor": 95258,
"candidates": 46,
"dropped": 45,
"stage": "old messages collapsed",
"state_tok": 24962,
"requests": 2,
"usd": 0.0023
},
{
"cycle": 3,
"at_msg": 820,
"before": 160418,
"after": 125852,
"freed_pct": 21.5,
"floor": 121184,
"candidates": 71,
"dropped": 71,
"stage": "old messages collapsed",
"state_tok": 24809,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 4,
"at_msg": 939,
"before": 162465,
"after": 133530,
"freed_pct": 17.8,
"floor": 126311,
"candidates": 52,
"dropped": 52,
"stage": "old messages collapsed",
"state_tok": 24815,
"requests": 2,
"usd": 0.0024
},
{
"cycle": 5,
"at_msg": 990,
"before": 160308,
"after": 146620,
"freed_pct": 8.5,
"floor": 133799,
"candidates": 16,
"dropped": 15,
"stage": "old messages collapsed",
"state_tok": 24915,
"requests": 1,
"usd": 0.0012
},
{
"cycle": 6,
"at_msg": 1023,
"before": 160097,
"after": 146959,
"freed_pct": 8.2,
"floor": 141253,
"candidates": 12,
"dropped": 12,
"stage": "old messages collapsed",
"state_tok": 24869,
"requests": 1,
"usd": 0.0012
},
{
"cycle": 7,
"at_msg": 1049,
"before": 162325,
"after": 154537,
"freed_pct": 4.8,
"floor": 143551,
"candidates": 9,
"dropped": 8,
"stage": "old messages collapsed",
"state_tok": 24787,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 8,
"at_msg": 1079,
"before": 160027,
"after": 150727,
"freed_pct": 5.8,
"floor": 146251,
"candidates": 6,
"dropped": 6,
"stage": "old messages collapsed",
"state_tok": 24914,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 9,
"at_msg": 1104,
"before": 160066,
"after": 158598,
"freed_pct": 0.9,
"floor": 149902,
"candidates": 6,
"dropped": 6,
"stage": "old messages collapsed",
"state_tok": 24834,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 10,
"at_msg": 1113,
"before": 160055,
"after": 158044,
"freed_pct": 1.3,
"floor": 151359,
"candidates": 3,
"dropped": 2,
"stage": "old messages collapsed",
"state_tok": 24921,
"requests": 1,
"usd": 0.0011
},
{
"cycle": 11,
"at_msg": 1115,
"before": 160235,
"after": 160235,
"freed_pct": 0.0,
"floor": 151359,
"candidates": 1,
"dropped": 0,
"stage": "old messages collapsed",
"state_tok": 24826,
"requests": 1,
"usd": 0.0011,
"stuck": true
}
],
"jev_usd_total": 0.02
}

570
evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-500k.json Normal file
View File

@@ -0,0 +1,570 @@
{
"transcript": "sigsegv",
"threshold": 500000,
"raw_tokens_consumed": 5233919,
"raw_total": 8553913,
"msgs_consumed": 11655,
"cycles": [
{
"cycle": 1,
"at_msg": 1287,
"before": 500038,
"after": 182626,
"freed_pct": 63.5,
"floor": 174991,
"candidates": 400,
"dropped": 400,
"stage": "old calls merged",
"state_tok": 17662,
"requests": 5,
"usd": 0.0053
},
{
"cycle": 2,
"at_msg": 1852,
"before": 500020,
"after": 207103,
"freed_pct": 58.6,
"floor": 183936,
"candidates": 278,
"dropped": 277,
"stage": "old messages left out",
"state_tok": 24991,
"requests": 8,
"usd": 0.0102
},
{
"cycle": 3,
"at_msg": 2693,
"before": 501211,
"after": 223256,
"freed_pct": 55.5,
"floor": 190801,
"candidates": 420,
"dropped": 420,
"stage": "old calls merged",
"state_tok": 19025,
"requests": 5,
"usd": 0.0059
},
{
"cycle": 4,
"at_msg": 3133,
"before": 502403,
"after": 246015,
"freed_pct": 51.0,
"floor": 198262,
"candidates": 214,
"dropped": 214,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 6,
"usd": 0.0077
},
{
"cycle": 5,
"at_msg": 3735,
"before": 500306,
"after": 270602,
"freed_pct": 45.9,
"floor": 218244,
"candidates": 295,
"dropped": 295,
"stage": "old messages left out",
"state_tok": 24989,
"requests": 8,
"usd": 0.0101
},
{
"cycle": 6,
"at_msg": 4342,
"before": 501091,
"after": 281554,
"freed_pct": 43.8,
"floor": 226344,
"candidates": 307,
"dropped": 304,
"stage": "old messages left out",
"state_tok": 24981,
"requests": 8,
"usd": 0.0101
},
{
"cycle": 7,
"at_msg": 4768,
"before": 509249,
"after": 307352,
"freed_pct": 39.6,
"floor": 231248,
"candidates": 220,
"dropped": 220,
"stage": "old messages left out",
"state_tok": 24976,
"requests": 6,
"usd": 0.0076
},
{
"cycle": 8,
"at_msg": 5269,
"before": 500887,
"after": 294854,
"freed_pct": 41.1,
"floor": 237531,
"candidates": 249,
"dropped": 249,
"stage": "old messages left out",
"state_tok": 24982,
"requests": 7,
"usd": 0.0088
},
{
"cycle": 9,
"at_msg": 5761,
"before": 501980,
"after": 301870,
"freed_pct": 39.9,
"floor": 244141,
"candidates": 249,
"dropped": 249,
"stage": "old messages left out",
"state_tok": 24993,
"requests": 7,
"usd": 0.0088
},
{
"cycle": 10,
"at_msg": 6220,
"before": 500834,
"after": 322952,
"freed_pct": 35.5,
"floor": 265593,
"candidates": 236,
"dropped": 236,
"stage": "old messages left out",
"state_tok": 24999,
"requests": 7,
"usd": 0.0089
},
{
"cycle": 11,
"at_msg": 6572,
"before": 502511,
"after": 335256,
"freed_pct": 33.3,
"floor": 274114,
"candidates": 185,
"dropped": 184,
"stage": "old messages left out",
"state_tok": 24987,
"requests": 5,
"usd": 0.0064
},
{
"cycle": 12,
"at_msg": 6875,
"before": 501130,
"after": 351857,
"freed_pct": 29.8,
"floor": 291380,
"candidates": 151,
"dropped": 151,
"stage": "old messages left out",
"state_tok": 24982,
"requests": 4,
"usd": 0.0052
},
{
"cycle": 13,
"at_msg": 7159,
"before": 500551,
"after": 362489,
"freed_pct": 27.6,
"floor": 297211,
"candidates": 145,
"dropped": 145,
"stage": "old messages left out",
"state_tok": 24996,
"requests": 4,
"usd": 0.0051
},
{
"cycle": 14,
"at_msg": 7515,
"before": 500283,
"after": 364245,
"freed_pct": 27.2,
"floor": 302243,
"candidates": 178,
"dropped": 178,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 5,
"usd": 0.0063
},
{
"cycle": 15,
"at_msg": 7819,
"before": 500074,
"after": 371163,
"freed_pct": 25.8,
"floor": 305408,
"candidates": 161,
"dropped": 158,
"stage": "old messages left out",
"state_tok": 24984,
"requests": 5,
"usd": 0.0063
},
{
"cycle": 16,
"at_msg": 8124,
"before": 500191,
"after": 376066,
"freed_pct": 24.8,
"floor": 307123,
"candidates": 156,
"dropped": 155,
"stage": "old messages left out",
"state_tok": 24995,
"requests": 4,
"usd": 0.0051
},
{
"cycle": 17,
"at_msg": 8344,
"before": 508392,
"after": 401559,
"freed_pct": 21.0,
"floor": 311053,
"candidates": 103,
"dropped": 103,
"stage": "old messages left out",
"state_tok": 24987,
"requests": 3,
"usd": 0.0038
},
{
"cycle": 18,
"at_msg": 8565,
"before": 501436,
"after": 395901,
"freed_pct": 21.0,
"floor": 313245,
"candidates": 108,
"dropped": 108,
"stage": "old messages left out",
"state_tok": 24973,
"requests": 3,
"usd": 0.0038
},
{
"cycle": 19,
"at_msg": 8730,
"before": 501934,
"after": 412806,
"freed_pct": 17.8,
"floor": 320186,
"candidates": 73,
"dropped": 73,
"stage": "old messages left out",
"state_tok": 24994,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 20,
"at_msg": 8897,
"before": 504766,
"after": 428240,
"freed_pct": 15.2,
"floor": 321876,
"candidates": 83,
"dropped": 83,
"stage": "old messages left out",
"state_tok": 24972,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 21,
"at_msg": 9057,
"before": 500496,
"after": 416652,
"freed_pct": 16.8,
"floor": 323635,
"candidates": 78,
"dropped": 77,
"stage": "old messages left out",
"state_tok": 24987,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 22,
"at_msg": 9234,
"before": 500162,
"after": 419587,
"freed_pct": 16.1,
"floor": 326454,
"candidates": 86,
"dropped": 86,
"stage": "old messages left out",
"state_tok": 24980,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 23,
"at_msg": 9372,
"before": 500372,
"after": 422027,
"freed_pct": 15.7,
"floor": 328514,
"candidates": 72,
"dropped": 72,
"stage": "old messages left out",
"state_tok": 24984,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 24,
"at_msg": 9590,
"before": 500061,
"after": 424811,
"freed_pct": 15.0,
"floor": 331120,
"candidates": 111,
"dropped": 110,
"stage": "old messages left out",
"state_tok": 24984,
"requests": 3,
"usd": 0.0039
},
{
"cycle": 25,
"at_msg": 9834,
"before": 500825,
"after": 431432,
"freed_pct": 13.9,
"floor": 333107,
"candidates": 119,
"dropped": 119,
"stage": "old messages left out",
"state_tok": 24997,
"requests": 4,
"usd": 0.005
},
{
"cycle": 26,
"at_msg": 9970,
"before": 502050,
"after": 438280,
"freed_pct": 12.7,
"floor": 335981,
"candidates": 67,
"dropped": 67,
"stage": "old messages left out",
"state_tok": 25000,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 27,
"at_msg": 10078,
"before": 502008,
"after": 434977,
"freed_pct": 13.4,
"floor": 336040,
"candidates": 60,
"dropped": 59,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 28,
"at_msg": 10231,
"before": 500000,
"after": 437925,
"freed_pct": 12.4,
"floor": 339747,
"candidates": 75,
"dropped": 75,
"stage": "old messages left out",
"state_tok": 24997,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 29,
"at_msg": 10362,
"before": 500680,
"after": 439569,
"freed_pct": 12.2,
"floor": 340881,
"candidates": 65,
"dropped": 63,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 30,
"at_msg": 10477,
"before": 500468,
"after": 447049,
"freed_pct": 10.7,
"floor": 343522,
"candidates": 60,
"dropped": 60,
"stage": "old messages left out",
"state_tok": 24980,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 31,
"at_msg": 10535,
"before": 500587,
"after": 447370,
"freed_pct": 10.6,
"floor": 343646,
"candidates": 31,
"dropped": 30,
"stage": "old messages left out",
"state_tok": 24970,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 32,
"at_msg": 10676,
"before": 500303,
"after": 442218,
"freed_pct": 11.6,
"floor": 344054,
"candidates": 70,
"dropped": 70,
"stage": "old messages left out",
"state_tok": 24984,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 33,
"at_msg": 10803,
"before": 500622,
"after": 448083,
"freed_pct": 10.5,
"floor": 346467,
"candidates": 62,
"dropped": 56,
"stage": "old messages left out",
"state_tok": 25000,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 34,
"at_msg": 10935,
"before": 500437,
"after": 448433,
"freed_pct": 10.4,
"floor": 348865,
"candidates": 72,
"dropped": 72,
"stage": "old messages left out",
"state_tok": 24970,
"requests": 2,
"usd": 0.0026
},
{
"cycle": 35,
"at_msg": 11103,
"before": 500346,
"after": 452133,
"freed_pct": 9.6,
"floor": 351123,
"candidates": 89,
"dropped": 88,
"stage": "old messages left out",
"state_tok": 24985,
"requests": 3,
"usd": 0.0038
},
{
"cycle": 36,
"at_msg": 11259,
"before": 500141,
"after": 450538,
"freed_pct": 9.9,
"floor": 351365,
"candidates": 82,
"dropped": 81,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 37,
"at_msg": 11349,
"before": 501953,
"after": 455933,
"freed_pct": 9.2,
"floor": 354478,
"candidates": 46,
"dropped": 46,
"stage": "old messages left out",
"state_tok": 24970,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 38,
"at_msg": 11458,
"before": 500610,
"after": 467843,
"freed_pct": 6.5,
"floor": 356508,
"candidates": 55,
"dropped": 55,
"stage": "old messages left out",
"state_tok": 24998,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 39,
"at_msg": 11540,
"before": 500396,
"after": 455754,
"freed_pct": 8.9,
"floor": 356786,
"candidates": 41,
"dropped": 41,
"stage": "old messages left out",
"state_tok": 24989,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 40,
"at_msg": 11655,
"before": 500495,
"after": 460570,
"freed_pct": 8.0,
"floor": 359496,
"candidates": 57,
"dropped": 57,
"stage": "old messages left out",
"state_tok": 24982,
"requests": 2,
"usd": 0.0025
}
],
"jev_usd_total": 0.1896
}

570
evals/compaction/results/jev-cycles-2026-09-19/cycles-sysprompt-500k.json Normal file
View File

@@ -0,0 +1,570 @@
{
"transcript": "sysprompt",
"threshold": 500000,
"raw_tokens_consumed": 8601919,
"raw_total": 23311571,
"msgs_consumed": 6663,
"cycles": [
{
"cycle": 1,
"at_msg": 508,
"before": 500281,
"after": 122136,
"freed_pct": 75.6,
"floor": 91399,
"candidates": 200,
"dropped": 200,
"stage": "old messages collapsed",
"state_tok": 24958,
"requests": 6,
"usd": 0.007
},
{
"cycle": 2,
"at_msg": 731,
"before": 503628,
"after": 130821,
"freed_pct": 74.0,
"floor": 110321,
"candidates": 109,
"dropped": 109,
"stage": "old messages collapsed",
"state_tok": 24975,
"requests": 3,
"usd": 0.0035
},
{
"cycle": 3,
"at_msg": 1006,
"before": 500183,
"after": 134695,
"freed_pct": 73.1,
"floor": 128713,
"candidates": 139,
"dropped": 139,
"stage": "old messages collapsed",
"state_tok": 24952,
"requests": 4,
"usd": 0.0047
},
{
"cycle": 4,
"at_msg": 1229,
"before": 500076,
"after": 158048,
"freed_pct": 68.4,
"floor": 151031,
"candidates": 111,
"dropped": 111,
"stage": "old messages collapsed",
"state_tok": 24870,
"requests": 3,
"usd": 0.0035
},
{
"cycle": 5,
"at_msg": 1459,
"before": 501400,
"after": 179777,
"freed_pct": 64.1,
"floor": 171380,
"candidates": 115,
"dropped": 115,
"stage": "old messages collapsed",
"state_tok": 24894,
"requests": 3,
"usd": 0.0035
},
{
"cycle": 6,
"at_msg": 1759,
"before": 501647,
"after": 199215,
"freed_pct": 60.3,
"floor": 182724,
"candidates": 143,
"dropped": 143,
"stage": "old messages collapsed",
"state_tok": 24976,
"requests": 4,
"usd": 0.0048
},
{
"cycle": 7,
"at_msg": 2164,
"before": 500531,
"after": 206717,
"freed_pct": 58.7,
"floor": 189181,
"candidates": 235,
"dropped": 235,
"stage": "old calls compacted",
"state_tok": 24878,
"requests": 6,
"usd": 0.0076
},
{
"cycle": 8,
"at_msg": 2515,
"before": 508820,
"after": 223053,
"freed_pct": 56.2,
"floor": 194669,
"candidates": 199,
"dropped": 199,
"stage": "old calls compacted",
"state_tok": 24953,
"requests": 6,
"usd": 0.0075
},
{
"cycle": 9,
"at_msg": 2861,
"before": 500524,
"after": 229101,
"freed_pct": 54.2,
"floor": 199926,
"candidates": 182,
"dropped": 182,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 5,
"usd": 0.0064
},
{
"cycle": 10,
"at_msg": 3062,
"before": 500764,
"after": 237294,
"freed_pct": 52.6,
"floor": 202656,
"candidates": 107,
"dropped": 107,
"stage": "old calls compacted",
"state_tok": 24998,
"requests": 3,
"usd": 0.0038
},
{
"cycle": 11,
"at_msg": 3228,
"before": 500037,
"after": 255078,
"freed_pct": 49.0,
"floor": 203299,
"candidates": 119,
"dropped": 119,
"stage": "old calls compacted",
"state_tok": 24948,
"requests": 4,
"usd": 0.005
},
{
"cycle": 12,
"at_msg": 3471,
"before": 501341,
"after": 244372,
"freed_pct": 51.3,
"floor": 205213,
"candidates": 118,
"dropped": 118,
"stage": "old messages left out",
"state_tok": 24971,
"requests": 4,
"usd": 0.005
},
{
"cycle": 13,
"at_msg": 3621,
"before": 502278,
"after": 266125,
"freed_pct": 47.0,
"floor": 207160,
"candidates": 70,
"dropped": 70,
"stage": "old calls compacted",
"state_tok": 24969,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 14,
"at_msg": 3769,
"before": 500791,
"after": 264091,
"freed_pct": 47.3,
"floor": 209372,
"candidates": 79,
"dropped": 79,
"stage": "old calls compacted",
"state_tok": 24980,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 15,
"at_msg": 3946,
"before": 500330,
"after": 279988,
"freed_pct": 44.0,
"floor": 211711,
"candidates": 115,
"dropped": 115,
"stage": "old messages left out",
"state_tok": 24977,
"requests": 3,
"usd": 0.0039
},
{
"cycle": 16,
"at_msg": 4055,
"before": 505381,
"after": 268923,
"freed_pct": 46.8,
"floor": 213362,
"candidates": 65,
"dropped": 65,
"stage": "old calls compacted",
"state_tok": 24969,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 17,
"at_msg": 4223,
"before": 501906,
"after": 310108,
"freed_pct": 38.2,
"floor": 214263,
"candidates": 82,
"dropped": 82,
"stage": "old messages left out",
"state_tok": 24983,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 18,
"at_msg": 4359,
"before": 500152,
"after": 282762,
"freed_pct": 43.5,
"floor": 215250,
"candidates": 66,
"dropped": 66,
"stage": "old messages left out",
"state_tok": 24997,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 19,
"at_msg": 4506,
"before": 504446,
"after": 305300,
"freed_pct": 39.5,
"floor": 217046,
"candidates": 69,
"dropped": 69,
"stage": "old messages left out",
"state_tok": 24978,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 20,
"at_msg": 4660,
"before": 501723,
"after": 293927,
"freed_pct": 41.4,
"floor": 218023,
"candidates": 79,
"dropped": 79,
"stage": "old messages left out",
"state_tok": 24990,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 21,
"at_msg": 4844,
"before": 505792,
"after": 306179,
"freed_pct": 39.5,
"floor": 219729,
"candidates": 86,
"dropped": 86,
"stage": "old messages left out",
"state_tok": 24992,
"requests": 3,
"usd": 0.0037
},
{
"cycle": 22,
"at_msg": 5081,
"before": 500256,
"after": 308778,
"freed_pct": 38.3,
"floor": 220875,
"candidates": 116,
"dropped": 116,
"stage": "old messages left out",
"state_tok": 24982,
"requests": 3,
"usd": 0.0038
},
{
"cycle": 23,
"at_msg": 5202,
"before": 500337,
"after": 330910,
"freed_pct": 33.9,
"floor": 222726,
"candidates": 55,
"dropped": 55,
"stage": "old messages left out",
"state_tok": 24970,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 24,
"at_msg": 5281,
"before": 505966,
"after": 344354,
"freed_pct": 31.9,
"floor": 223692,
"candidates": 37,
"dropped": 37,
"stage": "old messages left out",
"state_tok": 24984,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 25,
"at_msg": 5358,
"before": 500802,
"after": 360397,
"freed_pct": 28.0,
"floor": 225423,
"candidates": 35,
"dropped": 35,
"stage": "old messages left out",
"state_tok": 24994,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 26,
"at_msg": 5459,
"before": 502896,
"after": 352048,
"freed_pct": 30.0,
"floor": 227627,
"candidates": 51,
"dropped": 51,
"stage": "old messages left out",
"state_tok": 24994,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 27,
"at_msg": 5551,
"before": 502348,
"after": 361513,
"freed_pct": 28.0,
"floor": 228455,
"candidates": 60,
"dropped": 60,
"stage": "old messages left out",
"state_tok": 24969,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 28,
"at_msg": 5665,
"before": 501853,
"after": 347094,
"freed_pct": 30.8,
"floor": 229579,
"candidates": 54,
"dropped": 54,
"stage": "old messages left out",
"state_tok": 24979,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 29,
"at_msg": 5772,
"before": 502020,
"after": 349120,
"freed_pct": 30.5,
"floor": 230704,
"candidates": 52,
"dropped": 52,
"stage": "old messages left out",
"state_tok": 24987,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 30,
"at_msg": 5839,
"before": 500681,
"after": 359578,
"freed_pct": 28.2,
"floor": 230907,
"candidates": 31,
"dropped": 31,
"stage": "old messages left out",
"state_tok": 24992,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 31,
"at_msg": 5919,
"before": 501642,
"after": 372640,
"freed_pct": 25.7,
"floor": 232045,
"candidates": 39,
"dropped": 39,
"stage": "old messages left out",
"state_tok": 24994,
"requests": 2,
"usd": 0.0024
},
{
"cycle": 32,
"at_msg": 6017,
"before": 502268,
"after": 380565,
"freed_pct": 24.2,
"floor": 232644,
"candidates": 58,
"dropped": 58,
"stage": "old messages left out",
"state_tok": 24991,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 33,
"at_msg": 6108,
"before": 501239,
"after": 379645,
"freed_pct": 24.3,
"floor": 233193,
"candidates": 43,
"dropped": 43,
"stage": "old messages left out",
"state_tok": 24980,
"requests": 2,
"usd": 0.0024
},
{
"cycle": 34,
"at_msg": 6178,
"before": 502893,
"after": 380806,
"freed_pct": 24.3,
"floor": 234168,
"candidates": 33,
"dropped": 33,
"stage": "old messages left out",
"state_tok": 24994,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 35,
"at_msg": 6310,
"before": 502521,
"after": 376822,
"freed_pct": 25.0,
"floor": 236234,
"candidates": 68,
"dropped": 68,
"stage": "old messages left out",
"state_tok": 24972,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 36,
"at_msg": 6374,
"before": 500684,
"after": 380201,
"freed_pct": 24.1,
"floor": 236266,
"candidates": 45,
"dropped": 45,
"stage": "old messages left out",
"state_tok": 24989,
"requests": 2,
"usd": 0.0025
},
{
"cycle": 37,
"at_msg": 6456,
"before": 506443,
"after": 402465,
"freed_pct": 20.5,
"floor": 237354,
"candidates": 40,
"dropped": 40,
"stage": "old messages left out",
"state_tok": 24992,
"requests": 2,
"usd": 0.0024
},
{
"cycle": 38,
"at_msg": 6516,
"before": 502959,
"after": 390405,
"freed_pct": 22.4,
"floor": 237724,
"candidates": 28,
"dropped": 28,
"stage": "old messages left out",
"state_tok": 24996,
"requests": 1,
"usd": 0.0012
},
{
"cycle": 39,
"at_msg": 6588,
"before": 500087,
"after": 403242,
"freed_pct": 19.4,
"floor": 238371,
"candidates": 35,
"dropped": 35,
"stage": "old messages left out",
"state_tok": 24987,
"requests": 1,
"usd": 0.0013
},
{
"cycle": 40,
"at_msg": 6663,
"before": 500286,
"after": 401251,
"freed_pct": 19.8,
"floor": 239653,
"candidates": 44,
"dropped": 44,
"stage": "old messages left out",
"state_tok": 24989,
"requests": 2,
"usd": 0.0025
}
],
"jev_usd_total": 0.132
}

59
evals/compaction/scripts/jev_cycles.py Normal file
View File

@@ -0,0 +1,59 @@
#!/usr/bin/env python3
"""Simulate repeated Jev compaction over a growing session.
Usage: jev_cycles.py <lineage_json> <threshold_tokens> <max_cycles>
Feed a lineage chronologically; whenever the estimated context crosses the
threshold, compact with the Jev arm and record the cycle. Stops when the
transcript ends, Jev cannot fit its state (plugin fallback), or a compaction
frees nothing.
"""
import json
import sys
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
from evals.compaction.fixtures import estimate_tokens, load_transcript # noqa: E402
from evals.compaction.jev_arm import JevCompactor, JevOptions # noqa: E402
path, threshold, max_cycles = sys.argv[1], int(sys.argv[2]), int(sys.argv[3])
name = Path(path).stem
msgs = load_transcript(path)
ctx, i, cycles, total_jev_usd = [], 0, [], 0.0
def tokens(ms):
return sum(estimate_tokens(m) for m in ms)
def text_floor(ms):
return sum(estimate_tokens(m) for m in ms if m.get("role") != "tool" and not m.get("tool_calls"))
while i < len(msgs) and len(cycles) < max_cycles:
ctx.append(msgs[i]); i += 1
if tokens(ctx) < threshold or msgs[i - 1].get("tool_calls"):
continue # only compact on a well-formed boundary (result rows present)
jc = JevCompactor(options=JevOptions())
before = tokens(ctx)
try:
out = jc.compress(ctx)
except ValueError as e:
cycles.append({"cycle": len(cycles) + 1, "at_msg": i, "before": before, "fallback": str(e)[:90],
"floor": text_floor(ctx), "calls": len(jc.decisions)})
break
total_jev_usd += jc.usage.cost_usd
after = tokens(out)
cycles.append({"cycle": len(cycles) + 1, "at_msg": i, "before": before, "after": after,
"freed_pct": round(100 * (before - after) / before, 1), "floor": text_floor(out),
"candidates": len(jc.decisions) - jc.stats["pinned"], "dropped": jc.stats["calls_dropped"],
"stage": jc.stats["state_stage"], "state_tok": jc.stats["state_tokens"],
"requests": jc.stats["requests"], "usd": round(jc.usage.cost_usd, 4)})
if after >= threshold:
cycles[-1]["stuck"] = True
break
ctx = out
print(json.dumps({"transcript": name, "threshold": threshold, "raw_tokens_consumed": tokens(msgs[:i]),
"raw_total": tokens(msgs), "msgs_consumed": i, "cycles": cycles,
"jev_usd_total": round(total_jev_usd, 4)}, indent=1))