diff --git a/evals/compaction/results/SCORECARD-2026-09-19-jev.md b/evals/compaction/results/SCORECARD-2026-09-19-jev.md index 71523eb9d3..bb6b3c8f55 100644 --- a/evals/compaction/results/SCORECARD-2026-09-19-jev.md +++ b/evals/compaction/results/SCORECARD-2026-09-19-jev.md @@ -1,5 +1,28 @@ # fast-jev-compaction vs Hermes compaction — 3-transcript scorecard (2026-09-19) +## Verdict + +**Do not adopt Jev as Hermes' compaction. Do adopt the retention rule it demonstrates.** + +- The +32 pt recall win is entirely "keep user/assistant text verbatim, delete old tool + output". Jev itself dropped 100% of 851 candidates at its default threshold and, at a + matched token budget, ranked no better than plain recency (77.8 vs 77.8). +- Jev-only compaction is a one-way ratchet: the text floor grows every cycle and is never + reduced, so freed-per-compaction decays (89% → 55%, 76% → 20%, 63% → 8% over 32–40 + cycles) and a text-heavy session on a 200K-window host hit a hard stop after 0.4M tokens + of work (floor ≥ threshold, 0% freed). A summary path is still required; Jev could only be + a pre-pass. +- The 32K request window forces the whole history into 25K tokens: in every measured cycle + the ceiling was binding and Jev decided on tool name + 60-char input + result size with + message texts replaced by `[… N chars omitted …]`. >~500 tool calls between compactions + cannot fit at all (1 of 4 transcripts never got a first compaction). +- What it does buy: 1.4 s and < 1¢ per compaction vs 37 s and 6¢, and verbatim retention + of everything not a tool result. + +Recommended follow-up (not this PR): change the compressor's retention posture to prune +tool results first and summarise only the remainder, keeping assistant text verbatim within +the tail budget. Zero external dependency, captures most of the measured gain. + Question asked: does https://github.com/tamaratran/fast-jev-compaction ("replace the compaction summary with Jev decisions: score every tool call/result, drop or truncate stale ones, keep everything else verbatim") beat our compressor on remaining tokens, @@ -75,6 +98,45 @@ posture would capture most of Jev's gain at zero extra cost. If a scoring model for the prune ranking, Jev is fast and cheap enough (1.5 s, < 1¢) but this data shows no signal over recency at equal budget; re-test before wiring it in. +## Repeated compaction: how many cycles does Jev-only compaction survive? + +`scripts/jev_cycles.py` feeds a lineage chronologically and compacts with Jev (plugin +defaults) every time the estimate crosses the threshold; 500K ≈ our 1M-window posture, 160K +≈ a 200K-window host. Raw JSON per run in `results/jev-cycles-2026-09-19/`. Jev spend for all +six runs: $0.60. + +| run | cycles | raw session consumed | freed per cycle | text floor | end state | +|---|---|---|---|---|---| +| prreview @500K | 32 | 11.4M (whole lineage) | 89% → 55% | 43K → 139K | still working | +| sysprompt @500K | 40 | 8.6M of 23.3M | 76% → 20% | 91K → 240K | degrading | +| sigsegv @500K | 40 | 5.2M of 8.6M | 63% → 8% | 175K → 359K | 40K freed/cycle; wall ≈ 8M | +| prreview @160K | 60 | 3.9M of 11.4M | 87% → 22% | 12K → 87K | compacting every ~14 rows | +| sigsegv @160K | 11 | 0.42M of 8.6M | 52% → 0% | 85K → 151K | **STUCK** (floor ≥ threshold) | +| afff57 @500K | 0 | — | — | — | fallback on cycle 1 (541 calls) | + +Reading: every cycle has candidates (new tool calls arrive between compactions and Jev +drops ~all of them; 200+ cycles, zero orphaned call/result pairs), so "nothing left to +remove" never happens. What runs out is headroom: each cycle frees at most +`threshold − floor`, and the floor (all user/assistant rows) only grows. Well before the +hard stop the session runs permanently near the window: sigsegv @500K at cycle 40 compacts +every ~40K tokens of new work with every turn billed at ~460K input. + +## The 32K window in practice + +- `state_tok` was 24,8xx–25,000 in every one of ~180 measured cycles: the ceiling always + binds. Fitting stages reached at 500K: "old messages collapsed" (texts → `[… N chars + omitted …]`), "old calls compacted" (one line per call, input cut to 60 chars), "old + messages left out" (text-only old rows removed from the state), "old calls merged". Jev + never sees tool result contents (by design) and, at these stages, barely sees message + text either — it decides on tool name, input stub, result size and the last 3 user prompts. +- Hard limit: ~500+ unpinned tool calls between compactions cannot fit (afff57, 541 calls); + the plugin throws and the host falls back to its summary. The text floor does not affect + fit (old text rows are left out of the state); the call count does. +- Requests: full state resent per batch of ~40–80 calls → 5–8 concurrent requests per + cycle at 500K, 1–2 s, $0.005–0.008. +- Untouchable content: anything in assistant text (pasted logs, long analyses) is never + compacted, which is exactly what makes the floor grow. + ## Method notes - Transcripts reconstructed with `scripts/reconstruct_lineage.py` from a state.db copy; diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-afff57-500k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-afff57-500k.json new file mode 100644 index 0000000000..78fced42ee --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-afff57-500k.json @@ -0,0 +1,18 @@ +{ + "transcript": "afff57", + "threshold": 500000, + "raw_tokens_consumed": 500301, + "raw_total": 7357845, + "msgs_consumed": 1108, + "cycles": [ + { + "cycle": 1, + "at_msg": 1108, + "before": 500301, + "fallback": "history too large for Jev (~25972 tokens after truncation, limit 25000)", + "floor": 22650, + "calls": 0 + } + ], + "jev_usd_total": 0.0 +} diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-160k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-160k.json new file mode 100644 index 0000000000..46c0b7a731 --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-160k.json @@ -0,0 +1,850 @@ +{ + "transcript": "prreview", + "threshold": 160000, + "raw_tokens_consumed": 3908081, + "raw_total": 11361152, + "msgs_consumed": 3778, + "cycles": [ + { + "cycle": 1, + "at_msg": 84, + "before": 161753, + "after": 20516, + "freed_pct": 87.3, + "floor": 11651, + "candidates": 43, + "dropped": 38, + "stage": "full", + "state_tok": 24216, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 2, + "at_msg": 204, + "before": 160345, + "after": 25821, + "freed_pct": 83.9, + "floor": 20754, + "candidates": 69, + "dropped": 67, + "stage": "texts abridged", + "state_tok": 18878, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 3, + "at_msg": 417, + "before": 160545, + "after": 43910, + "freed_pct": 72.6, + "floor": 38723, + "candidates": 97, + "dropped": 95, + "stage": "texts abridged", + "state_tok": 16544, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 4, + "at_msg": 603, + "before": 161273, + "after": 62761, + "freed_pct": 61.1, + "floor": 50469, + "candidates": 82, + "dropped": 78, + "stage": "texts abridged", + "state_tok": 23411, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 5, + "at_msg": 770, + "before": 161973, + "after": 77531, + "freed_pct": 52.1, + "floor": 61000, + "candidates": 76, + "dropped": 66, + "stage": "texts abridged", + "state_tok": 23272, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 6, + "at_msg": 864, + "before": 161232, + "after": 104685, + "freed_pct": 35.1, + "floor": 65705, + "candidates": 48, + "dropped": 46, + "stage": "texts abridged", + "state_tok": 23088, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 7, + "at_msg": 905, + "before": 169559, + "after": 81925, + "freed_pct": 51.7, + "floor": 66307, + "candidates": 30, + "dropped": 29, + "stage": "texts abridged", + "state_tok": 21283, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 8, + "at_msg": 948, + "before": 164874, + "after": 90444, + "freed_pct": 45.1, + "floor": 66783, + "candidates": 24, + "dropped": 23, + "stage": "texts abridged", + "state_tok": 21077, + "requests": 1, + "usd": 0.0009 + }, + { + "cycle": 9, + "at_msg": 961, + "before": 165153, + "after": 112109, + "freed_pct": 32.1, + "floor": 66783, + "candidates": 9, + "dropped": 8, + "stage": "texts abridged", + "state_tok": 19750, + "requests": 1, + "usd": 0.0008 + }, + { + "cycle": 10, + "at_msg": 995, + "before": 166475, + "after": 90196, + "freed_pct": 45.8, + "floor": 67678, + "candidates": 25, + "dropped": 25, + "stage": "texts abridged", + "state_tok": 21316, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 11, + "at_msg": 1035, + "before": 161835, + "after": 90088, + "freed_pct": 44.3, + "floor": 68354, + "candidates": 20, + "dropped": 20, + "stage": "texts abridged", + "state_tok": 21796, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 12, + "at_msg": 1101, + "before": 160756, + "after": 76512, + "freed_pct": 52.4, + "floor": 68354, + "candidates": 42, + "dropped": 42, + "stage": "texts abridged", + "state_tok": 23610, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 13, + "at_msg": 1175, + "before": 162050, + "after": 81482, + "freed_pct": 49.7, + "floor": 69504, + "candidates": 43, + "dropped": 41, + "stage": "texts abridged", + "state_tok": 24232, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 14, + "at_msg": 1269, + "before": 160807, + "after": 83741, + "freed_pct": 47.9, + "floor": 69679, + "candidates": 62, + "dropped": 61, + "stage": "old messages collapsed", + "state_tok": 24863, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 15, + "at_msg": 1346, + "before": 162548, + "after": 78873, + "freed_pct": 51.5, + "floor": 70824, + "candidates": 47, + "dropped": 47, + "stage": "texts abridged", + "state_tok": 24955, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 16, + "at_msg": 1433, + "before": 160045, + "after": 80210, + "freed_pct": 49.9, + "floor": 70824, + "candidates": 41, + "dropped": 41, + "stage": "texts abridged", + "state_tok": 24696, + "requests": 1, + "usd": 0.0012 + }, + { + "cycle": 17, + "at_msg": 1481, + "before": 164436, + "after": 102906, + "freed_pct": 37.4, + "floor": 71719, + "candidates": 23, + "dropped": 23, + "stage": "texts abridged", + "state_tok": 23269, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 18, + "at_msg": 1514, + "before": 169351, + "after": 98544, + "freed_pct": 41.8, + "floor": 71719, + "candidates": 21, + "dropped": 21, + "stage": "texts abridged", + "state_tok": 22944, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 19, + "at_msg": 1606, + "before": 169524, + "after": 99953, + "freed_pct": 41.0, + "floor": 72852, + "candidates": 59, + "dropped": 59, + "stage": "old messages collapsed", + "state_tok": 24973, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 20, + "at_msg": 1627, + "before": 161132, + "after": 92569, + "freed_pct": 42.6, + "floor": 72852, + "candidates": 14, + "dropped": 14, + "stage": "texts abridged", + "state_tok": 22610, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 21, + "at_msg": 1690, + "before": 160119, + "after": 80377, + "freed_pct": 49.8, + "floor": 72852, + "candidates": 40, + "dropped": 40, + "stage": "texts abridged", + "state_tok": 24904, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 22, + "at_msg": 1754, + "before": 160964, + "after": 88796, + "freed_pct": 44.8, + "floor": 73786, + "candidates": 38, + "dropped": 38, + "stage": "old messages collapsed", + "state_tok": 24978, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 23, + "at_msg": 1843, + "before": 160168, + "after": 87896, + "freed_pct": 45.1, + "floor": 73786, + "candidates": 53, + "dropped": 53, + "stage": "old messages collapsed", + "state_tok": 24870, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 24, + "at_msg": 1871, + "before": 165096, + "after": 95841, + "freed_pct": 41.9, + "floor": 74776, + "candidates": 15, + "dropped": 15, + "stage": "texts abridged", + "state_tok": 23510, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 25, + "at_msg": 1923, + "before": 160843, + "after": 110837, + "freed_pct": 31.1, + "floor": 75759, + "candidates": 34, + "dropped": 34, + "stage": "old messages collapsed", + "state_tok": 24877, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 26, + "at_msg": 1953, + "before": 160363, + "after": 88775, + "freed_pct": 44.6, + "floor": 75759, + "candidates": 16, + "dropped": 14, + "stage": "texts abridged", + "state_tok": 24027, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 27, + "at_msg": 2047, + "before": 171545, + "after": 115659, + "freed_pct": 32.6, + "floor": 76397, + "candidates": 45, + "dropped": 44, + "stage": "old messages collapsed", + "state_tok": 24930, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 28, + "at_msg": 2093, + "before": 162639, + "after": 91133, + "freed_pct": 44.0, + "floor": 76397, + "candidates": 24, + "dropped": 23, + "stage": "old messages collapsed", + "state_tok": 24991, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 29, + "at_msg": 2239, + "before": 160686, + "after": 92844, + "freed_pct": 42.2, + "floor": 76989, + "candidates": 70, + "dropped": 70, + "stage": "old messages collapsed", + "state_tok": 24842, + "requests": 2, + "usd": 0.0023 + }, + { + "cycle": 30, + "at_msg": 2273, + "before": 160421, + "after": 94811, + "freed_pct": 40.9, + "floor": 76989, + "candidates": 15, + "dropped": 14, + "stage": "texts abridged", + "state_tok": 24605, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 31, + "at_msg": 2357, + "before": 171760, + "after": 128557, + "freed_pct": 25.2, + "floor": 77549, + "candidates": 38, + "dropped": 38, + "stage": "old messages collapsed", + "state_tok": 24981, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 32, + "at_msg": 2369, + "before": 160215, + "after": 114061, + "freed_pct": 28.8, + "floor": 77549, + "candidates": 7, + "dropped": 6, + "stage": "texts abridged", + "state_tok": 24087, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 33, + "at_msg": 2415, + "before": 167023, + "after": 121299, + "freed_pct": 27.4, + "floor": 78211, + "candidates": 22, + "dropped": 21, + "stage": "old messages collapsed", + "state_tok": 24971, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 34, + "at_msg": 2442, + "before": 160572, + "after": 96497, + "freed_pct": 39.9, + "floor": 78211, + "candidates": 16, + "dropped": 15, + "stage": "old messages collapsed", + "state_tok": 24963, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 35, + "at_msg": 2498, + "before": 161248, + "after": 132997, + "freed_pct": 17.5, + "floor": 78779, + "candidates": 26, + "dropped": 26, + "stage": "old messages collapsed", + "state_tok": 24924, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 36, + "at_msg": 2523, + "before": 160787, + "after": 103239, + "freed_pct": 35.8, + "floor": 78779, + "candidates": 16, + "dropped": 15, + "stage": "old messages collapsed", + "state_tok": 24941, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 37, + "at_msg": 2625, + "before": 160431, + "after": 96660, + "freed_pct": 39.7, + "floor": 78779, + "candidates": 57, + "dropped": 57, + "stage": "old messages collapsed", + "state_tok": 24891, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 38, + "at_msg": 2665, + "before": 161961, + "after": 123182, + "freed_pct": 23.9, + "floor": 79463, + "candidates": 17, + "dropped": 16, + "stage": "old messages collapsed", + "state_tok": 24942, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 39, + "at_msg": 2692, + "before": 160484, + "after": 101551, + "freed_pct": 36.7, + "floor": 80148, + "candidates": 14, + "dropped": 14, + "stage": "old messages collapsed", + "state_tok": 24926, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 40, + "at_msg": 2711, + "before": 169093, + "after": 118561, + "freed_pct": 29.9, + "floor": 80148, + "candidates": 9, + "dropped": 8, + "stage": "old messages collapsed", + "state_tok": 24919, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 41, + "at_msg": 2770, + "before": 160083, + "after": 99987, + "freed_pct": 37.5, + "floor": 80148, + "candidates": 30, + "dropped": 29, + "stage": "old messages collapsed", + "state_tok": 24997, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 42, + "at_msg": 2805, + "before": 162263, + "after": 124566, + "freed_pct": 23.2, + "floor": 80797, + "candidates": 19, + "dropped": 17, + "stage": "old messages collapsed", + "state_tok": 24898, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 43, + "at_msg": 2849, + "before": 160207, + "after": 101501, + "freed_pct": 36.6, + "floor": 80797, + "candidates": 24, + "dropped": 24, + "stage": "old messages collapsed", + "state_tok": 24900, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 44, + "at_msg": 2986, + "before": 160607, + "after": 101684, + "freed_pct": 36.7, + "floor": 80797, + "candidates": 70, + "dropped": 69, + "stage": "old messages collapsed", + "state_tok": 24947, + "requests": 2, + "usd": 0.0023 + }, + { + "cycle": 45, + "at_msg": 3022, + "before": 162446, + "after": 105814, + "freed_pct": 34.9, + "floor": 81538, + "candidates": 17, + "dropped": 17, + "stage": "old messages collapsed", + "state_tok": 24930, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 46, + "at_msg": 3111, + "before": 160129, + "after": 115971, + "freed_pct": 27.6, + "floor": 82171, + "candidates": 42, + "dropped": 42, + "stage": "old messages collapsed", + "state_tok": 24841, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 47, + "at_msg": 3130, + "before": 163879, + "after": 113877, + "freed_pct": 30.5, + "floor": 82171, + "candidates": 9, + "dropped": 9, + "stage": "old messages collapsed", + "state_tok": 24981, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 48, + "at_msg": 3235, + "before": 161073, + "after": 105216, + "freed_pct": 34.7, + "floor": 82732, + "candidates": 52, + "dropped": 52, + "stage": "old messages collapsed", + "state_tok": 24982, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 49, + "at_msg": 3262, + "before": 165994, + "after": 113169, + "freed_pct": 31.8, + "floor": 84296, + "candidates": 16, + "dropped": 16, + "stage": "old messages collapsed", + "state_tok": 24878, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 50, + "at_msg": 3295, + "before": 162010, + "after": 110995, + "freed_pct": 31.5, + "floor": 84296, + "candidates": 18, + "dropped": 18, + "stage": "old messages collapsed", + "state_tok": 24936, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 51, + "at_msg": 3383, + "before": 160170, + "after": 107831, + "freed_pct": 32.7, + "floor": 84296, + "candidates": 42, + "dropped": 42, + "stage": "old messages collapsed", + "state_tok": 24977, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 52, + "at_msg": 3495, + "before": 161271, + "after": 111319, + "freed_pct": 31.0, + "floor": 84897, + "candidates": 64, + "dropped": 64, + "stage": "old messages collapsed", + "state_tok": 24975, + "requests": 2, + "usd": 0.0022 + }, + { + "cycle": 53, + "at_msg": 3525, + "before": 160973, + "after": 116263, + "freed_pct": 27.8, + "floor": 85505, + "candidates": 19, + "dropped": 19, + "stage": "old messages collapsed", + "state_tok": 24905, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 54, + "at_msg": 3567, + "before": 165268, + "after": 118425, + "freed_pct": 28.3, + "floor": 85782, + "candidates": 23, + "dropped": 23, + "stage": "old messages collapsed", + "state_tok": 24996, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 55, + "at_msg": 3616, + "before": 162401, + "after": 123737, + "freed_pct": 23.8, + "floor": 86298, + "candidates": 20, + "dropped": 20, + "stage": "old messages collapsed", + "state_tok": 24838, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 56, + "at_msg": 3632, + "before": 160426, + "after": 123165, + "freed_pct": 23.2, + "floor": 86298, + "candidates": 7, + "dropped": 6, + "stage": "old messages collapsed", + "state_tok": 24944, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 57, + "at_msg": 3690, + "before": 161487, + "after": 118670, + "freed_pct": 26.5, + "floor": 86298, + "candidates": 29, + "dropped": 29, + "stage": "old messages collapsed", + "state_tok": 24911, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 58, + "at_msg": 3755, + "before": 165439, + "after": 123947, + "freed_pct": 25.1, + "floor": 86924, + "candidates": 31, + "dropped": 31, + "stage": "old messages collapsed", + "state_tok": 24879, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 59, + "at_msg": 3764, + "before": 160849, + "after": 139191, + "freed_pct": 13.5, + "floor": 86924, + "candidates": 3, + "dropped": 3, + "stage": "old messages collapsed", + "state_tok": 24852, + "requests": 1, + "usd": 0.001 + }, + { + "cycle": 60, + "at_msg": 3778, + "before": 162699, + "after": 126592, + "freed_pct": 22.2, + "floor": 86924, + "candidates": 11, + "dropped": 11, + "stage": "old messages collapsed", + "state_tok": 24871, + "requests": 1, + "usd": 0.001 + } + ], + "jev_usd_total": 0.0792 +} diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-500k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-500k.json new file mode 100644 index 0000000000..4604a04aac --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-prreview-500k.json @@ -0,0 +1,458 @@ +{ + "transcript": "prreview", + "threshold": 500000, + "raw_tokens_consumed": 11361152, + "raw_total": 11361152, + "msgs_consumed": 10074, + "cycles": [ + { + "cycle": 1, + "at_msg": 531, + "before": 500120, + "after": 52660, + "freed_pct": 89.5, + "floor": 43449, + "candidates": 252, + "dropped": 252, + "stage": "old calls compacted", + "state_tok": 24975, + "requests": 7, + "usd": 0.0083 + }, + { + "cycle": 2, + "at_msg": 961, + "before": 511654, + "after": 111265, + "freed_pct": 78.3, + "floor": 66783, + "candidates": 199, + "dropped": 199, + "stage": "old messages collapsed", + "state_tok": 24916, + "requests": 6, + "usd": 0.0071 + }, + { + "cycle": 3, + "at_msg": 1313, + "before": 502998, + "after": 88652, + "freed_pct": 82.4, + "floor": 70824, + "candidates": 216, + "dropped": 216, + "stage": "old messages collapsed", + "state_tok": 24882, + "requests": 6, + "usd": 0.0071 + }, + { + "cycle": 4, + "at_msg": 1627, + "before": 501264, + "after": 92569, + "freed_pct": 81.5, + "floor": 72852, + "candidates": 177, + "dropped": 177, + "stage": "old messages collapsed", + "state_tok": 24921, + "requests": 5, + "usd": 0.0059 + }, + { + "cycle": 5, + "at_msg": 1947, + "before": 500375, + "after": 94749, + "freed_pct": 81.1, + "floor": 75759, + "candidates": 193, + "dropped": 191, + "stage": "old messages collapsed", + "state_tok": 24852, + "requests": 5, + "usd": 0.0059 + }, + { + "cycle": 6, + "at_msg": 2411, + "before": 519129, + "after": 114932, + "freed_pct": 77.9, + "floor": 78211, + "candidates": 219, + "dropped": 218, + "stage": "old calls compacted", + "state_tok": 24999, + "requests": 6, + "usd": 0.0072 + }, + { + "cycle": 7, + "at_msg": 2741, + "before": 502448, + "after": 104732, + "freed_pct": 79.2, + "floor": 80148, + "candidates": 168, + "dropped": 167, + "stage": "old messages collapsed", + "state_tok": 24954, + "requests": 5, + "usd": 0.0059 + }, + { + "cycle": 8, + "at_msg": 3236, + "before": 504872, + "after": 117119, + "freed_pct": 76.8, + "floor": 82732, + "candidates": 245, + "dropped": 245, + "stage": "old calls compacted", + "state_tok": 24979, + "requests": 7, + "usd": 0.0084 + }, + { + "cycle": 9, + "at_msg": 3636, + "before": 500075, + "after": 119512, + "freed_pct": 76.1, + "floor": 86298, + "candidates": 210, + "dropped": 209, + "stage": "old calls compacted", + "state_tok": 24973, + "requests": 6, + "usd": 0.0072 + }, + { + "cycle": 10, + "at_msg": 4000, + "before": 502270, + "after": 128473, + "freed_pct": 74.4, + "floor": 88838, + "candidates": 181, + "dropped": 181, + "stage": "old messages collapsed", + "state_tok": 24956, + "requests": 5, + "usd": 0.006 + }, + { + "cycle": 11, + "at_msg": 4354, + "before": 501400, + "after": 151425, + "freed_pct": 69.8, + "floor": 91026, + "candidates": 192, + "dropped": 192, + "stage": "old messages collapsed", + "state_tok": 24842, + "requests": 5, + "usd": 0.006 + }, + { + "cycle": 12, + "at_msg": 4592, + "before": 511090, + "after": 158483, + "freed_pct": 69.0, + "floor": 93179, + "candidates": 128, + "dropped": 127, + "stage": "old messages collapsed", + "state_tok": 24897, + "requests": 4, + "usd": 0.0046 + }, + { + "cycle": 13, + "at_msg": 4806, + "before": 501974, + "after": 142234, + "freed_pct": 71.7, + "floor": 95299, + "candidates": 110, + "dropped": 110, + "stage": "old messages collapsed", + "state_tok": 24984, + "requests": 3, + "usd": 0.0035 + }, + { + "cycle": 14, + "at_msg": 5124, + "before": 501941, + "after": 156309, + "freed_pct": 68.9, + "floor": 98017, + "candidates": 161, + "dropped": 160, + "stage": "old messages collapsed", + "state_tok": 24877, + "requests": 5, + "usd": 0.0059 + }, + { + "cycle": 15, + "at_msg": 5463, + "before": 500937, + "after": 154241, + "freed_pct": 69.2, + "floor": 101588, + "candidates": 176, + "dropped": 176, + "stage": "old messages collapsed", + "state_tok": 24813, + "requests": 5, + "usd": 0.006 + }, + { + "cycle": 16, + "at_msg": 5819, + "before": 503417, + "after": 162498, + "freed_pct": 67.7, + "floor": 102640, + "candidates": 233, + "dropped": 233, + "stage": "old calls compacted", + "state_tok": 24996, + "requests": 7, + "usd": 0.0083 + }, + { + "cycle": 17, + "at_msg": 6033, + "before": 507005, + "after": 188509, + "freed_pct": 62.8, + "floor": 104655, + "candidates": 110, + "dropped": 110, + "stage": "old messages collapsed", + "state_tok": 24876, + "requests": 3, + "usd": 0.0035 + }, + { + "cycle": 18, + "at_msg": 6314, + "before": 501223, + "after": 163564, + "freed_pct": 67.4, + "floor": 105926, + "candidates": 143, + "dropped": 143, + "stage": "old messages collapsed", + "state_tok": 24865, + "requests": 4, + "usd": 0.0047 + }, + { + "cycle": 19, + "at_msg": 6646, + "before": 501654, + "after": 178869, + "freed_pct": 64.3, + "floor": 109116, + "candidates": 170, + "dropped": 170, + "stage": "old calls compacted", + "state_tok": 24992, + "requests": 5, + "usd": 0.006 + }, + { + "cycle": 20, + "at_msg": 6883, + "before": 501207, + "after": 172627, + "freed_pct": 65.6, + "floor": 111444, + "candidates": 126, + "dropped": 126, + "stage": "old messages collapsed", + "state_tok": 24957, + "requests": 4, + "usd": 0.0047 + }, + { + "cycle": 21, + "at_msg": 7016, + "before": 506054, + "after": 218132, + "freed_pct": 56.9, + "floor": 114654, + "candidates": 73, + "dropped": 73, + "stage": "old messages collapsed", + "state_tok": 24816, + "requests": 2, + "usd": 0.0023 + }, + { + "cycle": 22, + "at_msg": 7283, + "before": 503596, + "after": 204002, + "freed_pct": 59.5, + "floor": 116369, + "candidates": 129, + "dropped": 125, + "stage": "old messages collapsed", + "state_tok": 24970, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 23, + "at_msg": 7503, + "before": 504496, + "after": 191152, + "freed_pct": 62.1, + "floor": 117845, + "candidates": 109, + "dropped": 109, + "stage": "old messages collapsed", + "state_tok": 24917, + "requests": 3, + "usd": 0.0036 + }, + { + "cycle": 24, + "at_msg": 7749, + "before": 501038, + "after": 205540, + "freed_pct": 59.0, + "floor": 119715, + "candidates": 120, + "dropped": 119, + "stage": "old messages collapsed", + "state_tok": 24988, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 25, + "at_msg": 8050, + "before": 515776, + "after": 211866, + "freed_pct": 58.9, + "floor": 122519, + "candidates": 155, + "dropped": 155, + "stage": "old calls compacted", + "state_tok": 24996, + "requests": 5, + "usd": 0.006 + }, + { + "cycle": 26, + "at_msg": 8314, + "before": 504042, + "after": 216111, + "freed_pct": 57.1, + "floor": 125001, + "candidates": 132, + "dropped": 132, + "stage": "old messages collapsed", + "state_tok": 24980, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 27, + "at_msg": 8533, + "before": 500651, + "after": 205303, + "freed_pct": 59.0, + "floor": 126706, + "candidates": 110, + "dropped": 105, + "stage": "old messages collapsed", + "state_tok": 25000, + "requests": 3, + "usd": 0.0036 + }, + { + "cycle": 28, + "at_msg": 8819, + "before": 502475, + "after": 251171, + "freed_pct": 50.0, + "floor": 129085, + "candidates": 143, + "dropped": 138, + "stage": "old messages collapsed", + "state_tok": 24993, + "requests": 4, + "usd": 0.0049 + }, + { + "cycle": 29, + "at_msg": 9083, + "before": 507185, + "after": 215329, + "freed_pct": 57.5, + "floor": 131182, + "candidates": 137, + "dropped": 137, + "stage": "old messages collapsed", + "state_tok": 24856, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 30, + "at_msg": 9432, + "before": 500044, + "after": 213241, + "freed_pct": 57.4, + "floor": 133547, + "candidates": 178, + "dropped": 173, + "stage": "old calls compacted", + "state_tok": 24983, + "requests": 5, + "usd": 0.0063 + }, + { + "cycle": 31, + "at_msg": 9654, + "before": 501024, + "after": 222477, + "freed_pct": 55.6, + "floor": 136282, + "candidates": 112, + "dropped": 112, + "stage": "old messages collapsed", + "state_tok": 24876, + "requests": 3, + "usd": 0.0036 + }, + { + "cycle": 32, + "at_msg": 9856, + "before": 505769, + "after": 228548, + "freed_pct": 54.8, + "floor": 138773, + "candidates": 98, + "dropped": 96, + "stage": "old messages collapsed", + "state_tok": 24861, + "requests": 3, + "usd": 0.0036 + } + ], + "jev_usd_total": 0.1755 +} diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-160k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-160k.json new file mode 100644 index 0000000000..a4d91cee23 --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-160k.json @@ -0,0 +1,165 @@ +{ + "transcript": "sigsegv", + "threshold": 160000, + "raw_tokens_consumed": 417257, + "raw_total": 8553913, + "msgs_consumed": 1115, + "cycles": [ + { + "cycle": 1, + "at_msg": 449, + "before": 181861, + "after": 88112, + "freed_pct": 51.5, + "floor": 84762, + "candidates": 140, + "dropped": 140, + "stage": "old messages collapsed", + "state_tok": 24961, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 2, + "at_msg": 581, + "before": 160149, + "after": 107770, + "freed_pct": 32.7, + "floor": 95258, + "candidates": 46, + "dropped": 45, + "stage": "old messages collapsed", + "state_tok": 24962, + "requests": 2, + "usd": 0.0023 + }, + { + "cycle": 3, + "at_msg": 820, + "before": 160418, + "after": 125852, + "freed_pct": 21.5, + "floor": 121184, + "candidates": 71, + "dropped": 71, + "stage": "old messages collapsed", + "state_tok": 24809, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 4, + "at_msg": 939, + "before": 162465, + "after": 133530, + "freed_pct": 17.8, + "floor": 126311, + "candidates": 52, + "dropped": 52, + "stage": "old messages collapsed", + "state_tok": 24815, + "requests": 2, + "usd": 0.0024 + }, + { + "cycle": 5, + "at_msg": 990, + "before": 160308, + "after": 146620, + "freed_pct": 8.5, + "floor": 133799, + "candidates": 16, + "dropped": 15, + "stage": "old messages collapsed", + "state_tok": 24915, + "requests": 1, + "usd": 0.0012 + }, + { + "cycle": 6, + "at_msg": 1023, + "before": 160097, + "after": 146959, + "freed_pct": 8.2, + "floor": 141253, + "candidates": 12, + "dropped": 12, + "stage": "old messages collapsed", + "state_tok": 24869, + "requests": 1, + "usd": 0.0012 + }, + { + "cycle": 7, + "at_msg": 1049, + "before": 162325, + "after": 154537, + "freed_pct": 4.8, + "floor": 143551, + "candidates": 9, + "dropped": 8, + "stage": "old messages collapsed", + "state_tok": 24787, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 8, + "at_msg": 1079, + "before": 160027, + "after": 150727, + "freed_pct": 5.8, + "floor": 146251, + "candidates": 6, + "dropped": 6, + "stage": "old messages collapsed", + "state_tok": 24914, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 9, + "at_msg": 1104, + "before": 160066, + "after": 158598, + "freed_pct": 0.9, + "floor": 149902, + "candidates": 6, + "dropped": 6, + "stage": "old messages collapsed", + "state_tok": 24834, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 10, + "at_msg": 1113, + "before": 160055, + "after": 158044, + "freed_pct": 1.3, + "floor": 151359, + "candidates": 3, + "dropped": 2, + "stage": "old messages collapsed", + "state_tok": 24921, + "requests": 1, + "usd": 0.0011 + }, + { + "cycle": 11, + "at_msg": 1115, + "before": 160235, + "after": 160235, + "freed_pct": 0.0, + "floor": 151359, + "candidates": 1, + "dropped": 0, + "stage": "old messages collapsed", + "state_tok": 24826, + "requests": 1, + "usd": 0.0011, + "stuck": true + } + ], + "jev_usd_total": 0.02 +} diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-500k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-500k.json new file mode 100644 index 0000000000..ac8ae58006 --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sigsegv-500k.json @@ -0,0 +1,570 @@ +{ + "transcript": "sigsegv", + "threshold": 500000, + "raw_tokens_consumed": 5233919, + "raw_total": 8553913, + "msgs_consumed": 11655, + "cycles": [ + { + "cycle": 1, + "at_msg": 1287, + "before": 500038, + "after": 182626, + "freed_pct": 63.5, + "floor": 174991, + "candidates": 400, + "dropped": 400, + "stage": "old calls merged", + "state_tok": 17662, + "requests": 5, + "usd": 0.0053 + }, + { + "cycle": 2, + "at_msg": 1852, + "before": 500020, + "after": 207103, + "freed_pct": 58.6, + "floor": 183936, + "candidates": 278, + "dropped": 277, + "stage": "old messages left out", + "state_tok": 24991, + "requests": 8, + "usd": 0.0102 + }, + { + "cycle": 3, + "at_msg": 2693, + "before": 501211, + "after": 223256, + "freed_pct": 55.5, + "floor": 190801, + "candidates": 420, + "dropped": 420, + "stage": "old calls merged", + "state_tok": 19025, + "requests": 5, + "usd": 0.0059 + }, + { + "cycle": 4, + "at_msg": 3133, + "before": 502403, + "after": 246015, + "freed_pct": 51.0, + "floor": 198262, + "candidates": 214, + "dropped": 214, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 6, + "usd": 0.0077 + }, + { + "cycle": 5, + "at_msg": 3735, + "before": 500306, + "after": 270602, + "freed_pct": 45.9, + "floor": 218244, + "candidates": 295, + "dropped": 295, + "stage": "old messages left out", + "state_tok": 24989, + "requests": 8, + "usd": 0.0101 + }, + { + "cycle": 6, + "at_msg": 4342, + "before": 501091, + "after": 281554, + "freed_pct": 43.8, + "floor": 226344, + "candidates": 307, + "dropped": 304, + "stage": "old messages left out", + "state_tok": 24981, + "requests": 8, + "usd": 0.0101 + }, + { + "cycle": 7, + "at_msg": 4768, + "before": 509249, + "after": 307352, + "freed_pct": 39.6, + "floor": 231248, + "candidates": 220, + "dropped": 220, + "stage": "old messages left out", + "state_tok": 24976, + "requests": 6, + "usd": 0.0076 + }, + { + "cycle": 8, + "at_msg": 5269, + "before": 500887, + "after": 294854, + "freed_pct": 41.1, + "floor": 237531, + "candidates": 249, + "dropped": 249, + "stage": "old messages left out", + "state_tok": 24982, + "requests": 7, + "usd": 0.0088 + }, + { + "cycle": 9, + "at_msg": 5761, + "before": 501980, + "after": 301870, + "freed_pct": 39.9, + "floor": 244141, + "candidates": 249, + "dropped": 249, + "stage": "old messages left out", + "state_tok": 24993, + "requests": 7, + "usd": 0.0088 + }, + { + "cycle": 10, + "at_msg": 6220, + "before": 500834, + "after": 322952, + "freed_pct": 35.5, + "floor": 265593, + "candidates": 236, + "dropped": 236, + "stage": "old messages left out", + "state_tok": 24999, + "requests": 7, + "usd": 0.0089 + }, + { + "cycle": 11, + "at_msg": 6572, + "before": 502511, + "after": 335256, + "freed_pct": 33.3, + "floor": 274114, + "candidates": 185, + "dropped": 184, + "stage": "old messages left out", + "state_tok": 24987, + "requests": 5, + "usd": 0.0064 + }, + { + "cycle": 12, + "at_msg": 6875, + "before": 501130, + "after": 351857, + "freed_pct": 29.8, + "floor": 291380, + "candidates": 151, + "dropped": 151, + "stage": "old messages left out", + "state_tok": 24982, + "requests": 4, + "usd": 0.0052 + }, + { + "cycle": 13, + "at_msg": 7159, + "before": 500551, + "after": 362489, + "freed_pct": 27.6, + "floor": 297211, + "candidates": 145, + "dropped": 145, + "stage": "old messages left out", + "state_tok": 24996, + "requests": 4, + "usd": 0.0051 + }, + { + "cycle": 14, + "at_msg": 7515, + "before": 500283, + "after": 364245, + "freed_pct": 27.2, + "floor": 302243, + "candidates": 178, + "dropped": 178, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 5, + "usd": 0.0063 + }, + { + "cycle": 15, + "at_msg": 7819, + "before": 500074, + "after": 371163, + "freed_pct": 25.8, + "floor": 305408, + "candidates": 161, + "dropped": 158, + "stage": "old messages left out", + "state_tok": 24984, + "requests": 5, + "usd": 0.0063 + }, + { + "cycle": 16, + "at_msg": 8124, + "before": 500191, + "after": 376066, + "freed_pct": 24.8, + "floor": 307123, + "candidates": 156, + "dropped": 155, + "stage": "old messages left out", + "state_tok": 24995, + "requests": 4, + "usd": 0.0051 + }, + { + "cycle": 17, + "at_msg": 8344, + "before": 508392, + "after": 401559, + "freed_pct": 21.0, + "floor": 311053, + "candidates": 103, + "dropped": 103, + "stage": "old messages left out", + "state_tok": 24987, + "requests": 3, + "usd": 0.0038 + }, + { + "cycle": 18, + "at_msg": 8565, + "before": 501436, + "after": 395901, + "freed_pct": 21.0, + "floor": 313245, + "candidates": 108, + "dropped": 108, + "stage": "old messages left out", + "state_tok": 24973, + "requests": 3, + "usd": 0.0038 + }, + { + "cycle": 19, + "at_msg": 8730, + "before": 501934, + "after": 412806, + "freed_pct": 17.8, + "floor": 320186, + "candidates": 73, + "dropped": 73, + "stage": "old messages left out", + "state_tok": 24994, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 20, + "at_msg": 8897, + "before": 504766, + "after": 428240, + "freed_pct": 15.2, + "floor": 321876, + "candidates": 83, + "dropped": 83, + "stage": "old messages left out", + "state_tok": 24972, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 21, + "at_msg": 9057, + "before": 500496, + "after": 416652, + "freed_pct": 16.8, + "floor": 323635, + "candidates": 78, + "dropped": 77, + "stage": "old messages left out", + "state_tok": 24987, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 22, + "at_msg": 9234, + "before": 500162, + "after": 419587, + "freed_pct": 16.1, + "floor": 326454, + "candidates": 86, + "dropped": 86, + "stage": "old messages left out", + "state_tok": 24980, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 23, + "at_msg": 9372, + "before": 500372, + "after": 422027, + "freed_pct": 15.7, + "floor": 328514, + "candidates": 72, + "dropped": 72, + "stage": "old messages left out", + "state_tok": 24984, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 24, + "at_msg": 9590, + "before": 500061, + "after": 424811, + "freed_pct": 15.0, + "floor": 331120, + "candidates": 111, + "dropped": 110, + "stage": "old messages left out", + "state_tok": 24984, + "requests": 3, + "usd": 0.0039 + }, + { + "cycle": 25, + "at_msg": 9834, + "before": 500825, + "after": 431432, + "freed_pct": 13.9, + "floor": 333107, + "candidates": 119, + "dropped": 119, + "stage": "old messages left out", + "state_tok": 24997, + "requests": 4, + "usd": 0.005 + }, + { + "cycle": 26, + "at_msg": 9970, + "before": 502050, + "after": 438280, + "freed_pct": 12.7, + "floor": 335981, + "candidates": 67, + "dropped": 67, + "stage": "old messages left out", + "state_tok": 25000, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 27, + "at_msg": 10078, + "before": 502008, + "after": 434977, + "freed_pct": 13.4, + "floor": 336040, + "candidates": 60, + "dropped": 59, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 28, + "at_msg": 10231, + "before": 500000, + "after": 437925, + "freed_pct": 12.4, + "floor": 339747, + "candidates": 75, + "dropped": 75, + "stage": "old messages left out", + "state_tok": 24997, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 29, + "at_msg": 10362, + "before": 500680, + "after": 439569, + "freed_pct": 12.2, + "floor": 340881, + "candidates": 65, + "dropped": 63, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 30, + "at_msg": 10477, + "before": 500468, + "after": 447049, + "freed_pct": 10.7, + "floor": 343522, + "candidates": 60, + "dropped": 60, + "stage": "old messages left out", + "state_tok": 24980, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 31, + "at_msg": 10535, + "before": 500587, + "after": 447370, + "freed_pct": 10.6, + "floor": 343646, + "candidates": 31, + "dropped": 30, + "stage": "old messages left out", + "state_tok": 24970, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 32, + "at_msg": 10676, + "before": 500303, + "after": 442218, + "freed_pct": 11.6, + "floor": 344054, + "candidates": 70, + "dropped": 70, + "stage": "old messages left out", + "state_tok": 24984, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 33, + "at_msg": 10803, + "before": 500622, + "after": 448083, + "freed_pct": 10.5, + "floor": 346467, + "candidates": 62, + "dropped": 56, + "stage": "old messages left out", + "state_tok": 25000, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 34, + "at_msg": 10935, + "before": 500437, + "after": 448433, + "freed_pct": 10.4, + "floor": 348865, + "candidates": 72, + "dropped": 72, + "stage": "old messages left out", + "state_tok": 24970, + "requests": 2, + "usd": 0.0026 + }, + { + "cycle": 35, + "at_msg": 11103, + "before": 500346, + "after": 452133, + "freed_pct": 9.6, + "floor": 351123, + "candidates": 89, + "dropped": 88, + "stage": "old messages left out", + "state_tok": 24985, + "requests": 3, + "usd": 0.0038 + }, + { + "cycle": 36, + "at_msg": 11259, + "before": 500141, + "after": 450538, + "freed_pct": 9.9, + "floor": 351365, + "candidates": 82, + "dropped": 81, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 37, + "at_msg": 11349, + "before": 501953, + "after": 455933, + "freed_pct": 9.2, + "floor": 354478, + "candidates": 46, + "dropped": 46, + "stage": "old messages left out", + "state_tok": 24970, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 38, + "at_msg": 11458, + "before": 500610, + "after": 467843, + "freed_pct": 6.5, + "floor": 356508, + "candidates": 55, + "dropped": 55, + "stage": "old messages left out", + "state_tok": 24998, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 39, + "at_msg": 11540, + "before": 500396, + "after": 455754, + "freed_pct": 8.9, + "floor": 356786, + "candidates": 41, + "dropped": 41, + "stage": "old messages left out", + "state_tok": 24989, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 40, + "at_msg": 11655, + "before": 500495, + "after": 460570, + "freed_pct": 8.0, + "floor": 359496, + "candidates": 57, + "dropped": 57, + "stage": "old messages left out", + "state_tok": 24982, + "requests": 2, + "usd": 0.0025 + } + ], + "jev_usd_total": 0.1896 +} diff --git a/evals/compaction/results/jev-cycles-2026-09-19/cycles-sysprompt-500k.json b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sysprompt-500k.json new file mode 100644 index 0000000000..a22ff92056 --- /dev/null +++ b/evals/compaction/results/jev-cycles-2026-09-19/cycles-sysprompt-500k.json @@ -0,0 +1,570 @@ +{ + "transcript": "sysprompt", + "threshold": 500000, + "raw_tokens_consumed": 8601919, + "raw_total": 23311571, + "msgs_consumed": 6663, + "cycles": [ + { + "cycle": 1, + "at_msg": 508, + "before": 500281, + "after": 122136, + "freed_pct": 75.6, + "floor": 91399, + "candidates": 200, + "dropped": 200, + "stage": "old messages collapsed", + "state_tok": 24958, + "requests": 6, + "usd": 0.007 + }, + { + "cycle": 2, + "at_msg": 731, + "before": 503628, + "after": 130821, + "freed_pct": 74.0, + "floor": 110321, + "candidates": 109, + "dropped": 109, + "stage": "old messages collapsed", + "state_tok": 24975, + "requests": 3, + "usd": 0.0035 + }, + { + "cycle": 3, + "at_msg": 1006, + "before": 500183, + "after": 134695, + "freed_pct": 73.1, + "floor": 128713, + "candidates": 139, + "dropped": 139, + "stage": "old messages collapsed", + "state_tok": 24952, + "requests": 4, + "usd": 0.0047 + }, + { + "cycle": 4, + "at_msg": 1229, + "before": 500076, + "after": 158048, + "freed_pct": 68.4, + "floor": 151031, + "candidates": 111, + "dropped": 111, + "stage": "old messages collapsed", + "state_tok": 24870, + "requests": 3, + "usd": 0.0035 + }, + { + "cycle": 5, + "at_msg": 1459, + "before": 501400, + "after": 179777, + "freed_pct": 64.1, + "floor": 171380, + "candidates": 115, + "dropped": 115, + "stage": "old messages collapsed", + "state_tok": 24894, + "requests": 3, + "usd": 0.0035 + }, + { + "cycle": 6, + "at_msg": 1759, + "before": 501647, + "after": 199215, + "freed_pct": 60.3, + "floor": 182724, + "candidates": 143, + "dropped": 143, + "stage": "old messages collapsed", + "state_tok": 24976, + "requests": 4, + "usd": 0.0048 + }, + { + "cycle": 7, + "at_msg": 2164, + "before": 500531, + "after": 206717, + "freed_pct": 58.7, + "floor": 189181, + "candidates": 235, + "dropped": 235, + "stage": "old calls compacted", + "state_tok": 24878, + "requests": 6, + "usd": 0.0076 + }, + { + "cycle": 8, + "at_msg": 2515, + "before": 508820, + "after": 223053, + "freed_pct": 56.2, + "floor": 194669, + "candidates": 199, + "dropped": 199, + "stage": "old calls compacted", + "state_tok": 24953, + "requests": 6, + "usd": 0.0075 + }, + { + "cycle": 9, + "at_msg": 2861, + "before": 500524, + "after": 229101, + "freed_pct": 54.2, + "floor": 199926, + "candidates": 182, + "dropped": 182, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 5, + "usd": 0.0064 + }, + { + "cycle": 10, + "at_msg": 3062, + "before": 500764, + "after": 237294, + "freed_pct": 52.6, + "floor": 202656, + "candidates": 107, + "dropped": 107, + "stage": "old calls compacted", + "state_tok": 24998, + "requests": 3, + "usd": 0.0038 + }, + { + "cycle": 11, + "at_msg": 3228, + "before": 500037, + "after": 255078, + "freed_pct": 49.0, + "floor": 203299, + "candidates": 119, + "dropped": 119, + "stage": "old calls compacted", + "state_tok": 24948, + "requests": 4, + "usd": 0.005 + }, + { + "cycle": 12, + "at_msg": 3471, + "before": 501341, + "after": 244372, + "freed_pct": 51.3, + "floor": 205213, + "candidates": 118, + "dropped": 118, + "stage": "old messages left out", + "state_tok": 24971, + "requests": 4, + "usd": 0.005 + }, + { + "cycle": 13, + "at_msg": 3621, + "before": 502278, + "after": 266125, + "freed_pct": 47.0, + "floor": 207160, + "candidates": 70, + "dropped": 70, + "stage": "old calls compacted", + "state_tok": 24969, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 14, + "at_msg": 3769, + "before": 500791, + "after": 264091, + "freed_pct": 47.3, + "floor": 209372, + "candidates": 79, + "dropped": 79, + "stage": "old calls compacted", + "state_tok": 24980, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 15, + "at_msg": 3946, + "before": 500330, + "after": 279988, + "freed_pct": 44.0, + "floor": 211711, + "candidates": 115, + "dropped": 115, + "stage": "old messages left out", + "state_tok": 24977, + "requests": 3, + "usd": 0.0039 + }, + { + "cycle": 16, + "at_msg": 4055, + "before": 505381, + "after": 268923, + "freed_pct": 46.8, + "floor": 213362, + "candidates": 65, + "dropped": 65, + "stage": "old calls compacted", + "state_tok": 24969, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 17, + "at_msg": 4223, + "before": 501906, + "after": 310108, + "freed_pct": 38.2, + "floor": 214263, + "candidates": 82, + "dropped": 82, + "stage": "old messages left out", + "state_tok": 24983, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 18, + "at_msg": 4359, + "before": 500152, + "after": 282762, + "freed_pct": 43.5, + "floor": 215250, + "candidates": 66, + "dropped": 66, + "stage": "old messages left out", + "state_tok": 24997, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 19, + "at_msg": 4506, + "before": 504446, + "after": 305300, + "freed_pct": 39.5, + "floor": 217046, + "candidates": 69, + "dropped": 69, + "stage": "old messages left out", + "state_tok": 24978, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 20, + "at_msg": 4660, + "before": 501723, + "after": 293927, + "freed_pct": 41.4, + "floor": 218023, + "candidates": 79, + "dropped": 79, + "stage": "old messages left out", + "state_tok": 24990, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 21, + "at_msg": 4844, + "before": 505792, + "after": 306179, + "freed_pct": 39.5, + "floor": 219729, + "candidates": 86, + "dropped": 86, + "stage": "old messages left out", + "state_tok": 24992, + "requests": 3, + "usd": 0.0037 + }, + { + "cycle": 22, + "at_msg": 5081, + "before": 500256, + "after": 308778, + "freed_pct": 38.3, + "floor": 220875, + "candidates": 116, + "dropped": 116, + "stage": "old messages left out", + "state_tok": 24982, + "requests": 3, + "usd": 0.0038 + }, + { + "cycle": 23, + "at_msg": 5202, + "before": 500337, + "after": 330910, + "freed_pct": 33.9, + "floor": 222726, + "candidates": 55, + "dropped": 55, + "stage": "old messages left out", + "state_tok": 24970, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 24, + "at_msg": 5281, + "before": 505966, + "after": 344354, + "freed_pct": 31.9, + "floor": 223692, + "candidates": 37, + "dropped": 37, + "stage": "old messages left out", + "state_tok": 24984, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 25, + "at_msg": 5358, + "before": 500802, + "after": 360397, + "freed_pct": 28.0, + "floor": 225423, + "candidates": 35, + "dropped": 35, + "stage": "old messages left out", + "state_tok": 24994, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 26, + "at_msg": 5459, + "before": 502896, + "after": 352048, + "freed_pct": 30.0, + "floor": 227627, + "candidates": 51, + "dropped": 51, + "stage": "old messages left out", + "state_tok": 24994, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 27, + "at_msg": 5551, + "before": 502348, + "after": 361513, + "freed_pct": 28.0, + "floor": 228455, + "candidates": 60, + "dropped": 60, + "stage": "old messages left out", + "state_tok": 24969, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 28, + "at_msg": 5665, + "before": 501853, + "after": 347094, + "freed_pct": 30.8, + "floor": 229579, + "candidates": 54, + "dropped": 54, + "stage": "old messages left out", + "state_tok": 24979, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 29, + "at_msg": 5772, + "before": 502020, + "after": 349120, + "freed_pct": 30.5, + "floor": 230704, + "candidates": 52, + "dropped": 52, + "stage": "old messages left out", + "state_tok": 24987, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 30, + "at_msg": 5839, + "before": 500681, + "after": 359578, + "freed_pct": 28.2, + "floor": 230907, + "candidates": 31, + "dropped": 31, + "stage": "old messages left out", + "state_tok": 24992, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 31, + "at_msg": 5919, + "before": 501642, + "after": 372640, + "freed_pct": 25.7, + "floor": 232045, + "candidates": 39, + "dropped": 39, + "stage": "old messages left out", + "state_tok": 24994, + "requests": 2, + "usd": 0.0024 + }, + { + "cycle": 32, + "at_msg": 6017, + "before": 502268, + "after": 380565, + "freed_pct": 24.2, + "floor": 232644, + "candidates": 58, + "dropped": 58, + "stage": "old messages left out", + "state_tok": 24991, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 33, + "at_msg": 6108, + "before": 501239, + "after": 379645, + "freed_pct": 24.3, + "floor": 233193, + "candidates": 43, + "dropped": 43, + "stage": "old messages left out", + "state_tok": 24980, + "requests": 2, + "usd": 0.0024 + }, + { + "cycle": 34, + "at_msg": 6178, + "before": 502893, + "after": 380806, + "freed_pct": 24.3, + "floor": 234168, + "candidates": 33, + "dropped": 33, + "stage": "old messages left out", + "state_tok": 24994, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 35, + "at_msg": 6310, + "before": 502521, + "after": 376822, + "freed_pct": 25.0, + "floor": 236234, + "candidates": 68, + "dropped": 68, + "stage": "old messages left out", + "state_tok": 24972, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 36, + "at_msg": 6374, + "before": 500684, + "after": 380201, + "freed_pct": 24.1, + "floor": 236266, + "candidates": 45, + "dropped": 45, + "stage": "old messages left out", + "state_tok": 24989, + "requests": 2, + "usd": 0.0025 + }, + { + "cycle": 37, + "at_msg": 6456, + "before": 506443, + "after": 402465, + "freed_pct": 20.5, + "floor": 237354, + "candidates": 40, + "dropped": 40, + "stage": "old messages left out", + "state_tok": 24992, + "requests": 2, + "usd": 0.0024 + }, + { + "cycle": 38, + "at_msg": 6516, + "before": 502959, + "after": 390405, + "freed_pct": 22.4, + "floor": 237724, + "candidates": 28, + "dropped": 28, + "stage": "old messages left out", + "state_tok": 24996, + "requests": 1, + "usd": 0.0012 + }, + { + "cycle": 39, + "at_msg": 6588, + "before": 500087, + "after": 403242, + "freed_pct": 19.4, + "floor": 238371, + "candidates": 35, + "dropped": 35, + "stage": "old messages left out", + "state_tok": 24987, + "requests": 1, + "usd": 0.0013 + }, + { + "cycle": 40, + "at_msg": 6663, + "before": 500286, + "after": 401251, + "freed_pct": 19.8, + "floor": 239653, + "candidates": 44, + "dropped": 44, + "stage": "old messages left out", + "state_tok": 24989, + "requests": 2, + "usd": 0.0025 + } + ], + "jev_usd_total": 0.132 +} diff --git a/evals/compaction/scripts/jev_cycles.py b/evals/compaction/scripts/jev_cycles.py new file mode 100644 index 0000000000..983854f50c --- /dev/null +++ b/evals/compaction/scripts/jev_cycles.py @@ -0,0 +1,59 @@ +#!/usr/bin/env python3 +"""Simulate repeated Jev compaction over a growing session. + +Usage: jev_cycles.py + +Feed a lineage chronologically; whenever the estimated context crosses the +threshold, compact with the Jev arm and record the cycle. Stops when the +transcript ends, Jev cannot fit its state (plugin fallback), or a compaction +frees nothing. +""" +import json +import sys +from pathlib import Path + +sys.path.insert(0, str(Path(__file__).resolve().parents[3])) +from evals.compaction.fixtures import estimate_tokens, load_transcript # noqa: E402 +from evals.compaction.jev_arm import JevCompactor, JevOptions # noqa: E402 + +path, threshold, max_cycles = sys.argv[1], int(sys.argv[2]), int(sys.argv[3]) +name = Path(path).stem +msgs = load_transcript(path) +ctx, i, cycles, total_jev_usd = [], 0, [], 0.0 + + +def tokens(ms): + return sum(estimate_tokens(m) for m in ms) + + +def text_floor(ms): + return sum(estimate_tokens(m) for m in ms if m.get("role") != "tool" and not m.get("tool_calls")) + + +while i < len(msgs) and len(cycles) < max_cycles: + ctx.append(msgs[i]); i += 1 + if tokens(ctx) < threshold or msgs[i - 1].get("tool_calls"): + continue # only compact on a well-formed boundary (result rows present) + jc = JevCompactor(options=JevOptions()) + before = tokens(ctx) + try: + out = jc.compress(ctx) + except ValueError as e: + cycles.append({"cycle": len(cycles) + 1, "at_msg": i, "before": before, "fallback": str(e)[:90], + "floor": text_floor(ctx), "calls": len(jc.decisions)}) + break + total_jev_usd += jc.usage.cost_usd + after = tokens(out) + cycles.append({"cycle": len(cycles) + 1, "at_msg": i, "before": before, "after": after, + "freed_pct": round(100 * (before - after) / before, 1), "floor": text_floor(out), + "candidates": len(jc.decisions) - jc.stats["pinned"], "dropped": jc.stats["calls_dropped"], + "stage": jc.stats["state_stage"], "state_tok": jc.stats["state_tokens"], + "requests": jc.stats["requests"], "usd": round(jc.usage.cost_usd, 4)}) + if after >= threshold: + cycles[-1]["stuck"] = True + break + ctx = out + +print(json.dumps({"transcript": name, "threshold": threshold, "raw_tokens_consumed": tokens(msgs[:i]), + "raw_total": tokens(msgs), "msgs_consumed": i, "cycles": cycles, + "jev_usd_total": round(total_jev_usd, 4)}, indent=1))