evals(compaction): default arm is current+recovery, the production compaction path

Closed-book arms scored the summary with its session_search pointer unused
(43% vs 79% on the same banks) and made external compactors look like wins.
Bare policy names stay available as an explicit opt-in floor.
This commit is contained in:
teknium1
2026-09-19 11:47:34 -07:00
committed by Teknium
parent dba031b5cb
commit 085ed46b63
2 changed files with 9 additions and 2 deletions

View File

@@ -19,7 +19,7 @@ Measures what context compaction actually costs in *recall*, not just tokens.
# from repo root, venv active
python evals/compaction/runner.py \
--transcript /path/to/lineage.json \
--policies current,aggressive,floor10k \
--policies current+recovery,lean+recovery \
--questions 15 \
--out evals/compaction/results/run1
python evals/compaction/report.py evals/compaction/results/run1
@@ -99,3 +99,9 @@ token bill.
does not.
- `--also-uncompacted` adds a control arm that answers from the full
original transcript — the recall ceiling.
- **Default arm is `current+recovery`: the production path.** Compaction in
Hermes is the summary *plus* the session_search pointer it carries, so the
answerer gets one search round-trip over the archived region (same FTS5+BM25
engine as production). A bare policy name (`current`) is closed-book — the
summary with its recovery pointer unused — and scores 30+ pts lower on
needle questions. Use it only when you specifically want that floor.

View File

@@ -412,7 +412,8 @@ def main():
ap = argparse.ArgumentParser()
ap.add_argument("--transcript", required=True)
ap.add_argument("--cap-tokens", type=int, default=500_000)
ap.add_argument("--policies", default="current,tail25k,codex_style")
ap.add_argument("--policies", default="current+recovery",
help="comma-separated arms; <name>+recovery = production path (summary + one session_search round-trip). Bare <name> is closed-book, opt-in only.")
ap.add_argument("--questions", type=int, default=15)
ap.add_argument("--out", required=True)
ap.add_argument("--also-uncompacted", action="store_true")