evals(compaction): default arm is current+recovery, the production compaction path
Closed-book arms scored the summary with its session_search pointer unused (43% vs 79% on the same banks) and made external compactors look like wins. Bare policy names stay available as an explicit opt-in floor.
This commit is contained in:
@@ -19,7 +19,7 @@ Measures what context compaction actually costs in *recall*, not just tokens.
|
||||
# from repo root, venv active
|
||||
python evals/compaction/runner.py \
|
||||
--transcript /path/to/lineage.json \
|
||||
--policies current,aggressive,floor10k \
|
||||
--policies current+recovery,lean+recovery \
|
||||
--questions 15 \
|
||||
--out evals/compaction/results/run1
|
||||
python evals/compaction/report.py evals/compaction/results/run1
|
||||
@@ -99,3 +99,9 @@ token bill.
|
||||
does not.
|
||||
- `--also-uncompacted` adds a control arm that answers from the full
|
||||
original transcript — the recall ceiling.
|
||||
- **Default arm is `current+recovery`: the production path.** Compaction in
|
||||
Hermes is the summary *plus* the session_search pointer it carries, so the
|
||||
answerer gets one search round-trip over the archived region (same FTS5+BM25
|
||||
engine as production). A bare policy name (`current`) is closed-book — the
|
||||
summary with its recovery pointer unused — and scores 30+ pts lower on
|
||||
needle questions. Use it only when you specifically want that floor.
|
||||
|
||||
@@ -412,7 +412,8 @@ def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
ap.add_argument("--transcript", required=True)
|
||||
ap.add_argument("--cap-tokens", type=int, default=500_000)
|
||||
ap.add_argument("--policies", default="current,tail25k,codex_style")
|
||||
ap.add_argument("--policies", default="current+recovery",
|
||||
help="comma-separated arms; <name>+recovery = production path (summary + one session_search round-trip). Bare <name> is closed-book, opt-in only.")
|
||||
ap.add_argument("--questions", type=int, default=15)
|
||||
ap.add_argument("--out", required=True)
|
||||
ap.add_argument("--also-uncompacted", action="store_true")
|
||||
|
||||
Reference in New Issue
Block a user