docs(evals): real Codex CLI head-to-head arm + results
scripts/codex_arm.py drives OpenAI Codex CLI end-to-end on the same transcripts: chunk-file reads until its REAL auto-compaction fires (verified via compacted events in the rollout jsonl; peak 455-483K vs its 258K window), then quizzes post-compaction with the identical question banks and judge. Results (results/codex-arm-2026-08-15/): codex 36.7% avg vs lean closed-book 40.0% vs lean+recovery 68.3%. Codex has no runtime re-access over its rollout history — the session_search differentiator, measured.
This commit is contained in:
Teknium
@@ -46,6 +46,41 @@ lean+recovery 70.0 @ 62K 80.0 @ 41K 43.3 @ 45K 80.0 @ 50K 68.
|
||||
(+20-43pts measured); better accuracy AND more savings than current ✓
|
||||
(+22.5pts at 0.30x tokens).
|
||||
|
||||
|
||||
## Codex CLI head-to-head (same transcripts, same exams, same judge)
|
||||
|
||||
Real OpenAI Codex CLI (v0.147.0, gpt-5.6-sol, 258K window) run end-to-end on
|
||||
the identical four transcripts: chunk files read via `codex exec` until its
|
||||
REAL auto-compaction fired (verified `compacted` event in the rollout jsonl;
|
||||
peak context 455-483K), then quizzed post-compaction from memory with the
|
||||
same 15-question banks and scored by the same judge.
|
||||
|
||||
policy sweep gui prmerge acp AVG retained state
|
||||
codex (real, post-cmp) 26.7% 40.0% 43.3% 36.7% 36.7% ~4.5K (opaque blob + user msgs)
|
||||
hermes current 93.3%* 26.7%* 33.3% 30.0% 45.8% ~162K
|
||||
hermes lean closed-book 40.0% 60.0% 23.3% 36.7% 40.0% ~49K
|
||||
hermes lean+recovery 70.0% 80.0% 43.3% 80.0% 68.3% ~49K
|
||||
|
||||
Notes:
|
||||
- codex answers from its own post-compaction session — the honest analog of
|
||||
our closed-book arms. It has NO session_search equivalent (its rollout is
|
||||
on disk but the agent cannot search it at runtime), so recovery has no
|
||||
codex counterpart; that gap is exactly the differentiator lean leans on.
|
||||
- Apples-to-apples closed-book: lean 40.0% vs codex 36.7% — parity-plus at
|
||||
10x codex's retained state but 0.30x current's. With recovery: +31.6pts
|
||||
over codex.
|
||||
- codex ties lean+recovery on prmerge (43.3%) — the dense multi-PR campaign
|
||||
is the hardest transcript for every policy and the clearest iteration
|
||||
target.
|
||||
- Methodology caveats: codex ingested transcripts as FILE READS (tool
|
||||
outputs), not native conversation — this matches how its compaction
|
||||
treats tool output (drops it all into the server-side summary) but is not
|
||||
byte-identical to a native session. Its model (gpt-5.6-sol) also differs
|
||||
from the answering model in our arms; scores compare COMPACTION PIPELINES
|
||||
end-to-end, not models in isolation. One codex quiz reply was also
|
||||
capped short (~1K chars for 15 answers), which its terse post-compaction
|
||||
style invites.
|
||||
|
||||
## Recommendation
|
||||
|
||||
Ship lean as opt-in (compression.tail_mode: lean, legacy default), harness as
|
||||
|
||||
134
evals/compaction/results/codex-arm-2026-08-15/acp.json
Normal file
134
evals/compaction/results/codex-arm-2026-08-15/acp.json
Normal file
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 36.7,
|
||||
"scores": [
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
2
|
||||
],
|
||||
"read_seconds": 229,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 454931,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-35-51-01a00835-fef4-7c31-944d-9d0d8ba933be.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What was the specific reason Teknium gave for reverting PR #30179 in July 2026?",
|
||||
"gold": "WTF??? REVERT! DAMMIT",
|
||||
"answer": "UNKNOWN \u2014 best guess: it caused a regression on current `main` and had to be reverted until a safer implementation was available.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a logical guess about a regression, but it fails to capture the specific, idiosyncratic reason/quote provided in the gold answer ('WTF??? REVERT! DAMMIT')."
|
||||
},
|
||||
{
|
||||
"q": "On which specific PR did Teknium say, 'tf are you saying to me. Stop giving me such random verbose details'?",
|
||||
"gold": "PR #6391",
|
||||
"answer": "UNKNOWN \u2014 best guess: PR #37865.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold answer (PR #37865 vs PR #6391)."
|
||||
},
|
||||
{
|
||||
"q": "Which file path should be checked for the canonical list of provider models?",
|
||||
"gold": "hermes_cli/models.py",
|
||||
"answer": "UNKNOWN \u2014 best guess: `hermes_cli/models.py`.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the correct file path as its best guess, which matches the gold answer exactly."
|
||||
},
|
||||
{
|
||||
"q": "What was the identified bug in PR #2314 regarding provider names?",
|
||||
"gold": "checking for \"alibaba-coding-plan\"",
|
||||
"answer": "UNKNOWN \u2014 best guess: it used inconsistent/noncanonical provider names, so provider matching and routing failed.",
|
||||
"score": 1,
|
||||
"why": "The answer identifies the core issue (provider name mismatch/inconsistency) which led to the specific bug, but it does not mention the specific string 'alibaba-coding-plan' identified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What is the mandatory line limit for PR reviews requested by Teknium?",
|
||||
"gold": "<= 15 lines",
|
||||
"answer": "Maximum 5 lines.",
|
||||
"score": 0,
|
||||
"why": "The answer states a limit of 5 lines, which contradicts the gold answer of 15 lines."
|
||||
},
|
||||
{
|
||||
"q": "What exact error message did the agent receive when attempting to checkout a worktree while in the live source directory?",
|
||||
"gold": "Blocked: `git checkout` would rewrite Hermes's live source checkout (/home/teknium/.hermes/hermes-agent) and can mix module versions in this running process.",
|
||||
"answer": "UNKNOWN \u2014 best guess: `fatal: '<branch>' is already checked out at '<worktree path>'`.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a standard Git error message as a guess, which does not match the specific custom error message provided in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "Why was PR #74658 necessary to fix Slack 'broken on main'?",
|
||||
"gold": "SlackResponse isn't a dict subclass, so every gate is always False.",
|
||||
"answer": "Real Slack SDK calls return `SlackResponse` objects, but the main code treated them like dictionaries and called dictionary-style accessors.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that SlackResponse objects were being treated as dictionaries when they are not, which matches the core issue described in the gold answer that their lack of dictionary subclassing caused attribute/gate checks to fail."
|
||||
},
|
||||
{
|
||||
"q": "What was the final merge commit SHA for the Slack SDK response fix on main?",
|
||||
"gold": "24ba86627515ad5fda69a39ef338c365713448bc",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The gold answer provides a specific commit SHA, while the answer states 'UNKNOWN', failing to provide the requested factual information."
|
||||
},
|
||||
{
|
||||
"q": "In the 'Pop-laboratory' style infographic for the Auxiliary Client fix, what were the two specific outcomes shown in cell 2?",
|
||||
"gold": "Messages wrapper keeps /anthropic and OpenAI fallback keeps /v1",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The answer 'UNKNOWN' is a refusal to provide the information required by the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific SQL update was added to the migration path in hermes_cli/kanban_db.py to prevent losing active wake on upgrade?",
|
||||
"gold": "UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'",
|
||||
"answer": "`UPDATE kanban_notify_subs SET delivery_mode = 'notify+wake' WHERE platform != 'tui'` \u2014 applied only during first migration/addition of the column so existing choices were not overwritten.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the exact SQL statement specified in the gold answer and correctly contextualizes its purpose."
|
||||
},
|
||||
{
|
||||
"q": "Which test failed in CI slice 5/12 for the kanban delivery modes PR?",
|
||||
"gold": "tests/gateway/test_kanban_notifier_apiserver_wake.py::test_apiserver_sub_wakes_real_session_via_self_post",
|
||||
"answer": "UNKNOWN \u2014 best guess: a relay-metrics concurrency test, unrelated to the delivery-mode change.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies an unrelated test (relay-metrics) rather than the specific kanban notifier test mentioned in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "According to the transcript, why is squash merging banned as of July 2026?",
|
||||
"gold": "DevOps policy",
|
||||
"answer": "Squash merging destroys the contributor-authored commit history and therefore loses reliable authorship/credit.",
|
||||
"score": 0,
|
||||
"why": "The answer explains a general rationale for why someone might dislike squash merging, but it fails to provide the specific reason cited in the text ('DevOps policy')."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor authored the first fix for issue #73030 in July?",
|
||||
"gold": "@Tranquil-Flow",
|
||||
"answer": "UNKNOWN.",
|
||||
"score": 0,
|
||||
"why": "The answer states 'UNKNOWN' while the gold answer provides a specific contributor name (@Tranquil-Flow)."
|
||||
},
|
||||
{
|
||||
"q": "What was the 'Superman-style' shield error in the first generation of the Kanban infographic?",
|
||||
"gold": "red \"S\" inside the diamond shield",
|
||||
"answer": "The generated shield contained an emblem resembling Superman\u2019s \u201cS,\u201d rather than a neutral Kanban/routing symbol.",
|
||||
"score": 2,
|
||||
"why": "The answer accurately identifies that the shield contained a symbol resembling Superman's 'S', which matches the core fact of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific file was modified to add the 'scope_id_for_chat' method for Slack?",
|
||||
"gold": "plugins/platforms/slack/adapter.py",
|
||||
"answer": "`plugins/platforms/slack/adapter.py`.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer exactly, including the full file path."
|
||||
}
|
||||
]
|
||||
}
|
||||
134
evals/compaction/results/codex-arm-2026-08-15/gui.json
Normal file
134
evals/compaction/results/codex-arm-2026-08-15/gui.json
Normal file
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 40.0,
|
||||
"scores": [
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 207,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 482525,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-28-36-01a0082f-5b69-7ff0-977d-998523a4b3fc.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What is the PR number for the authored fix addressing mid-turn message ordering bugs in Hermes Desktop?",
|
||||
"gold": "#86617",
|
||||
"answer": "PR **#86617**.",
|
||||
"score": 2,
|
||||
"why": "The answer provides the exact PR number specified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "According to the contribution rubric in AGENTS.md, which type of config belongs in '.env' and which belongs in 'config.yaml'?",
|
||||
"gold": ".env is for secrets only (API keys, tokens, passwords). All behavioral settings... go in config.yaml.",
|
||||
"answer": "`.env` is for secrets and deployment-specific values; `config.yaml` is for non-secret user-facing runtime configuration.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that .env is for secrets and config.yaml is for non-secret/behavioral settings, which matches the core distinction in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What specific file and line number were identified as the cause of an AssertionError (assert 56 == 55) in the Python tests?",
|
||||
"gold": "tests/hermes_cli/test_session_recovery_lost_and_found.py:327",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: a schema-width assertion in a Python schema/config test, expecting 55 fields but receiving 56 after `git_metadata_generation` was added.",
|
||||
"score": 0,
|
||||
"why": "The answer fails to provide the specific file and line number requested, which are present in the gold answer. It instead provides a guess about the context of the error without identifying the location."
|
||||
},
|
||||
{
|
||||
"q": "What was the root cause of issue #73793 regarding mid-turn message rendering?",
|
||||
"gold": "redirect/steer paths spliced the mid-turn user bubble BEFORE the active assistant stream row",
|
||||
"answer": "Two live paths handled mid-turn corrections independently, allowing corrections and assistant stream output to be inserted in the wrong causal order.",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the same root cause as the gold answer: the independent handling of paths caused the user bubble (correction) and assistant stream to be rendered in the wrong causal order."
|
||||
},
|
||||
{
|
||||
"q": "Which PR was verified to already be on 'main', resulting in nothing needing to be salvaged for it?",
|
||||
"gold": "#84287",
|
||||
"answer": "**PR #86299**.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies PR #86299, but the gold answer is PR #84287. The two are different pull requests."
|
||||
},
|
||||
{
|
||||
"q": "In the Desktop virtualized-scrolling cluster, what was the fix for issue #79157 (scrollbar unclickable)?",
|
||||
"gold": "pane sash grab band made asymmetric 1px/7px",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: make the scrollbar gutter/overlay accept pointer events instead of letting the virtualized content layer cover it.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a guess that contradicts the gold answer; the fix was making the pane sash grab band asymmetric (1px/7px), not changing pointer events on the gutter/overlay."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor's email was mapped to 'baihemax' during the attribution audit of PR #86588?",
|
||||
"gold": "602028@ky-tech.com.cn",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: the email associated with contributor `hkfiberlaser-svg`.",
|
||||
"score": 0,
|
||||
"why": "The answer states the answer is unknown and provides a guess that does not match the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What error message does the Hermes terminal tool return when a git command is blocked to prevent rewriting the live source checkout?",
|
||||
"gold": "Blocked: `git <op>` would rewrite Hermes's live source checkout",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: an error stating that the git command was blocked because it would rewrite the live source checkout.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the core substance of the error message (blocked git command because it would rewrite the live source checkout) which matches the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What is the core design principle regarding 'Narrow Waist' in Hermes development?",
|
||||
"gold": "The core is a narrow waist; capability lives at the edges.",
|
||||
"answer": "Keep a small, stable core interface\u2014the \u201cnarrow waist\u201d\u2014through which different clients, providers, and integrations communicate, avoiding duplicated business logic across surfaces.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the core principle of a 'narrow waist' and explains that the logic/interfaces are centralized there while clients and providers interact at the edges, matching the substance of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What was the result of the rebase-merge attempt for PR #86589?",
|
||||
"gold": "GraphQL: Pull Request has merge conflicts (mergePullRequest)",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: the rebase merge could not proceed because the PR had conflicts and was not mergeable.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies that the rebase-merge could not proceed due to merge conflicts, which matches the factual content of the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "In the infographic style picker, what vibe is associated with the 'designers-republic' style?",
|
||||
"gold": "The Designers Republic: flat orange+violet vector schematic on pewter grey",
|
||||
"answer": "A bold, experimental, anti-corporate graphic-design vibe\u2014brutalist/techno typography inspired by The Designers Republic.",
|
||||
"score": 0,
|
||||
"why": "The answer describes the real-world aesthetic of 'The Designers Republic' studio, but fails to identify the specific visual elements (flat orange+violet vector schematic on pewter grey) defined for this style within the infographic picker context."
|
||||
},
|
||||
{
|
||||
"q": "Why was PR #76286 excluded from the compaction/compression transcript-visibility cluster?",
|
||||
"gold": "conflicts with main in 4 files and introduces a second competing display-dedupe scheme",
|
||||
"answer": "It did not address transcript visibility after compaction/compression; its changes concerned a different part of the compaction flow.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a completely different reason than the gold answer, which specifies merge conflicts and a competing deduplication scheme as the reasons for exclusion."
|
||||
},
|
||||
{
|
||||
"q": "What is the 'Provenance note' date for the pr-infographic-workflow.md reference file?",
|
||||
"gold": "May 23 2026",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **2026-08-12**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' that is factually incorrect compared to the gold date (2026-08-12 vs. May 23 2026)."
|
||||
},
|
||||
{
|
||||
"q": "What specific TypeScript error caused PR #86772 to fail CI linting after a rebase?",
|
||||
"gold": "Property 'onToggleUnread' is missing in type",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: a TypeScript unresolved-name/import error introduced by the rebase.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies as UNKNOWN and provides an incorrect guess. The gold answer specifies a 'property is missing' error, while the provided answer guesses an 'unresolved-name/import' error."
|
||||
},
|
||||
{
|
||||
"q": "According to the Desktop Engineering Guide, who is the authority for process lifecycle and the native filesystem?",
|
||||
"gold": "Electron",
|
||||
"answer": "The **Rust/Tauri backend** is authoritative for process lifecycle and the native filesystem.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the Rust/Tauri backend as the authority, which directly contradicts the gold answer stating Electron is the authority."
|
||||
}
|
||||
]
|
||||
}
|
||||
134
evals/compaction/results/codex-arm-2026-08-15/prmerge.json
Normal file
134
evals/compaction/results/codex-arm-2026-08-15/prmerge.json
Normal file
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 43.3,
|
||||
"scores": [
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
1,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 154,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 465131,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-32-45-01a00833-2968-7453-8646-d5bb7f5e8400.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "Which PR number added the public subagent lifecycle API?",
|
||||
"gold": "#63359",
|
||||
"answer": "PR **#63359**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold PR number exactly."
|
||||
},
|
||||
{
|
||||
"q": "What is the name of the typed service added to PluginContext for launching and monitoring child sessions?",
|
||||
"gold": "subagent_lifecycle",
|
||||
"answer": "`SubagentService` (`ctx.subagents`).",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the correct property on the context (ctx.subagents), which is the typed service used for the task, even though it uses a slightly different name for the service itself."
|
||||
},
|
||||
{
|
||||
"q": "How many contract and security tests were included with the subagent lifecycle API PR?",
|
||||
"gold": "42",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **24 tests**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect guess (24) compared to the gold answer (42) and frames the response as unknown."
|
||||
},
|
||||
{
|
||||
"q": "What specific gap was identified regarding the `ctx.inject_message()` function in gateway sessions?",
|
||||
"gold": "cannot currently trigger a turn in an existing gateway session",
|
||||
"answer": "`ctx.inject_message()` could not safely target an active gateway session; it lacked a profile/session-scoped `session_key`.",
|
||||
"score": 2,
|
||||
"why": "The answer identifies the core limitation (targeting active gateway sessions) and provides technical detail on why a turn cannot be triggered (lack of session-scoped key)."
|
||||
},
|
||||
{
|
||||
"q": "Which PR implements gateway-safe plugin injection by extending `ctx.inject_message()` with a keyword-only `session_key`?",
|
||||
"gold": "#64436",
|
||||
"answer": "PR **#64436**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer exactly."
|
||||
},
|
||||
{
|
||||
"q": "What are the two specific constraints placed on redaction patterns in the pattern registry to prevent exposing data?",
|
||||
"gold": "must compile, must start with \u22652 literal characters",
|
||||
"answer": "Patterns must be **bounded in length** and **validated against unsafe/catastrophic regex behavior** before registration.",
|
||||
"score": 0,
|
||||
"why": "The answer discusses general regex safety (length and performance), whereas the gold answer specifies two precise technical constraints: compilation and a minimum of two literal characters at the start."
|
||||
},
|
||||
{
|
||||
"q": "Which contributor authorized sustained help for the Phase 0\u20131 expansion track?",
|
||||
"gold": "Daniel",
|
||||
"answer": "**@asimons81**.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies @asimons81 as the contributor, whereas the gold answer states it was Daniel. These do not match."
|
||||
},
|
||||
{
|
||||
"q": "What is the issue number for the disposition gap concerning `pre_command` middleware and MCP tool access?",
|
||||
"gold": "#64204",
|
||||
"answer": "Issue **#64204**.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold issue number exactly."
|
||||
},
|
||||
{
|
||||
"q": "What configuration setting is required to opt-in to reasoning deltas in streaming output?",
|
||||
"gold": "plugins.stream_reasoning_deltas: true",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: `display.show_reasoning: true`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies itself as a guess and provides the incorrect configuration setting ('display.show_reasoning' vs the gold 'plugins.stream_reasoning_deltas')."
|
||||
},
|
||||
{
|
||||
"q": "How many additions and across how many files were made in PR #63359?",
|
||||
"gold": "650 additions across 4 files",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: roughly **1,000 additions across 12 files**.",
|
||||
"score": 0,
|
||||
"why": "The answer states the information is unknown and provides a guess that is factually incorrect regarding both the number of additions and the number of files."
|
||||
},
|
||||
{
|
||||
"q": "What is the name of the reference plugin shipped with the redaction pattern registry?",
|
||||
"gold": "nvapi-redaction",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: `redaction-patterns`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies itself as unknown and provides a guess ('redaction-patterns') that is incorrect compared to the gold answer ('nvapi-redaction')."
|
||||
},
|
||||
{
|
||||
"q": "List the four observer-only streaming output plugin hooks added in PR #64317.",
|
||||
"gold": "on_stream_start, on_stream_delta, on_stream_end, on_interim_message",
|
||||
"answer": "`on_stream_start`, `on_stream_delta`, `on_stream_end`, and `on_interim_message`.",
|
||||
"score": 2,
|
||||
"why": "The answer lists all four hooks correctly as specified in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "What was addressed in the update to PR #58541 regarding lifecycle hooks?",
|
||||
"gold": "created-hook timing and added kanban_task_promoted",
|
||||
"answer": "It updated the lifecycle hooks to match the current registry contract, including correct hook timing, payloads, and failure isolation.",
|
||||
"score": 1,
|
||||
"why": "The answer correctly identifies 'hook timing' as part of the update, but it fails to mention the specific addition of 'kanban_task_promoted' and instead provides a general description of other improvements like payloads and isolation not mentioned in the gold answer."
|
||||
},
|
||||
{
|
||||
"q": "Which sub-issue number is associated with the 'developer tooling' (scaffold + Plugin Doctor + test harness)?",
|
||||
"gold": "#64230",
|
||||
"answer": "**UNKNOWN** \u2014 best guess: **#64179**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect issue number (#64179) while the gold answer is #64230."
|
||||
},
|
||||
{
|
||||
"q": "What was the Round 3 review's outcome for PR #63359 and @asimons81?",
|
||||
"gold": "sub-issue #65447",
|
||||
"answer": "Round 3 **approved PR #63359 and authorized @asimons81 to continue sustained Phase 0\u20131 work**.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a general description of approval and authorization, but fails to mention the specific outcome required by the gold answer (the creation or reference of sub-issue #65447)."
|
||||
}
|
||||
]
|
||||
}
|
||||
134
evals/compaction/results/codex-arm-2026-08-15/sweep.json
Normal file
134
evals/compaction/results/codex-arm-2026-08-15/sweep.json
Normal file
@@ -0,0 +1,134 @@
|
||||
{
|
||||
"summary": {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": 26.7,
|
||||
"scores": [
|
||||
2,
|
||||
0,
|
||||
2,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
2,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0,
|
||||
0
|
||||
],
|
||||
"read_seconds": 420,
|
||||
"compaction_events": 1,
|
||||
"peak_context_tokens": 482248,
|
||||
"rollout": "/home/teknium/.codex/sessions/2026/08/15/rollout-2026-08-15T18-20-54-01a00828-4e75-70f0-9e6d-31c7aac8d71c.jsonl"
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"q": "What is the reason given for never using 'git checkout pr-branch -- <file>' on stale branches?",
|
||||
"gold": "the stale file version silently deletes newer main code",
|
||||
"answer": "It can silently replace current files with stale branch versions, discarding newer fixes and resurrecting deleted code; salvage must use commits and resolve conflicts explicitly.",
|
||||
"score": 2,
|
||||
"why": "The answer accurately reflects the gold answer's point that stale branch versions will overwrite newer code, while adding specific detail about resurrecting deleted code."
|
||||
},
|
||||
{
|
||||
"q": "According to the transcript, how much RSS memory does the gateway balloon to every ~2h in the regression reported in issue #81625?",
|
||||
"gold": "~60GB",
|
||||
"answer": "UNKNOWN \u2014 best guess: approximately 4 GB RSS.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a 'best guess' of 4 GB, which is factually incorrect and contradicts the gold answer of 60 GB."
|
||||
},
|
||||
{
|
||||
"q": "Which specific Electron setting is suspected of causing the Windows occlusion freeze in issue #83420?",
|
||||
"gold": "backgroundThrottling",
|
||||
"answer": "UNKNOWN \u2014 best guess: Electron\u2019s `backgroundThrottling: false`.",
|
||||
"score": 2,
|
||||
"why": "The answer correctly identifies the specific setting 'backgroundThrottling' as the suspected cause, despite the initial 'UNKNOWN' disclaimer."
|
||||
},
|
||||
{
|
||||
"q": "What exact error message is returned when 'gh pr merge --auto' is attempted on the NousResearch/hermes-agent repository?",
|
||||
"gold": "Auto merge is not allowed for this repository (enablePullRequestAutoMerge)",
|
||||
"answer": "`GraphQL: Pull request Auto merge is not allowed for this repository`",
|
||||
"score": 2,
|
||||
"why": "The answer matches the core message of the gold answer. The 'GraphQL:' prefix accurately reflects the technical origin of the error message when using the GitHub CLI."
|
||||
},
|
||||
{
|
||||
"q": "What is the specified 'Rule 0' that must be included in a subagent brief?",
|
||||
"gold": "load the skill first",
|
||||
"answer": "Rule 0: do not merge anything; only the primary agent/user is authorized to merge.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies a different 'Rule 0' regarding merging permissions, which does not match the gold answer's requirement to 'load the skill first'."
|
||||
},
|
||||
{
|
||||
"q": "In the July 2026 title-cluster sweep, what was the title of the missed first submitter PR #35416?",
|
||||
"gold": "add config gate for title generation",
|
||||
"answer": "UNKNOWN \u2014 best guess: \u201cfix: prevent duplicate message submission.\u201d",
|
||||
"score": 0,
|
||||
"why": "The answer provides an incorrect guess ('fix: prevent duplicate message submission.') which does not match the gold answer ('add config gate for title generation')."
|
||||
},
|
||||
{
|
||||
"q": "Which file path is noted as containing the #34034/#28149 manifest guard 'test_bundled_plugin_manifests_ship_in_both_wheel_and_sdist'?",
|
||||
"gold": "tests/test_packaging_metadata.py",
|
||||
"answer": "UNKNOWN \u2014 best guess: `tests/test_plugin_packaging.py`.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the wrong file (test_plugin_packaging.py vs the correct test_packaging_metadata.py)."
|
||||
},
|
||||
{
|
||||
"q": "What was the result of the 'npm ci' command run in /home/teknium/salv-desktop according to the background process notification?",
|
||||
"gold": "completed normally (exit code 0)",
|
||||
"answer": "`npm ci` completed successfully with exit code 0.",
|
||||
"score": 2,
|
||||
"why": "The answer matches the gold answer perfectly, confirming the successful completion and the exit code of 0."
|
||||
},
|
||||
{
|
||||
"q": "What was the 'Root Cause A' identified for why 'uv sync --extra all --locked' failed daily in issue #79434?",
|
||||
"gold": "relative exclude-newer makes the committed lock stale every day",
|
||||
"answer": "Root Cause A: the `all` extra referenced platform-specific/nonexistent dependency groups that were absent or inconsistent in the locked resolution.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies an issue with dependency groups, whereas the gold answer states the root cause was the use of a relative 'exclude-newer' value that caused the lockfile to become stale daily."
|
||||
},
|
||||
{
|
||||
"q": "How many tasks are reported as done in the 'fangliquanflq' desktop retry truncation PR #86605?",
|
||||
"gold": "13",
|
||||
"answer": "UNKNOWN \u2014 best guess: 7 tasks done.",
|
||||
"score": 0,
|
||||
"why": "The answer guesses 7 tasks, while the gold answer is 13. The answer is factually incorrect."
|
||||
},
|
||||
{
|
||||
"q": "In the 'salv-cron' worktree, what was the exit code when the agent tried to execute a 'BLOCKED (hardline)' command?",
|
||||
"gold": "-1",
|
||||
"answer": "Exit code 2.",
|
||||
"score": 0,
|
||||
"why": "The answer provides a specific exit code (2) that contradicts the gold answer (-1)."
|
||||
},
|
||||
{
|
||||
"q": "What is the full title block text for the technical schematic infographic generated for the Gateway Drain?",
|
||||
"gold": "GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT",
|
||||
"answer": "UNKNOWN \u2014 best guess: `GATEWAY DRAIN` / `INTERRUPT \u2192 DELIVER \u2192 DISCONNECT` / `GRACEFUL SHUTDOWN WITHOUT MESSAGE LOSS`.",
|
||||
"score": 0,
|
||||
"why": "The answer fails to provide the correct title block text specified in the gold answer. It incorrectly guesses metadata or slogans instead of the specific title 'GATEWAY DRAIN \u00d7 CRON \u2014 SHUTDOWN CONTRACT'."
|
||||
},
|
||||
{
|
||||
"q": "Which PR number's watcher reported '=== ALL GREEN (streak=1, checks=46) ===' at [03:56:19]?",
|
||||
"gold": "82980",
|
||||
"answer": "UNKNOWN \u2014 best guess: PR #86594.",
|
||||
"score": 0,
|
||||
"why": "The answer identifies the wrong PR number (86594 instead of 82980)."
|
||||
},
|
||||
{
|
||||
"q": "What is the specific Gist ID created for the PR infographic host in the cron cluster?",
|
||||
"gold": "ee33edd5804689243f974536ef7aecb9",
|
||||
"answer": "UNKNOWN \u2014 I do not retain the Gist ID.",
|
||||
"score": 0,
|
||||
"why": "The answer states it does not know the information, whereas the gold answer provides a specific Gist ID."
|
||||
},
|
||||
{
|
||||
"q": "What was the final merge SHA for Cluster D's Trigger-now PR #70638?",
|
||||
"gold": "f9d64b9a9d8b306f64851c1a13869d96ad5d7869",
|
||||
"answer": "UNKNOWN \u2014 I do not retain the final merge SHA for PR #70638.",
|
||||
"score": 0,
|
||||
"why": "The answer claims it does not know the information, while the gold answer provides the specific SHA requested."
|
||||
}
|
||||
]
|
||||
}
|
||||
211
evals/compaction/scripts/codex_arm.py
Normal file
211
evals/compaction/scripts/codex_arm.py
Normal file
@@ -0,0 +1,211 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run the codex CLI as an eval arm on the same transcripts + question banks.
|
||||
|
||||
Per transcript:
|
||||
1. Split the 500K-token prefix into ~150KB chunk files in a work dir.
|
||||
2. `codex exec` reads every file (2-3 sentence summary each) — the read
|
||||
volume exceeds codex's 258K window, so its auto-compaction fires
|
||||
naturally (verified via token_count drops / compacted events in the
|
||||
rollout jsonl).
|
||||
3. `codex exec resume --last` asks the SAME 15 exam questions; answers are
|
||||
judged by the same LLM judge against the same golds.
|
||||
|
||||
Usage: codex_arm.py <lineage_json> <questions_json> <workdir> <out_json>
|
||||
"""
|
||||
import glob
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[0] / "main-co"))
|
||||
|
||||
LINEAGE = sys.argv[1]
|
||||
QUESTIONS = sys.argv[2]
|
||||
WORKDIR = Path(sys.argv[3])
|
||||
OUT = sys.argv[4]
|
||||
|
||||
JUDGE_PROMPT = """Score this answer against the gold answer. Reply with STRICT JSON: {{"score": 2|1|0, "why": "..."}}.
|
||||
2 = factually matches gold (wording may differ)
|
||||
1 = partially correct or hedged-but-right
|
||||
0 = wrong, or refuses/says it doesn't know with a wrong/no guess
|
||||
|
||||
QUESTION: {question}
|
||||
GOLD: {gold}
|
||||
ANSWER: {answer}"""
|
||||
|
||||
|
||||
def prepare_chunks() -> int:
|
||||
from evals.compaction.fixtures import load_transcript
|
||||
|
||||
WORKDIR.mkdir(parents=True, exist_ok=True)
|
||||
msgs = load_transcript(LINEAGE, cap_tokens=500_000)
|
||||
chunk, size, idx = [], 0, 0
|
||||
for m in msgs:
|
||||
c = m.get("content") or ""
|
||||
if not isinstance(c, str) or not c:
|
||||
continue
|
||||
chunk.append(f"--- {m['role']} ---\n{c}\n")
|
||||
size += len(c)
|
||||
if size > 150_000:
|
||||
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
|
||||
"\n".join(chunk), encoding="utf-8")
|
||||
chunk, size = [], 0
|
||||
idx += 1
|
||||
if chunk:
|
||||
(WORKDIR / f"transcript_{idx:02d}.txt").write_text(
|
||||
"\n".join(chunk), encoding="utf-8")
|
||||
idx += 1
|
||||
return idx
|
||||
|
||||
|
||||
def newest_rollout() -> str:
|
||||
files = sorted(
|
||||
glob.glob(os.path.expanduser("~/.codex/sessions/*/*/*/rollout-*.jsonl")),
|
||||
key=os.path.getmtime,
|
||||
)
|
||||
return files[-1] if files else ""
|
||||
|
||||
|
||||
def rollout_session_id(path: str) -> str:
|
||||
for line in open(path, encoding="utf-8", errors="replace"):
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
if d.get("type") == "session_meta":
|
||||
return d.get("payload", {}).get("session_id", "")
|
||||
return ""
|
||||
|
||||
|
||||
def last_agent_message(path: str) -> str:
|
||||
msgs = []
|
||||
for line in open(path, encoding="utf-8", errors="replace"):
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
p = d.get("payload", {})
|
||||
if p.get("type") == "agent_message":
|
||||
msgs.append(p.get("message", ""))
|
||||
return msgs[-1] if msgs else ""
|
||||
|
||||
|
||||
def rollout_stats(path: str) -> dict:
|
||||
compacted = 0
|
||||
peak = 0
|
||||
for line in open(path, encoding="utf-8", errors="replace"):
|
||||
try:
|
||||
d = json.loads(line)
|
||||
except Exception:
|
||||
continue
|
||||
p = d.get("payload", {})
|
||||
if d.get("type") == "compacted" or p.get("type") == "compacted":
|
||||
compacted += 1
|
||||
if p.get("type") == "token_count" and p.get("info"):
|
||||
last = p["info"].get("last_token_usage") or {}
|
||||
ctx = last.get("input_tokens", 0) + last.get("cached_input_tokens", 0)
|
||||
peak = max(peak, ctx)
|
||||
return {"compaction_events": compacted, "peak_context_tokens": peak}
|
||||
|
||||
|
||||
def codex(args: list, prompt: str, timeout: int = 3600) -> str:
|
||||
proc = subprocess.run(
|
||||
["codex", "exec", *args, "--skip-git-repo-check", prompt],
|
||||
cwd=str(WORKDIR), capture_output=True, text=True, timeout=timeout,
|
||||
)
|
||||
return proc.stdout + proc.stderr
|
||||
|
||||
|
||||
def judge(question: str, gold: str, answer: str) -> dict:
|
||||
from agent.auxiliary_client import call_llm
|
||||
|
||||
resp = call_llm(
|
||||
messages=[{"role": "user", "content": JUDGE_PROMPT.format(
|
||||
question=question, gold=gold, answer=answer)}],
|
||||
task="compression", max_tokens=300,
|
||||
)
|
||||
text = resp.choices[0].message.content if hasattr(resp, "choices") else str(resp)
|
||||
m = re.search(r"\{.*\}", text, re.S)
|
||||
try:
|
||||
return json.loads(m.group(0))
|
||||
except Exception:
|
||||
return {"score": 0, "why": f"judge parse failure: {text[:80]}"}
|
||||
|
||||
|
||||
def main():
|
||||
n = prepare_chunks()
|
||||
print(f"[codex-arm] {WORKDIR.name}: {n} chunk files", flush=True)
|
||||
t0 = time.time()
|
||||
codex(
|
||||
["-s", "read-only"],
|
||||
f"This directory contains transcript_00.txt through transcript_{n-1:02d}.txt. "
|
||||
"Read EVERY file COMPLETELY one at a time using 'cat transcript_NN.txt' "
|
||||
"(full file, do not use head/tail/grep). After each file, write a 2-3 "
|
||||
"sentence summary of what happened in that portion. Do not skip any file.",
|
||||
)
|
||||
rollout = newest_rollout()
|
||||
session_id = rollout_session_id(rollout)
|
||||
stats = rollout_stats(rollout)
|
||||
# Codex auto-compacts at ~90% of its 258K window. If one read pass didn't
|
||||
# trigger it, re-read files in the SAME session until it does (max 3
|
||||
# extra passes) — the comparison requires post-compaction state.
|
||||
passes = 0
|
||||
while stats["compaction_events"] == 0 and passes < 3:
|
||||
passes += 1
|
||||
print(f"[codex-arm] no compaction yet (peak={stats['peak_context_tokens']:,}) — re-read pass {passes}", flush=True)
|
||||
codex(
|
||||
["resume", session_id],
|
||||
"Re-read ALL transcript files again completely with 'cat', one at a "
|
||||
"time, and refine each of your per-file summaries with any details "
|
||||
"you missed. Do not skip any file.",
|
||||
)
|
||||
stats = rollout_stats(rollout)
|
||||
read_s = time.time() - t0
|
||||
print(f"[codex-arm] read phase {read_s:.0f}s, {stats}", flush=True)
|
||||
if stats["compaction_events"] == 0:
|
||||
print("[codex-arm] WARNING: compaction never fired — arm invalid", flush=True)
|
||||
|
||||
questions = json.loads(Path(QUESTIONS).read_text(encoding="utf-8"))
|
||||
qlist = "\n".join(f"{i+1}. {q['q']}" for i, q in enumerate(questions))
|
||||
codex(
|
||||
["resume", session_id],
|
||||
"Based on everything you learned from the transcript files earlier in "
|
||||
"this session, answer the following questions from memory. Do NOT "
|
||||
"re-read any files — answer only from what you currently retain in "
|
||||
"context. If you don't know, say 'UNKNOWN' and give your best guess. "
|
||||
"Reply with a numbered list, one concise answer per question.\n\n" + qlist,
|
||||
)
|
||||
quiz_text = last_agent_message(rollout)
|
||||
print(f"[codex-arm] quiz reply: {len(quiz_text)} chars", flush=True)
|
||||
answers = {}
|
||||
for m in re.finditer(r"(?m)^\s*\**(\d{1,2})[.)]\**\s+(.+?)(?=^\s*\**\d{1,2}[.)]\**\s|\Z)",
|
||||
quiz_text, re.S):
|
||||
answers[int(m.group(1))] = m.group(2).strip()[:600]
|
||||
|
||||
results = []
|
||||
for i, q in enumerate(questions):
|
||||
ans = answers.get(i + 1, "(no answer parsed)")
|
||||
verdict = judge(q["q"], q["gold"], ans)
|
||||
results.append({"q": q["q"], "gold": q["gold"], "answer": ans, **verdict})
|
||||
print(f" Q{i+1}: {verdict['score']}", flush=True)
|
||||
|
||||
scored = [r["score"] for r in results]
|
||||
summary = {
|
||||
"policy": "codex_real",
|
||||
"recall_pct": round(100 * sum(scored) / (2 * len(scored)), 1),
|
||||
"scores": scored,
|
||||
"read_seconds": round(read_s),
|
||||
**stats,
|
||||
"rollout": rollout,
|
||||
}
|
||||
Path(OUT).write_text(json.dumps({"summary": summary, "results": results}, indent=1),
|
||||
encoding="utf-8")
|
||||
print(json.dumps(summary, indent=1), flush=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user