5eea87882a
* refactor(desktop): split clarify-tool.tsx into a clarify/ folder
Pure moves, no behaviour change. Parsing, the question-card core
(shell, choice rows, question block), the delivery watchdog, and each
pending/settled card get their own file so the core can be reused.
* feat(clarify): one questions[] shape with per-question status and one outcome
The clarify tool now takes only questions=[{question, choices?, multi_select?}].
A wrong shape is a tool error that names the right one, so a model that
sends the old top-level question/choices corrects itself on the next call.
Every result has the same shape on every surface: each response carries
status (answered, skipped, unanswered) with user_response null unless
answered, and the result carries one outcome (submitted, cancelled,
timed_out, undelivered) plus an optional surface-written notice. The
timeout and cancel sentences that used to pose as the user's answer are
gone, and the compressor reads status instead of matching their prefixes.
The callback contract is callback(questions) -> {answers, outcome, notice?}
(None = skipped, missing qid = unanswered). tui_gateway settles a batch the
same way for the last lock, a cancel, an interrupt and the deadline, and
clarify.lock keeps a null answer as a skip. A multi-select answer that is
not a JSON array counts as one typed ("Other") answer. The tool-row preview
reads the first question.
* feat(clarify): every surface asks through the one question card
CLI: the batch panel is the only panel; Enter on an empty field skips a
question, Ctrl+C cancels and keeps the locked answers, the deadline returns
timed_out. -q and -z return undelivered with a no-user notice.
Ink TUI: the single-question prompt is gone; the card supports multi-select
(Space or a digit toggles, Enter locks, typed Other text joins the picks),
an empty submit skips, Esc cancels, and "Batch" names are dropped.
Desktop: the single-question card is deleted and its keyboard handling moved
into the one card (arrows, letters and digits, auto-advance, Enter picks then
confirms). Confirm enables at one answer; blank questions lock null; Skip and
a composer message cancel; the settled card shows No answer for unanswered
questions. The bots room card follows the same contract.
Messaging: one card per question with a "Reply skip to skip" line in every
locale; timeouts and delivery failures map to timed_out and undelivered.
* fix(clarify): cancelled for a released messaging card, labels win over the skip word
A prose reply to a messaging choice card, /new and session end released
the wait with "", which read as timed_out. They now resolve with a
CANCELLED marker and the tool gets outcome cancelled.
A typed reply that matches a choice label ("Skip") now resolves to that
choice; the skip word applies only when no choice matches.
The bots room card sends the picked labels as they are; the tool already
strips the recommendation label. Unused OUTCOMES and a new docstring and
comment are removed.
* fix(tui): re-editing a multi-select answer keeps its picks
The Ink card stored a multi-select answer as display text ("A, B"), so
revisiting the question put the whole string into Other and re-locked it
as one item. The answers map now keeps the raw JSON array, the card
restores the picks on revisit, and only the display lines format it.
Comments that earlier edits reworded are restored to their original
words, minus the phrases the change made wrong. The compute-host clarify
lock accepts None for a skipped question.
* test(clarify): align three checks with the one-answer confirm and the regenerated keys
The desktop Cmd+Enter test now expects a send with one answer (the blank
question locks null), the compressor test expects the single-answer summary
the code gives, and locales/_keys.desktop.json is regenerated for the
removed and added clarify strings.
* test(tui): the one-question card header is singular
core_tool_deferral — live A/B harness for tool-visibility changes
Built for the PR #97979 maintainer battery (core-tool deferral behind the
tool_search bridge). Runs REAL in-process AIAgents from two pinned
checkouts and grades task outcomes programmatically — accuracy, api turns,
tokens, wall, bridge-call counts — across any set of models.
Original verdict + full numbers: results/SUMMARY.md and the PR #97979 body
(288 runs; gpt-5.6-terra / glm-5.3-flash / qwen3.8-27b).
Layout
tasks.py— 14-task battery: one task per deferred tool, multistep (todo discipline, GUI chains), long-range (session_search → backup → cron → todo), a destructive-ambiguity clarify trap, an eager-only control, and a false-discovery distractor. Each task carries fixtures, a programmatic grader (0–1 partial credit), and scripted user replies.worker.py— one (arm, model, task, rep) cell in an isolated subprocess: temp HERMES_HOME + workspace, hermetic env (only OPENROUTER_API_KEY survives), seeded session DB (targets + decoys), deterministic desktop-surface stubs (desktop_ui emitter + agent callbacks), computer_use/image_generate stubbed at the registry handler. Terminal/files/cron/process/session-DB are REAL. Exit 3 = infra/config error (never scored).orchestrator.py— battery runner: resume-safe, per-task wall timeouts, parallel cells, errored-record retry, 3-infra-abort fuse.report.py— per-task table both arms (score spread, turns, tok, wall, bridge calls), mean-of-task-means, noise/error accounting.
Running
# 1. Two plain checkouts pinned to the SHAs under test (never pip install -e)
git worktree add /tmp/abdefer-base <baseline-sha>
git worktree add /tmp/abdefer-pr <pr-sha>
export ABDEFER_BASE_TREE=/tmp/abdefer-base
export ABDEFER_PR_TREE=/tmp/abdefer-pr
export OPENROUTER_API_KEY=... # the only key the worker keeps
# 2. Smoke one cheap cell first
python3 worker.py base openai/gpt-5.6-terra config_grep_distractor 1 /tmp/smoke.json
# 3. Battery (per model; start with the STRONGEST model to validate variance)
python3 orchestrator.py openai/gpt-5.6-terra 3 --parallel=5
python3 orchestrator.py z-ai/glm-5.3-flash 3 --parallel=5
python3 orchestrator.py qwen/qwen3.8-27b 3 --parallel=5
# 4. Readout
python3 report.py
ABDEFER_PYTHON overrides the worker interpreter (defaults to the
orchestrator's own); ABDEFER_RESULTS overrides the results root.
Discipline (from the readtool/session_search harness lineage)
- Verify model slugs against the live OpenRouter list before launching.
- Interactive fairness: if the agent ends its turn with a plain-text
question, the worker sends the scripted reply (max 2, counted as
user_roundtrips) — without this, every clarify-shaped task scores 0 unfairly and the battery is poisoned (the first terra run was discarded for exactly this). - Same-denominator rule: errored runs score 0 and STAY in the accuracy denominator; they are excluded from efficiency means.
- Extend contested cells (score spread at n=3) to n=6 before concluding.
- For discovery-rate regressions, always check base-arm usage on the same tasks first — a tool models skip even when visible is not a deferral regression.
- Audit anomalous cells from
*.transcript.jsonbefore publishing.
results/ is gitignored except SUMMARY.md — rep JSONs are rebuildable,
verdicts are the artifact.