`recover_abandoned_delegations` only replays recorded children of a unit, and only
the >1-child join path ever called `record_unit_child` (c5594ec4b3 wired it into
`_run_children_parallel`). The single-child branch of `_execute_and_aggregate` ran
the child with no durable record at all, so a background `delegate_task` with one
goal — the default shape, and every ungrouped unit under `independent_completions` —
lost a finished child whenever the owner died between the child's return and the
durable completion write (OOM-kill on Linux, orphaning elsewhere, and everything
after the child returns is owner-lifetime: host-owned finalize, transcripts, manifest,
then the write). The parent got the bare "outcome unknown" and the work was gone (#116000).
`_record_finished_child` is now the one place that decides to durably record a finished
child of a detached unit; the join loop and the one-child branch both call it.
No new rows, no new consumer shape.
Tests: tests/tools/test_async_delegation.py::test_one_child_unit_keeps_its_finished_child_when_the_owner_dies
_run_children_parallel polled futures and fabricated interrupted entries
for pending children, but the executor's with exit then joined every
worker - a child parked in an uninterruptible call held the parent
thread forever, defeating the interrupt fast-path. Switch to an explicit
DaemonThreadPoolExecutor and shutdown(wait=not interrupted,
cancel_futures=interrupted) in a finally, matching the moa_loop shape;
abandoned children unwind on their own heartbeat and deferred-close
transport drain, and cancelled queued children never start.
Squashed integration of the user-facing message audit for this surface set.
Full per-finding receipts: /tmp/ux-audit/lanes/*-receipt.md (campaign artifacts).
When a child's final answer still missed its output_schema after the one
bounded retry, the result entry flipped to status=failed with the error
"Final answer does not satisfy the declared output_schema" — the completion
line printed ✗ and orchestrators read a finished audit as a failure. Five
audits of 413-4103 s were lost this way in the Sep 10-14 retrospective and
the parent had to mine the live transcripts; in four of them the "violation"
was a ```json fence around a valid array, which the candidate extractor
sliced to its first..last object.
Now: status stays completed, `summary` is the child's raw final text,
`schema_valid: false` + `schema_errors` carry the verdict and a `schema_note`
says the text is unvalidated; the sync completion line shows ⚠ with the
reason. The extractor tries the earliest-opening bracket span and keeps the
first that parses (fenced arrays validate). The OUTPUT CONTRACT the child
sees now says "ONLY the JSON value — no prose, no code fence" and what a miss
costs. One bounded retry is unchanged.
_attach_child now mirrors a pending parent stop with the same split
interrupt() uses for its own fan-out (hard -> hard_interrupt, soft ->
interrupt), so a redirect is not turned into a cancel on a child that was
attached late. _restore_parent_cancellation collapses to re-attaching the
rejected unit's children: the replay is the attach step's job now.
Test fixture: _Batch gained origin_session_history_delivery on main after
the salvaged PR was written.
Co-authored-by: illidan <noequal666@gmail.com>
Rebuilt against the post-refactor owners: the chat-completions route now
lives in gateway/platforms/api_server_openai_routes.py, the wake gate in
tools/delegate_tool_dispatch.py, and session_context.py was reshaped —
the original patch aimed at code main no longer has.
Header-less OpenAI-compatible clients get a fingerprint-derived session
id bound as the api_server chat_id. delegate_task's background gate
treated ANY bound session id as wake-capable and dispatched detached
subagents, but for derived ids the wake self-post lands in a session
whose history the client never reloads — the result is undeliverable by
construction.
Bind a wake_capable provenance flag at session-bind time, DEFAULT-DENY
at the central boundary (set_session_vars and _bind_api_server_session
both treat an omitted declaration as denied; a binder that says nothing
grants no wake authority): "1" only from audited producers whose client
can address the id again (explicit X-Hermes-Session-Id — 403-gated on
API_SERVER_KEY — native /api/sessions/{id} routes, /v1/runs); "" for
fingerprint-derived ids. The delegate gate requires the flag (fail-closed,
captured pre-child-construction alongside origin_wake_sid) and keeps the
forced-sync fallback with its honest note.
Reported-and-investigated-by: shojikumaru (Sho + Alpha) via #98619
Stress run with a real orchestrator subagent showed the per-task keys buried in results[] were not relayed
upward; the same prose lines now sit at the top of the sync delegate_task result. evals/subagent_process_handoff/
stress_handoff_live.py runs 12 real parent+child scenarios (handoff, orphan, unread, clean read, cap, exited refusal,
mixed fan-out, parent controlling the inherited process, sibling theft incl. adversarial, nested orchestrator,
5-way burst) and scores them against runtime accounting, never model prose.
Three orchestrator failures traced through the Sep 7 gpt-6-astra campaign sessions:
1. delegation.independent_completions (new, default false). #104299 made every
ungrouped task its own completion message, so a 15-task call woke the
orchestrator up to 15 times; one chain received 132 notices and answered
130 of them with "already incorporated". A multi-task call now returns as
ONE consolidated message unless the flag is on; `group` is inert until then.
2. Queued units were killed before they started. Units of one call share a
pool slot but the executor was still sized by slots, so with 15 units live
a new unit queued behind a full pool; the stale monitor's clock ran from
dispatch, interrupted it at 450 s, and the child exited `interrupted 0.02s`
when its thread finally came up (13 such lanes in one session). The
executor now grows to the number of live units and the stall clock arms
when the runner actually starts.
3. The tool text said "do not wait or poll — just continue" without saying
that completions are delivered only BETWEEN turns. A model that never ends
its turn (one 203-minute turn, 717 API calls) never received 40 finished
results. Tool description, dispatch note and completion header now say to
finish independent work, give a one-line status, and end the turn.
Since #104299 a background delegate_task call is split into completion units
(one per `group`, one per ungrouped task). A multi-child unit still joined on
all its children before anything was written durably, so an owner crash
between the first and last child lost the finished work and replayed the whole
unit as "outcome unknown" — the restart-granularity gap that #104233 (Xipong's
#76228/#76229 direction) solved with a second row per child.
Each finished child of a detached unit is now recorded on the unit's OWN row
(`record_unit_child` → result_json {results, partial}) as its future lands;
the real result overwrites it at finalize. `recover_abandoned_delegations`
replays recorded children with their real summaries and marks only the
unfinished ones unknown, naming the count. No new rows, no new consumer shape.
`task_indexes` is persisted so recovery knows a split unit's members.
A background delegate_task call used to be ONE async unit: the runner joined on
every child and a single consolidated message re-entered the conversation when
the SLOWEST finished. Fifteen independent PR reviews therefore waited on the
fifteenth before the parent could act on the first.
Each task now carries an optional `group`. `_units_of` partitions the call's
children into units — one per distinct group, one per ungrouped task — and each
unit is dispatched to the async registry on its own, so its results re-enter the
conversation as soon as THAT unit is done. Tasks that must be compared or merged
share a group and still return together.
Capacity is unchanged: every unit of one call joins the first unit's pool slot
(`slot_key` in `async_delegation._dispatch`), so splitting never consumes more of
`delegation.max_concurrent_children` than the call did. Unit ids suffix the
call's id (`deleg_xxxx-1`, `-2`, …) so live transcripts stay under one dir; the
completion block names the group and notes that sibling units report separately;
`active_task_count` counts a unit's own tasks.
`format_batch_tag` handed out `set N` ordinals from one process-wide table,
so every conversation on a shared backend and every child's nested fan-out
advanced the same counter. A user's second wave of 15 lanes rendered as
`[set 20 · 13/15]`, which reads like 20 batches were spawned.
Scope the ordinal table by the parent conversation (`parent_agent.session_id`)
and thread the parent through the three render sites (batch header,
completion lines, child tree-line prefix via the shared session_ref). The
first fan-out in a conversation is `set 1`, the next `set 2`; sibling
conversations and nested child batches no longer inflate it.
Invariant test proven red on origin/main, green with the fix.
Batches join on the slowest sibling before ONE consolidated block re-enters
(the design: one results block per fan-out). Failure is the case that
should not wait. In the 1,393-agent refactor run every wave-1 child died in
the 08:29 401 storm; the parent learned of it at 09:36, when the batch's
"unknown outcome" block finally arrived: 66 minutes of a dead wave with
nothing running, the single largest idle gap of the run.
_run_children_parallel, for DETACHED batches only (the sync path prints a
completion line the parent is already watching), pushes ONE
type="async_delegation" event with task_failure_notice=True and a
single-entry results list when a child ends in a failure status while
siblings are still pending. Same event shape and routing fields as the
batch result, so every drain/ownership/format path treats it identically;
the batch record is not finalized and its consolidated result still
arrives unchanged. The formatter renders
"[ASYNC DELEGATION TASK FAILED — <id>, task i/n]" with the task, status,
error and live transcript path, and says the batch result is still coming.
The TUI dedup key distinguishes a notice from the final result and from a
sibling's notice.
Live through the real detached dispatch (3 children, one fails at 0.1 s,
two succeed at 8 s): notice drained at t+0.3 s, batch final at t+8.4 s. On
main the parent hears nothing until t+8.4 s.
Tests (2): the notice reaches the queue with the record's routing fields
while the record stays running, and formats as an early warning; no notice
for a finished batch; dedup key differs from the final result.
Delegation/registry/notification suites (129 files) 1,344 passed.
tools/approval.py no longer re-exports sibling names (approval_context/prompt/floors/detection/
human_wait/smart/gateway_wait); it imports only what it uses. Siblings reference sibling-defined
names directly (module-attribute reads on tools.approval_context so patching the defining module
still works); only facade-owned state (_lock, _gateway_queues, _permanent_approved, _denied,
_denial_breaker_addendum, _gateway_notify_cb) is still read back through tools.approval.
approval_detection calls its own _command_detection_variants instead of late-binding through the facade.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
