* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
Review finding (major): ProcessRegistry.restore_completions() is once-per-process but read
async_delegation._db_path() (ContextVar-aware) under whatever scope the FIRST consumer ran in.
In the TUI gateway both first consumers (the session notification poller and the prompt_turn
drain) run inside _session_profile_runtime_scope(session), so under multi-profile
`hermes serve` the first session's profile ledger was replayed and the LAUNCH profile's
undelivered completions were never replayed for the life of the process (origin/main's
import-time replay always covered the launch profile).
Fix: restore_completions() clears the hermes-home override for the duration of the replay
(set_hermes_home_override(None) + reset), so the once-per-process replay always reads the
launch ledger whoever gets there first; the caller's scope is restored afterwards. Smaller
than a per-home restored-set plus a tui_gateway boot hook: it restores exactly main's
invariant with no new boot seam, and secondaries stay where they were (gateway
_restore_secondary_completion_ledgers). Docstring states the invariant.
Tests:
- tests/tools/test_process_registry_lazy_restore.py::test_first_drain_under_secondary_scope_replays_the_launch_ledger
(red on the PR head: replayed profiles/b/state.db; green now)
- tests/gateway/test_multiplex_unserved_shared_ingress.py::test_boot_replays_the_launch_ledger_before_secondaries_and_watchers
(minor: pins the gateway boot hook — launch-scope replay in _start_secondary_profiles, i.e.
before the secondary bind and before _async_delegation_watcher, the only production consumer
that reads completion_queue.get_nowait() without drain_notifications)
WHAT
- tools/process_registry.py: ProcessRegistry.__init__ no longer calls
restore_undelivered_completions(); a new once-per-process
restore_completions() does, invoked by the first consumer:
drain_notifications() (CLI process_loop / TUI prompt turn), the gateway
startup (run_startup._start_secondary_profiles, right before the secondary
ledgers are replayed) and the TUI session notification poller.
- tools/async_delegation.py: restore_undelivered_completions() returns 0 when
<HERMES_HOME>/state.db does not exist (a replay must not create or migrate
the ledger); _connect() creates the parent via mkdir_under_hermes_home()
instead of a bare mkdir so a late writer cannot resurrect a deleted
(tombstoned) or missing named profile.
- gateway/run_notifications.py: docstring no longer claims the import restores
the launch ledger.
WHY
`process_registry = ProcessRegistry()` runs at module import, and model_tools
imports it transitively (tools.registry -> close_terminal_tool), so
`python -c 'import model_tools'` (hermes doctor, any tool-registry consumer)
under a fresh/typo HERMES_HOME created the full profile skeleton + state.db,
recreated a tombstoned `profiles/.deleted/<name>` profile, and ran
reconcile_state_schema() against an existing store — bypassing the
assert_named_profile_home_live / mkdir_under_hermes_home guards (#97128,
#112592). Restoring on first consume keeps the replay for every process that
actually drains the queue while an import performs no state.db I/O.
Salvaged from #123348 (@webtecnica: lazy restore + missing-ledger guard) and
#123298 (@Wenfengcheng: mkdir_under_hermes_home in _connect), trimmed to the
minimal shape (no completion_queue property, no read-only probe).
Fixes#123265
Co-authored-by: funky-xamarin <30426178+Wenfengcheng@users.noreply.github.com>
(cherry picked from commit b77c4ff49062d2936271ded80821b48b0c11d368)
A `terminal(background=true, heartbeat=N)` tick queued a notification every N seconds
whether or not the process had printed anything, and every queued event costs the owning
session a full model turn. On Desktop and the TUI that turn painted the wake as a user
bubble ("[Background process ... heartbeat #9 ... (no new output since the last
heartbeat)]") followed by the model's "Still running normally." — over and over, for a
process whose row on the status stack already said it was running — and while the wake
held the session's turn, the user's own prompt sat queued behind it.
- `ProcessRegistry._emit_heartbeat` skips a tick with no new output. The sequence counts
delivered beats only; the "(no new output)" placeholder in the formatter is gone.
- TUI/Desktop type heartbeat rows `display_kind: hidden` (the kind both clients and the
transcript preview already honour); the CLI paints a one-line receipt and persists the
row hidden, so reopening the session in Desktop shows only the agent's reply.
- Desktop hydration drops heartbeat rows persisted by older backends the same way.
- `display.background_process_notifications: off` is honored by the TUI/Desktop poller and
the CLI drain, not just the messaging gateway. `off` mutes process-driven wakes only:
a finished `delegate_task(background=true)` still lands.
Supersedes #123123 (cherry-picked; scoped so `off` keeps subagent results) and #119202
(cherry-picked; `heartbeat: 0` is schema-valid so models that materialize every field
stop tripping the foreground guard).
Checkpoint refresh treated a failed start-time check as a collected exit
and queued a completion. A live PID whose start time matches, or whose
start time cannot be read, stays running. A reused PID is closed without
being signalled. A gone PID is pruned. A completion is emitted only when
an exit status was collected.
Background processes spawned with terminal(background=true) are killed from
three agent-lifecycle sweeps: agent release()'s kill_all, a gateway turn
timeout's kill_started_since, and agent close's owned-process loop. Jobs the
user explicitly wants to outlive the session (overnight batches, watchful
daemons) had no way to opt out.
Add terminal(background=true, persist_on_release=true):
- ProcessSession.persist_on_release, stamped by spawn_local/spawn_via_env,
carried in crash-recovery checkpoints and exposed via list_sessions()
- kill_all skips persisted sessions only for lifecycle sources
(_LIFECYCLE_KILL_SOURCES: kill_all, gateway_turn_timeout, agent_close);
explicit operator stops (process_manage kill, /stop slash + RPC mirror,
CLI /stop) now pass distinct sources so they still reach persisted jobs
- the agent_close owned-process loop in _close_task_resources skips
persisted sessions the same way
- gateway shutdown keeps killing persisted jobs (source=gateway_shutdown):
the host is going away and survivors would become PPID=1 orphans
Co-authored-by: salvaged from #109846 (persist_on_release plumbing) and
extended to the turn-timeout and agent_close paths.
utf-8-sig exists to tolerate BOMs that Windows tooling adds to files
users edit. /proc and /sys files are generated by the Linux kernel, never
BOM'd and absent on Windows, so -sig there only muddies the read/write
policy. Switch every literal /proc/ and /sys/ read to utf-8 and teach
the footgun read rule that string literals starting with /proc/ or
/sys/ are exempt (user-edited files keep utf-8-sig).
With ProcessSession defaulting owner_task_id to task_id at construction,
the remaining `owner_task_id or task_id` spellings in _new_session and the
three record emitters can never take the fallback. Also moves __post_init__
below the last dataclass field. Final-gate quality/reuse findings.
The ownership filters this stack moved onto owner_task_id each spelled the
same `(s.owner_task_id or s.task_id)` fallback, five copies beside the
strict `owner_task_id ==` form running_owned_by already uses. Every
production constructor stamps the owner; only hand-built sessions lacked it.
ProcessSession.__post_init__ now defaults owner_task_id to task_id, the
filters compare owner_task_id alone, and snapshot_running_ids reuses
running_owned_by.
wait_for_pending_completions(task_id) filtered notify_on_complete jobs by
ProcessSession.task_id, but the terminal tool stores the container key
there (`session:<key>`, `default`) and the spawning turn in owner_task_id
— the same mismatch the previous commit fixed in snapshot_running_ids,
kill_all and list_sessions. A caller scoping the linger to its own turn
would therefore wait on nothing. Match (owner_task_id or task_id) like
its siblings; the existing scoping test now spawns both jobs under one
shared container key so it fails on the old predicate.
Follow-up to #120631 (sibling site noted in review).
Co-authored-by: John Paul Soliva <soliva.johnpaul@icloud.com>
The terminal tool stores the container key in ProcessSession.task_id
(`session:<key>` on local/SSH, `default` / `profile:<name>` under
persistent Docker) and the spawning turn's id in owner_task_id. The
abandoned-turn cleanup from #76687 (/stop, /new, eviction, the
inactivity watchdog, API-server disconnect and /v1/runs stop) snapshots
and reaps by the turn's raw session id, and snapshot_running_ids /
kill_all compared it against task_id, so they never matched: the
turn's background jobs kept running after the turn was abandoned, and a
notify_on_complete job woke the session later. process_manage(list)
missed the session's own jobs for the same reason.
Match ownership on owner_task_id in snapshot_running_ids, kill_all
(and so kill_started_since) and list_sessions(task_id=), falling back
to task_id for entries without an owner. has_active_processes stays on
the container key: its caller passes container keys.
The two new tests spawn through the real terminal tool inside a session
context and drive the real /stop core and inactivity watchdog; both
fail on the previous code.
(cherry picked from commit 1735bbf84dae90ea2c68499a8971f6bed1570366)
Background jobs run under an interactive `bash -lic` wrapper, which ignores SIGTERM.
_terminate_host_pid snapshotted the tree once, BEFORE the grace windows; the shell kept
running its script through both windows, and every child it started meanwhile was
reparented to init when the shell was finally SIGKILLed -- while kill_process reported
"killed" and the post-kill survivor check (#115490) only re-probed the stale snapshot.
Deterministic repro: spawn_local('sleep 1; for i in 1 2 3; do sleep 3600 & done; wait')
and kill it within the first second -> 3 orphaned sleeps every time. Found by
tests/e2e/core/chaos (tool_background_swarm flaked under load, and
tool_background_spawns_while_killed is red without this).
Re-snapshot the descendants while the parent is still alive, right before escalation.
A tty-less `bash -lic` writes "cannot set terminal process group" and
"no job control in this shell" as two separate write() calls. The reader
cleaned shell noise from the FIRST chunk only, so whenever it woke between
the two writes (loaded CI runners) the second line landed in
output_buffer as the process's only "output".
Symptoms on main: tests/hermes_cli/test_process_dock.py painted
"last: bash: no job control..." instead of "starting" (3 main reds,
Sep 21-22) and tests/tools/test_process_registry_list_exit.py's probe
woke on that noise before the writer had printed, so the completion
event lacked "owner-output" (2 main FLAKY frames). The same leak reaches
users through process.list output_preview and the live-work dock.
Keep stripping leading noise from every chunk until the process has
produced non-blank output; after that, matching text is the process's
own. The list_exit probe now waits for the writer's marker rather than
for any bytes.
Live repro (stand-in shell reproducing bash's two writes with a 50 ms
gap, real spawn_local + select/read1 reader): base 10/10 buffers carry
"bash: no job control in this shell\n"; fixed 0/10.
The #70760 commit widened _redact_process_result(result, *, task_id=) and passed the
caller's task_id from _handle_process. tests/gateway/test_completion_delivery.py
monkeypatches the function with a one-arg stub (CI red on #118845), and the caller's id
is the wrong identity anyway: a poll/log/wait result belongs to the process OWNER, whose
task_id the session record already carries. Resolve it inside the function from
result["task_id"] or the session looked up by result["session_id"]; the test pins the
owner's id ("t1") instead of the caller's.
A plugin registered on transform_terminal_output only ever saw foreground
`terminal` output: tools/terminal_tool_result.py::_apply_output_transform_hook
runs from finalize_foreground_result and nowhere else. Background output
reached the model through a different seam — process_manage poll/wait/log/kill
results, `list` previews and the completion/heartbeat/watch notifications all
pass through tools/process_registry.py::_redact_process_result — which redacted
but never transformed, so a fleet redaction or summarising plugin silently did
nothing for backgrounded commands.
Apply the same hook helper at that shared seam (one new
transform_process_output wrapper) and at the two gateway agent-notify sites
that read session.output_buffer directly. The order matches the foreground
path and teknium1's review note on #71401: hook first, redaction after, so a
replacement the plugin returns is still masked. returncode is None while the
process runs; env_type is not recorded per process and is passed empty.
Not changed: the spawn acknowledgement ("Background process started") carries
no command output, and the persistent local shell already goes through
_run_foreground and was transformed — the issue's reading of that branch was
wrong; the real gap was the process_manage/notification seam.
Fixes#70760
Slim redo of #71401 (Christopher-Schulze): same seam and ordering, without
the ANSI-stripping relocation and render helper.
Co-authored-by: Christopher <210261288+Christopher-Schulze@users.noreply.github.com>
(cherry picked from commit 84661de52f78ccb84054e1ded92b07166e40546d)
List reconciliation could race the live stdout reader, publish a completion before buffered descendant output was ingested, then close the pipe underneath that reader. For selectable POSIX pipes, mark the direct child exited but ask the reader to perform the final drain and remain the sole completion publisher. Keep the existing fallback for readers that cannot be coordinated.
`terminal(background=true, heartbeat=N)` emits a `heartbeat` event every N seconds
(floor 60) carrying only the output produced since the previous one, plus the usual
completion notice. The agent stays current on a long bounded job (merge train, full
test suite, deploy) and reacts to a failure within N seconds instead of at exit.
Why: `notify=true` fires once at exit, so multi-hour merge trains ran silently — the
parent's prompt cache went cold and every "what's up" cost 5–10 rediscovery calls;
30 days of batch sessions show 4,930 hand status polls (14% of all tool calls).
`notify=[pattern]` cannot serve this: its lifetime cap (8) exists precisely to stop
periodic output from flooding the agent.
Mechanism: one daemon timer thread for every heartbeat session (reader threads block
on the pipe and cannot keep time); delta output via a total-ingested counter over the
rolling buffer; heartbeats only where the completion notice can be delivered (same
async-support/subagent gates), never after exit. `heartbeat_seconds` is checkpointed.
Rendered on CLI, gateway and TUI through the existing process-notification paths.
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
The live-work docks retire a finished background process ~60 s after it
ends; the registry only knew started_at, so the age of an exit was not
observable. _move_to_finished is the single choke point every exit path
(reader loop, reconcile, kill) passes through, so the stamp lives there.
completion_reason rides along so a killed process can read "killed"
instead of "exit -15".
process(action='wait'/'poll'), the already_exited and kill snapshots still used a
hard-coded 2000-char tail with no output_cut, so the polling fallback bot-mode.md
documents for api_server/one-shot senders delivered a silently truncated reply.
One _completion_output() helper now sizes every delivery route.
SIGKILL/taskkill are asynchronous; poll()/isalive() right after kill() still
say alive, so the exact #115490 scenario (a SIGTERM-ignoring child that needed
escalation) was reported as 'Kill incomplete' on every run (10/10 in a live
probe) and the session stayed in _running until the reader thread finalised it
as a plain exit. Re-probe survivors for up to 1s before deciding.
Post-kill, check the session tree (Popen poll, PTY aliveness,
PID-scope identity plus descendant scan). Survivors keep the session
running instead of persisting a false killed receipt and pruning it.
Fixes#115490.
A bot DM's reply comes back as the delivery process's completion notification,
which carried the last 2000 characters of that process's output — the right
tail for a build log, but a bot may send 16,000 characters (MESSAGE_MAX_CHARS)
and its teammate's answer routinely runs longer than 2000. The sender got the
tail of the reply, header included in the cut, with nothing saying so, and
relayed it as the whole answer.
The registry now sizes a completion per process (ProcessSession
.completion_output_chars, checkpointed) and declares a cut (output_cut) when
the output did not fit; the notice names the cut and the process log that has
the rest. tools/bot_mode_dm._spawn_delivery — the one spawn point for every
DM lane (local, live-owner, Desktop relay waiter) — asks for MESSAGE_MAX_CHARS
plus header room. Everything else keeps the 2000-char tail.
Fixes#115334
Resolved toward the branch: PM provisions uv/python (main's install.ps1 uv-shim
salvage + its test and workflow steps dropped), the shim re-exec stays retired,
package.json carries no electron-builder block (afterExtract identity stamp wired
into electron-builder.config.cjs instead; after-pack.mjs keeps signing only),
Desktop workspace-deps helpers stay retired. Main's scratch-dir bootstrap
(export_scratch_tmp_env) is taken and re-run after profile resolution.
Hermes now routes scratch space through HERMES_HOME/cache/scratch (exported as
TMPDIR), so every production path that still spelled out /tmp bypassed that and
kept teaching the agent the habit. Fallbacks in tool_result_storage,
code_execution_tool, process_registry, the ACP child HOME, mini_swe_runner's
local cwd, and the CI/profiling scripts now use tempfile.gettempdir(); shell
installers fall back to $TMPDIR (then HERMES_HOME) when mktemp is missing, and
repro/eval shells use `mktemp -d -t`. User-facing help text and sample payloads
(hermes send, approvals test, hooks test, voice-mode WSL hints, meet_bot debug
line) no longer suggest /tmp.
Container-side paths (mini_swe_runner docker cwd, sandbox base env, remote
sync tarballs) keep the literal because they name the sandbox filesystem,
not the host.
An agent-issued `taskkill /F /IM python.exe` (or `pkill -9 python3`, `killall python`,
`Stop-Process -Name python`, `taskkill /FI "IMAGENAME eq python.exe"`, `pgrep python | xargs
kill`) from inside the supervised gateway killed the gateway: every branch of
_GATEWAY_LIFECYCLE_PATTERN was anchored on a hermes/gateway token, and the supervised gateway is
literally a `python` process. Branch E is token-aware (not a line regex) so option values are
never read as targets, `-f` cmdline patterns are judged as patterns (`pkill -f 'python
my_script.py'` passes, `pkill -f 'python -m hermes_cli.main'` does not), and other image names
(`taskkill /F /IM agent-browser.exe`) stay killable. Numeric-PID kills stay out of scope: the
explicit PID / `proc_*` id is the ownership-scoped route the terminal rejection now names.
The guard never ran on the Windows Scheduled-Task topology either: the launcher exports only the
generalized HERMES_SUPERVISED_CHILD marker, which gateway/restart.py never read.
is_supervised_gateway_launch() reads it and gates the self-kill guards;
is_gateway_supervisor_process() deliberately keeps ignoring it because it also selects the
exit-75 restart route, which the task has no restart policy to honour (#113670).
Supersedes the narrow `/IM python.exe` regex from #113671 (kept for authorship); the Windows
spellings from #94379 (`hermes.exe gateway restart`, `taskkill`/`Stop-Process` on hermes-gateway
tokens) ride along.
Fixes#113667
Two defects at the same spawn boundary (hermes_cli/kanban_db_dispatch.py::
_restart_safe_worker_argv -> tools/process_registry.py::restart_safe_gateway_child_argv).
#114720 — a managed-gateway dispatcher whose user D-Bus session had gone away
raised the restart-safe-scope RuntimeError at spawn; the ordinary spawn
except-clause fed it to _record_task_failure, which advanced
consecutive_failures and at failure_limit parked the card as a bare `blocked`
(block_kind NULL) with a `gave_up` run. Nothing about the card had run.
- restart_safe_gateway_child_argv raises RestartSafeScopeUnavailable
(RuntimeError subclass) so the dispatcher can tell a host refusal from a
card failure.
- _record_task_failure(infrastructure=True): run + event carry
`infrastructure: true`, consecutive_failures is left alone, the breaker
never trips; the card stays `ready` with the linger remedy in
last_failure_error and a warning in the dispatcher log.
- check_respawn_guard returns `infrastructure_cooldown` while the latest run
is such a refusal inside the rate-limit cooldown window, so a dead bus is
retried spaced instead of every tick.
- require_restart_safe_scope=True keeps its hard-fail semantics under the
managed gateway.
#113612 — `hermes kanban dispatch` from an operator's Type=oneshot systemd unit
spawned unmanaged workers: the gate `_is_supervised_gateway_process()` is
False for a CLI dispatcher, so the helper returned the bare argv and the
workers died at the unit's cgroup teardown with empty logs, while the
`running` rows waited for reclaim.
- New outlives_parent=True (kanban only; cron blocks its caller and keeps the
old gate): under any systemd unit (INVOCATION_ID) a fire-and-forget worker
is scope-wrapped when the user bus is reachable. Without a bus it degrades
with a loud once-per-process warning naming the consequence and both
remedies (enable-linger, KillMode=process) rather than refusing — the
unit's KillMode/lifetime is unknowable and a long-lived sequencer without
linger must keep working.
Docs: kanban.md "Workers and systemd cgroups" + guard reasons; cron.md note.
Based on analysis from PR #113624.
Co-authored-by: Kevin Rajan <7121943+kvnloo@users.noreply.github.com>
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
scoped_spawn_lost_user_bus() re-derived the user bus from an empty base env,
so it only ever looked under /run/user/<uid>. The worker itself is launched
with systemd_user_bus_env(worker_env), which honours a configured
XDG_RUNTIME_DIR. On a host whose bus lives outside the default runtime dir,
any unrelated systemd-run exit was therefore misread as "bus gone": the job
error named a missing bus that was still there, the cached scope verdict
flipped to False, and the next 60s of cron fires dispatched without cgroup
isolation.
The check now takes the spawn env, drops the bus address the spawn already
carried, re-derives from that, and decides on the DBUS_SESSION_BUS_ADDRESS
key rather than on dict truthiness (a non-empty env with no bus was never
"bus present").
Review finding: scoped_spawn_lost_user_bus used systemd_user_bus_env({}) instead of the worker's spawn env, so a configured XDG_RUNTIME_DIR made unrelated wrapper exits flip the scope cache to unscoped dispatch.
One TTL for both probe verdicts (the success-TTL constant collapses into
`_SYSTEMD_SCOPE_PROBE_TTL_SECONDS`), and the ack-wait loop consults
`scoped_spawn_lost_user_bus()` when a scoped dispatch exits before the
worker acknowledged: with `/run/user/<uid>/bus` gone the job error names
the missing bus and the enable-linger remedy instead of the wrapper's bare
`exit 1`, and the cached True flips so the next fire degrades to a direct
external subprocess rather than consuming another occurrence on a dead
wrapper. Contributor test trimmed to the revalidation invariant.
The early-EOF reaper made _finish_reader return without publishing when
wait() raises, so the session stays tracked for later reconciliation.
That is right for the pipe path (_reconcile_local_exit can still reap via
session.process), but PTY sessions have no session.process: when
ptyprocess.wait raises (waitpid ECHILD after isalive() already reaped the
child) the exitstatus is known, yet poll() reported "running" forever.
Only leave the session tracked when exit_code() is still None; otherwise
record the known status and finish as before.
Review finding: PTY session whose pty.wait raises stays in _running forever (fail-open regression vs main)
`ProcessRegistry._move_to_finished` pops the session out of `_running`, then
saves the receipt, releases handles and writes the checkpoint, and only THEN
enqueues the completion and sets `_completion_event`. A quiet one-shot parent
whose turn ends inside that window called `wait_for_pending_completions`,
found nothing in `_running`, drained an empty queue and exited without the
follow-up turn. That is the CI flake in
tests/tools/test_completed_process_results.py::test_headless_terminal_result_survives_cli_exit
(`follow_ups == []`), which also hit unrelated branches.
Consider `_finished` sessions whose event is not yet set as pending too.
Repro: a 6s sleep before the enqueue plus a 2s delay before the parent's first
wait fails the E2E 2/2 on main and passes 2/2 with this change.
_release_finished_handles read the dataclass fields through getattr
fallbacks and swallowed every exception; use the real attributes, suppress
only OSError/ValueError on the pipe close (an stdin flush can hit EPIPE),
and rely on ptyprocess/pywinpty close() idempotence for the master fd.
The call-site comment claimed the reader had always drained the pipe;
on kill_process/_reconcile_local_exit the reader may still be reading,
so state what actually happens (its next read raises, the loop exits).
Widen #75162: _prune_if_needed() drops finished sessions (TTL expiry and
oldest-finished eviction at MAX_PROCESSES) — release their Popen/PTY
handles there too, covering sessions inserted into _finished without
passing through _move_to_finished(). The release helper is idempotent,
so double-close on the normal path is a no-op. Adds two tests: prune
releases handles of dropped sessions, and a still-running session's
pipe stays open.
Finished sessions retained their subprocess.Popen pipe objects (and PTY
masters) until the finished-process TTL (FINISHED_TTL_SECONDS, default 30
minutes) elapsed. Under heavy background churn — deployments, archivers,
watchers — finished-but-unpruned sessions accumulated one open pipe FD
each, exhausting the gateway process's file descriptor budget and
surfacing as a 'file descriptor limit' error on new background spawns.
The registry never rejects spawns (it prunes oldest-finished at
MAX_PROCESSES), so the real defect was the retained-handle leak, not a
registry-cap rejection. The fix closes each finished session's Popen
stdout/stderr/stdin streams and PTY master in _move_to_finished(), right
after the reader loop drains EOF. poll()/wait()/read_log() serve output
from the buffered output_buffer — never from the pipe — so the release is
lossless.
Tests: 4 new cases in TestFinishedHandleRelease — Popen pipes closed,
PTY closed, no-handle sessions safe, and poll() still serves buffered
output after the release. All 4 fail on main (reproduction) and pass
with the fix.
Programs run via terminal(background=true, pty=true) can block forever when
they probe their terminal — device-status (ESC[5n), window-size (ESC[18t),
cursor-position (ESC[6n), or DEC private-mode (ESC[?N$p) queries — because
nothing on the PTY master side answers, and the raw query bytes leak into
captured output.
- tools/pty_query_responder.py: incremental byte scanner that strips the
handled queries from PTY output (chunk splits included) and produces
bounded replies; everything else passes through untouched.
- tools/process_registry.py: wire the responder into _pty_reader_loop
(POSIX only — ConPTY answers its own queries); flush partial escape
tails at end-of-stream.
- tests mirror the codex fixtures plus a live-PTY E2E where a subprocess
blocks on ESC[6n until answered.
Carve the direct-child list reconciliation from Indigo Karasu's earliest
PR #60506 (2a96ae2cbf806ccdc3e9b584911774f32622f421), corroborated by
fangliquanflq's narrow #81385 (50ffd243d92627e4a03a3ee8427ad0f5e090c3ab).
Run the existing helper after task/session filtering and reuse the
idempotent owner-stamped completion path. Do not import cross-session
disclosure, bare-PID healing, forget RPCs, or reader rewrites.
A real-child regression fails before this change and passes after it:
the direct child exits while its descendant keeps writing to stdout;
listing reports exit without consuming the result or waiting for EOF.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
