A foreground bot open against a full 3-slot local pool waited 30s for a slot
and timed out (production: 7,122 slot waits, 6,305 timeouts, ~790 cancelled
starts in 10 days on a 17-profile host). Nothing could free a slot: a child's
hard lease lives until the child exits (main.ts child 'exit' handler,
teardownFailedLocalBackend, stopPoolBackend after the bounded SIGTERM->SIGKILL
exit), while LRU eviction and the idle reaper key off lastActiveAt, which the
renderer refreshes every 60s for every open socket. A bot-tile-pinned resident
is keepalive-fresh forever. Occupied is not busy.
New `electron/pool-retire.ts` (pure, DI-testable like pool-spawn-coordinator):
a foreground dial that finds `activeCount >= poolMaxBackends()` may retire ONE
resident, under these rules:
* Proof is the backend's. Each LRU candidate is probed over
`/api/health/idle` (running sessions + running cron jobs + prompts waiting on
a human). Only `true` is idle; `false` and `null` (older runtime 404, probe
error, unreadable ledger) are busy. The renderer-published `activeTurn`
lease is an early skip, never the proof; entries no longer initialise it to
false, so a fresh backend is not evictable by default.
* Admission fence: concurrent foreground dials share one retirement and one
probe; the coordinator hands the freed slot to the ticket that queued first.
* Identity recheck (`pool.get(key) === entry`, no new turn lease) after every
await, and a re-probe immediately before the stop.
* The waiter's `coordinator.request()` is issued BEFORE the stop so it sits at
the queue head; the slot is released only by the retired child's real exit
through the existing stopPoolBackend -> releaseLocalBackendSlot path.
* `hermes:pool:retiring` is broadcast to every window before the SIGTERM so
the renderer parks the scope instead of redialing into the vacated slot.
main.ts grows only the trigger point, the HTTP probe, the broadcast and the
retirer construction. Kept from #104871 with credit: the trigger point before
`coordinator.request`, the `touchBackend(profile, options)` IPC widening
(preload.ts / global.d.ts), and the LRU-among-eligible selector shape.
Tests (pool-retire.test.ts): fence with two concurrent tickets over a real
LocalBackendSpawnCoordinator (one probe, one SIGTERM, slot granted only after
the simulated exit, exactly one waiter served); identity recheck aborts on a
swapped entry or a lease published after the probe; idle null / cron-running
ineligible with fall-through to the queue; renderer activeTurn:false loses to
a backend re-probe.
Co-authored-by: bounce12340 <128559392+bounce12340@users.noreply.github.com>
5fef73015a