956732d4a8
Eval probes wrote fixtures, receipts and evidence to hard-coded /tmp paths and two live A/B tasks literally instructed the model to work in /tmp. They now derive locations from tempfile.gettempdir() / os.tmpdir() (env overrides kept), usage examples use relative output names, and the desktop e2e screenshot dirs, perf scripts and the short-session repro fixture stop naming /tmp. Also adds the explicit encoding= the windows-footgun check wants in the touched files.
Native Bot Mode delivery probe
Real Electron, production Python backend and tool execution, disposable HOME/HERMES_HOME, loopback scripted inference (no paid model). Linux seat fixture; run from repository root:
npm ci --no-audit --no-fund
cp evals/botmode-dm-delivery/probe-dm-delivery.spec.ts apps/desktop/e2e/
git apply evals/botmode-dm-delivery/mock-trigger.patch
(cd apps/desktop && npm run build)
(cd apps/desktop && DISPLAY=:0 XAUTHORITY=/run/user/1000/xauth_cnpsqU \
XDG_RUNTIME_DIR=/run/user/1000 VIRTUAL_ENV="$VIRTUAL_ENV" \
HERMES_DESKTOP_CDP_PORT=off npx playwright test e2e/probe-dm-delivery.spec.ts --reporter=list)
git apply -R evals/botmode-dm-delivery/mock-trigger.patch
rm apps/desktop/e2e/probe-dm-delivery.spec.ts
Use the current seat's actual Xauthority path and an existing runtime venv.
Artifacts default to /tmp/botmode-dm-review/native (override with BOT_DM_EVIDENCE); sandbox path is printed. The
fixture's generated hermes shim pins every child to this checkout, not an installed launcher.
Verified results
Base cf35e7351e770: nested beta quiet CLI executes real message_agent to the named
alpha Desktop owner. Admission, execution and reply succeed, and the quiet sender
receives its completion. A reload then shows the attributed incoming message. The
live pre-reload view showed the reply but omitted the incoming row in this fixture;
that renderer-refresh behavior is not fixed by this cron change.
Base CLI-owner case: separate cron producer returns exact SESSION_NOT_OWNED refusal.
Release owner, run scheduler tick, open Beta in Desktop: no cron output, assertion red.
Fixed case: queued receipt; release owner; real synchronous scheduler tick drains;
Beta Desktop renders CLI_OWNER_CRON_SENTINEL and its reply once. Final two-case
native run: 2 passed (1.2m). Supported-owner nested case also passed independently
on base (1 passed (40.0s)).
This does not establish native Windows parity, general retry after unowned CLI
failure, or correction of the issue #105460 --in ~ premise. CLI title resolution
is profile-DB-based, not workspace selection; the named live-owner route is positive.
The queue is deliberately at-most-once after claim. A crash before spawning but after claiming remains inspectable as claimed; it is not retried automatically.
Independent review follow-up
The probe now admits the named Beta destination from the default scheduler, then
changes the ticker's HOME to a different existing directory before drain.
Published head c91dfcbfe810c fails with Profile 'beta' does not exist, leaving
zero sentinel inputs in Beta. The follow-up pins the admitted home and ID; Beta
renders one input and reply. Set BOT_DM_CORRUPT=1 to add one malformed JSON
record alongside the valid admission: before the follow-up the real tick raises
JSONDecodeError; afterward the damaged record stays on disk while Beta delivers.
Fresh built native run with both root change and corruption: 2 passed (1.4m),
including the existing nested message_agent control. Receipts/screenshots:
/tmp/botmode-dm-review/{native-red2,corrupt-red,final-native} and matching .log
files. The first follow-up run also exposed a fixture mistake (the changed HOME
was not created, so --in ~ correctly refused); that failed receipt is retained
in native-green.log, and both source legs were rerun with an existing HOME.
Prior /tmp/botmode-dm-recovery* evidence remains untouched.
Per-record delivery exception isolation
Set BOT_DM_EXCEPTION=1 and select -g "cron output" for the controlled native
exception probe. exception-tick.py raises PermissionError at the actual
post-discovery target Path.is_dir() boundary, not at the delivery helper.
The same exception was first reproduced with real directory traversal permission
loss on Python 3.11/Linux; the retained test uses a portable controlled fault.
Before the guard, native tick exits 1, leaving the head claimed and sibling queued.
Afterward, the head is ambiguous, the sibling settles and renders once, and a
second real tick replays neither. Logs: /tmp/botmode-dm-exception-{red,green}.log;
receipts and screenshot: /tmp/botmode-dm-exception/{red,green}/.
The review's repeated-head starvation claim is not reachable: the claim commits before delivery and later scans skip every non-queued record. One failed tick is real; recurring replay of that same head is not. Indefinite queued/payload retention is intentional, with no TTL or automatic ambiguous retry introduced here.
Ordinary custom-root fallback (#104066 / #104055)
probe-cron-root.spec.ts adds the never-deferred sibling: copy it to
apps/desktop/e2e/ and run with the same native fixture (no mock-trigger patch
needed for this case). It keeps default's real Desktop Bot Chat lease, submits
ordinary cron output to unowned Alpha from a separate Python producer under a
custom Hermes root, and holds the real quiet CLI child at loopback inference.
The child shim PID must match Alpha's real CLI lease; default's lease is unchanged.
After release, Alpha has exactly one input and Desktop renders the output.
The same case removes unused Beta and verifies delivery neither recreates Beta nor
creates a second .hermes root under HOME.
Both origin/main's scheduler and pre-follow-up c827ae179d67c fail with
Profile 'alpha' does not exist before any recipient turn. Fixed native run:
1 passed (46.3s). Two invariant tests exercise the actual CLI startup resolver
across named/default/own destinations with a changed active profile, and refusal
when the destination is missing initially or disappears during discovery:
5 failed before, 5 passed after. The old env-clearing test is replaced by these
behavior checks rather than retaining the broken expectation.
Evidence: /tmp/botmode-cron-root/{before2,origin-main,after}.log,
after/{owners.json,children.log,rows.json,result.json,missing.json,ordinary-recipient.png}.
The first fixture attempt (before.log) used the wrong default row label; the
actual Desktop label is Hermes. No production failure is claimed for that attempt.
Full cron directory: 1346 passed, 1 skipped across 116 files; sibling mailbox,
DM, gateway consumer and profile tests: 124 passed, 3 skipped across 4 files.
Credit @fangliquanflq's #104066 for the root-boundary diagnosis and anchoring fix;
this combined branch reuses its already-resolved destination instead of repeating
name resolution. No retry or receipt semantics change.