Greptile on 0ce63424a06:
- Only the first four images of a result were prepared; a damaged or
already-in-context image among them hid a valid fifth. Images now
prepare in waves sized to the free slots (bounded at 16 candidates), so
a skip hands its slot to the next image.
- The attach check compared against the 20 MB hard cap, not
`vision.embed_target_bytes`; a best-effort resize that bottomed out
above a low budget rode history anyway. It now checks the configured
target (clamped to the hard cap).
- `tools/computer_use/vision_routing.py` had become a one-line negation
of `_native_tool_result_images`. The capture boundary calls the shared
gate directly; the module and its tautological agreement test are gone,
and the route-decision cases move into the capture-routing tests where
they drive `_should_route_through_aux_vision` itself.
Independent review follow-up:
- The shared resizer is best-effort: its 64 px short-edge floor keeps a
60000x64 image at 60000 px, and the MCP path checked only bytes, so an
over-limit part could enter history (providers reject >8000 px per
turn and the reactive shrink cannot fix it). The attach path now
refuses any image whose output long edge is still above 1568 px; the
result stays the MEDIA: text.
- A resized screenshot lost its coordinate mapping. The MCP path now
passes scale_out and lists `_build_scale_note` per MEDIA: path, as
vision_analyze does, so a cua-driver-over-MCP click lands on the screen
pixel the model saw.
- computer_use cached its route per (home, provider, model) for the
process, so editing `image_input_mode` (or the aux backend) never
reached the next capture. The cache is gone: the decision reads the
signature-cached `load_config_readonly` per capture. Pre-existing on
main (12ad13ddca), fixed here because this PR claims every image tool
honours the setting; the perf test that pinned the stale cache becomes
the "config change applies without restart" invariant.
The capture route only checked "does the provider carry tool-result
media", never `agent.image_input_mode` or a catalog verdict that the
model cannot see. deepseek-v3.2 on OpenRouter therefore went native, the
agent backstop swapped the screenshot for a "switch to a vision-capable
model" error, and the configured auxiliary vision describer never ran;
`image_input_mode: text` was ignored for screenshots too.
`tools.vision_tools._native_tool_result_images(provider, model, cfg)` is
now the one gate for every tool that hands the model pixels
(vision_analyze, browser screenshots, computer_use captures, MCP images):
image routing resolves to native AND the route accepts tool-result
images. The capture route is its negation, which drops three local
helpers that duplicated image_routing. Proxy aliases the catalog does
not know still go native under `image_input_mode: native` (#115248's
case); `auto` sends them to aux, matching vision_analyze.
Tests: test_computer_use_vision_routing.py is rewritten to drive the
real gate (the old cases patched the removed helpers). Changed
assertions: "whitelisted provider + catalog-unknown model" now requires
image_input_mode: native (it carried that cfg already); new cases pin
catalog text-only -> aux and image_input_mode: text -> aux.
kokhlo's measurement on the trimmed PR: the zero-delivery gate also catches
type_text_synthesis_budget_exceeded, whose delivered_chars is pinned to 0
because the bounded synthesis budget declined to emit at all. That target is
not dropping synthetic keystrokes, and re-routing it to set_value replaces the
driver's own chunk recommendation with a different write path and a wrong
explanation. The gate now keys on the driver's own partial-delivery verdict;
the hint also names set_value as an action rather than a delivery rung.
When a type action delivers 0 of the requested characters, the field is
dropping synthetic keystrokes at the source (trusted-event checks on web
inputs). No delivery rung can fix that; the AX set_value path writes the
value directly and bypasses event filtering, so the escalation verdict
now recommends it instead of the generic px/foreground ladder.
Trimmed from #103157 / #103174 to the unique zero-delivery delta; the
snapshot_id element addressing landed via #111313 (element_token) and
#116367 (snapshot_id carrier).
Co-authored-by: Konstantin Khlopkov <konstantin.khlopkov93@gmail.com>
Third-party drivers (e.g. a Codex-style computer-use reimplementation) could
only replace cua-driver by patching core. computer_use now follows the
single-select provider model Hermes uses for memory providers, context
engines and cron schedulers:
- plugins/computer_use/__init__.py discovers bundled
plugins/computer_use/<name>/ then user $HERMES_HOME/plugins/<name>/
(marker: register_computer_use_provider / ComputerUseProvider) and loads
ONLY the provider computer_use.backend names (default "cua"), read per
call so each profile gets its own. Unselected providers are never
imported; listing reads plugin.yaml only. An unknown/unloadable name
returns an error naming it and the installed providers; no fallback.
- The built-in cua-driver is an ordinary bundled provider at
plugins/computer_use/cua/ (driver implementation stays in
tools/computer_use). tools/computer_use/backend.py gains
ComputerUseProvider (create_backend(permission_mode=...), is_available(),
optional doctor()).
- The tool's check_fn and backend construction go through the selected
provider; the model-facing schema is unchanged for every provider
(prompt-cache invariant). The test-only HERMES_COMPUTER_USE_BACKEND env
switch is gone (tests patch _new_backend).
- General plugin discovery treats computer-use providers as exclusive
(not eagerly imported, not listed as toggleable) and skips the bundled
computer_use/ category, same as memory/, context_engine/, cron_providers/.
- hermes tools and the Desktop toolset panel list installed providers next
to cua; computer-use doctor/status/permissions and the Desktop status
payload skip cua-driver/TCC probes when another provider is selected.
Chromium debug profile, media-cache and vision/computer-use cache dirs were
created with bare makedirs inheriting the umask (0755); their files landed
0644. Under the documented HERMES_HOME_MODE=0701 traversal hatch those stores
(Cookies, Login Data, screenshots, inbound media) were readable by every
other local account. Dirs are now created owner-only (0700) with the mode
passed to mkdir (no post-chmod window) and reconciled via the house
hermes_cli.config._secure_dir / _secure_file policy helpers, which keep the
managed/NixOS group-share carve-out and HERMES_UID/HERMES_GID ownership.
Cache files are written 0600 via os.open.
Deleting a session also left its journaled in-flight turn tail (user prompt
+ tool calls) in localStorage to age out on its own; purgeInFlightTurnJournals
now runs on both delete surfaces (sidebar removeSession and Settings
permanent delete), draining every id the delete holds.
Refs #77486
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
* feat(docker): publish nousresearch/hermes-sandbox:desktop for terminal backends
The terminal backends (docker, modal, daytona, singularity) all default to
nikolaik/python-nodejs:python3.11-nodejs20, a bare Python+Node base. For Bot
Screen, computer_use and the browser to run INSIDE that sandbox instead of on
the gateway host, the sandbox image needs the display stack.
docker/sandbox-desktop.Dockerfile is that base plus:
- the everyday tools it lacked (jq, ripgrep, fd, tmux, less, nano, vim,
zip, rsync, tree, procps, htop, sudo for the base's uid-1000 `pn`)
- the exact package set the Hermes -desktop image installs (TigerVNC,
Xfce components, dbus, xauth, fonts)
- Playwright's headed Chromium (same build as the -desktop image)
- cua-driver 0.28.2 from its pinned release tarball
No Hermes inside; the default user stays root like the base so nothing
changes for people who just switch docker_image. Desktop processes run as
`pn`. 4.27 GB on amd64.
docker.yml gains a `sandbox` variant with its own cache scope and repository
(nousresearch/hermes-sandbox:desktop, :main-desktop, :<release>-desktop);
the docker-integration suite is skipped for it (no Hermes to test) and
docker/sandbox-desktop-smoke.sh runs instead: as `pn`, every launcher and
cua-driver binary resolves, the real launcher.sh publishes :20, the RFB
socket completes the 3.8 handshake relayed over `docker exec -i` stdio, and
a headed Chromium maps a window on that display. hadolint lints the new
Dockerfile in docker-lint.yml.
* feat(docker): sandbox desktop base on python3.13-nodejs26
Matches the Hermes image (Python 3.13 / Node 26) and the top of requires-python;
the default docker_image tag it inherited was Python 3.11 / Node 20. Same pn
uid 1000, Debian 13; smoke (launcher, RFB relay, headed Chromium) passes.
* feat(docker): bake agent-browser into hermes-sandbox:desktop
The browser tools drive the agent-browser CLI; when the browser follows the
terminal backend that CLI has to exist inside the sandbox. Pinned to the same
^0.26.0 range the gateway resolves, --ignore-scripts like the gateway's npx path.
* feat(bot_desktop): the screen, computer_use and the browser follow the terminal backend
A user who sandboxes `terminal` (docker/ssh/singularity) had the agent's
screen, cua-driver and Chromium running on the gateway HOST beside that
sandbox: Bot Screen gave a headless host a display, the Xfce panel carries
xfce4-terminal, and `computer_use` could open a shell outside the boundary
the sandbox exists for.
Now the desktop lives where the terminal lives:
- tools/environments/streams.py: one primitive per spawn-per-call backend, the
local argv prefix that runs its remainder inside the sandbox with stdio open
(`docker exec -i`, `ssh`, `apptainer exec`). SDK backends (modal, daytona,
vercel) have none and report so.
- tools/bot_desktop/sandbox_host.py: launcher.sh runs inside the sandbox as
the image's `pn`; the pane's RFB bytes ride a 12-line python relay over that
prefix; `cua-driver mcp` is the prefix + the sandbox image's own driver.
- tools/bot_desktop/placement.py + `bot_desktop.placement` (auto|terminal|
gateway). `auto` follows the backend; a sandbox that cannot host a screen
REFUSES with the opt-in named instead of silently using the host.
- runtime.start/stop/status/published_env branch on placement; the pane,
lease, epoch fencing and CLI are unchanged.
- cua_backend: the MCP invocation is the sandbox one when the screen is
there; the host driver's runtime contract is irrelevant then; check_fn is
true under a terminal placement without a host binary.
- browser_tool_session: agent-browser invocations are wrapped in the prefix
with the daemon, socket dir and profile inside the sandbox; screenshots are
fetched back so MEDIA: paths keep working; recycle closes the sandbox
daemon.
- web_routers/display.py: the bridge pumps a relay's stdio when the screen is
in a sandbox, a unix socket otherwise.
Live on docker with nousresearch/hermes-sandbox:desktop: start/observe/RFB
handshake through the dashboard bridge, human takeover fences the agent
(HumanHasControl) and keystrokes reach the sandbox Xvnc, handback restores,
three start/stop rounds leave zero desktop processes; computer_use capture
and list_windows see only the sandbox's Xfce; browser_navigate/snapshot/
vision run with Chromium and agent-browser inside the container and zero
host processes on the bot profile; modal + auto refuses naming the opt-in.
* feat(desktop): Screen pane shows where a sandbox-placed screen runs; Install is host-only
DesktopStatus gains placement ('gateway' | 'terminal:<backend>'). A sandbox
image lacking the stack is a blocker naming hermes-sandbox:desktop, shown in
place of Start; install_command stays None there because the pane's Install
button runs the package manager on the gateway host, the wrong machine, and
display.install refuses for the same reason. The pane header carries
'Screen runs inside the docker sandbox, with the terminal' (4 locales).
* fix(bot_desktop): "is the screen in the sandbox" is a disk check on hot paths, never a config read
Every browser command and CUA spawn asked in_sandbox(), which resolves placement by
loading config, which initializes HERMES_HOME. Under a test's fake home that raised
HomeInitializationError from _run_browser_command; on a real host it read config per
click. Hot paths now ask sandbox_screen_running(): the start marker on disk, written
only by a sandbox start. Policy (in_sandbox) stays for start/install, where config is
the question. display.observe gates on "an RFB endpoint exists" for either placement.
* test(moa): late-accounting sink test asserts the wedged slot's row, not sink order
Under CI load the poll loop can see the interrupt before collecting the fast slot, so the
fast slot also arrives late and first; the test then failed on sink_calls[0]. The
contract is that the wedged slot's real usage reaches the sink.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* fix(bot_desktop): a sandbox that died under a live screen fails loudly, never falls to the host
sandbox_screen_running() drops a start marker whose terminal environment is no longer
registered (stale after a process restart). When the environment object outlives its
container, the browser's sandbox wrap now checks the published DISPLAY and raises
"the screen inside the terminal backend's sandbox is gone; start it again" instead of
KeyError('AGENT_BROWSER_PROFILE'). Live: fresh sandbox navigate ok; docker rm -f the
container; next navigate returns that error; zero host Chromium either way.
* feat(terminal): nousresearch/hermes-sandbox:desktop is the default container sandbox
Every container backend (docker, modal, daytona, singularity) now defaults to the
sandbox image with the desktop stack, so Bot Screen, computer_use and the browser
run inside the sandbox for everyone who never chose an image; Python 3.13 / Node 26
match the Hermes image. One constant (DEFAULT_SANDBOX_IMAGE) replaces six copies of
the old literal. Migration 47 moves saved configs still holding the OLD default and
never touches an image the user pinned. Docker reuse recreates a container built
from another image, or the flip would silently never take effect for anyone with a
persisted container (live: old container removed, new one on 3.13 / Node 26 with
Xvnc, cua-driver, agent-browser present).
* chore: retrigger CI (zero-job dispatch failure)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore(config): template stamps v47 and shows the new default sandbox image
The template is what install.sh / docker / doctor --fix seed; a stamp behind
DEFAULT_CONFIG makes every fresh install migrate on first run.
* feat(sandbox): the default image change is a decision, not a surprise
A persisted Docker sandbox on another image is kept when docker_image is unset;
only a written docker_image (an explicit pin) recreates it. The pin verdict
travels as TERMINAL_DOCKER_IMAGE_PINNED through both terminal bridges (process
env and per-profile scope) and the container-config allowlist.
Approval surfaces, all through hermes_cli.sandbox_image_switch: the interactive
CLI asks once at startup (y = pin the new image, n = pin the current one, Enter =
ask later); the Screen pane shows the same choice with Switch / Keep buttons via
display.switchSandboxImage; `hermes config set terminal.docker_image …` is the
same answer from any shell. Gateways and cron never decide: they keep the sandbox
and log the notice.
Migration 47 now unsets a saved image equal to the OLD default instead of
rewriting it to the new one — that value was the template copied, not a pin, and
rewriting it would have made the runtime recreate existing sandboxes unasked.
Modal restores its snapshot and Daytona reuses its labeled sandbox regardless of
the configured image, so existing sandboxes there were already untouched.
* fix(config): both plain defaults that preceded the desktop sandbox image are template copies
main pinned nikolaik/python-nodejs:python3.14-nodejs22 (cd0f97f833) without a migration;
a saved config holding either literal is unset by migration 47, so it follows the default
and existing sandboxes get the keep-or-switch decision instead of a silent recreate.
* ci(docker): build the sandbox image on release/dispatch, not every main push
Leaves docker.yml exactly as on main. The sandbox image carries no Hermes code,
so two 4 GB multi-arch builds per merge bought nothing. sandbox-image.yml builds
and smokes on a PR that edits its own Dockerfile/smoke, and publishes only on a
release or a manual dispatch with publish=true. Stable tag stays :desktop.
* fix(config): keep main's config.py/config_defaults.py edits under the sandbox-image delta
The rebase resolved both files wholesale with the branch side, dropping main's move to
hermes_yaml (the 3.14 runtime venv has no PyYAML) and the 3.14 base pin. This is main's
version plus exactly the branch's own changes: DEFAULT_SANDBOX_IMAGE, the pin verdict in the
env bridge, placement defaults and the v47 stamp.
* test: sandbox-image tests read config.yaml through hermes_yaml (no PyYAML on the 3.14 runtime)
* fix(bot_desktop): read the sandbox marker BOM-tolerantly (windows footgun lint)
* fix(bot_desktop): placement is the authority; sandbox screen survives restarts
Review findings on the sandbox-hosted Bot Screen, each reproduced live first.
Authority. The browser preflight and the CUA invocation keyed off screen
LIVENESS, so `placement: terminal` with the screen not yet up handed the tool an
unchanged host command. `runtime.tool_placement()` is now the one resolver:
terminal placement starts the sandbox screen on demand (no auto_start opt-in
inside the user's own sandbox), refused placement raises its reason, and neither
ever yields the host. placement.resolve() answers a local backend from env alone
so the common case costs no config load on the spawn path.
Restart. sandbox_screen_running() deleted the marker whenever the process-local
terminal registry was empty, i.e. after every gateway restart, while Xvnc kept
running in the container; stop() then returned False and left it. The marker
now records the owning container; liveness comes from `docker inspect` on it,
stop/status re-attach to the recorded owner (even after the placement setting
moved), and only a container that is gone drops the marker.
SSH. remote_argv emitted `bash -c <script>` as three words; OpenSSH joins them
and the remote login shell ran `bash -c export` and the rest itself. The script
travels as one quoted word for ssh (remote_command knows the backend); docker
and apptainer keep argv.
CDP reach. agent-browser inside the sandbox reports the sandbox's loopback;
the Browser Use harness, browser_exec and the vault supervisor connect from the
host and got connection refused. streams.forward_port() proxies a local port
over the exec stream (same relay as the RFB bridge) and the CDP URL is rewritten
to the local end.
pids limit. --pids-limit 256 counts threads; measured on the desktop image the
desktop stack is 44, one Chromium tab 212, the agent's browser with two tabs
488. Past the cap every further docker exec died with "procReady not received".
Default is 2048 with the measurements in the comment.
Replacement. An approved image switch force-removed the old container before
`docker run` tried the new image; a private tag or registry outage left nothing.
The image is inspected/pulled first and the old container kept on failure.
Desktop integration. The sandbox start never passed the dock's browser launcher
(no Browser icon) and the thumbnail needed a host launcher pid + host ImageGrab
(always None). The dock runs the sandbox's Playwright Chromium on the shared
profile; the thumbnail is grabbed inside the sandbox (Pillow baked into the
image). The browser profile moves from /tmp — a 512 MB tmpfs emptied on every
container stop — to the desktop user's home, so logins follow the container.
Pin provenance. A TERMINAL_DOCKER_IMAGE written in a routed profile's .env is a
pin even when it spells the default; the scope compared values before.
* docs(bot-screen): no literal tmp path in the profile-location note
* fix(bot_desktop): docker inspect liveness probe closes stdin (TUI subprocess guard)
* fix(bot_desktop): adopting a screen the sandbox kept records the marker
Live ssh probe: after the host's state was lost while the sandbox kept its
Xvnc, start() took the idempotent early return (display already published)
and never wrote the host marker, so status/thumbnail/stop lost the screen.
Record the adopted display like a fresh launch.
Docs: what an ssh host of your own must carry, and why a Dockerfile ENV is
not enough for a login session (PLAYWRIGHT_BROWSERS_PATH via /etc/environment).
* docker(sandbox-desktop): login sessions find the browser (PLAYWRIGHT_BROWSERS_PATH via /etc/environment)
* docs(bot-screen): what the Apptainer path inherits from the image and what it does not
* chore(config): sandbox-image migration is 47→48 (main took 47 for compression.threshold_tokens)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
- tools/browser_tool_install.py: keep pm-clean's frozen old-updater stub; main's
UTF-8 decode fix touched only the npx prefetch body it replaces.
- tests/hermes_cli/test_update_scoped_reconciliation.py: keep pm-clean's test
subset (catch-up rides the PM completion owner) and take main's gateway-less
host evidence (#120740): the updated seed that holds the host at a running
gateway, and the two gateway-less matrices for the source change that merged
cleanly into update_cmd_fleet.py.
The Desktop serve backend polls /api/fs/default-cwd, whose git-branch probe
captured git output with text=True and no encoding. subprocess then decodes
with locale.getencoding() - cp936 on zh-CN Windows - so a UTF-8 branch name or
git's localized stderr raised UnicodeDecodeError inside communicate()'s
_readerthread on every poll ([gateway-crash] ... 'gbk' codec can't decode).
Pin encoding='utf-8', errors='replace' on the remaining in-process captures
whose children emit UTF-8: the git branch probe, the self-repo guard's git
alias read, the Bot Mode DM transport (a Hermes CLI child, stdio forced to
UTF-8; builds on the salvaged errors='replace'), the agent-browser npx probe,
the lightpanda help probe, and the cua-driver stderr drain thread.
Fixes#83851
b5a9f44829 switched the shared computer_use config reader to
load_config_readonly. That reader also serves the daemon-launch paths
(overlay flag, Wayland child env), and tests/computer_use pins
hermes_cli.config.load_config as the seam at 13 sites; the deepcopy it
saves is noise next to an 80-500 ms capture. The ledger half of that
commit stays.
PROOF: tests/computer_use/test_cua_wayland_env.py and
test_cua_no_overlay.py::test_linux_x11_explicit_false_overrides_auto_detect
red on the previous head, green now (macOS-only
test_serve_process_disables_overlay fails identically on origin/main).
Drop the `_ax_max_elements_sent` side channel, its `getattr` default and the
`contextlib.suppress(Exception)` around the lazy `cua_backend` import:
`_cua_configured_ax_max_elements()` already swallows config errors, and the
same lazy import is bare in `_resolve_capture_windows`. `capture()` now reads
`_gws_args().get("max_elements", 0)` directly (one more cached
`load_config_readonly` hit); the vision lane still reports 0.
PROOF: tests/tools/test_computer_use_ax_walk_bound.py 5 passed; hardcoding
`ax_max_elements=0` in capture() turns `stub.capture("ax").ax_max_elements == 350`
red (observed), restored green. ruff clean; check-windows-footguns clean; real import ok.
tool.py inferred 'walk capped' by re-reading cua's private config through
_cua_configured_ax_max_elements() — a leaky abstraction that also cost a
config load per capture. The backend now records the max_elements it put on
the get_window_state call and returns it on CaptureResult.ax_max_elements
(defaulted field, 0 = unbounded / vision); _capture_view compares
len(cap.elements) >= cap.ax_max_elements > 0 and the getattr guards go
since the field is always set. Hint text and line order unchanged.
PROOF: tests/tools/test_computer_use_ax_walk_bound.py 5 passed (hint tests
now build CaptureResult(ax_max_elements=bound)); the new
_ax_max_elements_sent assertion is red with cua_backend_capture.py
reverted to HEAD~1 ('AttributeError: ... no attribute _ax_max_elements_sent',
1 failed 4 passed). Probe: 200/200 -> capped hint + 'full ' dropped;
199/200 and 8/0 -> full tree promised. check-windows-footguns --all clean.
_max_ledger_bytes() runs on every ledger append (right after ledger_enabled()
already deep-copied the config) and _computer_use_cfg() now runs twice per
computer-use capture; neither mutates the result, so use
load_config_readonly() as hermes_cli/config.py documents for read-only hot
paths (skips the deepcopy, ~half the cache-hit cost). Tests that patched
load_config now patch load_config_readonly as well (test-only edit).
PROOF: real import under PYTHONSAFEPATH shows load_config() ==
load_config_readonly() (same dict shape; defaults 5242880 / 200 resolve);
tests/tools/test_skill_ledger.py + test_computer_use_ax_walk_bound.py +
test_skill_ledger_delta.py -> 39 passed, 0 failed.
`_capture_view`'s hint promised a "full element tree saved to elements_file",
but once `computer_use.ax_max_elements` bounds the driver's walk the spill
holds only the first N nodes. When the capture returned at least the bound
(bound > 0), drop "full" and add "accessibility walk capped at N elements;
pass app= to narrow" so the model knows how to reach the rest.
`capture()` sent `get_window_state` with only `pid`, `window_id` and `session`, so the driver
walked the target's entire accessibility tree before Hermes trimmed the surfaced element list
to `_DEFAULT_MAX_ELEMENTS` (100) and spilled the rest to a cache file. Every node past the
first ~100 was paid for and discarded.
Measured on macOS (cua-driver 0.28.2, M-series), same window, bound fixed at 200:
| Target | Unbounded walk | Bounded |
|---|---|---|
| 1,444-node Chrome window | 540 ms | 83 ms |
| 456-node Finder window (pathologically slow AX surface) | 6.9 s | 0.6 s |
The bound is lossless for the response: the bounded element list is a *prefix* of the
unbounded walk (checked by role/label/depth at 200/400/600/1000), so nothing the model sees
changes.
The bound is internal and config-driven — `computer_use.ax_max_elements`, default 200,
`0` disables (driver default) — and deliberately NOT a model-facing schema parameter:
`max_elements` was removed from the tool schema on purpose, so a capture's surfaced window
stays at its fixed default.
(cherry picked from commit 9f07ff428bd1fcfbcc0e5089d834e1f33d8d1b76)
The first cut appended the recovery only in `_tcc_row`, which is built by the
0.10 fallback probes. Drivers that serve health_report (0.22+, the ones a
stale row actually bites) render their own tcc_* rows untouched, so `doctor`
never showed it. Apply the hint at the report seam like the display-count
guard so both paths get it.
Also: one CUA_DRIVER_BUNDLE_ID (permissions.py is the leaf; the daemon
imports it), the CLI derives the field set from the same table, and the
unverified "--upgrade repairs stale rows" claim is dropped from hint + docs
(the user hitting this is already on a current driver).
`hermes computer-use permissions status` and the doctor's fallback `tcc_*`
rows told a user whose System Settings toggle already showed CuaDriver ON to
"grant it in System Settings" — the one step that cannot help. macOS keys the
TCC row to the app's code-signing requirement; a row written for an earlier
CuaDriver build stops matching after a driver update and flipping the toggle
does not rewrite it (trycua/cua#3170, repaired by the cua-driver >= 0.22
installer on update). The daemon then reports Accessibility / Screen Recording
false while the pane shows ON, and every click silently no-ops (#99732).
`permissions.stale_tcc_grant_hint(*missing)` renders the reset for exactly
the missing services (`tccutil reset Accessibility|ScreenCapture
com.trycua.driver`, then `hermes computer-use permissions grant`); both
surfaces append it only when a grant is actually reported False. Docs: the
computer-use page still said bounded/unrestricted daemons run "under the
Hermes host identity" — they launch through CuaDriver.app since #95381 — and
gains the stale-row troubleshooting entry.
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
A hand-written Linux daemon unit (systemd user unit / XDG autostart entry
running `cua-driver serve`) can be dead for days — crash loop, stopped, never
started — while `hermes computer-use doctor` and `status` report a healthy
binary: the runtime contract only checks the binary (`manifest`), and nothing
ever connected to the daemon socket (#114748).
- tools/computer_use/cua_backend.py::cua_daemon_listening — socket-level
liveness via `cua-driver status [--socket PATH]` (rc 0 = a daemon answered,
"not running" = dead, anything else = unknown). Never raises.
- tools/computer_use/doctor.py::cua_daemon_units — one scan of the units that
run cua-driver (kind, unit, exec target, runs `serve`, `--socket` path with
`%h` expanded); the pruned-Exec guard now filters that list instead of
re-scanning.
- doctor.py::_apply_daemon_liveness_guard — per `serve` unit: `pass` when its
socket answers, `fail` (degrading `ok`) when not, with the hint that a driver
reinstall does not start the daemon. Unconfigured daemons are never probed:
on Linux the MCP runtime needs none, so a silent default socket is normal.
- `hermes computer-use status` prints the dead-daemon line and exits 1.
- docs: the Linux daemon-unit paragraph under the doctor section.
Not changed: _maybe_repair_runtime_contract. The repair is gated on the
binary-level contract only; a dead daemon never enters it, so a reinstall was
never triggered by the daemon (the `.release_installed/<version>` marker the
reporter saw is written by cua-driver itself on any first run of the binary —
observed via strace of `cua-driver status` under a fresh HOME).
Supersedes #114928 (@Finn763): same finding, ~600-line implementation with a
new module and a repair-gate rewrite; this is the ~90-line version on the
existing doctor seams.
systemd --user and XDG autostart honour $XDG_CONFIG_HOME; scanning only
~/.config left hosts that set it with the same silent green doctor the PR
removes. One invariant test, red before.
Linux has no managed cua-driver autostart, so daemon units are hand-written
against a concrete release directory. The installer prunes all but the last
five release dirs, so a versioned Exec reference crash-loops with 203/EXEC
after every upgrade while every binary-level check stays green (#114748).
doctor now scans systemd user units and XDG autostart entries for cua-driver
Exec targets under packages/releases/<version>/ that no longer exist and
appends a fail check with the packages/current recovery hint, downgrading an
ok overall to degraded (#114748).
After the shared _accepts_tool_result_images gate the capture route still
demanded _lookup_supports_vision(...) is True, so a whitelisted provider with a
catalog-unknown model (proxy alias) went native in vision_analyze and aux in
computer_use. The route is now the negation of the shared gate; the redundant
catalog lookup helpers are gone.
The capture route (`tools/computer_use/vision_routing.py`) opened the native
multimodal lane only for providers on `_TOOL_RESULT_MEDIA_PROVIDERS` (or a
profile `supports_vision` flag), while the `vision_analyze` fast path also
accepted a catalog vision hit. For deepseek/deepseek-flash the two gates
returned opposite verdicts, so a screenshot paid a same-model auxiliary round
trip and the main model acted on prose instead of pixels, while vision_analyze
embedded the image natively (#115248).
`tools.vision_tools._accepts_tool_result_images(provider, model, cfg)` is now
the single predicate (profile veto first, then provider tool-result media OR
capability lookup) and both lanes call it. The lookup keeps the full
`_lookup_supports_vision` order (config override -> catalog -> local probes ->
profile), so Ollama/local VLMs are unaffected.
Supersedes #115276 (@Tranquil-Flow), whose direction this follows; its
models.dev-only lookup in the fast path would have dropped the local-runtime
and Ollama probes.
hermes_cli.runtime_paths (venv generations, selection, activation) moves to
pm.environments, and gains venv_bin_dir / venv_python / project_python. Every
in-tree caller asks pm for an interpreter now; pm no longer reaches back into
hermes_cli for its own environment layout (pm.packages, pm.extras, pm.ensure,
pm.paths imported hermes_cli.runtime_paths). The three open-coded
"Scripts/python.exe or bin/python" ladders in pm collapse onto venv_python.
hermes_constants.venv_python_path / venv_bin_dir and hermes_cli.runtime_paths
stay as frozen-updater-surface shims only (tests/compat/old_updater_surface.json).
To keep the boot path light, pm/__init__ resolves its facade lazily (PEP 562)
and pm.registry loads the built-in package definitions on first read instead of
at import: `import hermes_bootstrap` now loads pm + pm.environments only (25ms,
was 37ms with the eager facade dragging in the downloader). The stripped-payload
fixtures that ship only pre-import files keep working for the same reason.
Also restores two frozen-surface re-exports the F401 sweep dropped
(banner._github_compare_behind, cua_backend.resolve_cua_driver_cmd).
- 15 `MERGE-CHECK:` conflict-resolution comments removed from prod code (two were
TODOs already done: the utf-8-sig sessions.json read lives in session_persistence,
the pm-aware cron script helpers in scheduler_script).
- 49 imports the branch left unused (ruff F401, none present at the merge base,
none inside PLUGIN-COMPAT blocks). update_cmd's frozen-surface re-exports are
trimmed to the names tests/compat/old_updater_surface.json actually lists under
hermes_cli.update_cmd; the rest resolve through hermes_cli.main.__getattr__.
- tools/environments/local_gitbash_probe.py: nothing imported it once _find_bash
delegated to pm.shell().
- Three try/except wrappers around calls that cannot raise (install_truststore,
get_hermes_home, and a duplicated except clause in supermemory).
Extract only backend admission/lifetime protection from #114565 (484b5d7), adapted to the existing DISPLAY identity and human-lease fences in #108914. No WSL selection, installer, config or UI changes.
Reconcile plugin declarations and validation through PM's atomic generation publication; preserve external runtimes, target markers, and conflict refusal. Keep one source-update completion owner and port upstream lifecycle changes to the PM desktop/runtime paths.
Conflicts (all keep-both): tools/browser_tool.py takes main's scope-bound
passthrough + loopback NO_PROXY and still routes the env through the Bot
Desktop's desktop_env; i18n gains main's sudoDesc/sudoCommandUnavailable
beside our sudoInstallDesc; test_profiles keeps both sides' new tests.
Main replaced every *.request/*.respond event pair with real JSON-RPC server→client requests and
made Python the single source of the wire (tui_gateway/contracts → generated TS). Bot Screen now
speaks that dialect:
- tui_gateway/contracts/display.py declares the eight display.* methods, the display.install.sudo
server request and the four display.* events; the generated TS/OpenRPC is regenerated.
- The install sudo card is a `display.install.sudo` server request (app-level, empty session);
the desktop answers it through respondToServerRequest, so the respondMethod/origin plumbing
that pinned a reply to its socket is gone: a JSON-RPC response cannot land elsewhere.
- screen-connection.ts consumes the generated DisplayStatus/DisplayLease instead of hand-typed
copies (holder named by viewer_hash only; epoch always present).
- computer_use: the lease fence and main's screenshot-dedup session key ride the same read
handlers; the fence fires before a frame reaches the dedup cache.
- server.py keeps the display module registration a naive --theirs would have dropped.
On Windows the Hermes venv interpreter cannot CreateProcess a binary under
C:\Program Files\WindowsApps (WinError 5) even though the shell resolves it,
so `hermes computer-use doctor` died with a raw PermissionError traceback
from _open_mcp. Catch the spawn OSError and print what failed, why the tool
may still work (PATH resolves another copy), and the fix (reinstall outside
WindowsApps or HERMES_CUA_DRIVER_CMD), exit 2.
cua-driver 0.21 refuses a bare element_index:
click: bare element_index is not accepted; pass element_token,
or snapshot_id together with element_index
_maybe_attach_element_token gated solely on the trycua/cua#1961 capability
vocabulary. 0.21 stopped publishing per-tool capability sets — every tool
reports an empty set — while still accepting element_token in its input
schema. The gate therefore fails closed on 0.21.x and we send the bare
index, so the driver rejects the call.
The effect is total: every element-targeted click is refused, leaving
agents with only blind pixel coordinates. Observed against cua-driver
0.21.0 on X11, where a capture returned 762 elements with 762 tokens
cached and every subsequent click still failed with snapshot_id_required.
Check the live input schema first — supports_input_property() already
exists for exactly this, and its docstring notes it "deliberately inspects
tools/list rather than ... requiring a capability token the driver never
shipped". The capability check is retained as a fallback so drivers that
did ship the vocabulary are unaffected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The "screen unchanged" result points the model at its previous capture. After
context compression that capture may be summarized away, so the note would refer
to pixels no longer in context. Mirror read_file's reset_file_dedup: the
compaction boundary (both the summary path and the codex app-server path) now
clears the session's screenshot digest, and the first capture afterwards delivers
the image again even when the screen is byte-identical.
Key the screenshot-dedup state by the same profile-scoped session id the
backend cache uses, so two multiplexed profiles sharing a session id (or a
DISPLAY) never dedup against each other's frames, and forget the state in
release_computer_use_session so a re-created session's first capture
always delivers pixels. Reword the unchanged note to cover the aux-vision
path (where the prior result was an analysis, not an image). Tests: the
dispatch path (explicit capture + capture_after) honours the streak cap;
release forgets state.
Port from openclaw/openclaw#129924: a capture whose pixels are
byte-identical to the previous capture of the same target in the same
session returns its full text metadata (element index included) plus an
explicit 'screen unchanged' note instead of the multimodal image block.
Adapted for Hermes: openclaw gates dedup on per-frame context-epoch
tracking; Hermes bounds staleness with a consecutive-omission streak cap
(2) so full pixels are re-delivered before compaction could evict the
referenced image. Dedup state is per-session (no cross-session leaks),
append-only (no history rewrites — prompt cache prefixes untouched),
and skipped entirely when no session_id is present.
