Third-party drivers (e.g. a Codex-style computer-use reimplementation) could
only replace cua-driver by patching core. computer_use now follows the
single-select provider model Hermes uses for memory providers, context
engines and cron schedulers:
- plugins/computer_use/__init__.py discovers bundled
plugins/computer_use/<name>/ then user $HERMES_HOME/plugins/<name>/
(marker: register_computer_use_provider / ComputerUseProvider) and loads
ONLY the provider computer_use.backend names (default "cua"), read per
call so each profile gets its own. Unselected providers are never
imported; listing reads plugin.yaml only. An unknown/unloadable name
returns an error naming it and the installed providers; no fallback.
- The built-in cua-driver is an ordinary bundled provider at
plugins/computer_use/cua/ (driver implementation stays in
tools/computer_use). tools/computer_use/backend.py gains
ComputerUseProvider (create_backend(permission_mode=...), is_available(),
optional doctor()).
- The tool's check_fn and backend construction go through the selected
provider; the model-facing schema is unchanged for every provider
(prompt-cache invariant). The test-only HERMES_COMPUTER_USE_BACKEND env
switch is gone (tests patch _new_backend).
- General plugin discovery treats computer-use providers as exclusive
(not eagerly imported, not listed as toggleable) and skips the bundled
computer_use/ category, same as memory/, context_engine/, cron_providers/.
- hermes tools and the Desktop toolset panel list installed providers next
to cua; computer-use doctor/status/permissions and the Desktop status
payload skip cua-driver/TCC probes when another provider is selected.
- test_code_kernel::TestModelFacingReset: reset stays in the schema under a stale
kernel_mode and a model-shaped reset call really discards kernel state (#96787).
- test_computer_use::TestStartupTimeoutPhaseDetail: cua-driver ready timeout names
the wedged phase + doctor hint, and the session stays un-started (#57025/#69372).
- test_file_operations::TestEscapeNativeToolArg (windows_only, was FAKE_OS): rg
search and the node --check linter get native C:/ paths, not /c/ (#84303).
- test_file_tools::TestSSHConfigWriteGate (2): a ~/.ssh/config write goes through
the real approval gate, BLOCKED with nobody present / in -q mode, nothing written (#93201).
- test_lazy_deps: a newer compatible plugin SDK is satisfied, never re-pinned down
(#98407, #86992); windows_only real Matrix probe reports unsupported.
- test_mcp_empty_error_message: empty/whitespace exception messages still yield a
diagnostic (#19417).
- test_mcp_stdio_children_dead: the fast-fail probe never calls the watcher on the
plain path, no un-awaited coroutine (#96044).
- test_subprocess_utf8_encoding: real _op_whoami survives invalid UTF-8 child output (#53428/#55339).
- test_tts_output_timestamp: same-second TTS calls get distinct output files (#43911).
- test_windows_agent_loop_papercuts::TestAutocompleteDevicePaths: a relpath
ValueError on a device path is skipped, not raised (#42016).
- test_windows_native_support::TestCronSchedulerBashResolution (2): cron .sh uses
bash from PATH; missing bash is an actionable error, not WinError 2.
Change-detectors, tautologies, source-reading tests, redundant duplicates,
mock-echo tests and dead/unrunnable tests. Per-test rationale in the lane
ledger (category + reason for every removal).
Extract only backend admission/lifetime protection from #114565 (484b5d7), adapted to the existing DISPLAY identity and human-lease fences in #108914. No WSL selection, installer, config or UI changes.
The compressor pass and the Anthropic wire pass each carried their own copy of the
limit/batch/floor constants (under different names) and of the retire-count loop.
Retuning one alone would make the wire pass re-evict on a second frontier after the
compressor pass — the exact prefix-rewrite class #113517 fixes.
- agent/image_eviction_policy.py: stdlib-only leaf holding the constants and
outbound_image_retire_count(); the wire converter stays a leaf (no compaction import).
- evict_stale_outbound_tool_images loses `keep_newest`: no caller passed it, and its
default was the COMPACTION window constant the comments say the send path must not use.
- _retire_stale_tool_result_images (compaction) back to its main body; behaviour was
unchanged, so the rewrite was churn.
- One `_tool_result_parts` unwrap instead of three copies; one `_image_payload` walk
per message instead of two.
- Wire pass counts blocks via `_block_type` (non-dict inner blocks no longer raise).
Tests import the constants from the policy module; the two "stops at the floor" tests
assert the invariant (newest floor survives, request under the limit) instead of the
exact frontier. test_user_uploads_are_not_evicted now builds a request that actually
trips eviction — at 6 blocks it never fired and proved nothing.
Both eviction passes retire the oldest image-bearing tool results in whole
batches (8) and let the floor shelter a breach only when reserved uploads alone
exceed the ceiling. When uploads fill most of the ceiling without exceeding it,
the first batch step overshoots past the newest frames: fifteen uploads plus six
one-frame tool results retired all six (the model blind on the frame it had just
asked for) although keeping the newest three already clears the 20-block limit.
The batch step is now cut back to `total - floor` when it would cross the floor
and keeping the floor fits; a floor that does not fit (one 25-block tool_result,
byte pressure) still retires past it, and uploads alone over the ceiling still
shelter the floor. Pure-screenshot sessions are unchanged: same batch-aligned
frontier at 21/29/37 images.
Review-fix on #113519 (found while checking @ehz0ah's mixed-session boundary on
the sibling PR #113619).
267d42b3 made the compressor pass shelter the keep-newest floor only when
retiring every tool image still cannot bring the request inside the ceiling.
The Anthropic wire pass (_evict_old_screenshots) kept the unconditional
`len(carriers) - _MAX_KEEP_SCREENSHOTS` floor, so a single tool_result
carrying 25 image blocks retired nothing there. The compressor pass masks it
on the main loop, but auxiliary_wire.py reaches convert_messages_to_anthropic
without that pass, so the wire pass has to hold the invariant on its own.
Mirror the rule: when reserved uploads alone fit the ceiling the breach is
fixable and every carrier is retireable; only reserved uploads over the
ceiling keep the floor. One invariant test, red on 267d42b3.
Parallel tool calls land as sibling tool_result blocks inside one user
message, oldest first. The outer walk is newest-first but the inner walk
was forward, so a batch boundary that fell inside a message retired that
message's NEWEST frames instead of its oldest (21 siblings in one message:
survivors were t0..t12, the frames the model was just asked about were
the ones removed). Reverse the inner walk so the carrier list honours the
newest-first contract the batch tail relies on.
Same defect and fix as #103217 (credit @Maero47).
Review fix on top of the limit-triggered eviction. Both send-path stages ran
statelessly on a fresh clone every request, and each retired a FIXED single
batch (`min(batch, count - keep)`) once over the limit. That caps total
eviction at 8 per stage forever: once a session passes 20 + 8 (+ 8 from the
adapter) images, the outbound request grows unbounded again (24 image blocks
at 40 vision calls, 44 at 60 in the issue's own replay), so the documented
20-image threshold and the 24 MB byte budget were not enforced above the
first window. The PR's 40/60-row "3 collapses" result was this saturation,
not a held frontier under the limit.
Retire `ceil(overshoot / batch) * batch` instead, extending by whole batches
while the surviving payload still exceeds the byte budget. The frontier is
still a step function of the image count (one move per batch), and the
request now stays at or below the limit at every size:
images collapses max outbound images (PR head -> fixed)
40 3 -> 4 24 -> 20
60 3 -> 6 44 -> 20
2 MB x 30 3 -> 4 40 MB -> 22 MB
Both frontier tests now span three batch windows and assert the outbound
count never exceeds the limit; both are red on the previous head.
Both send-path image evictions retired the oldest payload as soon as more than
three images were present. Retiring an image edits a message the provider has
already cached, and Anthropic matches its prompt cache on an exact byte prefix,
so each retirement re-wrote the entire conversation. Because the window was a
count, one more message retired on every new image and nearly every turn became
a full-prefix miss.
Observed on a real 104-request session: requests reported cache_read=16,076 --
system prompt plus tool schemas and nothing else -- with cache_write up to
127,720, against a steady-state read of ~145,000. Those requests produced 62
percent of the session's entire cache-write volume.
Trigger eviction on the API's own per-request image and byte limits instead, and
retire a batch when it fires. Under the limit nothing is rewritten and the
request is append-only, so the cached prefix survives; over it, the cost is one
slower turn per batch rather than one per image. This matches the documented
behaviour of the reference Anthropic client.
Simulated across session sizes, prefix collapses fall from 3/8/18/38/58 to
1/1/1/3/3 at 5/10/20/40/60 vision calls.
Compaction keeps the count-based window: it commits its rewrite into the
canonical transcript once and has no prefix to preserve.
computer_use kept its own approval decision: two module dicts
(_session_auto_approve / _always_allow) mirroring tools.approval's
session store and _persist_choice, a private verdict vocabulary
(approve_once/approve_session/always_approve) that hermes_cli mapped
back to once/session/always, and — the real problem — `if
_approval_callback is None: return None`. Only the interactive CLI ever
installed that callback, so every other host (gateway turns, cron,
api_server, tui_gateway, ACP) ran destructive desktop input with no
approval at all, ignoring cron_mode / unattended_mode / the permanent
allowlist, and "always" grants were invisible to `is_approved`,
`clear_session` and the messaging-platform approval buttons.
_request_approval now calls tools.approval._run_approval_gate with
pattern_key `cua:<action>:<background|foreground>` (the old scope shape,
so a background grant still never covers the visible foreground variant)
and fail_closed_when_no_human=True, the same posture as
request_tool_approval / the SSH-config write gate. The private dicts,
their release/atexit clearing, the verdict mapping in
hermes_cli/cli_modal_mixin.py and the extra callback install in cli.py
are deleted: the CLI's terminal_tool callback answers computer_use
prompts like any other tool. set_approval_callback stays as an optional
explicit-callback hook with the shared callback contract
(cb(command, description, **kw) -> once|session|always|deny|timeout);
no in-tree host uses it.
Behavior change:
- No approval callback and no gateway (cron, api_server/webhook,
headless -q, plain library use): destructive actions are now REFUSED
with a BLOCKED error and never reach the backend. Previously they
silently ran. cron honors approvals.cron_mode, unattended platforms
approvals.unattended_mode, -q approvals.single_query_mode.
- --yolo / gateway /yolo / approvals.mode: off still allow (unchanged).
- Gateway sessions (Telegram/Discord/Slack/...) now get a real pending
approval with once/session/always buttons instead of default-allow.
- session/always grants live in tools.approval's store; "always" is one
command_allowlist entry (`cua:click:background`) and is scoped to that
action+mode — the old blanket "always_approve unlocks everything for
the session" no longer exists.
- Denial wording is the shared gate's ("BLOCKED: User denied ...",
"BLOCKED: Action timed out ..."); the error JSON keeps `action`.
Tests: tests/tools/test_computer_use_approval_isolation.py
::test_no_callback_refuses_unless_yolo (blocked + no backend call, then
yolo executes) and ::test_always_grant_lands_in_the_shared_store
(is_approved sees the cua:<action>:<mode> key; second call served from
the store). Sabotage: restoring the `callback is None -> allow`
short-circuit fails the first; swapping the shared gate for a private
grant set fails the second plus the three delivery-ladder scope tests.
tests/tools/conftest.py gains `grant_computer_use_approvals` for
dispatch tests that only care about routing.
Sweep of `ruff check . --select F821 --target-version py311`: 2,234 hits. 2,201 are left
alone on purpose: tui_gateway (2,169; bind_module rebinds bodies onto server.py globals,
all names verified to resolve there), the Feishu adapter (27; globals().update() SDK
binding) and the godmode script (5; dead standalone script). The other 33 were all
genuine defects. No lint config change; no TYPE_CHECKING escape hatches — every
annotation names a real, imported type; ty on the touched files: 0 new diagnostics.
- gateway/slash_commands.py: HISTORY_UNREADABLE never imported after #102117
→ NameError on the /btw error branch (same one-liner as #102952).
- gateway/platforms/whatsapp_common.py: `-> Path` return annotation with no Path
import (the body uses `_Path`). Never raised at runtime thanks to
`from __future__ import annotations`, but `typing.get_type_hints()` and ty
both fail on it.
- gateway/run.py: ActivityProvenance imported at module level
(agent.session_activity has no gateway deps); stringly annotation and the
lazy in-function import are gone.
- tools/patch_parser.py: PatchResult imported at module level; real return
annotation. The "avoid circular import" lazy import guarded a cycle that
does not exist (file_operations_common never imports patch_parser).
- gateway/platforms/helpers.py: base.py imports helpers at module level, so
MessageEvent cannot be named here; TextBatchAggregator only reads .text and
.source, so it is typed by a BatchableEvent Protocol that MessageEvent
satisfies structurally.
- tools/mcp_tool_sampling.py: mcp_tool imports this module, so MCPServerTask
cannot be named here; ElicitationHandler only reads
owner._pending_call_context, typed by an ElicitationOwner Protocol.
- plugins/platforms/sms/adapter.py: aiohttp is an optional dep ([messaging] extra) →
module-level try/except ImportError binding `aiohttp = web = None`, the pattern the
homeassistant / webhook / whatsapp_cloud adapters already use. Retires three lazy
in-function imports and the `_aiohttp_available()` wrapper; `_handle_webhook` typed
`web.Request -> web.Response`.
- plugins/platforms/teams/summary_writer.py: plain module-level `import httpx` — httpx is a
hard core dependency (pyproject `httpx[socks]==0.28.1`), so the lazy import and the
"imported on every CLI start" docstring premise were both wrong (plugin discovery never
imports this module; it is reached only via the Teams adapter / meeting pipeline).
Tests:
- tests/hermes_cli/test_config.py: a test body orphaned by the wave-1 prune
(6b81590c55) sat inside the class as dead code with self/tmp_path unbound
— header restored, so the v11→12 custom_providers migration is covered.
- tests/tools/test_mcp_tool.py: @staticmethod recursing on `self` in the
win32 branch; call portalocker directly.
- tests/test_background_review_list_shapes.py: main() still ran 3 pruned tests.
- tests/agent/test_cursor_optimizations_parity.py: bench() used names only
imported inside a sibling test.
- GatewayRunner / FeishuAdapter / Dict / Optional: missing imports.
main guarded scroll's x/y independently (coordinate=[null,100] -> y=100),
while click treats a coordinate without x as no point. The shared _xy()
applied click semantics to scroll; split out _scroll_xy() with main's
per-axis guard and pin both behaviors in a test.
Real-profile browsing routes all in-page browser work through the Browser Use
CLI (browser_exec), which obsoletes the cua-driver typed-browser surface baked
into computer_use. Remove it so computer_use is a pure DESKTOP-control tool
(screenshots / mouse / keyboard / window management) and every call's schema
drops ~24 browser-only params + 9 actions.
- schema.py: 9 cua_browser_* actions and the typed-browser param block removed;
14 desktop actions + shared params kept; description drops the browser rung.
- tool.py: cua_browser entries out of _SAFE/_DESTRUCTIVE_ACTIONS; the whole
cua_browser dispatch block deleted; {"type","cua_browser_type"} → "type"
(desktop typing untouched); _config_preauthorized (browser-prepare-only, a
no-op for every desktop action) and the browser-page escalation hint removed.
- browser_route.py deleted (no importers outside the package); cua_backend.py
drops the import + typed_browser_* methods; backend.py drops the non-abstract
defaults.
- tests: browser-route/contract suites removed; browser assertions trimmed.
Desktop control unchanged. 233 computer_use tests pass; the 1 remaining failure
(test_gateway_session_key_yolo_maps_to_unrestricted_mode) is a pre-existing
cross-test state leak — fails identically on origin/main, passes in isolation.
An MCP call_tool deadline hit left the cua-driver session wedged for
all later computer-use calls. Mark the session suspect on a
concurrent.futures.TimeoutError and tear down + recreate it before the
next non-lifecycle call; healthy sessions are never restarted.
Fail-closed: the timed-out action may still have taken effect on the
remote screen, so it is never silently replayed — the error result
carries structuredContent.code=timeout_outcome_unknown with
next_step=fresh_state.
Informed by #74877 by BlackishGreen33.
Co-authored-by: BlackishGreen33 <s5460703@gmail.com>
mcp 2.0.0 implements MCP revision 2026-07-28 and makes three breaking
changes Hermes sits on top of: `mcp.server.fastmcp` is gone, every model
field is renamed to snake_case (camelCase survives only as a
serialization alias, which pydantic does not expose to attribute
access), and the SDK's own HTTP stack moved from `httpx` to `httpx2`.
Bump the pin across the dev/mcp/computer-use extras and port the tree:
- `mcp_serve.py` and `agent/transports/hermes_tools_mcp_server.py` move
from `FastMCP` to `mcp.server.MCPServer`, which has the same
decorator/add_tool surface. The hermes-tools server already
synthesised `__signature__` from Hermes' JSON Schema, which is exactly
what 2.0's `add_tool` reads.
- SDK model reads go through `mcp_field(obj, snake, camel)`, which reads
both spellings. A single-spelling read fails *silently* on the other
generation — empty tool schemas, dropped structured content, tool
results vanishing from sampling conversations — and `mcp` is an
optional extra users install at their own version.
- `sdk_httpx()` resolves the httpx flavour from the SDK's own transport
module, so objects handed to `streamable_http_client`, the `sse_client`
factory, and the OAuth metadata helpers come from the module the
installed SDK actually imports.
- HTTP support is gated on either streamable-HTTP entry point, not just
the deprecated alias 2.0 removed.
- OAuth: `OAuthClientProvider` lost its `timeout` argument (the
configured `oauth.timeout` now bounds the callback waiter's own poll
loop, where the browser round-trip was always awaited), and
`callback_handler` must return `AuthorizationCodeResult` rather than a
tuple. 2.0 also validates the RFC 9207 `iss` parameter, so the
callback handler and paste fallback capture it.
`mcp`/`mcp-types` 2.0.0 are inside the 14-day `exclude-newer` window, so
two narrow `exclude-newer-package` entries unblock `uv lock`, annotated
for removal on or after 2026-08-11. `httpx2` needs no exemption: 2.7.0 is
already outside the window and satisfies mcp's floor.
Refs #69931
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A same-day version-floor bump (0.20 runtime contract) left every install
with an older cua-driver hard-failing on all computer_use calls: the
start() gate fails closed, while the `hermes update` refresh defers to the
driver's own check-update verb — whose ~20h cache routinely answers "no
update available" right after we raise the floor. Hermes knew it required
0.20+ but never acted on that knowledge.
Two changes:
- tools_config.install_cua_driver(): a contract-failed installed driver is
repaired on the upgrade=True path too (previously only upgrade=False).
The contract failure itself is the confirmation, so the
require_confirmed_update gate and the check-update short-circuit are
bypassed for repairs — an indeterminate or stale-cached check can no
longer pin users on an unusable driver.
- cua_backend.CuaDriverBackend.start(): when the contract gate fails on an
installed binary, attempt one automatic repair per process via the
standard install path, then re-probe. HERMES_CUA_DRIVER_CMD overrides
are never repaired (explicit override is authoritative even when broken)
and a missing binary still just reports the install hint. A failing
installer can't loop: the second start() surfaces the original error.
Tests: contract-repair coverage in test_computer_use.py (auto-repair
success, failed repair surfaces the original error, once-per-process
guard, override never repaired, missing binary never repaired) and
test_install_cua_driver.py (incompatible driver repairs despite an
indeterminate check-update, check-update not consulted). All new tests
verified to fail against the unfixed source (sabotage run).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
`_canon_key_combo` (the `_BLOCKED_KEY_COMBOS` gate in
`handle_computer_use`) split key strings on `+` only, but the cua-driver
backend's `_parse_key_combo` splits on both `+` and `-`. So a model could
issue `{"action":"key","keys":"ctrl-alt-delete"}` (or `alt-f4`,
`cmd-shift-q`): the gate saw a single unknown token and let it through
while the backend executed the real destructive shortcut.
Split the gate on both `+` and `-` so it canonicalizes combos the same
way the backend does. Non-destructive hyphen combos (`cmd-c`) and the
literal `-` zoom key (`cmd+-`) are unaffected.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
cua-driver 0.7.x can return list_windows/list_apps payloads under
structuredContent.windows, data.windows, data._legacy_windows, or
top-level windows/_legacy_windows (direct CLI responses). The wrapper
only read structuredContent.windows, so discovery came back empty
(capture 0x0, list_apps []) while raw cua-driver calls worked.
- add _windows_from_tool_result(): walks the known envelope shapes in
priority order, skipping empty higher-priority envelopes
- route _load_windows() (MCP + CLI re-fetch paths) through the helper,
covering capture() and focus_app()
- harden _ingest_windows(): skip non-dict members, normalize untrusted
app_name/title/z_index fields
- list_apps(): prefer structuredContent.apps, fall through populated
data/top-level envelopes, derive unique apps from window-shaped
payloads via _apps_from_windows(), keep the text-line fallback last
- tests for every envelope shape, precedence, malformed records, and
app derivation
Salvaged from #63037 (Reaper-Legion), which itself preserved the
original implementation from #57961 (kohoj); #73007 (umi008)
independently proposed the same normalization later.
Fixes#57905
Co-authored-by: Reaper <248977840+Reaper-Forge@users.noreply.github.com>
Co-authored-by: Ulises Millan Guerrero <ulises.millanguerrero@gmail.com>
_start_lifecycle_locked reassigns a fresh threading.Event() at line
786, so patching the pre-made instance's wait() is lost and the real
30s wait races pytest-timeout. Patch threading.Event itself with a
FakeEvent whose wait() returns False immediately (#69372).
Treat cua-driver's Linux `is_on_screen: null` as unknown instead of
off-screen, and skip GNOME Shell desktop/backdrop helper windows
(ding "Desktop Icons", @!x,y;BDHF) when selecting the default capture
target — they are targetable X11 windows but capture as empty.
Reconciled with the _NET_ACTIVE_WINDOW fallback from #58030: helper
windows are filtered out of the candidate pool first, then the tied
z-order active-window probe runs on the remaining real app windows.
Also falls back to the requested app name for _last_app when Linux
windows carry no app_name.
Salvaged from #54173 by @dnth.
cua_driver_update_check now short-circuits to None when no driver
resolves; CI has none installed, so pin resolve_cua_driver_cmd in the
update-check and env-sanitization tests.
- Add autouse fixture to TestMcpInvocationResolution to disable
--no-overlay flag so existing tests assert baseline args
- Make test_config_load_failure_fails_safe and test_missing_section_enables
platform-aware (Linux auto-detect returns True, macOS/Windows False)
* fix(computer-use): target Linux app windows reliably
Resolve app filters through the canonical cua-driver MCP app metadata and join running app PIDs back to windows. Preserve an exact selected window across capture_after, support direct capture by pid/window_id, and send the active window ID for coordinate pointer actions on Linux.
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
* fix(computer-use): address review on PR #63725
Address three review comments from @f-trycua:
1. type_text, press_key, and hotkey now carry _active_window_id and
fail closed when it is missing. Previously they sent only the PID,
so CUA Driver fell back to the first window for that PID — input
could reach the wrong window in multi-window apps.
2. Coordinate scroll x/y are now capability-gated behind
input.scroll.coordinates. CUA Driver 0.7.1 Linux schema rejects
x/y on scroll; omitting them when the driver doesn't advertise
support avoids the schema rejection while still routing via
window_id.
3. Windows are sorted by z_index descending (higher = front, per CUA
Driver semantics) instead of ascending. Null z_index (Wayland) is
coerced to 0 in _ingest_windows so it doesn't crash the sort and
sorts to the back instead of being selected as the capture target.
---------
Co-authored-by: LeonSGP43 <cine.dreamer.one@gmail.com>
Co-authored-by: annguyenNous <annguyenNous@users.noreply.github.com>
Co-authored-by: grimmjoww578 <willies578@gmail.com>
Co-authored-by: ai-ag2026 <261867348+ai-ag2026@users.noreply.github.com>
The SOM/AX element list dropped labels for two extremely common cua-driver
render forms, leaving the model unable to target elements by name:
- [79] AXButton (Dark) -> parenthesised label
- [4] AXStaticText = "Wi-Fi" -> = "value" form
- [92] AXPopUpButton = "Automatic" -> = "value" form
The old regex only matched quoted "label" and id=Label, so System Settings
buttons/text/popups all surfaced with empty labels. That's why selecting the
macOS Appearance 'Dark' button by element index required guessing — the
labels weren't available to aim with.
Fix: extend _ELEMENT_LINE_RE to capture all four label forms (= "value",
"quoted", (parenthesised), id=Label), skipping a pure-digit (N) order number
in favour of the id= label. Verified live against System Settings: the
Appearance buttons now surface as Auto/Light/Dark.
Adds a regression test covering all label forms. Full suite: 84 passed.
The first fix handled the EAGAIN McpError path. But the persistent MCP
session (long-running gateway/desktop worker) has a second failure mode:
list_windows or get_window_state 'succeed' over MCP yet return a
degenerate/empty payload (no windows, or no screenshot + blank tree)
WITHOUT raising — typically when the bridge reconnected mid-call and
dropped the heavy response. That surfaced to the model as a silent 0x0
capture with no error and no fallback firing (0.00s empty return).
Fix: detect empty results in capture() and re-fetch over the CLI
transport before giving up:
- empty list_windows -> CLI re-fetch the window list
- empty get_window_state (som/ax) -> CLI re-fetch the AX tree + screenshot
- empty screenshot (vision) -> CLI re-fetch get_window_state for the PNG
Adds 2 regression tests. Full suite: 83 passed.
The cua-driver MCP stdio bridge intermittently (and on some machines
persistently) fails to forward heavier calls like get_window_state to
the daemon with POSIX EAGAIN — 'daemon transport error forwarding
get_window_state: Resource temporarily unavailable (os error 35)'.
The wrapper surfaced this as an empty 0x0 capture, so computer_use
returned blank screenshots even though the display, permissions, and
the daemon were all healthy (the direct 'cua-driver call' CLI path
worked fine throughout).
Fix: when the MCP path raises the transient/transport error, fall back
to the 'cua-driver call' subprocess transport, which talks to the
daemon over a different socket. The CLI fallback routes get_window_state
screenshots to a temp file via screenshot_out_file (tiny JSON response
instead of a multi-MB base64 blob that congests the socket), reads the
PNG back, retries with backoff, and remaps the JSON into the same
{data, images, structuredContent, isError} shape the MCP path produces
so capture()/_action() are transport-agnostic.
Adds _is_transient_daemon_error() classifier and 3 regression tests.
Verified live: captures that returned 0x0 now return full
1567x905 screenshots with the AX element tree.
