hermes-admin/hermes-agent
1
0
Fork 0
Code Issues Pull Requests Actions 28 Packages Projects Releases Wiki Activity

fix(computer_use): attach element_token when the driver schema accepts it

Browse Source 
cua-driver 0.21 refuses a bare element_index:

    click: bare element_index is not accepted; pass element_token,
    or snapshot_id together with element_index

_maybe_attach_element_token gated solely on the trycua/cua#1961 capability
vocabulary. 0.21 stopped publishing per-tool capability sets — every tool
reports an empty set — while still accepting element_token in its input
schema. The gate therefore fails closed on 0.21.x and we send the bare
index, so the driver rejects the call.

The effect is total: every element-targeted click is refused, leaving
agents with only blind pixel coordinates. Observed against cua-driver
0.21.0 on X11, where a capture returned 762 elements with 762 tokens
cached and every subsequent click still failed with snapshot_id_required.

Check the live input schema first — supports_input_property() already
exists for exactly this, and its docstring notes it "deliberately inspects
tools/list rather than ... requiring a capability token the driver never
shipped". The capability check is retained as a fallback so drivers that
did ship the vocabulary are unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This commit is contained in:
Jeff J Hunter
2026-08-23 04:17:34 +00:00
committed by Teknium
parent 9a6eb21a88
commit 65625dfe87
1 changed files with 7 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files

9
tools/computer_use/cua_backend.py
View File

@@ -354,10 +354,15 @@ class CuaDriverBackend(_CaptureMixin, _InputMixin, ComputerUseBackend):
def _action(self, name: str, args: Dict[str, Any], *, inject_session: bool = True) -> ActionResult:
# Attach the snapshot's `element_token` to an `element_index` call so a superseded snapshot yields an explicit
# 'stale' error. Gated on the per-tool capability: older drivers (`additionalProperties: false`) must never see it.
# 'stale' error. Two ways to establish support, the live input schema first: cua-driver 0.21+ stopped
# publishing per-tool `capabilities[]` while still accepting `element_token` in its schema, and it REFUSES a
# bare `element_index` (`snapshot_id_required`) — gating on the capability alone broke EVERY element click and
# left only pixel clicks working. The capability check stays so older drivers that shipped the vocabulary keep
# working; drivers advertising neither (`additionalProperties: false`) must never see the property.
idx = args.get("element_index")
token = self._snapshot_tokens.get(idx) if isinstance(idx, int) else None
if token and self._session.supports_capability("accessibility.element_tokens", tool=name):
if token and (self._session.supports_input_property(name, "element_token")
or self._session.supports_capability("accessibility.element_tokens", tool=name)):
args["element_token"] = token
if inject_session: # setdefault preserves any explicit session a caller already supplied
args.setdefault("session", self._session_id)