Commit Graph

227 Commits

Author SHA1 Message Date
teknium1
a4d474777f fix(computer-use): doctor diagnoses a denied cua-driver spawn instead of crashing
On Windows the Hermes venv interpreter cannot CreateProcess a binary under
C:\Program Files\WindowsApps (WinError 5) even though the shell resolves it,
so `hermes computer-use doctor` died with a raw PermissionError traceback
from _open_mcp. Catch the spawn OSError and print what failed, why the tool
may still work (PATH resolves another copy), and the fix (reinstall outside
WindowsApps or HERMES_CUA_DRIVER_CMD), exit 2.
2026-09-14 17:34:31 -07:00
Jeff J Hunter
65625dfe87 fix(computer_use): attach element_token when the driver schema accepts it
cua-driver 0.21 refuses a bare element_index:

    click: bare element_index is not accepted; pass element_token,
    or snapshot_id together with element_index

_maybe_attach_element_token gated solely on the trycua/cua#1961 capability
vocabulary. 0.21 stopped publishing per-tool capability sets — every tool
reports an empty set — while still accepting element_token in its input
schema. The gate therefore fails closed on 0.21.x and we send the bare
index, so the driver rejects the call.

The effect is total: every element-targeted click is refused, leaving
agents with only blind pixel coordinates. Observed against cua-driver
0.21.0 on X11, where a capture returned 762 elements with 762 tokens
cached and every subsequent click still failed with snapshot_id_required.

Check the live input schema first — supports_input_property() already
exists for exactly this, and its docstring notes it "deliberately inspects
tools/list rather than ... requiring a capability token the driver never
shipped". The capability check is retained as a fallback so drivers that
did ship the vocabulary are unaffected.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-14 17:34:31 -07:00
teknium1
d28938d3da fix(computer-use): screenshot dedup forgets its last frame at a compaction boundary
The "screen unchanged" result points the model at its previous capture. After
context compression that capture may be summarized away, so the note would refer
to pixels no longer in context. Mirror read_file's reset_file_dedup: the
compaction boundary (both the summary path and the codex app-server path) now
clears the session's screenshot digest, and the first capture afterwards delivers
the image again even when the screen is byte-identical.
2026-09-14 07:21:18 -07:00
teknium1
31964ff4c6 fix(computer-use): dedup keyed by the scoped session, cleared on release
Key the screenshot-dedup state by the same profile-scoped session id the
backend cache uses, so two multiplexed profiles sharing a session id (or a
DISPLAY) never dedup against each other's frames, and forget the state in
release_computer_use_session so a re-created session's first capture
always delivers pixels. Reword the unchanged note to cover the aux-vision
path (where the prior result was an analysis, not an image). Tests: the
dispatch path (explicit capture + capture_after) honours the streak cap;
release forgets state.
2026-09-14 07:21:18 -07:00
Teknium
682b973b32 feat(computer-use): stop resending unchanged screenshots
Port from openclaw/openclaw#129924: a capture whose pixels are
byte-identical to the previous capture of the same target in the same
session returns its full text metadata (element index included) plus an
explicit 'screen unchanged' note instead of the multimodal image block.

Adapted for Hermes: openclaw gates dedup on per-frame context-epoch
tracking; Hermes bounds staleness with a consecutive-omission streak cap
(2) so full pixels are re-delivered before compaction could evict the
referenced image. Dedup state is per-session (no cross-session leaks),
append-only (no history rewrites — prompt cache prefixes untouched),
and skipped entirely when no session_id is present.
2026-09-14 07:21:18 -07:00
teknium1
71cebc6348 fix(tools): browser_exec and computer_use caches are namespaced by the served profile
Both process-global caches were keyed by the caller's session/task id alone, so under
gateway.multiplex_profiles two profiles using the same id — a shared `browser_exec session=`
name, or two Hermes sessions whose screens report the same DISPLAY — resolved to the FIRST
profile's cloud browser / cua-driver, and a command issued in one bot's chat could act on
another bot's screen.

The key now carries the routed profile's home key whenever a served-profile scope is active
(`get_hermes_home_override()` set), the same shape `tools/approval.py::_baseline_key` and the
camofox/cloud caches already use; outside a scope every key is byte-identical to before. The
computer_use lookup, install and release paths all go through one `_scoped_sid`, so a release
under profile B never stops profile A's driver; approval-bypass state keeps the bare session id.

Fixes #110032 (report by @wolfyy970, from @vandaimer's manual test on #108914).
2026-09-13 14:41:26 -07:00
teknium1
3e066dfedd fix(computer_use): approval goes through the shared gate; no callback now fails closed
computer_use kept its own approval decision: two module dicts
(_session_auto_approve / _always_allow) mirroring tools.approval's
session store and _persist_choice, a private verdict vocabulary
(approve_once/approve_session/always_approve) that hermes_cli mapped
back to once/session/always, and — the real problem — `if
_approval_callback is None: return None`. Only the interactive CLI ever
installed that callback, so every other host (gateway turns, cron,
api_server, tui_gateway, ACP) ran destructive desktop input with no
approval at all, ignoring cron_mode / unattended_mode / the permanent
allowlist, and "always" grants were invisible to `is_approved`,
`clear_session` and the messaging-platform approval buttons.

_request_approval now calls tools.approval._run_approval_gate with
pattern_key `cua:<action>:<background|foreground>` (the old scope shape,
so a background grant still never covers the visible foreground variant)
and fail_closed_when_no_human=True, the same posture as
request_tool_approval / the SSH-config write gate. The private dicts,
their release/atexit clearing, the verdict mapping in
hermes_cli/cli_modal_mixin.py and the extra callback install in cli.py
are deleted: the CLI's terminal_tool callback answers computer_use
prompts like any other tool. set_approval_callback stays as an optional
explicit-callback hook with the shared callback contract
(cb(command, description, **kw) -> once|session|always|deny|timeout);
no in-tree host uses it.

Behavior change:
- No approval callback and no gateway (cron, api_server/webhook,
  headless -q, plain library use): destructive actions are now REFUSED
  with a BLOCKED error and never reach the backend. Previously they
  silently ran. cron honors approvals.cron_mode, unattended platforms
  approvals.unattended_mode, -q approvals.single_query_mode.
- --yolo / gateway /yolo / approvals.mode: off still allow (unchanged).
- Gateway sessions (Telegram/Discord/Slack/...) now get a real pending
  approval with once/session/always buttons instead of default-allow.
- session/always grants live in tools.approval's store; "always" is one
  command_allowlist entry (`cua:click:background`) and is scoped to that
  action+mode — the old blanket "always_approve unlocks everything for
  the session" no longer exists.
- Denial wording is the shared gate's ("BLOCKED: User denied ...",
  "BLOCKED: Action timed out ..."); the error JSON keeps `action`.

Tests: tests/tools/test_computer_use_approval_isolation.py
::test_no_callback_refuses_unless_yolo (blocked + no backend call, then
yolo executes) and ::test_always_grant_lands_in_the_shared_store
(is_approved sees the cua:<action>:<mode> key; second call served from
the store). Sabotage: restoring the `callback is None -> allow`
short-circuit fails the first; swapping the shared gate for a private
grant set fails the second plus the three delivery-ladder scope tests.
tests/tools/conftest.py gains `grant_computer_use_approvals` for
dispatch tests that only care about routing.
2026-09-13 05:21:02 -07:00
Teknium
4b8c01f691 fix(multiplex): key tool-side and agent-side memos by profile home
Camofox VNC one-shot, computer-use aux-vision verdict, tirith binary path, MCP
discovery lock path, remote-backend probe text, learned image token costs,
auxiliary per-task semaphores and the custom-endpoint /models memo all held one
profile's config-derived value for the whole process. The skill-sync debounce
Timer ran with empty ContextVars, so a secondary's write pushed as the launch
profile (and cancelled its pending push).

Each memo is now keyed by hermes_home_key() (or credential fingerprint for the
per-key catalog) under an override; the timer is per home and runs its callback
inside the scheduling turn's copied context. Unscoped slots are unchanged.
2026-09-12 01:35:05 -07:00
Teknium
facb9eb4f1 fix: trim computer use tool schema guidance 2026-09-08 17:03:01 -07:00
Teknium
d63e380324 compat(plugins): warn once per name when a plugin resolves an old import path; lint step restored in CI
Every PLUGIN-COMPAT __getattr__ now calls hermes_cli.plugin_compat.warn_once(facade, name, target) before
resolving, emitting a HermesPluginCompatWarning (FutureWarning) once per process per name: old path, new
path, removal target. Importing a facade for its live API stays silent; only resolving a moved name warns.
COMPAT_MANIFEST.md documents the warning and how to silence it during migration.

Verified the runtime never routes through a pointer: every entry point (run_agent, cli, hermes_cli.main,
gateway.run, tui_gateway.server, web_server, model_tools + tool discovery, hermes_state, cron.scheduler,
browser_tool, mcp_tool, kanban, auth) imports clean and `hermes doctor` runs end to end with the warning
promoted to an error.

Also restores the check_compat_pointers CI step to .github/workflows/lint.yml, which a0be177aac dropped
when the compat layer was regenerated (the lint script itself was present; the workflow step was not).

hermes_cli/plugin_compat.py, tests/test_plugin_compat_warning.py and the two-line insert per facade are
part of the compat layer and go away with it.
2026-09-04 00:15:16 -07:00
Teknium
2776813df3 compat(plugins): temporary import-path shims for external plugins — ONE commit, revert on schedule
The Sep 2026 decomposition (PR #102117) makes internal import paths a non-API: names now live in
the focused modules that define them. This commit is the ONLY thing keeping the old paths alive,
so external plugins have time to update. It is deliberately a single, unsquashed commit:

    git revert <this sha>

removes every shim, stub and manifest at once on the announced date. Nothing in-tree may depend on
these pointers: scripts/check_compat_pointers.py (wired into lint.yml) fails CI if it does.

What it adds (see COMPAT_MANIFEST.md, compat_manifest.json):
- 332 facade modules get one delimited `PLUGIN-COMPAT` block appended at the end of the file
- 1,172 moved names resolved lazily via a module `__getattr__` (PEP 562) — never a top-level import,
  so no import cycles; facades that already had `__getattr__` get a chained one
- 592 third-party/stdlib names the old modules used to expose, with their original import statements
- 266 public definitions that had been deleted as unused, restored byte-for-byte from the pre-decomposition
  tree (+40 private helpers and 16 imports pulled in only because a restored definition needs them)
- 3 deleted modules recreated as re-export stubs (gateway/startup_watchdog, hermes_cli/observability/
  relay_runtime, tools/environments/modal_utils)
- private names (`_x`) get no pointer: they were never API (3,792 skipped)

Verified: all 335 touched modules import under a fresh HERMES_HOME and every manifest name resolves;
the lint reports zero in-tree uses; ruff clean; targeted suites unchanged.
2026-09-03 17:13:22 -07:00
Teknium
eb8a30cc2f simplify(compat): skills_sync/skill_manager/kanban/computer_use — drop 37 re-exports/aliases, repoint 8 callers + 9 test files 2026-09-03 13:27:01 -07:00
Teknium
5c3db6750f simplify(compat): tools/computer_use/cua_backend — drop 24 re-exports + shutil re-import, repoint 5 callers + 14 test files 2026-09-03 13:22:51 -07:00
Teknium
75f8dd3293 review-fix(suppress-audit): tools/computer_use/tool.py — restore BASE exception semantics in _bounds_hints (int() failures propagate as on main) 2026-09-03 09:47:10 -07:00
Teknium
e83816a4d1 review-fix(comments): restore lost #NNNN rationale comments across non-test source (mechanical sweep, condensed, code unchanged)
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
2026-09-03 09:44:26 -07:00
Teknium
febbe6ae12 review-fix(computer_use): restore per-axis scroll coordinate semantics
main guarded scroll's x/y independently (coordinate=[null,100] -> y=100),
while click treats a coordinate without x as no point. The shared _xy()
applied click semantics to scroll; split out _scroll_xy() with main's
per-axis guard and pin both behaviors in a test.
2026-09-03 09:32:38 -07:00
Teknium
0071ba9965 Merge origin/main (561b053f79) into simp/forwardport: forward-port 220 main commits into the simplified tree 2026-09-03 03:31:03 -07:00
Teknium
23b9ffc4fa fix(integration): restore subprocess stdin=DEVNULL / utf-8 encoding guards and windows-footgun gates dropped by round-3 compaction
Repo scanners (check_subprocess_stdin, check-windows-footguns --all) flagged 21 sites where
the r3 single-line collapses lost stdin=DEVNULL, encoding='utf-8'/errors='replace', the
'# windows-footgun: ok' same-line marker, or the getattr(os, 'geteuid') gate. Each guard is
restored at the call site (real portability/hang fixes, not suppressions).
2026-09-03 02:46:19 -07:00
Teknium
75934b89c8 Merge branch 'simp/r2-tools-b' (tools wave 2 net-deletion: approval/delegate/skills_hub/terminal/computer_use) into simp/integration3
Conflicts (union): computer_use/{cua_backend,cua_backend_daemon,doctor}.py — kept r2-tools-b's
_run_quiet/_run_driver/_cli_text helpers and the integration3 stdin=DEVNULL fix (06fb390212).
2026-09-03 00:44:12 -07:00
Teknium
6b393f70eb Merge branch 'simp/r2-tools-b-delegate2' (tools wave 2: approval/delegate/skills_hub/terminal/computer_use) into simp/integration3 2026-09-03 00:41:31 -07:00
Teknium
72f5578b3a refactor(computer_use): restore leading comments on capture-cap constants; wrap _write_cache_file signature (legibility) 2026-09-02 23:49:02 -07:00
Teknium
5080b1024a Merge branch 'simp/r2-tools-b-cua2-sess2' into simp/r2-tools-b-cua2 2026-09-02 23:38:27 -07:00
Teknium
de2c49b0d6 refactor(computer_use): compact section headers, wrapped literals, and one-shot walrus guards 2026-09-02 23:35:38 -07:00
Teknium
1fcb6f521d refactor(computer_use): atexit decorator, list-comprehension _format_elements, tighter signatures 2026-09-02 23:30:17 -07:00
Teknium
b8f6a8693a refactor(computer_use): set_value/focus_app handlers as table lambdas 2026-09-02 23:23:59 -07:00
Teknium
06c0243180 refactor(computer_use): lambda writer for element spill; single prefix assignment in follow capture 2026-09-02 23:22:16 -07:00
Teknium
0bd806bd22 refactor(computer_use): drop cap aliases in capture builders, single-line noop __init__, compact section spacing 2026-09-02 23:21:06 -07:00
Teknium
eebddc22e8 refactor(computer_use): drop dead re-raise branch in _stop_backend; compact _do_capture target forwarding 2026-09-02 23:19:21 -07:00
Teknium
775f1d77dc refactor(computer_use): inline _multimodal_capture, drop dead non-JSON fallback in follow capture, _xy as kwargs 2026-09-02 23:17:04 -07:00
Teknium
b651b77af5 refactor(computer_use): inline _view into its two builders; fold constant comments and _VERDICT_FIELDS 2026-09-02 23:12:47 -07:00
Teknium
230524b5d1 refactor(computer_use): _NoopBackend capture/focus_app via _noop_stub result factory 2026-09-02 23:10:01 -07:00
Teknium
b569633449 refactor(computer_use): fold _configured_permission_mode, _listing, _input into callers; merge capture MIME/ext helpers 2026-09-02 23:08:33 -07:00
Teknium
796a7e4829 refactor(computer_use): inline single-use session wrappers, hoist result classifiers to module level, fold list_windows 2026-09-02 22:50:24 -07:00
Teknium
c890cab0d5 Merge branch 'simp/r2-tools-b-cua2-drv' into simp/r2-tools-b-cua2 2026-09-02 21:51:32 -07:00
Teknium
298304fe58 Merge branch 'simp/r2-tools-b-cua2-doc' into simp/r2-tools-b-cua2 2026-09-02 21:51:31 -07:00
Teknium
89f08dfdb6 Merge branch 'simp/r2-tools-b-cua2-capt' into simp/r2-tools-b-cua2 2026-09-02 21:51:31 -07:00
Teknium
26862482c5 Merge branch 'simp/r2-tools-b-cua2-sess' into simp/r2-tools-b-cua2 2026-09-02 21:51:30 -07:00
Zane Chee
b1bf099f14 fix(computer-use): distinguish doctor and gateway environments 2026-09-02 21:49:23 -07:00
Teknium
fa0ff15614 refactor(computer_use): final cua backend/driver/daemon folds — inline discovery reasons, contextlib suppress, one-line probes; strings frozen 2026-09-02 21:47:13 -07:00
Teknium
303a8cafb0 refactor(computer_use): compact capture/input/backend — drop dead UIElement.center, section banners, reflow docstrings to 118 cols 2026-09-02 21:28:25 -07:00
Teknium
982f22fd56 refactor(computer_use): suppress-based bounds parsing; inline scale note into summary notes 2026-09-02 21:07:03 -07:00
Teknium
e368f42a0f refactor(computer_use): table-lookup summarize default, inline listing lambda and app-guard walrus 2026-09-02 21:02:45 -07:00
Teknium
22def3f4a9 refactor(computer_use): contextlib rollback/suppress in cua backend + daemon; _json_object helper; daemon startup-failure helper; tighten prose 2026-09-02 21:02:05 -07:00
Teknium
a791dfbc63 refactor(computer_use): _stop_backend on_error hook replaces three try/except blocks; _install_backend owns the hook mirror 2026-09-02 20:58:47 -07:00
Teknium
29fb6e8ea9 refactor(computer_use): namedtuple _ActionSpec, walrus cache write, suppress-based vision analysis parse 2026-09-02 20:55:01 -07:00
Teknium
e5f3927b04 refactor(computer_use): compact doctor row/overall helpers 2026-09-02 20:54:07 -07:00
Teknium
a11b6bae91 refactor(computer_use): fold doctor probe assignments and vision_routing header 2026-09-02 20:47:51 -07:00
Teknium
7390fa2929 refactor(computer_use): suppress-based bypass check, single denial payload, cache glob prune without branch 2026-09-02 20:45:36 -07:00
Teknium
74dccc1eaa refactor(computer_use): _view namespace builder replaces _CaptureView dataclass; suppress-based config readers; exact-target dict in follow capture 2026-09-02 20:41:54 -07:00
Teknium
95b0ae9121 refactor(computer_use): compact cua_backend/driver/daemon docstrings, banners and imports; fold contract-reason logic 2026-09-02 20:37:04 -07:00