tool.py inferred 'walk capped' by re-reading cua's private config through
_cua_configured_ax_max_elements() — a leaky abstraction that also cost a
config load per capture. The backend now records the max_elements it put on
the get_window_state call and returns it on CaptureResult.ax_max_elements
(defaulted field, 0 = unbounded / vision); _capture_view compares
len(cap.elements) >= cap.ax_max_elements > 0 and the getattr guards go
since the field is always set. Hint text and line order unchanged.
PROOF: tests/tools/test_computer_use_ax_walk_bound.py 5 passed (hint tests
now build CaptureResult(ax_max_elements=bound)); the new
_ax_max_elements_sent assertion is red with cua_backend_capture.py
reverted to HEAD~1 ('AttributeError: ... no attribute _ax_max_elements_sent',
1 failed 4 passed). Probe: 200/200 -> capped hint + 'full ' dropped;
199/200 and 8/0 -> full tree promised. check-windows-footguns --all clean.
`_capture_view`'s hint promised a "full element tree saved to elements_file",
but once `computer_use.ax_max_elements` bounds the driver's walk the spill
holds only the first N nodes. When the capture returned at least the bound
(bound > 0), drop "full" and add "accessibility walk capped at N elements;
pass app= to narrow" so the model knows how to reach the rest.
Extract only backend admission/lifetime protection from #114565 (484b5d7), adapted to the existing DISPLAY identity and human-lease fences in #108914. No WSL selection, installer, config or UI changes.
Main replaced every *.request/*.respond event pair with real JSON-RPC server→client requests and
made Python the single source of the wire (tui_gateway/contracts → generated TS). Bot Screen now
speaks that dialect:
- tui_gateway/contracts/display.py declares the eight display.* methods, the display.install.sudo
server request and the four display.* events; the generated TS/OpenRPC is regenerated.
- The install sudo card is a `display.install.sudo` server request (app-level, empty session);
the desktop answers it through respondToServerRequest, so the respondMethod/origin plumbing
that pinned a reply to its socket is gone: a JSON-RPC response cannot land elsewhere.
- screen-connection.ts consumes the generated DisplayStatus/DisplayLease instead of hand-typed
copies (holder named by viewer_hash only; epoch always present).
- computer_use: the lease fence and main's screenshot-dedup session key ride the same read
handlers; the fence fires before a frame reaches the dedup cache.
- server.py keeps the display module registration a naive --theirs would have dropped.
The "screen unchanged" result points the model at its previous capture. After
context compression that capture may be summarized away, so the note would refer
to pixels no longer in context. Mirror read_file's reset_file_dedup: the
compaction boundary (both the summary path and the codex app-server path) now
clears the session's screenshot digest, and the first capture afterwards delivers
the image again even when the screen is byte-identical.
Key the screenshot-dedup state by the same profile-scoped session id the
backend cache uses, so two multiplexed profiles sharing a session id (or a
DISPLAY) never dedup against each other's frames, and forget the state in
release_computer_use_session so a re-created session's first capture
always delivers pixels. Reword the unchanged note to cover the aux-vision
path (where the prior result was an analysis, not an image). Tests: the
dispatch path (explicit capture + capture_after) honours the streak cap;
release forgets state.
Port from openclaw/openclaw#129924: a capture whose pixels are
byte-identical to the previous capture of the same target in the same
session returns its full text metadata (element index included) plus an
explicit 'screen unchanged' note instead of the multimodal image block.
Adapted for Hermes: openclaw gates dedup on per-frame context-epoch
tracking; Hermes bounds staleness with a consecutive-omission streak cap
(2) so full pixels are re-delivered before compaction could evict the
referenced image. Dedup state is per-session (no cross-session leaks),
append-only (no history rewrites — prompt cache prefixes untouched),
and skipped entirely when no session_id is present.
Both process-global caches were keyed by the caller's session/task id alone, so under
gateway.multiplex_profiles two profiles using the same id — a shared `browser_exec session=`
name, or two Hermes sessions whose screens report the same DISPLAY — resolved to the FIRST
profile's cloud browser / cua-driver, and a command issued in one bot's chat could act on
another bot's screen.
The key now carries the routed profile's home key whenever a served-profile scope is active
(`get_hermes_home_override()` set), the same shape `tools/approval.py::_baseline_key` and the
camofox/cloud caches already use; outside a scope every key is byte-identical to before. The
computer_use lookup, install and release paths all go through one `_scoped_sid`, so a release
under profile B never stops profile A's driver; approval-bypass state keeps the bare session id.
Fixes#110032 (report by @wolfyy970, from @vandaimer's manual test on #108914).
`computer_use` carried two Bot Screen actions, `request_handoff` and `wait_for_human`, that let
the model ask for the screen and then block a tool call until the human handed it back. Both only
make sense when a person is guaranteed to be watching the Desktop pane; from Telegram, the CLI or
a cron worker the request lands nowhere and the wait burns minutes before returning. The blocking
wait also fought the sequential tool deadline (600 s default vs 420 s), so the model saw a timeout
before the wait returned while the thread stayed parked.
The agent now simply says what it needs in its reply and ends the turn; the user takes over from
the pane, does the step, hands back and tells it to continue. Take over / hand back and every fence
(actions refused with human_has_control, epoch-voided results, suppressed thumbnails) are
unchanged. Removes the handoff module, the schema entries and their host-conditional rewriter,
the lease's pending_handoff field and wait helpers, and the "Bot needs you" badge in the pane.
computer_use kept its own approval decision: two module dicts
(_session_auto_approve / _always_allow) mirroring tools.approval's
session store and _persist_choice, a private verdict vocabulary
(approve_once/approve_session/always_approve) that hermes_cli mapped
back to once/session/always, and — the real problem — `if
_approval_callback is None: return None`. Only the interactive CLI ever
installed that callback, so every other host (gateway turns, cron,
api_server, tui_gateway, ACP) ran destructive desktop input with no
approval at all, ignoring cron_mode / unattended_mode / the permanent
allowlist, and "always" grants were invisible to `is_approved`,
`clear_session` and the messaging-platform approval buttons.
_request_approval now calls tools.approval._run_approval_gate with
pattern_key `cua:<action>:<background|foreground>` (the old scope shape,
so a background grant still never covers the visible foreground variant)
and fail_closed_when_no_human=True, the same posture as
request_tool_approval / the SSH-config write gate. The private dicts,
their release/atexit clearing, the verdict mapping in
hermes_cli/cli_modal_mixin.py and the extra callback install in cli.py
are deleted: the CLI's terminal_tool callback answers computer_use
prompts like any other tool. set_approval_callback stays as an optional
explicit-callback hook with the shared callback contract
(cb(command, description, **kw) -> once|session|always|deny|timeout);
no in-tree host uses it.
Behavior change:
- No approval callback and no gateway (cron, api_server/webhook,
headless -q, plain library use): destructive actions are now REFUSED
with a BLOCKED error and never reach the backend. Previously they
silently ran. cron honors approvals.cron_mode, unattended platforms
approvals.unattended_mode, -q approvals.single_query_mode.
- --yolo / gateway /yolo / approvals.mode: off still allow (unchanged).
- Gateway sessions (Telegram/Discord/Slack/...) now get a real pending
approval with once/session/always buttons instead of default-allow.
- session/always grants live in tools.approval's store; "always" is one
command_allowlist entry (`cua:click:background`) and is scoped to that
action+mode — the old blanket "always_approve unlocks everything for
the session" no longer exists.
- Denial wording is the shared gate's ("BLOCKED: User denied ...",
"BLOCKED: Action timed out ..."); the error JSON keeps `action`.
Tests: tests/tools/test_computer_use_approval_isolation.py
::test_no_callback_refuses_unless_yolo (blocked + no backend call, then
yolo executes) and ::test_always_grant_lands_in_the_shared_store
(is_approved sees the cua:<action>:<mode> key; second call served from
the store). Sabotage: restoring the `callback is None -> allow`
short-circuit fails the first; swapping the shared gate for a private
grant set fails the second plus the three delivery-ladder scope tests.
tests/tools/conftest.py gains `grant_computer_use_approvals` for
dispatch tests that only care about routing.
Input actions never receive fence=, so a take-over then hand-back during
approval still clicks: assert_agent_may_act only asks if the holder is
human now. Compare admitted.epoch immediately before _dispatch.
Test uses a recording backend and does not patch _dispatch.
Addresses kvnloo on #108914.
(cherry picked from commit 97ce5d5a579ccc797c7ef50b6bb92465955a3277)
The epoch check ran only after _dispatch() returned, by which point the
capture path had already written the PNG to the media cache, spilled the
element tree to disk and routed the frame through auxiliary vision — the
human's screen had left the process before being "discarded". A fence
callable is threaded into _dispatch; capture and capture_after call it
the moment backend.capture() returns, and the post-dispatch check stays
for every other action.
(cherry picked from commit 70c09e5fad89d2817f13aea772bd845ab7cc4c8a)
Independent review of the PR head found the control boundary only held inside
one process and several claims the code did not back. Each item below was
reproduced, fixed, covered by an invariant test proven red without the fix, and
re-verified live on a real Xvnc/Xfce screen.
- Lease authority on disk. `lease.json` under an fcntl lock in the profile's
bot-desktop dir; every read goes to the file. `hermes serve` (viewer bridge),
the messaging gateway, a CLI turn and isolated workers now agree. Live: a
takeover in process A made `computer_use capture` in process B return
human_has_control; release in C made B work again.
- Takeover fences admitted actions. `handle_computer_use` re-checks the lease
under the dispatch lock and discards a result produced after the lease epoch
changed, so an action admitted before a takeover cannot picture what the human
typed during approval / backend start-up waits.
- Sudo reply pinned to its origin. `SudoRequest.origin` records the
(connection, profile) the card came from; SudoDialog answers through
`requestGatewayForAgent` on that socket, never the foreground gateway. A
password typed for host A can no longer reach host B. `sudo.expire` and
`display.install.sudo.expire` now tear the card down (the Desktop never
handled sudo.expire).
- Dock Browser IS the bot's browser. `tools/bot_desktop/browser.py` resolves one
identity — the Chromium agent-browser drives + a persistent per-profile
user-data-dir (`bot-desktop/browser-profile`) — and both sides use it: the
agent env gets AGENT_BROWSER_EXECUTABLE_PATH / AGENT_BROWSER_PROFILE, the
dock launcher gets the same exe + --user-data-dir. Live: the bot wrote
localStorage on http://127.0.0.1:8765 through agent-browser; a dock click and
a typed URL on the screen showed BOT-WROTE-THIS in Chrome for Testing.
- Safe display reuse. Allocation under a host-wide lock; a recorded number is
reused only when no live server holds it; the launcher never unlinks a lock
whose pid is alive. Live: A stopped, B took :20, A restarted on :21, B kept
running.
- Install worker keeps the caller's profile scope (copy_context carries the
HERMES_HOME override and the transport); the done event carries the requested
profile's status.
- Honest scope: `bot_desktop.auto_start` defaults to false (opt-in; Start lives
in the Screen pane); request_handoff no longer claims a Telegram/Discord
message was sent — the model relays the ask in its reply; docs match.
A bot running on a headless Linux gateway now gets its own desktop (TigerVNC
Xvnc + a minimal Xfce session, one per profile) that Hermes Desktop streams
live. The user can watch the bot work, take over to type a login / 2FA code /
CAPTCHA, and hand control back; the bot refuses every computer_use action
(screenshots included) while a human holds the screen, then resumes with the
session cookies the human just created.
Why this shape:
- The screen lives on the machine Hermes runs on, not in a vendor cloud browser,
so it works for any app the bot drives and keeps the session on the user's host.
- Xfce components are launched individually (xfwm4, xfce4-panel, xfdesktop,
xfsettingsd) under a private dbus session rather than xfce4-session/the
metapackage: no screensaver, power manager or polkit agent to lock or prompt
a headless desktop.
- Transport is raw RFB over a WebSocket beside /api/ws, authenticated with a
one-shot ticket minted through the already-authenticated RPC channel; noVNC
runs in the Electron renderer. The bridge parses the RFB client stream and
drops keyboard/pointer/clipboard (incl. QEMU Extended KeyEvent, which noVNC
switches to once Xvnc advertises it) from any viewer that does not hold the
lease, so viewOnly is enforced server-side, not by the client.
- One lease per profile (agent | human viewer) is the single truth for the RFB
bridge, the computer_use tool and the Desktop UI; taking control evicts other
viewers' input with close code 4000 control-taken.
- Auto-start happens only at the computer_use tool boundary (headless host,
packages present, bot_desktop.auto_start=true); env builders stay pure so
status probes and tests never spawn X servers. tests/tools/conftest.py pins
the binaries to "missing" for the same reason the browser-use fixture does.
Surfaces: Desktop (Bots → right-click → Open Screen; Take over / Hand back),
CLI (`hermes computer-use screen status|start|stop|install-deps`), tool
(`computer_use` actions request_handoff / wait_for_human), gateway RPCs
(display.status/start/stop/observe/lease.acquire/lease.release + display.lease
event), docs page user-guide/features/bot-screen.
Camofox VNC one-shot, computer-use aux-vision verdict, tirith binary path, MCP
discovery lock path, remote-backend probe text, learned image token costs,
auxiliary per-task semaphores and the custom-endpoint /models memo all held one
profile's config-derived value for the whole process. The skill-sync debounce
Timer ran with empty ContextVars, so a secondary's write pushed as the launch
profile (and cancelled its pending push).
Each memo is now keyed by hermes_home_key() (or credential fingerprint for the
per-key catalog) under an override; the timer is per home and runs its callback
inside the scheduling turn's copied context. Unscoped slots are unchanged.
The Sep 2026 decomposition (PR #102117) makes internal import paths a non-API: names now live in
the focused modules that define them. This commit is the ONLY thing keeping the old paths alive,
so external plugins have time to update. It is deliberately a single, unsquashed commit:
git revert <this sha>
removes every shim, stub and manifest at once on the announced date. Nothing in-tree may depend on
these pointers: scripts/check_compat_pointers.py (wired into lint.yml) fails CI if it does.
What it adds (see COMPAT_MANIFEST.md, compat_manifest.json):
- 332 facade modules get one delimited `PLUGIN-COMPAT` block appended at the end of the file
- 1,172 moved names resolved lazily via a module `__getattr__` (PEP 562) — never a top-level import,
so no import cycles; facades that already had `__getattr__` get a chained one
- 592 third-party/stdlib names the old modules used to expose, with their original import statements
- 266 public definitions that had been deleted as unused, restored byte-for-byte from the pre-decomposition
tree (+40 private helpers and 16 imports pulled in only because a restored definition needs them)
- 3 deleted modules recreated as re-export stubs (gateway/startup_watchdog, hermes_cli/observability/
relay_runtime, tools/environments/modal_utils)
- private names (`_x`) get no pointer: they were never API (3,792 skipped)
Verified: all 335 touched modules import under a fresh HERMES_HOME and every manifest name resolves;
the lint reports zero in-tree uses; ruff clean; targeted suites unchanged.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
main guarded scroll's x/y independently (coordinate=[null,100] -> y=100),
while click treats a coordinate without x as no point. The shared _xy()
applied click semantics to scroll; split out _scroll_xy() with main's
per-axis guard and pin both behaviors in a test.
