* feat(docker): publish nousresearch/hermes-sandbox:desktop for terminal backends
The terminal backends (docker, modal, daytona, singularity) all default to
nikolaik/python-nodejs:python3.11-nodejs20, a bare Python+Node base. For Bot
Screen, computer_use and the browser to run INSIDE that sandbox instead of on
the gateway host, the sandbox image needs the display stack.
docker/sandbox-desktop.Dockerfile is that base plus:
- the everyday tools it lacked (jq, ripgrep, fd, tmux, less, nano, vim,
zip, rsync, tree, procps, htop, sudo for the base's uid-1000 `pn`)
- the exact package set the Hermes -desktop image installs (TigerVNC,
Xfce components, dbus, xauth, fonts)
- Playwright's headed Chromium (same build as the -desktop image)
- cua-driver 0.28.2 from its pinned release tarball
No Hermes inside; the default user stays root like the base so nothing
changes for people who just switch docker_image. Desktop processes run as
`pn`. 4.27 GB on amd64.
docker.yml gains a `sandbox` variant with its own cache scope and repository
(nousresearch/hermes-sandbox:desktop, :main-desktop, :<release>-desktop);
the docker-integration suite is skipped for it (no Hermes to test) and
docker/sandbox-desktop-smoke.sh runs instead: as `pn`, every launcher and
cua-driver binary resolves, the real launcher.sh publishes :20, the RFB
socket completes the 3.8 handshake relayed over `docker exec -i` stdio, and
a headed Chromium maps a window on that display. hadolint lints the new
Dockerfile in docker-lint.yml.
* feat(docker): sandbox desktop base on python3.13-nodejs26
Matches the Hermes image (Python 3.13 / Node 26) and the top of requires-python;
the default docker_image tag it inherited was Python 3.11 / Node 20. Same pn
uid 1000, Debian 13; smoke (launcher, RFB relay, headed Chromium) passes.
* feat(docker): bake agent-browser into hermes-sandbox:desktop
The browser tools drive the agent-browser CLI; when the browser follows the
terminal backend that CLI has to exist inside the sandbox. Pinned to the same
^0.26.0 range the gateway resolves, --ignore-scripts like the gateway's npx path.
* feat(bot_desktop): the screen, computer_use and the browser follow the terminal backend
A user who sandboxes `terminal` (docker/ssh/singularity) had the agent's
screen, cua-driver and Chromium running on the gateway HOST beside that
sandbox: Bot Screen gave a headless host a display, the Xfce panel carries
xfce4-terminal, and `computer_use` could open a shell outside the boundary
the sandbox exists for.
Now the desktop lives where the terminal lives:
- tools/environments/streams.py: one primitive per spawn-per-call backend, the
local argv prefix that runs its remainder inside the sandbox with stdio open
(`docker exec -i`, `ssh`, `apptainer exec`). SDK backends (modal, daytona,
vercel) have none and report so.
- tools/bot_desktop/sandbox_host.py: launcher.sh runs inside the sandbox as
the image's `pn`; the pane's RFB bytes ride a 12-line python relay over that
prefix; `cua-driver mcp` is the prefix + the sandbox image's own driver.
- tools/bot_desktop/placement.py + `bot_desktop.placement` (auto|terminal|
gateway). `auto` follows the backend; a sandbox that cannot host a screen
REFUSES with the opt-in named instead of silently using the host.
- runtime.start/stop/status/published_env branch on placement; the pane,
lease, epoch fencing and CLI are unchanged.
- cua_backend: the MCP invocation is the sandbox one when the screen is
there; the host driver's runtime contract is irrelevant then; check_fn is
true under a terminal placement without a host binary.
- browser_tool_session: agent-browser invocations are wrapped in the prefix
with the daemon, socket dir and profile inside the sandbox; screenshots are
fetched back so MEDIA: paths keep working; recycle closes the sandbox
daemon.
- web_routers/display.py: the bridge pumps a relay's stdio when the screen is
in a sandbox, a unix socket otherwise.
Live on docker with nousresearch/hermes-sandbox:desktop: start/observe/RFB
handshake through the dashboard bridge, human takeover fences the agent
(HumanHasControl) and keystrokes reach the sandbox Xvnc, handback restores,
three start/stop rounds leave zero desktop processes; computer_use capture
and list_windows see only the sandbox's Xfce; browser_navigate/snapshot/
vision run with Chromium and agent-browser inside the container and zero
host processes on the bot profile; modal + auto refuses naming the opt-in.
* feat(desktop): Screen pane shows where a sandbox-placed screen runs; Install is host-only
DesktopStatus gains placement ('gateway' | 'terminal:<backend>'). A sandbox
image lacking the stack is a blocker naming hermes-sandbox:desktop, shown in
place of Start; install_command stays None there because the pane's Install
button runs the package manager on the gateway host, the wrong machine, and
display.install refuses for the same reason. The pane header carries
'Screen runs inside the docker sandbox, with the terminal' (4 locales).
* fix(bot_desktop): "is the screen in the sandbox" is a disk check on hot paths, never a config read
Every browser command and CUA spawn asked in_sandbox(), which resolves placement by
loading config, which initializes HERMES_HOME. Under a test's fake home that raised
HomeInitializationError from _run_browser_command; on a real host it read config per
click. Hot paths now ask sandbox_screen_running(): the start marker on disk, written
only by a sandbox start. Policy (in_sandbox) stays for start/install, where config is
the question. display.observe gates on "an RFB endpoint exists" for either placement.
* test(moa): late-accounting sink test asserts the wedged slot's row, not sink order
Under CI load the poll loop can see the interrupt before collecting the fast slot, so the
fast slot also arrives late and first; the test then failed on sink_calls[0]. The
contract is that the wedged slot's real usage reaches the sink.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* fix(bot_desktop): a sandbox that died under a live screen fails loudly, never falls to the host
sandbox_screen_running() drops a start marker whose terminal environment is no longer
registered (stale after a process restart). When the environment object outlives its
container, the browser's sandbox wrap now checks the published DISPLAY and raises
"the screen inside the terminal backend's sandbox is gone; start it again" instead of
KeyError('AGENT_BROWSER_PROFILE'). Live: fresh sandbox navigate ok; docker rm -f the
container; next navigate returns that error; zero host Chromium either way.
* feat(terminal): nousresearch/hermes-sandbox:desktop is the default container sandbox
Every container backend (docker, modal, daytona, singularity) now defaults to the
sandbox image with the desktop stack, so Bot Screen, computer_use and the browser
run inside the sandbox for everyone who never chose an image; Python 3.13 / Node 26
match the Hermes image. One constant (DEFAULT_SANDBOX_IMAGE) replaces six copies of
the old literal. Migration 47 moves saved configs still holding the OLD default and
never touches an image the user pinned. Docker reuse recreates a container built
from another image, or the flip would silently never take effect for anyone with a
persisted container (live: old container removed, new one on 3.13 / Node 26 with
Xvnc, cua-driver, agent-browser present).
* chore: retrigger CI (zero-job dispatch failure)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore(config): template stamps v47 and shows the new default sandbox image
The template is what install.sh / docker / doctor --fix seed; a stamp behind
DEFAULT_CONFIG makes every fresh install migrate on first run.
* feat(sandbox): the default image change is a decision, not a surprise
A persisted Docker sandbox on another image is kept when docker_image is unset;
only a written docker_image (an explicit pin) recreates it. The pin verdict
travels as TERMINAL_DOCKER_IMAGE_PINNED through both terminal bridges (process
env and per-profile scope) and the container-config allowlist.
Approval surfaces, all through hermes_cli.sandbox_image_switch: the interactive
CLI asks once at startup (y = pin the new image, n = pin the current one, Enter =
ask later); the Screen pane shows the same choice with Switch / Keep buttons via
display.switchSandboxImage; `hermes config set terminal.docker_image …` is the
same answer from any shell. Gateways and cron never decide: they keep the sandbox
and log the notice.
Migration 47 now unsets a saved image equal to the OLD default instead of
rewriting it to the new one — that value was the template copied, not a pin, and
rewriting it would have made the runtime recreate existing sandboxes unasked.
Modal restores its snapshot and Daytona reuses its labeled sandbox regardless of
the configured image, so existing sandboxes there were already untouched.
* fix(config): both plain defaults that preceded the desktop sandbox image are template copies
main pinned nikolaik/python-nodejs:python3.14-nodejs22 (cd0f97f833) without a migration;
a saved config holding either literal is unset by migration 47, so it follows the default
and existing sandboxes get the keep-or-switch decision instead of a silent recreate.
* ci(docker): build the sandbox image on release/dispatch, not every main push
Leaves docker.yml exactly as on main. The sandbox image carries no Hermes code,
so two 4 GB multi-arch builds per merge bought nothing. sandbox-image.yml builds
and smokes on a PR that edits its own Dockerfile/smoke, and publishes only on a
release or a manual dispatch with publish=true. Stable tag stays :desktop.
* fix(config): keep main's config.py/config_defaults.py edits under the sandbox-image delta
The rebase resolved both files wholesale with the branch side, dropping main's move to
hermes_yaml (the 3.14 runtime venv has no PyYAML) and the 3.14 base pin. This is main's
version plus exactly the branch's own changes: DEFAULT_SANDBOX_IMAGE, the pin verdict in the
env bridge, placement defaults and the v47 stamp.
* test: sandbox-image tests read config.yaml through hermes_yaml (no PyYAML on the 3.14 runtime)
* fix(bot_desktop): read the sandbox marker BOM-tolerantly (windows footgun lint)
* fix(bot_desktop): placement is the authority; sandbox screen survives restarts
Review findings on the sandbox-hosted Bot Screen, each reproduced live first.
Authority. The browser preflight and the CUA invocation keyed off screen
LIVENESS, so `placement: terminal` with the screen not yet up handed the tool an
unchanged host command. `runtime.tool_placement()` is now the one resolver:
terminal placement starts the sandbox screen on demand (no auto_start opt-in
inside the user's own sandbox), refused placement raises its reason, and neither
ever yields the host. placement.resolve() answers a local backend from env alone
so the common case costs no config load on the spawn path.
Restart. sandbox_screen_running() deleted the marker whenever the process-local
terminal registry was empty, i.e. after every gateway restart, while Xvnc kept
running in the container; stop() then returned False and left it. The marker
now records the owning container; liveness comes from `docker inspect` on it,
stop/status re-attach to the recorded owner (even after the placement setting
moved), and only a container that is gone drops the marker.
SSH. remote_argv emitted `bash -c <script>` as three words; OpenSSH joins them
and the remote login shell ran `bash -c export` and the rest itself. The script
travels as one quoted word for ssh (remote_command knows the backend); docker
and apptainer keep argv.
CDP reach. agent-browser inside the sandbox reports the sandbox's loopback;
the Browser Use harness, browser_exec and the vault supervisor connect from the
host and got connection refused. streams.forward_port() proxies a local port
over the exec stream (same relay as the RFB bridge) and the CDP URL is rewritten
to the local end.
pids limit. --pids-limit 256 counts threads; measured on the desktop image the
desktop stack is 44, one Chromium tab 212, the agent's browser with two tabs
488. Past the cap every further docker exec died with "procReady not received".
Default is 2048 with the measurements in the comment.
Replacement. An approved image switch force-removed the old container before
`docker run` tried the new image; a private tag or registry outage left nothing.
The image is inspected/pulled first and the old container kept on failure.
Desktop integration. The sandbox start never passed the dock's browser launcher
(no Browser icon) and the thumbnail needed a host launcher pid + host ImageGrab
(always None). The dock runs the sandbox's Playwright Chromium on the shared
profile; the thumbnail is grabbed inside the sandbox (Pillow baked into the
image). The browser profile moves from /tmp — a 512 MB tmpfs emptied on every
container stop — to the desktop user's home, so logins follow the container.
Pin provenance. A TERMINAL_DOCKER_IMAGE written in a routed profile's .env is a
pin even when it spells the default; the scope compared values before.
* docs(bot-screen): no literal tmp path in the profile-location note
* fix(bot_desktop): docker inspect liveness probe closes stdin (TUI subprocess guard)
* fix(bot_desktop): adopting a screen the sandbox kept records the marker
Live ssh probe: after the host's state was lost while the sandbox kept its
Xvnc, start() took the idempotent early return (display already published)
and never wrote the host marker, so status/thumbnail/stop lost the screen.
Record the adopted display like a fresh launch.
Docs: what an ssh host of your own must carry, and why a Dockerfile ENV is
not enough for a login session (PLAYWRIGHT_BROWSERS_PATH via /etc/environment).
* docker(sandbox-desktop): login sessions find the browser (PLAYWRIGHT_BROWSERS_PATH via /etc/environment)
* docs(bot-screen): what the Apptainer path inherits from the image and what it does not
* chore(config): sandbox-image migration is 47→48 (main took 47 for compression.threshold_tokens)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
The published image had no Xvnc/Xfce because nothing set the Dockerfile's
HERMES_BOT_DESKTOP argument, and a hosted instance (unprivileged, no sudo,
sealed /opt/hermes) cannot install at run time. The image layer is the only
delivery path.
- docker.yml: variant axis [slim, desktop]. :latest / :main / :v* stay the
image they are today; :latest-desktop / :main-desktop / :v*-desktop carry
the packages plus Playwright's headed Chromium. Slim owns the build cache
scope; one manifest per variant so a desktop publish failure never skips
slim's :latest.
- Dockerfile / stage2-hook.sh: XDG_RUNTIME_DIR=/tmp/hermes-runtime seeded
0700 as hermes (containers have no logind; the $HOME/.cache fallback was
the shared /opt/data volume), refused when foreign-owned; deterministic
Chromium discovery exporting the headless shell for ordinary browsing.
- bot_desktop: memory gate reads the cgroup working set (usage minus
inactive_file) so it cannot tighten over uptime and refuse to restart a
screen idle-stop just stopped; installable() gives three distinct dead-end
messages instead of a sudo line nobody there can run; env_for_agent
replaces a headless-shell pin so agent and dock share one Chromium.
Squash of IAvecilla/hermes-agent:bot-desktop-cloud-image (#112381, 13
commits), which GitHub auto-closed when its base branch merged as #108914.
Review fixes from pefontana (cache scope, per-variant merge, red browser
test) are included.
Co-authored-by: pefontana <pefontana@users.noreply.github.com>
Main now rejects literal /tmp paths in favour of the scratch-dir resolver. The
X11 protocol fixes .X<n>-lock and .X11-unix/X<n> under /tmp; these lines detect
or document that location, they do not pick scratch space.
_reap_orphaned_server only knew the orphan via /tmp/.X<n>-lock, so a launcher
SIGKILLed while a /tmp cleaner also took the lock (case B of #109941) — or a
failed start() that dropped <sd>/display — left one live Xvnc leaked per
occurrence; the allocator merely moved to the next number. When the lock names
nothing alive, the reaper now scans for the Xvnc whose command line binds this
profile's rfb.sock (the same ownership proof it already required), and skips the
lock unlink when no number is recorded.
_build_browser_env() is shared by the npx cache warmer (hermes update / doctor
--fix), the lazy Chromium auto-installer, the Lightpanda engine's env and
browser_use_cli. Hooking bot_desktop.auto_start there made every one of those
block up to 15 s spawning Xvnc+Xfce with browser.headed on, against
desktop_env()'s own "never starts anything" contract.
The hook now sits where a headed Chromium is actually spawned for a tool
action, mirroring computer_use dispatch: _spawn_and_collect (the first
agent-browser command forks the daemon; Lightpanda engine excluded), the Chrome
fallback from Lightpanda, and the real-profile Chrome launch. The regression
test asserts both halves: the env builder never starts the screen, the headed
Chromium spawn does, a headless or Lightpanda spawn does not.
_display_in_use() keyed only on /tmp/.X<n>-lock naming a live pid. A /tmp reaper
removes that lock while Xvnc keeps running; the allocator then handed the number
out, the new launcher died with "server already running", and because the
profile's recorded `display` file was never cleared, _pick_display() returned the
same number on every retry — the profile wedged until someone killed Xvnc by hand.
- _x_socket_bound(): parse /proc/net/unix for a bound (@)/tmp/.X11-unix/X<n>
socket, which the kernel holds for the server's whole life, and OR it into
_display_in_use(). Module-level _X_UNIX_TABLE so tests point it at a fixture.
- start(): a failed launch (launcher exit / readiness timeout) unlinks the
recorded `display` so the next attempt allocates afresh.
Live-verified with a real Xvnc :77: after `rm /tmp/.X77-lock` the base module
reported _display_in_use(77) False while Xvnc was alive; this head reports True,
and False again once Xvnc exits.
Closes#109941
Hosted instances come in 4 GB and 8 GB packages; a headed desktop plus the
browser a human opens during a takeover costs 1.1-1.5 GB on top of the
gateway (measured in the official image: gateway 304 MiB, +Xvnc/Xfce 520,
+Chromium on one page 1,073). Left unguarded, the loser of that squeeze is
Chromium mid-login or the gateway itself.
- runtime.start() refuses while the host, or its cgroup when tighter, has
less than bot_desktop.min_free_memory_mb free (default 1536); status()
carries `blocker` plus memory_available_mb/memory_limit_mb so the pane
shows the reason in place of Start and the CLI prints it. A running
screen is never reported blocked: the gate guards the allocation.
- A screen nobody used for bot_desktop.idle_stop_minutes (default 30) is
stopped by the gateway's display watcher and comes back on the next
use. "Use" = a computer_use dispatch, a browser/cua spawn onto the
screen, an attached viewer (stamped from the RFB bridge at most once a
minute), or start itself; a human holding the lease is never timed out.
Both knobs are 0-disableable; the contract gains the three status fields.
start() held _ALLOC_LOCK across the whole Xfce bring-up (up to wait_seconds), serializing every
profile's start behind one desktop launch and letting a hung launcher block them all. The lock
exists for the pick -> /tmp/.X<n>-lock window; it is dropped the moment the X lock appears.
The launcher is Popen'd by the gateway that started it; a gateway started later holds no Popen for it,
so after stop() the dead leader sat as a zombie in our table and killpg(pgid, 0) kept reporting the
group alive. waitpid(WNOHANG) on the leader during and after the kill sequence.
_kill_group_then_wait (runtime) and _kill_group (install) returned once the leader had exited, so a
TERM-ignoring descendant in the same group (a stuck dpkg, a display client) survived the KILL round
with the display or the dpkg lock. Both now poll killpg(pgid, 0) until nobody is left. The installer
also kills the group when the drain raises (a failing output sink), since the slot is released either way.
launcher.sh received the dock browser as one shell line and recovered the
executable with `${3%% *}`: a Chromium under '/opt/Google Chrome/' or a
profile dir under a HERMES_HOME with a space split at the first blank, the
existence check failed or Exec= became garbage, and the dock had no working
Browser icon.
Python now hands the launcher the bare executable (HERMES_BD_BROWSER_EXEC, for
the `command -v` check) and a ready-made Exec= line
(HERMES_BD_BROWSER_EXEC_LINE) built by `browser.dock_exec_line`: each argument
double-quoted, reserved characters backslash-escaped inside the quotes and the
backslashes string-escaped once more, per the Desktop Entry spec. `dock_argv`
is the single source of the dock's arguments (incl. the root sandbox flags).
Test: tests/tools/test_bot_desktop_browser.py — a spaced executable and a
spaced, quote-bearing profile dir produce a correctly quoted Exec= line
(AttributeError on the previous commit); the launcher seed test keeps running
the real script (`bash -n` clean).
`browser.executable()` always preferred Playwright's bundled Chromium. Its
`chrome_sandbox` is not setuid, so for a non-root user on Ubuntu 23.10+
(`kernel.apparmor_restrict_unprivileged_userns=1`) the dock's Browser icon died
with `FATAL: No usable sandbox!` even when a distro chromium with the sandbox
helper was installed. And when the only bundle is chromium_headless_shell (the
official image) `executable()` is None and launcher.sh silently skipped the
dock entry — nothing anywhere said "no headed browser".
Now: non-root under the userns restriction tries a system chrome/chromium first
and falls back to the Playwright build (no --no-sandbox for non-root, by
ruling: a loud failure beats a sandbox-less browser). `DesktopStatus.browser`
carries the resolved executable (None = no headed browser) through `as_dict()`
so the pane and `hermes computer-use screen status --json` can show it.
The root sandbox-bypass flags are ONE list, `browser_tool_session.
CHROMIUM_SANDBOX_BYPASS_ARGS`: agent-browser gets it via AGENT_BROWSER_ARGS and
the dock command appends the same flags as root, so the human's click and the
agent's launch start the same binary the same way. The sysctl reader is shared
as `apparmor_restricts_unprivileged_userns()`.
Tests: tests/tools/test_bot_desktop_browser.py — restricted non-root prefers
the system chromium and keeps Playwright's when alone (AttributeError on
bc36ddb5f9); root dock args are a superset of the agent's (dock lacked
--no-sandbox); status exposes browser / None (field missing).
Four small exposures, none a same-UID boundary (which the design does not claim), all
about other local users and wrong install hints:
- The host-wide display-allocation lock sat at a predictable name in world-writable
/tmp, where any local user could pre-create or squat it. It now lives under
XDG_RUNTIME_DIR (else ~/.cache), created 0700; still host-wide, outside every profile
home, because all profiles allocate from one band.
- lease.py hand-rolled mkdir + plain writes, so a takeover recorded before start() ever
ran left bot-desktop/ 0755 and lease.json / lease.lock 0644 under the umask — who
holds the screen and the lock the RFB bridge serialises on, readable by everyone. It
now uses secure_parent_dir and creates every file 0600 via an opener; the fcntl-less
(Windows) fallback is untouched.
- launcher.sh passed the X cookie on xauth's argv, visible in ps to other local UIDs.
The cookie is now fed on stdin through `xauth source -`.
- The Fedora map named tigervnc-server-minimal (only a Provides of tigervnc-x11-server,
which actually ships Xvnc) and dbus-x11 for dbus-run-session (dbus-daemon owns it;
dbus-x11 ships dbus-launch). Both BINARY_PACKAGES and PACKAGES are corrected together.
Two dead ends on hosts without a sudo binary. As root (the official Docker
image is uid 0, no sudo): `runtime.install_command()` hardcoded `sudo <pm>`,
`_run` asserted argv[0] == "sudo", `sudo -n true` failed, so the password card
was raised for a user who already IS root and the install was cancelled. As an
unprivileged user in a minimal container the same card was raised although no
password could ever satisfy it.
`install_command()` now omits `sudo` when euid is 0 (one builder, so the pane's
shown command and the executed command stay the same line); `_run` runs the
package manager directly when the command carries no sudo, and when it does but
`shutil.which("sudo")` is None it streams the exact command to run on the host
as root and returns `install.NO_SUDO` instead of asking. `hermes computer-use
screen install` prints that command.
Tests: tests/tools/test_bot_desktop_install.py — euid 0: ask_password is never
called and argv[0] is the package manager; euid 1000 + no sudo: NO_SUDO, no
Popen, the apt-get line is in the stream (both raised the password card on
bc36ddb5f9).
_spawn_and_wait raised on the readiness deadline and walked away from the launcher it
had just spawned. Xvnc and Xfce kept coming up; a moment later the launcher published
DISPLAY and rfb.sock, and a screen whose start() had reported failure was now "running"
under a pid the caller never learned about. A retry then saw it as already up and returned
it, so the failure message described a state that no longer existed.
On timeout the launch's process group (the launcher is its own session leader, so the
group is exactly this launch) gets SIGTERM, then SIGKILL after a grace period, the child
is reaped, and only that launch's launcher.pid / env / rfb.sock are removed before the
error propagates.
status() and stop() keyed only on the launcher pid. When the launcher was SIGKILLed
(OOM, a stray kill, a crashed supervisor) its Xvnc kept running in the launcher's
session, holding the display number and rfb.sock, while status() reported stopped.
The next start() then allocated a fresh display and spawned a second server whose
launcher unlinked and re-bound the same rfb.sock path — two X servers, one socket,
and the orphan leaked forever because stop() never reached it.
The X lock of the recorded display names that server. When no live launcher exists,
start() and stop() now check it: a process in the dead launcher's process group, or
an Xvnc whose command line binds THIS profile's rfb.sock, is ours and is killed
(SIGTERM, then SIGKILL) before the stale state is dropped and a new launch proceeds.
A server that merely reused our display number is never touched. _pid_alive also
stops treating a zombie as alive: the gateway reaps a killed launcher lazily, and the
zombie made status() claim a running screen whose server nobody owned.
The dock's Browser launched raw Chromium on the shared user-data-dir with no automation
endpoint. When the human opened it first and handed back, agent-browser's own launch was
forwarded into their instance by Chromium's ProcessSingleton and exited 21 without a
DevToolsActivePort, so every browser_navigate failed until the human closed their window.
The reverse order worked, which is why it slipped through.
- The dock command carries --remote-debugging-port=0, so a human-started instance advertises a
port in <user-data-dir>/DevToolsActivePort (browser.dock_command; runtime.start reads it).
- browser.running_instance_cdp_port() trusts that file only when SingletonLock's pid is alive
AND the port accepts a connection (both files outlive a closed Chromium), and never for the
instance the calling agent-browser session launched itself: handing that daemon --cdp makes
it treat the launch as a config change, close its browser and attach to the port that just
died with it (seen live).
- The local argv builder appends --cdp <port> to the --session launch when such an instance
exists, so the same daemon (and its snapshot refs) drives the human's window.
Live, HERMES_HOME=/tmp/bs-f-home on this host: human-first — dock instance up, navigate x2
succeeded, one Chromium main process (same pid) throughout; agent-first — navigate, dock
click, navigate x2 succeeded, one process throughout. Before the fix human-first returned
"Chrome exited early (exit code: 21) ... Failed to create SingletonLock".
(cherry picked from commit d732fad0af005ffd38eca153dd29c0f7e9dd6bc7)
The log was opened in append mode and nothing rotated it; a long-lived
gateway restarting its screen grew it without bound and the tail
start() reports on failure mixed launches. Each start now owns the file.
(cherry picked from commit ea1e3ac737b3d0b76afc5ec5373c5d3248207820)
_allocate_display released the host-wide flock as soon as it picked a
number, but Xvnc writes /tmp/.X<n>-lock only once it is up. Two profiles
cold-starting together both picked n; the loser's Xvnc failed and its
launcher's stale-lock cleanup could unlink the winner's socket. There was
also no per-profile lock at all: two start() calls for one profile
spawned two launchers, the second overwrote launcher.pid and the first was
orphaned.
start() now takes <state_dir>/start.lock (per profile) around the
running-check, and the allocation lock around pick + spawn + publish
wait (bounded by wait_seconds, default 15s, so a stuck launcher cannot
wedge other profiles for long). stop() takes the same per-profile lock so
a stop cannot interleave with a start. The launcher does not inherit the
lock fds (close_fds=True), so the locks fall with the caller.
(cherry picked from commit 1eee1a331e7ac6a8a2860dd7ff2df0150fe12a8b)
pid_exists alone trusts a recycled pid: after a reboot or a long-lived
gateway, an unrelated process can own the recorded number and status()
reports the screen running while stop() SIGTERMs that stranger's whole
process group. The pid file now stores "<pid> <psutil create_time>" and
_launcher_pid() requires both to match; the pre-identity single-number
format, a dead pid and an out-of-range digit string all read as not
running (the safe direction: worst case is a spurious start()).
(cherry picked from commit 2f9e0e40a764f5a6bb38c02194081dc34c4c8f0b)
Fedora split xorg-x11-server-utils and xorg-x11-utils into per-binary
packages (F35) and retired the umbrellas; dnf5 refuses the whole
transaction on one unknown name, so `hermes computer-use screen install`
was a no-op on Fedora. The dnf list now names xsetroot/xset/xdpyinfo/
xprop/setxkbmap directly. xprop (the launcher's WM-ready wait) joins
REQUIRED_BINARIES and every distro list; apt gains x11-xkb-utils
(setxkbmap) and pacman gains dbus (dbus-run-session), both previously
reached only via transitive deps. BINARY_PACKAGES documents which package
ships each binary per manager so a test can hold the lists to it.
(cherry picked from commit dc5c4d70f7a0fd355fe21c96071513d8b5ae3744)
Independent review of the PR head found the control boundary only held inside
one process and several claims the code did not back. Each item below was
reproduced, fixed, covered by an invariant test proven red without the fix, and
re-verified live on a real Xvnc/Xfce screen.
- Lease authority on disk. `lease.json` under an fcntl lock in the profile's
bot-desktop dir; every read goes to the file. `hermes serve` (viewer bridge),
the messaging gateway, a CLI turn and isolated workers now agree. Live: a
takeover in process A made `computer_use capture` in process B return
human_has_control; release in C made B work again.
- Takeover fences admitted actions. `handle_computer_use` re-checks the lease
under the dispatch lock and discards a result produced after the lease epoch
changed, so an action admitted before a takeover cannot picture what the human
typed during approval / backend start-up waits.
- Sudo reply pinned to its origin. `SudoRequest.origin` records the
(connection, profile) the card came from; SudoDialog answers through
`requestGatewayForAgent` on that socket, never the foreground gateway. A
password typed for host A can no longer reach host B. `sudo.expire` and
`display.install.sudo.expire` now tear the card down (the Desktop never
handled sudo.expire).
- Dock Browser IS the bot's browser. `tools/bot_desktop/browser.py` resolves one
identity — the Chromium agent-browser drives + a persistent per-profile
user-data-dir (`bot-desktop/browser-profile`) — and both sides use it: the
agent env gets AGENT_BROWSER_EXECUTABLE_PATH / AGENT_BROWSER_PROFILE, the
dock launcher gets the same exe + --user-data-dir. Live: the bot wrote
localStorage on http://127.0.0.1:8765 through agent-browser; a dock click and
a typed URL on the screen showed BOT-WROTE-THIS in Chrome for Testing.
- Safe display reuse. Allocation under a host-wide lock; a recorded number is
reused only when no live server holds it; the launcher never unlinks a lock
whose pid is alive. Live: A stopped, B took :20, A restarted on :21, B kept
running.
- Install worker keeps the caller's profile scope (copy_context carries the
HERMES_HOME override and the transport); the done event carries the requested
profile's status.
- Honest scope: `bot_desktop.auto_start` defaults to false (opt-in; Start lives
in the Screen pane); request_handoff no longer claims a Telegram/Discord
message was sent — the model relays the ask in its reply; docs match.
A bot running on a headless Linux gateway now gets its own desktop (TigerVNC
Xvnc + a minimal Xfce session, one per profile) that Hermes Desktop streams
live. The user can watch the bot work, take over to type a login / 2FA code /
CAPTCHA, and hand control back; the bot refuses every computer_use action
(screenshots included) while a human holds the screen, then resumes with the
session cookies the human just created.
Why this shape:
- The screen lives on the machine Hermes runs on, not in a vendor cloud browser,
so it works for any app the bot drives and keeps the session on the user's host.
- Xfce components are launched individually (xfwm4, xfce4-panel, xfdesktop,
xfsettingsd) under a private dbus session rather than xfce4-session/the
metapackage: no screensaver, power manager or polkit agent to lock or prompt
a headless desktop.
- Transport is raw RFB over a WebSocket beside /api/ws, authenticated with a
one-shot ticket minted through the already-authenticated RPC channel; noVNC
runs in the Electron renderer. The bridge parses the RFB client stream and
drops keyboard/pointer/clipboard (incl. QEMU Extended KeyEvent, which noVNC
switches to once Xvnc advertises it) from any viewer that does not hold the
lease, so viewOnly is enforced server-side, not by the client.
- One lease per profile (agent | human viewer) is the single truth for the RFB
bridge, the computer_use tool and the Desktop UI; taking control evicts other
viewers' input with close code 4000 control-taken.
- Auto-start happens only at the computer_use tool boundary (headless host,
packages present, bot_desktop.auto_start=true); env builders stay pure so
status probes and tests never spawn X servers. tests/tools/conftest.py pins
the binaries to "missing" for the same reason the browser-use fixture does.
Surfaces: Desktop (Bots → right-click → Open Screen; Take over / Hand back),
CLI (`hermes computer-use screen status|start|stop|install-deps`), tool
(`computer_use` actions request_handoff / wait_for_human), gateway RPCs
(display.status/start/stop/observe/lease.acquire/lease.release + display.lease
event), docs page user-guide/features/bot-screen.
