Five opt-in counters that tune the agent loop itself, recorded through
record_process_mark (disabled config => zero rows, zero files):
- hermes.file_edit.count: tool, mode, outcome, match_strategy. The fuzzy
matcher reports the strategy that landed (or no_match / ambiguous) into a
context-local probe opened only around the patch/write_file handlers, so we
learn which strategies earn their keep without counting internal callers.
- hermes.loop_guard.count: provider, model, signal, detector. Hooked where
the guardrail already warns/blocks/halts and at turn end for the iteration
budget; latched once per turn per (signal, detector).
- hermes.tool_recovery.count: provider, model, tool, next_tool,
next_outcome. One row per failed tool call, resolved against the model's
next round (same tool first) or the turn end (no_tool_call / gave_up).
- hermes.terminal.outcome.count: backend, command_kind, outcome. The kind is
a table lookup on the first program word, never the text. Hermes' own
deadline/interrupt now set hermes_timed_out / hermes_interrupted on the
backend result, so a command's own `exit 124` reads nonzero, not timeout.
Disjoint dims from hermes.execution_backend.count (served vs not).
- hermes.model_reply_issue.count: provider, model, issue. Refusal and
truncation only from the structured finish reason; empty/reasoning_only
from the normalized message; issue=none per response as the denominator
(model_route counts attempts and files empty replies as failures).
Background review / curator loops and _host_local / unmetered terminal
calls never count. Helpers live in shared_metrics_harness.py; call sites
are one or two lines each.
* feat(vercel): start fresh sandboxes from a managed image instead of the deprecated runtime
Vercel deprecated Sandbox runtimes (node24/node22/python3.13) in Aug 2026 in favour of
images, and rejects runtime+image together and runtime with a snapshot source. New
terminal.vercel_image (default vercel/sandbox/universal:latest, Node 24 + Python 3.14)
picks the image for fresh sandboxes; a pinned terminal.vercel_runtime still works, wins
over the image and logs a deprecation warning; snapshot restores send neither.
Setup wizard prompts for the image, dashboard exposes both keys, status/config show the
effective choice, TERMINAL_VERCEL_IMAGE bridges config to the tool like its siblings.
* feat(config): migration 49 drops the seeded node24 Vercel runtime pin
Every pre-49 config.yaml carries terminal.vercel_runtime: node24 (the template default) and
the setup wizard mirrored it into .env as TERMINAL_VERCEL_RUNTIME. Both are the default
copied, not a choice, so the migration drops them and fresh sandboxes follow vercel_image;
node22 / python3.13 pins are the user's and survive. Persisted sandboxes are unaffected:
a snapshot restore never sends a runtime or an image.
* fix(modal): keep persistent-sandbox snapshots past the SDK's 30-day TTL
modal>=1.5 gives Sandbox.snapshot_filesystem() a default ttl of 30 days, so an idle
persistent Modal sandbox silently lost its filesystem and restarted from the base image.
Pass ttl=None (retain until deleted) and bump the modal extra from 1.3.4 (no ttl
parameter; legacy RPC) to 1.5.5 so the kwarg exists on every install.
* fix(modal): drop the dead modal.Mount credential-mount block
modal.Mount left the public API in modal 1.0, so _modal.Mount.from_local_file raised
AttributeError into the surrounding except on every sandbox start and the block never
mounted anything. The FileSyncManager created right after already uploads the same
credential, skills and cache files (iter_sync_files), so delete the duplicate; the test
fake stops exporting a Mount the real SDK does not have.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
MAJOR: `hermes -p X gateway restart --all` with no installed service re-entered
`run_gateway(replace=True)` IN the CLI process. `_home_env` swapped only HERMES_HOME
while os.environ already held X's dotenv (loaded at CLI startup) and gateway/run.py's
root .env load does not clear inherited keys, so the default gateway came up with X's
TELEGRAM_BOT_TOKEN / OPENAI_API_KEY. And any child spawned inside the swap (Windows
`start --all`, the launchd/detached fallback, migrate's secondary spawns) read the
swapped home as the launch profile, so `served_profile_child_env` neither stripped nor
scrubbed X's env.
- `_home_env` pins the process's launch home for the swap (`pin_process_hermes_home`),
so routed-home decisions inside it keep the real identity; `strip_launch_profile_env`
reads the residue list from that pinned home too.
- `_restart_all_as_host` spawns the root detached (`_spawn_detached_gateway`, scrubbed
base env + the root's own secret scope, `gateway run --replace`) when the CLI runs
under a named profile; the default profile keeps its foreground run. The Desktop
update hand-off's `gateway start --all` under a named active profile is still served.
- test: `test_named_profile_restart_all_with_host_down_restarts_the_default_root` now
asserts the child env carries none of the named profile's keys and `run_gateway` is
never entered (red on the PR head: "the root ran IN the named profile's process").
MINORS:
- (a) `gateway_migrate._service_op('enable')` raises on a non-zero `systemctl enable`;
a fresh apply treats it as a preflight refusal before the flag/manifest write, a
resume reports it and converges. New `test_migrate_refuses_when_the_survivor_cannot_
be_enabled` (red on head: exit 0 with the ⚠ line).
- (b) `systemd_install` already-current branch warns when `systemctl enable` fails.
- (c) `remove_system_systemd_unit` reports a failed `systemctl stop` instead of ✓.
* feat(docker): publish nousresearch/hermes-sandbox:desktop for terminal backends
The terminal backends (docker, modal, daytona, singularity) all default to
nikolaik/python-nodejs:python3.11-nodejs20, a bare Python+Node base. For Bot
Screen, computer_use and the browser to run INSIDE that sandbox instead of on
the gateway host, the sandbox image needs the display stack.
docker/sandbox-desktop.Dockerfile is that base plus:
- the everyday tools it lacked (jq, ripgrep, fd, tmux, less, nano, vim,
zip, rsync, tree, procps, htop, sudo for the base's uid-1000 `pn`)
- the exact package set the Hermes -desktop image installs (TigerVNC,
Xfce components, dbus, xauth, fonts)
- Playwright's headed Chromium (same build as the -desktop image)
- cua-driver 0.28.2 from its pinned release tarball
No Hermes inside; the default user stays root like the base so nothing
changes for people who just switch docker_image. Desktop processes run as
`pn`. 4.27 GB on amd64.
docker.yml gains a `sandbox` variant with its own cache scope and repository
(nousresearch/hermes-sandbox:desktop, :main-desktop, :<release>-desktop);
the docker-integration suite is skipped for it (no Hermes to test) and
docker/sandbox-desktop-smoke.sh runs instead: as `pn`, every launcher and
cua-driver binary resolves, the real launcher.sh publishes :20, the RFB
socket completes the 3.8 handshake relayed over `docker exec -i` stdio, and
a headed Chromium maps a window on that display. hadolint lints the new
Dockerfile in docker-lint.yml.
* feat(docker): sandbox desktop base on python3.13-nodejs26
Matches the Hermes image (Python 3.13 / Node 26) and the top of requires-python;
the default docker_image tag it inherited was Python 3.11 / Node 20. Same pn
uid 1000, Debian 13; smoke (launcher, RFB relay, headed Chromium) passes.
* feat(docker): bake agent-browser into hermes-sandbox:desktop
The browser tools drive the agent-browser CLI; when the browser follows the
terminal backend that CLI has to exist inside the sandbox. Pinned to the same
^0.26.0 range the gateway resolves, --ignore-scripts like the gateway's npx path.
* feat(bot_desktop): the screen, computer_use and the browser follow the terminal backend
A user who sandboxes `terminal` (docker/ssh/singularity) had the agent's
screen, cua-driver and Chromium running on the gateway HOST beside that
sandbox: Bot Screen gave a headless host a display, the Xfce panel carries
xfce4-terminal, and `computer_use` could open a shell outside the boundary
the sandbox exists for.
Now the desktop lives where the terminal lives:
- tools/environments/streams.py: one primitive per spawn-per-call backend, the
local argv prefix that runs its remainder inside the sandbox with stdio open
(`docker exec -i`, `ssh`, `apptainer exec`). SDK backends (modal, daytona,
vercel) have none and report so.
- tools/bot_desktop/sandbox_host.py: launcher.sh runs inside the sandbox as
the image's `pn`; the pane's RFB bytes ride a 12-line python relay over that
prefix; `cua-driver mcp` is the prefix + the sandbox image's own driver.
- tools/bot_desktop/placement.py + `bot_desktop.placement` (auto|terminal|
gateway). `auto` follows the backend; a sandbox that cannot host a screen
REFUSES with the opt-in named instead of silently using the host.
- runtime.start/stop/status/published_env branch on placement; the pane,
lease, epoch fencing and CLI are unchanged.
- cua_backend: the MCP invocation is the sandbox one when the screen is
there; the host driver's runtime contract is irrelevant then; check_fn is
true under a terminal placement without a host binary.
- browser_tool_session: agent-browser invocations are wrapped in the prefix
with the daemon, socket dir and profile inside the sandbox; screenshots are
fetched back so MEDIA: paths keep working; recycle closes the sandbox
daemon.
- web_routers/display.py: the bridge pumps a relay's stdio when the screen is
in a sandbox, a unix socket otherwise.
Live on docker with nousresearch/hermes-sandbox:desktop: start/observe/RFB
handshake through the dashboard bridge, human takeover fences the agent
(HumanHasControl) and keystrokes reach the sandbox Xvnc, handback restores,
three start/stop rounds leave zero desktop processes; computer_use capture
and list_windows see only the sandbox's Xfce; browser_navigate/snapshot/
vision run with Chromium and agent-browser inside the container and zero
host processes on the bot profile; modal + auto refuses naming the opt-in.
* feat(desktop): Screen pane shows where a sandbox-placed screen runs; Install is host-only
DesktopStatus gains placement ('gateway' | 'terminal:<backend>'). A sandbox
image lacking the stack is a blocker naming hermes-sandbox:desktop, shown in
place of Start; install_command stays None there because the pane's Install
button runs the package manager on the gateway host, the wrong machine, and
display.install refuses for the same reason. The pane header carries
'Screen runs inside the docker sandbox, with the terminal' (4 locales).
* fix(bot_desktop): "is the screen in the sandbox" is a disk check on hot paths, never a config read
Every browser command and CUA spawn asked in_sandbox(), which resolves placement by
loading config, which initializes HERMES_HOME. Under a test's fake home that raised
HomeInitializationError from _run_browser_command; on a real host it read config per
click. Hot paths now ask sandbox_screen_running(): the start marker on disk, written
only by a sandbox start. Policy (in_sandbox) stays for start/install, where config is
the question. display.observe gates on "an RFB endpoint exists" for either placement.
* test(moa): late-accounting sink test asserts the wedged slot's row, not sink order
Under CI load the poll loop can see the interrupt before collecting the fast slot, so the
fast slot also arrives late and first; the test then failed on sink_calls[0]. The
contract is that the wedged slot's real usage reaches the sink.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* fix(bot_desktop): a sandbox that died under a live screen fails loudly, never falls to the host
sandbox_screen_running() drops a start marker whose terminal environment is no longer
registered (stale after a process restart). When the environment object outlives its
container, the browser's sandbox wrap now checks the published DISPLAY and raises
"the screen inside the terminal backend's sandbox is gone; start it again" instead of
KeyError('AGENT_BROWSER_PROFILE'). Live: fresh sandbox navigate ok; docker rm -f the
container; next navigate returns that error; zero host Chromium either way.
* feat(terminal): nousresearch/hermes-sandbox:desktop is the default container sandbox
Every container backend (docker, modal, daytona, singularity) now defaults to the
sandbox image with the desktop stack, so Bot Screen, computer_use and the browser
run inside the sandbox for everyone who never chose an image; Python 3.13 / Node 26
match the Hermes image. One constant (DEFAULT_SANDBOX_IMAGE) replaces six copies of
the old literal. Migration 47 moves saved configs still holding the OLD default and
never touches an image the user pinned. Docker reuse recreates a container built
from another image, or the flip would silently never take effect for anyone with a
persisted container (live: old container removed, new one on 3.13 / Node 26 with
Xvnc, cua-driver, agent-browser present).
* chore: retrigger CI (zero-job dispatch failure)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore(config): template stamps v47 and shows the new default sandbox image
The template is what install.sh / docker / doctor --fix seed; a stamp behind
DEFAULT_CONFIG makes every fresh install migrate on first run.
* feat(sandbox): the default image change is a decision, not a surprise
A persisted Docker sandbox on another image is kept when docker_image is unset;
only a written docker_image (an explicit pin) recreates it. The pin verdict
travels as TERMINAL_DOCKER_IMAGE_PINNED through both terminal bridges (process
env and per-profile scope) and the container-config allowlist.
Approval surfaces, all through hermes_cli.sandbox_image_switch: the interactive
CLI asks once at startup (y = pin the new image, n = pin the current one, Enter =
ask later); the Screen pane shows the same choice with Switch / Keep buttons via
display.switchSandboxImage; `hermes config set terminal.docker_image …` is the
same answer from any shell. Gateways and cron never decide: they keep the sandbox
and log the notice.
Migration 47 now unsets a saved image equal to the OLD default instead of
rewriting it to the new one — that value was the template copied, not a pin, and
rewriting it would have made the runtime recreate existing sandboxes unasked.
Modal restores its snapshot and Daytona reuses its labeled sandbox regardless of
the configured image, so existing sandboxes there were already untouched.
* fix(config): both plain defaults that preceded the desktop sandbox image are template copies
main pinned nikolaik/python-nodejs:python3.14-nodejs22 (cd0f97f833) without a migration;
a saved config holding either literal is unset by migration 47, so it follows the default
and existing sandboxes get the keep-or-switch decision instead of a silent recreate.
* ci(docker): build the sandbox image on release/dispatch, not every main push
Leaves docker.yml exactly as on main. The sandbox image carries no Hermes code,
so two 4 GB multi-arch builds per merge bought nothing. sandbox-image.yml builds
and smokes on a PR that edits its own Dockerfile/smoke, and publishes only on a
release or a manual dispatch with publish=true. Stable tag stays :desktop.
* fix(config): keep main's config.py/config_defaults.py edits under the sandbox-image delta
The rebase resolved both files wholesale with the branch side, dropping main's move to
hermes_yaml (the 3.14 runtime venv has no PyYAML) and the 3.14 base pin. This is main's
version plus exactly the branch's own changes: DEFAULT_SANDBOX_IMAGE, the pin verdict in the
env bridge, placement defaults and the v47 stamp.
* test: sandbox-image tests read config.yaml through hermes_yaml (no PyYAML on the 3.14 runtime)
* fix(bot_desktop): read the sandbox marker BOM-tolerantly (windows footgun lint)
* fix(bot_desktop): placement is the authority; sandbox screen survives restarts
Review findings on the sandbox-hosted Bot Screen, each reproduced live first.
Authority. The browser preflight and the CUA invocation keyed off screen
LIVENESS, so `placement: terminal` with the screen not yet up handed the tool an
unchanged host command. `runtime.tool_placement()` is now the one resolver:
terminal placement starts the sandbox screen on demand (no auto_start opt-in
inside the user's own sandbox), refused placement raises its reason, and neither
ever yields the host. placement.resolve() answers a local backend from env alone
so the common case costs no config load on the spawn path.
Restart. sandbox_screen_running() deleted the marker whenever the process-local
terminal registry was empty, i.e. after every gateway restart, while Xvnc kept
running in the container; stop() then returned False and left it. The marker
now records the owning container; liveness comes from `docker inspect` on it,
stop/status re-attach to the recorded owner (even after the placement setting
moved), and only a container that is gone drops the marker.
SSH. remote_argv emitted `bash -c <script>` as three words; OpenSSH joins them
and the remote login shell ran `bash -c export` and the rest itself. The script
travels as one quoted word for ssh (remote_command knows the backend); docker
and apptainer keep argv.
CDP reach. agent-browser inside the sandbox reports the sandbox's loopback;
the Browser Use harness, browser_exec and the vault supervisor connect from the
host and got connection refused. streams.forward_port() proxies a local port
over the exec stream (same relay as the RFB bridge) and the CDP URL is rewritten
to the local end.
pids limit. --pids-limit 256 counts threads; measured on the desktop image the
desktop stack is 44, one Chromium tab 212, the agent's browser with two tabs
488. Past the cap every further docker exec died with "procReady not received".
Default is 2048 with the measurements in the comment.
Replacement. An approved image switch force-removed the old container before
`docker run` tried the new image; a private tag or registry outage left nothing.
The image is inspected/pulled first and the old container kept on failure.
Desktop integration. The sandbox start never passed the dock's browser launcher
(no Browser icon) and the thumbnail needed a host launcher pid + host ImageGrab
(always None). The dock runs the sandbox's Playwright Chromium on the shared
profile; the thumbnail is grabbed inside the sandbox (Pillow baked into the
image). The browser profile moves from /tmp — a 512 MB tmpfs emptied on every
container stop — to the desktop user's home, so logins follow the container.
Pin provenance. A TERMINAL_DOCKER_IMAGE written in a routed profile's .env is a
pin even when it spells the default; the scope compared values before.
* docs(bot-screen): no literal tmp path in the profile-location note
* fix(bot_desktop): docker inspect liveness probe closes stdin (TUI subprocess guard)
* fix(bot_desktop): adopting a screen the sandbox kept records the marker
Live ssh probe: after the host's state was lost while the sandbox kept its
Xvnc, start() took the idempotent early return (display already published)
and never wrote the host marker, so status/thumbnail/stop lost the screen.
Record the adopted display like a fresh launch.
Docs: what an ssh host of your own must carry, and why a Dockerfile ENV is
not enough for a login session (PLAYWRIGHT_BROWSERS_PATH via /etc/environment).
* docker(sandbox-desktop): login sessions find the browser (PLAYWRIGHT_BROWSERS_PATH via /etc/environment)
* docs(bot-screen): what the Apptainer path inherits from the image and what it does not
* chore(config): sandbox-image migration is 47→48 (main took 47 for compression.threshold_tokens)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
A volume that already owns /workspace skipped the configured working
directory, so tools treated that host path as unmounted. Bind it at a
second mount, or point tools at the volume that already has it, for any
drive path.
On shared remote backends the execute_code channel created kernel and
sandbox dirs under shared temp at the process umask (775 group-writable
under umask 002), wrote request/result files group-readable, and carried
HERMES_RPC_TOKEN on remote command lines where co-tenant users read argv
via ps for the whole run. A co-tenant could read tool arguments and
results, and on group-writable dirs forge RPC requests dispatched under
the user's approval context.
- All remote dirs are created owner-only (umask 077 + chmod 700, checked
fail-closed) and every Hermes file write is mode 600.
- The token travels in a sourced env file inside a subshell so the vars
never enter the backend's session-snapshot dump, and ships via stdin on
pipe-capable backends so it never enters argv at all.
- The RPC poll loop rejects non-int seq requests before dispatch instead
of replaying them every cycle.
- tool_result_storage gets the same owner-only treatment for archived
tool output.
(cherry picked from commit aef21731d7fb8a4e0a6ada4ff9889264df4a8893)
execute() can pass stdin_data="" (e.g. write_file of empty content). The
`is not None` guard then paid an upload (plus chmod on Daytona) and a
longer shell command just to feed zero bytes. Base heredoc mode skipped
empty stdin, and neither SDK exec attaches a stdin, so treating "" as no
stdin gives exactly the base command (checked: identical argv, no upload).
state["staged"] was set only after the upload returned. A kill() during
the upload therefore saw nothing to scrub, and exec_fn then hit the
cancelled gate and returned 130 without deleting. The staged file (which
can hold the sudo password line) stayed in the sandbox, whose filesystem
persists by default on both Daytona and Vercel.
Factor the scrub into a lock-held helper and call it from exec_fn's
cancelled branch as well as from cancel(). Also skip the upload entirely
when cancel already won before it started.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
Daytona and Vercel built the staged stdin path and the
`exec 0< f || exit $?; rm -f -- f || exit $?` prefix byte for byte the
same. That prefix is the security handoff (the shell takes ownership of the
payload and unlinks it before the user command runs), so it should live in
one place: two small BaseEnvironment helpers next to _embed_stdin_heredoc.
Upload and cancel lifecycle stay per-backend.
Also document the "payload" _stdin_mode; the old comment still claimed
Modal/Daytona use heredoc, which no built-in backend does now.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
The pinned SDK (daytona 0.155.0) upload_file() accepts bytes, so the host
NamedTemporaryFile round-trip was redundant and briefly wrote the merged
stdin (which can start with the sudo password line) to the host disk.
delete_file() is delete_file(path, recursive=False); the extra
request_timeout kwarg raised TypeError inside contextlib.suppress, so the
pre-dispatch cancel scrub silently never ran and the staged payload stayed
in the sandbox /tmp.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
The script already rm-s the staged file before running the user command,
so the success-path delete_file was a wasted round-trip, and post-cancel
deletes ran against a stopped sandbox. Only delete on kill() when the file
was uploaded but exec was never dispatched (checked under the env lock,
bounded request_timeout); exec_fn skips dispatch once cancelled.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
#122218 stopped the sandbox on ANY exec exception (killing background
processes on a transient SDK error, even for stdin-less commands) and
re-stopped/rm-ed on every cancel, failing the existing cancel test.
Track staged/dispatched under the env lock: kill() overwrites the staged
file only if it was uploaded but never dispatched, then stops once as
before. After dispatch the user shell opens and unlinks it itself, so no
extra command runs. exec_fn skips dispatch if cancel already won.
Co-authored-by: JoaoMarcos44 <joaomarcosdias444@gmail.com>
Follow-up to the ssh path fix salvaged from #121686.
- Read the ssh anchor raw (session record, override, TERMINAL_CWD): the
shared workspace-root helper expands ~ on the Hermes host, so
TERMINAL_CWD='~/proj' still resolved into the container home.
- Resolve ~ to the remote home the SSH environment detects at connect,
bringing the environment up through the file tools' own creator
(_get_file_ops, same cwd and cache) when none is live. A failed bring-up
is remembered per container for 30s so one call's several resolutions
don't each retry; a live environment is always used first. SSHEnvironment
now records whether the home was detected, and a guessed /home/<user>
(echo $HOME failed) is not used. Results are
absolute and stable from the first call (read tracking and staleness
checks key on them), and '..' normalizes to the real target: relative
traversal like ../../../etc/x from ~ was refused on main and slipped past
the sensitive-path guard on the PR head.
- If the remote home cannot be detected, an ssh ~-path that climbs above ~
cannot be classified; the write guard refuses it.
- ~user passes through for the remote shell instead of becoming ~/~user.
- coerce_ssh_remote_cwd maps paths under the host subprocess home onto ~/,
except when that home is the OS user's real home.
- The outside-workspace warning compares in the remote namespace (it fired
on every correct relative write when the anchor was ~).
- The backend type is looked up once per resolution again (the PR head did three per local path).
A heredoc body always ends in a newline, so on the heredoc-stdin backends
(Modal, Daytona, Vercel) every stdin payload arrived with one extra byte.
write_file and patch verify an exact sha256 of the written file, so even
with the command grouped every write failed verification and left
content + "\n" on disk.
Feed the heredoc through a process substitution that re-emits the body
minus that last character. The heredoc stays outside <( ) because bash
3.2 mis-parses heredoc bodies inside it, and the reader tolerates read's
EOF status under an inherited set -e.
The tool-result spill no longer needs its +1 byte allowance for heredoc
mode, so the size check is exact on every backend again.
A profile-scoped parent donated its environ to the host multiplexer, so a
named launcher was treated as the primary adapter owner and its platform
token became the primary claim. Spawn the host with served_profile_child_env
for the default root, mark multiplex active before that primary load, and
name the env-derived side in a duplicate-credential refusal.
The heartbeat that reports long-running command progress swallowed any
activity-callback exception with a bare pass, so a broken callback left
no trace. Log it at debug with exc_info like the sibling best-effort
handlers in the same module; the command still never fails because of
a progress hiccup.
Move retention, orphan pruning, status and clear operations into a topical sibling and route callers and tests directly to their defining module. Keep the shared store paths and git execution in checkpoint_manager; shorten the local child-env WHAT docstring.
Conflict resolutions and semantic fixups:
- tools/environments/base.py: main's hard-exit kill fence (kill a spawn the
fence missed, deregister from _live_foreground in a finally) wrapped around
pm-clean's output collector.
- pyproject.toml: pm-clean's marker list plus main's new `live` marker.
- hermes_cli/main.py: pm-clean runs startup recovery from hermes_bootstrap, so
the old early-recovery block stays gone; main's interrupted-pull restore
(auto-merged above it) runs right after bootstrap, as on main.
- hermes_cli/update_cmd.py: main's interrupted-pull marker now guards
pm-clean's first tree mutation (release-tag detach, ff-only, or reconcile)
and is cleared once git is done. The marker's target is the ref git actually
moves to (a release tag, not always origin/<branch>), since the restore
compares against it.
- hermes_cli/_early_recovery.py: restore `import subprocess`, which pm-clean
had dropped and main's auto-merged restore needs (NameError on the first
launch after a killed update; test_update_interrupted_pull red -> green).
- apps/desktop/src/i18n/{de,es,fr}.ts: main's new locales carry the full
settings.about block; trim it to `updates` as pm-clean's type and the other
overlays do (tsc: 27 errors -> 0).
- main's new e2e tests: `import yaml` -> hermes_yaml; wake-word import table
names pyopen_wakeword (pm-clean's wake-openwakeword extra); the anthropic
key-leak switch leg needs the SDK, and the api_server two-tenant test needs
aiohttp, both PM runtime extras the test env does not carry.
Re-review findings on the foreground-process registry:
- Spawn vs exit races. A child spawned but not yet registered when the hard-exit kill ran, or a
command launched after the kill took its snapshot, survived under init. The hard-exit kill now
raises a one-way exit fence and waits (bounded) for spawns already past it to register; a spawn
refuses once the fence is up, and one that registers after a timed-out wait kills itself.
- The immediate kill fell back to proc.kill() for every handle. On Modal/Daytona/Vercel that is a
blocking SDK cancel (an 8s cancel made _hard_exit take 8s). Popen handles are still killed
inline (killpg, never blocks); every other handle's kill runs on a daemon thread under one
shared 0.5s deadline.
- The registry lock was a plain Lock: a signal landing on a thread that held it deadlocked the
exit. It is now an RLock (via a Condition) and the hard-exit path only takes it with a timeout,
falling back to a lock-free copy.
- The kanban worker's SIGALRM deadman os._exit()ed without the kill; it now goes through it.
The SIGTERM handler arms a 1s os._exit timer, then runs _shutdown_sessions: a flush of up to
5s, then _stop_turns_before_exit, whose kill was the graceful TERM, wait 1s, KILL. A command
that ignores SIGTERM was still alive when the timer fired, and os._exit left it reparented to
init (live: `trap '' TERM; sleep 3600` survived a SIGTERM to `python -m tui_gateway.entry`).
- kill_live_foreground_processes(now=True): SIGKILL each in-flight foreground tree at once,
no TERM grace, no wait (BaseEnvironment._force_kill_process; LocalEnvironment kills the
recorded process group, never our own).
- The grace timer's exit (entry._hard_exit) runs it before os._exit.
- _stop_turns_before_exit SIGKILLs whatever is still alive halfway through its settle budget
(it ignored the interrupt's TERM), so the tool call still ends with a result the teardown
persists instead of a dangling tool_call in state.db.
- The other hard exits that skip cleanup do the same before os._exit: the serve parent-death
watchdog, the CLI exit watchdog, the kanban worker's SIGTERM path, and the messaging
gateway's shutdown and loop-liveness watchdogs.
- Deflake test_shutdown_mid_tool_kills_the_command_and_keeps_its_result: the 0.5s settle
budget was too tight under -n 40 (1 red in 9 runs); the join returns when the turn ends.
Foreground terminal commands run in their own session (start_new_session) so an
interrupt can kill the whole tree, which also puts them outside the host's process
group. When the tui_gateway left mid-command (client closed stdin, or SIGTERM) nothing
killed them: _shutdown_sessions closed the agents, the SIGTERM path hard-exits after a
1s grace, and the `bash -c` + child tree survived, reparented to init.
- tools/environments/base.py: execute() records every in-flight foreground command;
kill_live_foreground_processes() kills their trees through the backend's own
_kill_process (the same kill an interrupt uses).
- cleanup_all_environments() (the exit funnel of the CLI, one-shot, messaging gateway
and terminal_tool's atexit, so `hermes serve` too) kills them first.
- tui_gateway _shutdown_sessions (EOF atexit + SIGTERM handler) and the serve
SIGTERM/SIGINT exit-flush handler interrupt running turns, wait up to 0.5s for them
to settle so the tool call ends with a result the final persist records (no dangling
tool_call in state.db), then kill any foreground command still alive.
- ComputeHost.close() kills them too: every caller os._exit()s right after.
Covers the case of b9dac83d366c (Desktop quit: serve SIGTERM handler) on every host.
Symptom: a host that serves several profiles from one process and mirrors
the active turn's profile into `os.environ["HERMES_HOME"]` for legacy
readers (Hermes WebUI does this on every chat turn, next to the
context-local override) makes every launch-home decision see the served
profile as the launch profile. Two profiles that both configure `atlassian`
with different credentials share whichever MCP connection came first: a
READ_ONLY_MODE=false profile ends up calling a read-only server
(nesquena/hermes-webui#7721). The same misjudgement leaves the launch
residue in the served profile's child env, seeds the launch profile's
bridged allow-all grant into the served profile's secret scope, and lets
the served profile's `terminal.*` config bridge into the shared process env.
Cause: four launch-home checks compare the task's override with
`get_process_hermes_home()`, which reads `HERMES_HOME` live:
`agent.secret_scope.serves_routed_profile` (keys the MCP ledger via
`_mcp_registry_scope`, #108352 / #111481, and the check_fn cache, #111151),
`agent.secret_scope._is_process_home`, `tools.environments.local._is_routed_home`
and `hermes_cli.env_loader._process_hermes_home`. Under the mirror the two
sides are equal for every turn.
Change: `hermes_constants.pin_process_hermes_home(path | None)` lets the
host record the home it serves as its own; `get_routing_process_hermes_home()`
returns the pin when set, else `get_process_hermes_home()`; the four checks
compare against it. The pin is deliberately NOT folded into
`get_process_hermes_home()`: `get_hermes_home()` falls back to it for tasks
carrying no override (MCP loop, spawners), and the host's mirror exists
precisely so those readers see the served profile. Only "is this task
routed / is this the launch home" changes. Unpinned, behaviour is
byte-for-byte the old one; hosts that never mutate `HERMES_HOME` need not
call it. `activate_multi_profile_hosting()` is not the seam for this: it
flips `get_secret` fail-closed process-wide and freezes the launch env,
which an embedding host cannot adopt as a bug fix.
Tests (2 invariants, parametrized over the four checks plus the MCP ledger
key; red on main, green here): pinned + mirrored env -> the served home is
routed and the launch home is not, the MCP key is `(home_key, name)`,
`get_process_hermes_home()` still follows the env var; never pinned or
pinned-then-cleared -> old semantics, including "a mirrored env var IS the
launch home". `tests/conftest.py` resets the pin per test so the
module-global cannot leak between files.
Live repro (WebUI + a stdio FastMCP server named `atlassian` in two
profiles, one gated by READ_ONLY_MODE): base -> one ledger key
`'atlassian'`, the write profile lists only the read-only tools; fixed ->
`(<read_home_key>, 'atlassian')` and `(<write_home_key>, 'atlassian')`,
each profile lists its own tools.
Docs: `gateway/AGENTS.md` § Profile scope (one launch-home identity) and the
isolation table in `website/docs/user-guide/multi-profile-gateways.md`.
Also maps the author e-mail under contributors/emails/ (attribution check).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The profile_home stamp exists to let serves_routed_profile() see a foreign-home scope without the
HERMES_HOME override. Stamping the LAUNCH home from the own-home binders (launch_profile_runtime_scope,
_worker_profile_scope's launch arm, model_switch's launch arm) added nothing for production but, under
a test conftest where server._hermes_home differs from HERMES_HOME, flipped the launch profile to
'routed' and charged every turn a routed tool/plugin discovery. Own-home binders now leave the stamp
None; the foreign-home arms keep it.
is_profile_gate_env resolved the platform prefix set (Platform enum + bundled plugin scan + registry)
once per env key; strip_profile_gate_env walks the whole child env, so a spawn paid the scan
hundreds of times. The static half is lru_cached, the dynamic registry union is taken once per strip.
is_profile_gate_env matched any name containing _ALLOWED_ / _ALLOW_ALL_ / ... so an operator's own
DEMO_ALLOWED_SENDER (script data, not a Hermes authorization gate) was deleted from every routed
child env built with strip_launch_profile=True, breaking no_agent cron scripts under the host
gateway. The owner prefix now has to be a platform: built-in Platform values, bundled platform
plugins (directory names and manifest aliases), runtime-registered plugin adapters, plus GATEWAY_
(pairing) and QQ_ (qqbot). Every gate the adapters read still strips; operator variables survive.
Fixes#119539
7fe77f29a7 renamed cleanup_terminal_temp_cache's kwarg to max_idle_hours;
gateway/run.py::_run_media_cache_cleanup calls every registered cleanup
as fn(max_age_hours=24), so the terminal sweep raised TypeError inside
housekeeping on every tick. The kwarg is the shared cleanup_*_cache
signature (bot_relay notes the same parity); the idle semantics stay.
A test binds the loop's call shape against every registered cleanup.
Deleting an idle scratch entry left two things behind. Headless browsers
started by a lane's e2e run kept running for days with a "(deleted)" cwd
(about 20 on one host; the process registry never knew them because they
were grandchildren of a shell that had exited). Repos whose linked
worktree lived in the entry kept a dangling registration until someone
ran `git worktree prune` by hand (10 in one repo).
The prune now lives in hermes_constants_scratch (hermes_constants keeps the
entry point). Before an idle entry is removed, same-user processes whose
cwd is inside it, or inside any scratch path that no longer exists, are
TERMed then KILLed; the deleted-cwd sweep runs on every pass so orphans
from earlier deletions are caught too. `.git` files found in the entry
name their repo, which gets `git worktree prune` after the rmtree.
cache/terminal used its own 72h fixed-age sweep; it now shares the 24h
idle rule and the subtree check. `hermes doctor` warns about cache-root
directories over 1 GiB that neither pruner covers, since finished campaign
trees parked there sat for weeks (95 GB on one host). System-prompt
scratch line updated to match.
pm.shell.bash() returned shutil.which("bash") first on Windows. On most
machines that is C:\Windows\System32\bash.exe, the WSL launcher stub
(exits 1, "no installed distributions"), so a source install without the
staged git package lost its shell — the #116818 regression main fixed in
0abc3b04. The System32 guard survived only in local._windows_bash_candidates,
which had no callers after _find_bash collapsed onto pm.shell.
Move the ladder into pm.shell as a pure function (Program Files Git first,
then PATH minus the System32/WindowsApps stubs) so it is testable on any
host, and delete the dead copy in tools.environments.local.
Branch semantics kept where main and PM disagree: update_cmd_deps.py,
constraints-termux.txt, the Electron update-api-check module and the
post-swap hand-off test stay deleted; the pending-fleet-restart catch-up
and the local_runtime tag/download ladder stay retired (PM owns engines).
Ported from main onto the branch's shape: profile_scoped_chore for the
auto-archive and plugin-update housekeeping chores, the local-runtime
cross-process boot lock and residency cap, the checkpoint tmp_pack sweep,
the cua daemon-liveness status probe, the remote-served Desktop update
flag (posix.sh / windows.ps1), sign-in for env-pinned remote gateways
(urlDisabled on RemoteSetupFields), the uvloop extra split (uvicorn
without [standard]), and the umask-scoping spawn test.
uv.lock regenerated with pm.build_env --lock-only; new utf-8 reads from
main switched to utf-8-sig (check-windows-footguns).
A spawner that skipped setsid (the Darwin gateway's posix_spawn shim) leaves
tool children in the gateway's process group, and _kill_process_group_posix
then killpg()s the gateway itself — launchd logs 'Killed: 9' and KeepAlive
respawns it. Branch on pgid == os.getpgrp() as a plain if/else (not a raised
PermissionError routed into the EPERM handler) and signal the wrapper plus
its snapshotted descendants by PID; the EPERM fallback shares that helper.
Fixes#107029
On macOS a search that reaches its `limit` takes the early-stop branch that
TERMs rg's process group; rg has often already exited, and macOS answers
killpg on a zombie-only group with EPERM instead of ESRCH. The
PermissionError escaped _kill_process_group_posix, unwound to search_tool
and the drained matches were replaced by
`{"error": "[Errno 1] Operation not permitted"}` (#116855, same symptom
as #104696). The helper now treats EPERM like "nothing left to signal"
and falls back to killing the known PIDs so a live child cannot escape.
It also never killpg's the caller's own group (#107029): a child that
shares our pgid is torn down by PID.
Diagnosis credit: #116949 (@liuhao1024), redone slim.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
_windows_bash_candidates in tools/environments/local.py appended
shutil.which("bash") to the candidate list without filtering — when Git Bash
is installed but PATH resolves `bash` to WSL's stub (C:\Windows\System32\bash.exe)
or WindowsApps bash.exe, that stub enters the candidate list.
Fix: skip paths whose normalized form contains `system32` or `windowsapps`,
logging at debug level. Git Bash (already searched explicitly in the roots list)
remains the first usable candidate.
Fixes#116818
Switching local.py to the folded _is_provider_env_blocklisted helper dropped the
blocklist name from its import list, which removed the re-export that
tests/providers/test_auth_registry_import_order.py (and any out-of-tree caller)
reaches through tools.environments.local. Re-import it alongside the helper so
the module keeps exposing the set; behaviour is unchanged.
The provider-credential blocklist and every adjacent env-name check ran
case-sensitive membership tests while the Windows environment block
resolves names case-insensitively. A skill or terminal.env_passthrough
entry registering openai_api_key was accepted, then resolved to the real
OPENAI_API_KEY by os.getenv() and forwarded into SSH/Docker exec envs,
the same GHSA-rhgp-j443-p4rf tunnel the blocklist closes. The same gap
let variant-cased credentials survive the source-side strip
(_filter_secret_env, _scrub_credentials), evade the docker_forward_env
and docker_extra_args egress collision guards, and ride
strip_launch_profile_env residue into a routed sibling profile's child.
Add _is_provider_env_blocklisted (exact + folded membership) and apply
it at every layer: registration refusal, both scrub paths (Tier-1 and
plugin strip sets fold too; inherit_credentials still inherits), the
remote exec-env builder, and the docker egress collision checks. Fold
_matches_terminal_first_party_prefix symmetrically so a lowercase-stored
buzz_private_key keeps its terminal carve-out, and fold the launch-
residue pop (selection folds too so a lowercase path in .env stays
global). _HERMES_FORCE_* opt-in transport and docker_env literal
container names stay case-sensitive on purpose.
Conflicts resolved toward the PM model: main's lazy_deps/update_cmd_deps/npm
stamp machinery stays deleted (PM + scripts/build/node-deps.mjs own it), the
systemd ExecStop stop-mark rides the installation launcher, legacy
linux_only/macos_only/windows_only markers are rewritten to platforms(), and
finalize_update_receipt carries pending manual-serve obligations forward
again (lost when the ContextVar receipt rewrite crossed c0aa3ce354).
Test harness: the real-home I/O guard exempts /proc/<pid>/fd metadata reads
(deleted-WAL holder scans) and run_tests.sh drops ~/.hermes PATH entries so
shutil.which() cannot trip the tripwire.
reap_orphan_containers snapshots exited containers, then removes each
candidate with docker rm -f. The exited-only filter is only as fresh as
the docker ps snapshot: a sibling process can legitimately restart an
exited container between the snapshot and the rm (the reuse path calls
docker start on matching exited containers), and FinishedAt still
reports the previous exit, so the age check passes. The -f then kills
the sibling's live container and its in-container state.
A plain docker rm fails atomically in the daemon on a running container,
which is exactly the skip semantic the exited-only guard intends. Every
intended target of this sweep is already exited, so -f bought nothing.
The new test drives the real reaper with a simulated restart: rm argv
must not contain -f, and a daemon refusal on a running container is
skipped while the rest of the sweep continues.
Drop the nondeterminate-stderr fragment list: the rule is now simply "exit 0
caches True; a nonzero exit whose stderr names the probed capability
(cgroup / storage) caches False; everything else is uncached and retried on
the next spawn". A curated list of pull/daemon phrases would need to track
every docker CLI wording; the capability keyword is the property the cache
exists to record. Tests are trimmed to the invariants: the production
DockerEnvironment path recovers on the next spawn, a definitive cgroup
rejection is still cached, and the storage-opt probe caches only definitive
answers.
_cgroup_limits_available latched _cgroup_limits_ok=False on ANY probe
failure — a TimeoutExpired while docker run auto-pulls an uncached image,
a daemon cold-start, a manifest/pull error, or a missing docker binary —
permanently stripping --cpus/--memory/--pids-limit from every later
container in the process. The sibling _storage_opt_supported latched the
same way, and additionally read a failed `docker info` (returncode never
checked) as "not overlay2", disabling disk quota for the process.
Both probes now cache only definitive answers: exit 0 -> True; a daemon
rejection naming the probed capability -> False. Any other failure
(missing docker, pull/manifest/daemon errors, timeouts) degrades that
spawn and is retried on the next. A shared nondeterminate-stderr list is
checked before the capability keyword so an image named e.g.
"cgroup-tools" whose pull fails is not misread as a cgroup rejection.
Fold the salvaged parametrized positive/negative tables and the redundant
env-file/real-chain e2e tests into one parametrized invariant (every pflag
spelling of -e collides, legitimate args do not) plus one production-entry
test through DockerEnvironment. Args that do not start with "-" are
positionals under pflag and are never env flags, so the shorthand scanner
skips them instead of reading a container name like "de" as "-d -e".
The egress collision scan only recognized -e/--env/--env-file as standalone
tokens, so the combined-shorthand spellings docker accepts, -eNAME=v joined
and -iteNAME=v chained after boolean shorthands, injected critical env names
past the guard. Parse each arg the way pflag does (boolean ditPq shorthands
may precede the terminal value shorthand) and stop at the -- terminator.
Fixes#115887
