_run_foreground recorded outcome=timeout whenever a backend exception's
text contained "timeout" (an SSH connect timeout, a sandbox API timeout).
That command never reached an exit status, and the doc says timeout comes
only from Hermes' own deadline flag (hermes_timed_out), which
terminal_outcome() already reads. Owner ruling: never from exception text.
The exception path now records nothing; the model-facing result (exit 124)
is unchanged.
Probe (code-read finding; the new test drives the real terminal_tool with a
backend raising "... Connection timeout"):
before: terminal.outcome {backend: local, command_kind: shell_builtin, outcome: timeout} x1
after: no terminal.outcome row; tool result still exit_code 124
test_backend_exception_mentioning_timeout_is_not_a_terminal_timeout is RED on
base, green here.
Five opt-in counters that tune the agent loop itself, recorded through
record_process_mark (disabled config => zero rows, zero files):
- hermes.file_edit.count: tool, mode, outcome, match_strategy. The fuzzy
matcher reports the strategy that landed (or no_match / ambiguous) into a
context-local probe opened only around the patch/write_file handlers, so we
learn which strategies earn their keep without counting internal callers.
- hermes.loop_guard.count: provider, model, signal, detector. Hooked where
the guardrail already warns/blocks/halts and at turn end for the iteration
budget; latched once per turn per (signal, detector).
- hermes.tool_recovery.count: provider, model, tool, next_tool,
next_outcome. One row per failed tool call, resolved against the model's
next round (same tool first) or the turn end (no_tool_call / gave_up).
- hermes.terminal.outcome.count: backend, command_kind, outcome. The kind is
a table lookup on the first program word, never the text. Hermes' own
deadline/interrupt now set hermes_timed_out / hermes_interrupted on the
backend result, so a command's own `exit 124` reads nonzero, not timeout.
Disjoint dims from hermes.execution_backend.count (served vs not).
- hermes.model_reply_issue.count: provider, model, issue. Refusal and
truncation only from the structured finish reason; empty/reasoning_only
from the normalized message; issue=none per response as the denominator
(model_route counts attempts and files empty replies as failures).
Background review / curator loops and _host_local / unmetered terminal
calls never count. Helpers live in shared_metrics_harness.py; call sites
are one or two lines each.
- memory tool: one row per operation (a batch counts each op); gate/validation
refusals are "rejected", store-level non-application "failed". Memory provider
tools are counted in MemoryManager.handle_tool_call with the op read from the
action arg or tool-name verb.
- curator: run_curator_review records one row per pass (dry run = skipped) with
the before/after diff bucketed; a scheduled pass whose claim is held by another
process counts as skipped. The home is captured before the review thread starts.
- delegate_task: _run_batch opens the call, each joined unit folds its results in,
and the last unit emits the call's single row, so group-split background calls
are not counted per unit.
- terminal / execute_code / browser: counted per call that reached the backend;
the backend is resolved from the owning profile at call time (terminal plan
env_type, code local/remote, browser CDP > Camofox > cloud provider > engine, or
"extension" when the extension controller served the call). Guard refusals and
Hermes' own _host_local commands are not counted.
Tests read rows back from the real store; each call-site group is red with its
source reverted.
* feat(vercel): start fresh sandboxes from a managed image instead of the deprecated runtime
Vercel deprecated Sandbox runtimes (node24/node22/python3.13) in Aug 2026 in favour of
images, and rejects runtime+image together and runtime with a snapshot source. New
terminal.vercel_image (default vercel/sandbox/universal:latest, Node 24 + Python 3.14)
picks the image for fresh sandboxes; a pinned terminal.vercel_runtime still works, wins
over the image and logs a deprecation warning; snapshot restores send neither.
Setup wizard prompts for the image, dashboard exposes both keys, status/config show the
effective choice, TERMINAL_VERCEL_IMAGE bridges config to the tool like its siblings.
* feat(config): migration 49 drops the seeded node24 Vercel runtime pin
Every pre-49 config.yaml carries terminal.vercel_runtime: node24 (the template default) and
the setup wizard mirrored it into .env as TERMINAL_VERCEL_RUNTIME. Both are the default
copied, not a choice, so the migration drops them and fresh sandboxes follow vercel_image;
node22 / python3.13 pins are the user's and survive. Persisted sandboxes are unaffected:
a snapshot restore never sends a runtime or an image.
* refactor(plugins): remove the Sep 2026 decomposition compat layer on schedule
The PLUGIN-COMPAT layer (2776813df3 + d63e380324 + 0a5164cebe) kept pre-#102117 import paths
alive for external plugins until 2026-09-14. That window closed two weeks ago; since then the loader
has already been skipping plugins that use the old paths. This removes the layer itself:
- 328 appended `PLUGIN-COMPAT` blocks (lazy `__getattr__` pointer tables, re-exported third-party
names, restored dead definitions) and the three re-export stub modules
(gateway/startup_watchdog, hermes_cli/observability/relay_runtime, tools/environments/modal_utils)
- COMPAT_MANIFEST.md, compat_manifest.json, scripts/check_compat_pointers.py and its lint step
- the reporting surfaces: CLI banner notice, `hermes plugins compat`, the `hermes doctor` section,
the post-update notice, the Desktop one-time dialog, the loader's pre-import skip and the
`plugins.allow_deprecated_imports` escape hatch
An external plugin that still imports an old path now fails to load with its ImportError as the
reason in `hermes plugins list`, the same path as any broken plugin.
hermes_cli/plugin_compat.py stays as three inert stubs (compat_report, removal_in_effect,
summary_lines): an already-running pre-removal `hermes update` lazy-imports them after the checkout
swap (tests/compat/old_updater_surface.json).
In-tree fallout, both already dead: hermes_cli/setup.py::_check_espeak_ng (no callers; its
`shutil` came from a compat block) and gateway/config.py::SessionResetPolicy ("retained solely for
the scheduled plugin-compat window"). Two test_run_agent patches targeted the removed
`run_agent.handle_function_call` pointer; they now patch `model_tools.handle_function_call`, the
seam production reads, like every sibling test in that file.
* chore: retrigger CI (zero-job startup_failure phantom)
* test: drop resolution allowlist rows for the two deleted which() sites
hermes_cli/setup.py::_check_espeak_ng (dead) and tools/skillevaluator_scan.py::scanner_available
(a restored definition inside a PLUGIN-COMPAT block) no longer exist; the stale-row gate requires
their allowlist entries go with them.
Independent-review minors on #126157 (#123989 class):
- session.create is a plain @method, so tui_gateway/session_workdir.py::_register_session_cwd
wrote the cwd record under the RAW session key while the scoped turn read
profile:<p>:<key> and missed it until the first `cd`. The writer now binds the
session's own profile_home around register_task_env_overrides.
- tools/terminal_tool.py::_task_env_overrides stayed raw-keyed, so two profiles
registering the same task id (docker_image / cwd) still collided. Writer, clear
and both readers (_has_isolation_overrides, resolve_task_overrides) now use
_qualify_task_key; the isolation-override branch of _resolve_container_task_id
returns the qualified key so the env cache and the override record agree.
- gateway/platforms/api_server.py::_derive_chat_session_id prefixed the seed for
the NAMED LAUNCH profile addressed via /p/<launch>/, so prefixed and
un-prefixed requests on one profile derived different ids. The prefix is now
skipped when the routed name is the process's launch profile
(_names_launch_profile vs get_routing_process_hermes_home()).
Tests (red on 3720519199c, green here):
tests/tui_gateway/test_session_cwd_profile_key.py::test_secondary_profile_session_cwd_is_found_inside_its_scope
tests/gateway/test_api_server.py::TestDeriveChatSessionId::test_launch_profile_prefix_keeps_the_unprefixed_id
Fold the two salvaged fixes to the class and the salvage bar:
- tools/terminal_tool.py: `_qualify_task_key` reuses `_routed_home_task_key`
instead of a duplicate qualifier; branch 2 (session-isolated sandboxes) AND
branch 3's `session:<key>` (SSH-style backends) carry the routed profile, so a
session id two profiles share (header-less API fingerprint, a DM chat id served
by two bots) never resolves to one `_active_environments` slot. Cwd records
keep the symmetric qualification. No routed home → raw key, unchanged.
- gateway/platforms/api_server.py: `_derive_chat_session_id` namespaces only
named routed profiles, so default/standalone ids are byte-identical and live
Open WebUI conversations survive the upgrade.
- Tests trimmed to two invariants: A→B→A across two temp profile homes under
multiplex (distinct sandbox keys, alias follows the qualified parent, cwd
isolated per profile), and derived-id namespacing without moving default.
Both fail on 5912ed81ed, pass here.
A multiplexed host serves every profile in one process, and header-less API-server
clients derive their session id from a fingerprint of the system prompt + first
user message. Under per-session isolation (docker + container_persistent: false,
plus plugin backends declaring session_isolated_when_nonpersistent) branch 2 of
_resolve_container_task_id keyed the raw id, so two profiles whose conversations
open with the same text shared one _active_environments slot: the second profile's
turn attached to the first profile's sandbox, with the first profile's image,
mounts and exported environment. Verified live against real docker (both
directions on unmodified main; isolated after the fix).
Routed scopes (HERMES_HOME override: multiplex turns named AND default,
per-profile TUI/desktop RPC, cron ticks) now qualify the key with the profile
name or home realpath; CLI and single-profile gateways keep the historical raw
key. The session-cwd record/get/clear trio gets the same qualification so the
fingerprint collision cannot carry one profile's cd state into another's.
Regression tests in TestRoutedScopeQualification fail on pre-fix code (4/6) and
pass with the fix; persistent-Docker behavior is byte-identical (labels stay
profile:<name> / default).
* feat(docker): publish nousresearch/hermes-sandbox:desktop for terminal backends
The terminal backends (docker, modal, daytona, singularity) all default to
nikolaik/python-nodejs:python3.11-nodejs20, a bare Python+Node base. For Bot
Screen, computer_use and the browser to run INSIDE that sandbox instead of on
the gateway host, the sandbox image needs the display stack.
docker/sandbox-desktop.Dockerfile is that base plus:
- the everyday tools it lacked (jq, ripgrep, fd, tmux, less, nano, vim,
zip, rsync, tree, procps, htop, sudo for the base's uid-1000 `pn`)
- the exact package set the Hermes -desktop image installs (TigerVNC,
Xfce components, dbus, xauth, fonts)
- Playwright's headed Chromium (same build as the -desktop image)
- cua-driver 0.28.2 from its pinned release tarball
No Hermes inside; the default user stays root like the base so nothing
changes for people who just switch docker_image. Desktop processes run as
`pn`. 4.27 GB on amd64.
docker.yml gains a `sandbox` variant with its own cache scope and repository
(nousresearch/hermes-sandbox:desktop, :main-desktop, :<release>-desktop);
the docker-integration suite is skipped for it (no Hermes to test) and
docker/sandbox-desktop-smoke.sh runs instead: as `pn`, every launcher and
cua-driver binary resolves, the real launcher.sh publishes :20, the RFB
socket completes the 3.8 handshake relayed over `docker exec -i` stdio, and
a headed Chromium maps a window on that display. hadolint lints the new
Dockerfile in docker-lint.yml.
* feat(docker): sandbox desktop base on python3.13-nodejs26
Matches the Hermes image (Python 3.13 / Node 26) and the top of requires-python;
the default docker_image tag it inherited was Python 3.11 / Node 20. Same pn
uid 1000, Debian 13; smoke (launcher, RFB relay, headed Chromium) passes.
* feat(docker): bake agent-browser into hermes-sandbox:desktop
The browser tools drive the agent-browser CLI; when the browser follows the
terminal backend that CLI has to exist inside the sandbox. Pinned to the same
^0.26.0 range the gateway resolves, --ignore-scripts like the gateway's npx path.
* feat(bot_desktop): the screen, computer_use and the browser follow the terminal backend
A user who sandboxes `terminal` (docker/ssh/singularity) had the agent's
screen, cua-driver and Chromium running on the gateway HOST beside that
sandbox: Bot Screen gave a headless host a display, the Xfce panel carries
xfce4-terminal, and `computer_use` could open a shell outside the boundary
the sandbox exists for.
Now the desktop lives where the terminal lives:
- tools/environments/streams.py: one primitive per spawn-per-call backend, the
local argv prefix that runs its remainder inside the sandbox with stdio open
(`docker exec -i`, `ssh`, `apptainer exec`). SDK backends (modal, daytona,
vercel) have none and report so.
- tools/bot_desktop/sandbox_host.py: launcher.sh runs inside the sandbox as
the image's `pn`; the pane's RFB bytes ride a 12-line python relay over that
prefix; `cua-driver mcp` is the prefix + the sandbox image's own driver.
- tools/bot_desktop/placement.py + `bot_desktop.placement` (auto|terminal|
gateway). `auto` follows the backend; a sandbox that cannot host a screen
REFUSES with the opt-in named instead of silently using the host.
- runtime.start/stop/status/published_env branch on placement; the pane,
lease, epoch fencing and CLI are unchanged.
- cua_backend: the MCP invocation is the sandbox one when the screen is
there; the host driver's runtime contract is irrelevant then; check_fn is
true under a terminal placement without a host binary.
- browser_tool_session: agent-browser invocations are wrapped in the prefix
with the daemon, socket dir and profile inside the sandbox; screenshots are
fetched back so MEDIA: paths keep working; recycle closes the sandbox
daemon.
- web_routers/display.py: the bridge pumps a relay's stdio when the screen is
in a sandbox, a unix socket otherwise.
Live on docker with nousresearch/hermes-sandbox:desktop: start/observe/RFB
handshake through the dashboard bridge, human takeover fences the agent
(HumanHasControl) and keystrokes reach the sandbox Xvnc, handback restores,
three start/stop rounds leave zero desktop processes; computer_use capture
and list_windows see only the sandbox's Xfce; browser_navigate/snapshot/
vision run with Chromium and agent-browser inside the container and zero
host processes on the bot profile; modal + auto refuses naming the opt-in.
* feat(desktop): Screen pane shows where a sandbox-placed screen runs; Install is host-only
DesktopStatus gains placement ('gateway' | 'terminal:<backend>'). A sandbox
image lacking the stack is a blocker naming hermes-sandbox:desktop, shown in
place of Start; install_command stays None there because the pane's Install
button runs the package manager on the gateway host, the wrong machine, and
display.install refuses for the same reason. The pane header carries
'Screen runs inside the docker sandbox, with the terminal' (4 locales).
* fix(bot_desktop): "is the screen in the sandbox" is a disk check on hot paths, never a config read
Every browser command and CUA spawn asked in_sandbox(), which resolves placement by
loading config, which initializes HERMES_HOME. Under a test's fake home that raised
HomeInitializationError from _run_browser_command; on a real host it read config per
click. Hot paths now ask sandbox_screen_running(): the start marker on disk, written
only by a sandbox start. Policy (in_sandbox) stays for start/install, where config is
the question. display.observe gates on "an RFB endpoint exists" for either placement.
* test(moa): late-accounting sink test asserts the wedged slot's row, not sink order
Under CI load the poll loop can see the interrupt before collecting the fast slot, so the
fast slot also arrives late and first; the test then failed on sink_calls[0]. The
contract is that the wedged slot's real usage reaches the sink.
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* fix(bot_desktop): a sandbox that died under a live screen fails loudly, never falls to the host
sandbox_screen_running() drops a start marker whose terminal environment is no longer
registered (stale after a process restart). When the environment object outlives its
container, the browser's sandbox wrap now checks the published DISPLAY and raises
"the screen inside the terminal backend's sandbox is gone; start it again" instead of
KeyError('AGENT_BROWSER_PROFILE'). Live: fresh sandbox navigate ok; docker rm -f the
container; next navigate returns that error; zero host Chromium either way.
* feat(terminal): nousresearch/hermes-sandbox:desktop is the default container sandbox
Every container backend (docker, modal, daytona, singularity) now defaults to the
sandbox image with the desktop stack, so Bot Screen, computer_use and the browser
run inside the sandbox for everyone who never chose an image; Python 3.13 / Node 26
match the Hermes image. One constant (DEFAULT_SANDBOX_IMAGE) replaces six copies of
the old literal. Migration 47 moves saved configs still holding the OLD default and
never touches an image the user pinned. Docker reuse recreates a container built
from another image, or the flip would silently never take effect for anyone with a
persisted container (live: old container removed, new one on 3.13 / Node 26 with
Xvnc, cua-driver, agent-browser present).
* chore: retrigger CI (zero-job dispatch failure)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
* chore(config): template stamps v47 and shows the new default sandbox image
The template is what install.sh / docker / doctor --fix seed; a stamp behind
DEFAULT_CONFIG makes every fresh install migrate on first run.
* feat(sandbox): the default image change is a decision, not a surprise
A persisted Docker sandbox on another image is kept when docker_image is unset;
only a written docker_image (an explicit pin) recreates it. The pin verdict
travels as TERMINAL_DOCKER_IMAGE_PINNED through both terminal bridges (process
env and per-profile scope) and the container-config allowlist.
Approval surfaces, all through hermes_cli.sandbox_image_switch: the interactive
CLI asks once at startup (y = pin the new image, n = pin the current one, Enter =
ask later); the Screen pane shows the same choice with Switch / Keep buttons via
display.switchSandboxImage; `hermes config set terminal.docker_image …` is the
same answer from any shell. Gateways and cron never decide: they keep the sandbox
and log the notice.
Migration 47 now unsets a saved image equal to the OLD default instead of
rewriting it to the new one — that value was the template copied, not a pin, and
rewriting it would have made the runtime recreate existing sandboxes unasked.
Modal restores its snapshot and Daytona reuses its labeled sandbox regardless of
the configured image, so existing sandboxes there were already untouched.
* fix(config): both plain defaults that preceded the desktop sandbox image are template copies
main pinned nikolaik/python-nodejs:python3.14-nodejs22 (cd0f97f833) without a migration;
a saved config holding either literal is unset by migration 47, so it follows the default
and existing sandboxes get the keep-or-switch decision instead of a silent recreate.
* ci(docker): build the sandbox image on release/dispatch, not every main push
Leaves docker.yml exactly as on main. The sandbox image carries no Hermes code,
so two 4 GB multi-arch builds per merge bought nothing. sandbox-image.yml builds
and smokes on a PR that edits its own Dockerfile/smoke, and publishes only on a
release or a manual dispatch with publish=true. Stable tag stays :desktop.
* fix(config): keep main's config.py/config_defaults.py edits under the sandbox-image delta
The rebase resolved both files wholesale with the branch side, dropping main's move to
hermes_yaml (the 3.14 runtime venv has no PyYAML) and the 3.14 base pin. This is main's
version plus exactly the branch's own changes: DEFAULT_SANDBOX_IMAGE, the pin verdict in the
env bridge, placement defaults and the v47 stamp.
* test: sandbox-image tests read config.yaml through hermes_yaml (no PyYAML on the 3.14 runtime)
* fix(bot_desktop): read the sandbox marker BOM-tolerantly (windows footgun lint)
* fix(bot_desktop): placement is the authority; sandbox screen survives restarts
Review findings on the sandbox-hosted Bot Screen, each reproduced live first.
Authority. The browser preflight and the CUA invocation keyed off screen
LIVENESS, so `placement: terminal` with the screen not yet up handed the tool an
unchanged host command. `runtime.tool_placement()` is now the one resolver:
terminal placement starts the sandbox screen on demand (no auto_start opt-in
inside the user's own sandbox), refused placement raises its reason, and neither
ever yields the host. placement.resolve() answers a local backend from env alone
so the common case costs no config load on the spawn path.
Restart. sandbox_screen_running() deleted the marker whenever the process-local
terminal registry was empty, i.e. after every gateway restart, while Xvnc kept
running in the container; stop() then returned False and left it. The marker
now records the owning container; liveness comes from `docker inspect` on it,
stop/status re-attach to the recorded owner (even after the placement setting
moved), and only a container that is gone drops the marker.
SSH. remote_argv emitted `bash -c <script>` as three words; OpenSSH joins them
and the remote login shell ran `bash -c export` and the rest itself. The script
travels as one quoted word for ssh (remote_command knows the backend); docker
and apptainer keep argv.
CDP reach. agent-browser inside the sandbox reports the sandbox's loopback;
the Browser Use harness, browser_exec and the vault supervisor connect from the
host and got connection refused. streams.forward_port() proxies a local port
over the exec stream (same relay as the RFB bridge) and the CDP URL is rewritten
to the local end.
pids limit. --pids-limit 256 counts threads; measured on the desktop image the
desktop stack is 44, one Chromium tab 212, the agent's browser with two tabs
488. Past the cap every further docker exec died with "procReady not received".
Default is 2048 with the measurements in the comment.
Replacement. An approved image switch force-removed the old container before
`docker run` tried the new image; a private tag or registry outage left nothing.
The image is inspected/pulled first and the old container kept on failure.
Desktop integration. The sandbox start never passed the dock's browser launcher
(no Browser icon) and the thumbnail needed a host launcher pid + host ImageGrab
(always None). The dock runs the sandbox's Playwright Chromium on the shared
profile; the thumbnail is grabbed inside the sandbox (Pillow baked into the
image). The browser profile moves from /tmp — a 512 MB tmpfs emptied on every
container stop — to the desktop user's home, so logins follow the container.
Pin provenance. A TERMINAL_DOCKER_IMAGE written in a routed profile's .env is a
pin even when it spells the default; the scope compared values before.
* docs(bot-screen): no literal tmp path in the profile-location note
* fix(bot_desktop): docker inspect liveness probe closes stdin (TUI subprocess guard)
* fix(bot_desktop): adopting a screen the sandbox kept records the marker
Live ssh probe: after the host's state was lost while the sandbox kept its
Xvnc, start() took the idempotent early return (display already published)
and never wrote the host marker, so status/thumbnail/stop lost the screen.
Record the adopted display like a fresh launch.
Docs: what an ssh host of your own must carry, and why a Dockerfile ENV is
not enough for a login session (PLAYWRIGHT_BROWSERS_PATH via /etc/environment).
* docker(sandbox-desktop): login sessions find the browser (PLAYWRIGHT_BROWSERS_PATH via /etc/environment)
* docs(bot-screen): what the Apptainer path inherits from the image and what it does not
* chore(config): sandbox-image migration is 47→48 (main took 47 for compression.threshold_tokens)
* chore: retrigger CI (zero-job dispatch failure, auto-heal)
Since 4317ed0e71 the heartbeat schema allows 0 (disabled) and clamps
positive values to 60, but the validation error still said "min 60",
steering models away from the valid 0. Flagged on #119202.
A volume that already owns /workspace skipped the configured working
directory, so tools treated that host path as unmounted. Bind it at a
second mount, or point tools at the volume that already has it, for any
drive path.
A Desktop Docker session that registers an absolute host directory
such as /mnt/... or /srv/... as its cwd skipped the /Users|/home/
drive-letter heuristic, so the mount check never ran and commands
were wrapped with cd to that host path. Classify the mounted host
directory as unusable before that heuristic, and remap it to
/workspace in the live-env write and the per-command resolver.
The session cwd can already be the Hermes host's /opt/data/home. That
path is not on the SSH target, so the implicit cd exits 126 and the
command never runs.
(cherry picked from commit 4e99397bb4e18416b6a3e0b850d51604c0ad0d1d)
Background processes spawned with terminal(background=true) are killed from
three agent-lifecycle sweeps: agent release()'s kill_all, a gateway turn
timeout's kill_started_since, and agent close's owned-process loop. Jobs the
user explicitly wants to outlive the session (overnight batches, watchful
daemons) had no way to opt out.
Add terminal(background=true, persist_on_release=true):
- ProcessSession.persist_on_release, stamped by spawn_local/spawn_via_env,
carried in crash-recovery checkpoints and exposed via list_sessions()
- kill_all skips persisted sessions only for lifecycle sources
(_LIFECYCLE_KILL_SOURCES: kill_all, gateway_turn_timeout, agent_close);
explicit operator stops (process_manage kill, /stop slash + RPC mirror,
CLI /stop) now pass distinct sources so they still reach persisted jobs
- the agent_close owned-process loop in _close_task_resources skips
persisted sessions the same way
- gateway shutdown keeps killing persisted jobs (source=gateway_shutdown):
the host is going away and survivors would become PPID=1 orphans
Co-authored-by: salvaged from #109846 (persist_on_release plumbing) and
extended to the turn-timeout and agent_close paths.
Under persistent Docker, profile B's session-bound work keys profile:B but its
session-less (cron) work keyed home:<path>, so the same profile ran two
long-lived containers. The routed no-session branch now returns the same
profile key as branch 3 when Docker is persistent (profile-scoped); other
backends keep the per-home key.
A multiplexed host runs every profile's cron jobs without a session key, and
_resolve_container_task_id collapsed all of them onto the shared "default"
environment. Profile B's cron tool calls therefore reused the LocalEnvironment
the launch profile's job created: B's terminal subprocess saw the launch
profile's .env residue and bridged TERMINAL_* (TERMINAL_CWD, backend policy).
Found by tests/e2e/core/tenancy/test_two_tenant_gateway.py (C7 canary): alpha's
cron env snapshot carried default's TENANT_MARKER and TERMINAL_CWD. Session-less
work under a routed home override now keys home:<realpath>; the launch profile
and single-profile processes keep "default". The unit test that pinned the
shared key for a routed profile is updated.
Every tool schema rides every API call; the first draft read like docs (111 est. tokens).
Kept the two facts the model needs — when to use it and what it delivers — the rest lives in
tools-reference.md.
`terminal(background=true, heartbeat=N)` emits a `heartbeat` event every N seconds
(floor 60) carrying only the output produced since the previous one, plus the usual
completion notice. The agent stays current on a long bounded job (merge train, full
test suite, deploy) and reacts to a failure within N seconds instead of at exit.
Why: `notify=true` fires once at exit, so multi-hour merge trains ran silently — the
parent's prompt cache went cold and every "what's up" cost 5–10 rediscovery calls;
30 days of batch sessions show 4,930 hand status polls (14% of all tool calls).
`notify=[pattern]` cannot serve this: its lifetime cap (8) exists precisely to stop
periodic output from flooding the agent.
Mechanism: one daemon timer thread for every heartbeat session (reader threads block
on the pipe and cannot keep time); delta output via a total-ingested counter over the
rolling buffer; heartbeats only where the completion notice can be delivered (same
async-support/subagent gates), never after exit. `heartbeat_seconds` is checkpointed.
Rendered on CLI, gateway and TUI through the existing process-notification paths.
A bot DM's reply comes back as the delivery process's completion notification,
which carried the last 2000 characters of that process's output — the right
tail for a build log, but a bot may send 16,000 characters (MESSAGE_MAX_CHARS)
and its teammate's answer routinely runs longer than 2000. The sender got the
tail of the reply, header included in the cut, with nothing saying so, and
relayed it as the whole answer.
The registry now sizes a completion per process (ProcessSession
.completion_output_chars, checkpointed) and declares a cut (output_cut) when
the output did not fit; the notice names the cut and the process log that has
the rest. tools/bot_mode_dm._spawn_delivery — the one spawn point for every
DM lane (local, live-owner, Desktop relay waiter) — asks for MESSAGE_MAX_CHARS
plus header room. Everything else keeps the 2000-char tail.
Fixes#115334
A cwd override registered by a desktop/TUI surface is a raw host path. On
container backends it was written verbatim into the cached live env's cwd,
so every file-tools command wrapper did `builtin cd -- <host path> || exit 126`
and all writes outside the mapped workspace failed with an unrelated `cd:`
error while terminal commands kept working (their per-command resolver
already sanitizes).
The live-env write now mirrors the creation-path guards: a host/relative cwd
is remapped to /workspace when it is the directory mounted there (docker cwd
passthrough), otherwise not applied to the live env at all. The session cwd
record keeps the raw path and non-container backends apply overrides
verbatim, so ACP project-root switching is unchanged.
Fixes#113894
(cherry picked from commit d712c6b47d28f213773f40ebac81dd9a45c6e94f)
execute_code ran the same unbounded _is_supervised_gateway_process() probe
ahead of every cell, so the wedge #111922 bounds in terminal_tool still hung
an execute_code call (and its cron slot) forever: share the cell's deadline
and fail closed with a retryable error when the probe renders no verdict.
Moving the terminal pre-exec guard onto a deadline worker made it blind to
/stop, which keys on the tool thread's ident: record the acting-for tid in a
contextvar (copied into the worker by run_bounded_sync) so is_interrupted()
on the worker honours the tool thread's bit too.
Floor the guard's share of the deadline at 30s so a short command timeout
does not turn the guard's own cold-start cost (imports, git probes under
load) into a refusal — tests/tools/test_terminal_error_redaction.py was red
on the branch for exactly that.
The salvaged commit put `_pre_exec_block` behind the command's
`run_bounded_sync` deadline but let a timed-out guard fall through into
execution. The gateway-lifecycle, dangerous-workdir and self-repo checks
apply unconditionally (`force=True` cannot bypass them), so a guard that
never rendered a verdict must not let the command run unguarded: return
the terminal error envelope (`status: error`, "did not finish ... Retry
the call") instead, mirroring how the bounded `env.execute` path reports
its own expiry as a result rather than continuing.
Tests trimmed to the two invariants: a wedged guard returns a bounded
error without executing; a completed guard keeps its verdict (pass ->
execution, rejection -> its own blocked result).
Squashed integration of the user-facing message audit for this surface set.
Full per-finding receipts: /tmp/ux-audit/lanes/*-receipt.md (campaign artifacts).
Port from Kilo-Org/kilocode#13224: fixed waits belong in foreground terminal calls, while background mode is reserved for independently running processes.
A multiplexed dashboard can call _ensure_terminal_env_bridged while a
secondary profile's HERMES_HOME override is active. The one-shot latch then
wrote that profile's docker policy into process-global os.environ and poisoned
later unscoped launch-profile tool calls (#107422).
Skip the ambient bridge whenever a context-local home override is set —
ambient env is launch-profile authority only; routed profiles must use
terminal_scope (same rule as env_loader._reapply_terminal_config_bridge).
(cherry picked from commit 2050efb24fdc9d54a282b24d0042b90f47486c5a)
A message typed while the agent runs (CLI busy_input_mode=interrupt, gateway
priority redirect, ACP redirect) goes through AIAgent.redirect(). During tool
execution redirect() degrades to steer(), whose delivery rides the tool result
— so a long foreground command (a `sleep 285` CI poller, a build) parked the
user's message until it exited. The UI printed "Redirected current turn" while
nothing happened for minutes.
redirect() now also asks the tool workers to YIELD (tools/interrupt.request_yield).
The local terminal backend's wait loop honours it: the drain thread is stopped, the
still-running Popen is adopted by the process registry as a notify_on_complete
background session (ProcessRegistry.adopt_local — output so far seeds the buffer,
the registry reader continues from the pipe), and the tool returns immediately with
status "yielded_to_background" + session_id. The command is never killed; the
completion notification arrives as usual and process(poll/wait/log/kill) work on it.
Non-local backends and internal env.execute() consumers pass no yield_handler and
are unaffected; a stale yield bit is cleared with the interrupt bit per worker tid.
DockerEnvironment.cleanup() runs docker stop + docker rm -f on a daemon thread and
keeps the handle on the env. The idle reaper pops the env out of _active_environments
BEFORE calling cleanup(), and the atexit drain only iterated that registry — so a
detached env's worker was unreachable and died with the interpreter, leaving a
stopped (or, if the exit came fast enough, still running) labeled container behind
while the log said the environment was cleaned.
Every teardown worker is now also recorded in a module-level set in docker.py;
_atexit_cleanup joins that set after the registry pass via
DockerEnvironment.wait_for_all_teardowns (re-snapshotting each pass, since the
reaper can start a worker while the drain runs). Finished workers are dropped from
the set on each drain so it cannot grow across a long gateway life.
Mechanism from #86344 by @PRATHAMESH75; this is the slim redo — one shared set
plus one static drain hooked into the existing terminal_tool atexit, instead of a
second atexit registration in docker.py.
On hosts where Docker ships as a snap (Ubuntu cloud images / Azure VMs), the
snap's AppArmor confinement turns two sandbox hardening flags into a dead
container at start: `--init` fails with "exec /sbin/docker-init: operation not
permitted" and `--security-opt no-new-privileges` then fails every exec the
same way ("exec /usr/bin/sleep: operation not permitted"). This is snapd
LP#1908448 — not probeable from the client, and docker_extra_args cannot remove
flags we add.
`terminal.docker_snap_compat: true` drops exactly those two flags; cap-drop ALL,
the tmpfs hardening, PID limits and the privdrop caps are unchanged, and a
warning is logged at container start. Bridged everywhere the other docker_*
keys are (CLI env map, gateway env map, `hermes config set` sync, terminal_tool
env read, the shared container_config shaper, DEFAULT_CONFIG).
_HOST_CWD_PREFIXES only listed C:\ and C:/, so a D:\ or E:/ working directory on a
Windows host reached docker run -w and the container started in a directory that does
not exist. The two prefix scans now go through one _is_host_cwd predicate that matches
any drive letter with either slash.
Independent review: an over-cap timeout on `cmd &` was promoted, so the
tracked shell exited at once while the payload ran untracked (the exact
thing the '&' guidance exists to prevent); and the note promised a
notification even on finite sessions where async delivery is disabled and
the spawn had already cleared notify_on_complete. The detachment guidance
now runs before the promotion decision regardless of timeout, and the note
reads the spawn's actual notify_on_complete: notification wording when kept,
poll-only wording when the session cannot receive one.
Tests (2 new): '&' and nohup with an over-cap timeout still refuse; the note
matches the delivery capability.
`Foreground timeout Ns exceeds the maximum of 600s` was the second most
frequent tool-layer error in the 1,393-agent refactor run: 454 refusals
(285 asked 900 s, 41 asked 620, 20 asked 1800), 251 of them test-suite
invocations. Every retry was mechanical: lower the timeout (167), split into
sleeps (82) or re-send with background=true (86). The schema invited it
("set high for long tasks", max 600) while the suites take 10-30 min.
An over-cap foreground timeout is a bounded job the caller wants to wait
for. terminal_tool now runs it as a tracked background process with
notify_on_complete=true and says so in the result
(`promoted_from_foreground`, naming the requested and cap seconds, with
"do NOT re-run it"); the schema text describes the new behaviour. The
`&`/nohup/setsid and long-lived-server guidance stays a refusal: those need
the command itself rewritten, which the tool cannot do safely.
Live (real local backend): main refuses `sleep 1; echo LIVE_OK` at
timeout=900; branch returns a proc_* session with notify_on_complete and
the note, and the command runs once.
Tests: the promotion test runs a real command through the real registry and
asserts the result shape plus that it executed exactly once in the
background; `&` still refuses; schema text updated. Terminal/process suites
(27 files) 378 passed.
Every PLUGIN-COMPAT __getattr__ now calls hermes_cli.plugin_compat.warn_once(facade, name, target) before
resolving, emitting a HermesPluginCompatWarning (FutureWarning) once per process per name: old path, new
path, removal target. Importing a facade for its live API stays silent; only resolving a moved name warns.
COMPAT_MANIFEST.md documents the warning and how to silence it during migration.
Verified the runtime never routes through a pointer: every entry point (run_agent, cli, hermes_cli.main,
gateway.run, tui_gateway.server, web_server, model_tools + tool discovery, hermes_state, cron.scheduler,
browser_tool, mcp_tool, kanban, auth) imports clean and `hermes doctor` runs end to end with the warning
promoted to an error.
Also restores the check_compat_pointers CI step to .github/workflows/lint.yml, which a0be177aac dropped
when the compat layer was regenerated (the lint script itself was present; the workflow step was not).
hermes_cli/plugin_compat.py, tests/test_plugin_compat_warning.py and the two-line insert per facade are
part of the compat layer and go away with it.
The Sep 2026 decomposition (PR #102117) makes internal import paths a non-API: names now live in
the focused modules that define them. This commit is the ONLY thing keeping the old paths alive,
so external plugins have time to update. It is deliberately a single, unsquashed commit:
git revert <this sha>
removes every shim, stub and manifest at once on the announced date. Nothing in-tree may depend on
these pointers: scripts/check_compat_pointers.py (wired into lint.yml) fails CI if it does.
What it adds (see COMPAT_MANIFEST.md, compat_manifest.json):
- 332 facade modules get one delimited `PLUGIN-COMPAT` block appended at the end of the file
- 1,172 moved names resolved lazily via a module `__getattr__` (PEP 562) — never a top-level import,
so no import cycles; facades that already had `__getattr__` get a chained one
- 592 third-party/stdlib names the old modules used to expose, with their original import statements
- 266 public definitions that had been deleted as unused, restored byte-for-byte from the pre-decomposition
tree (+40 private helpers and 16 imports pulled in only because a restored definition needs them)
- 3 deleted modules recreated as re-export stubs (gateway/startup_watchdog, hermes_cli/observability/
relay_runtime, tools/environments/modal_utils)
- private names (`_x`) get no pointer: they were never API (3,792 skipped)
Verified: all 335 touched modules import under a fresh HERMES_HOME and every manifest name resolves;
the lint reports zero in-tree uses; ruff clean; targeted suites unchanged.
For each issue anchor present in BASE 63279301bc non-test .py and absent on HEAD, the BASE comment/docstring block was re-attached at the HEAD location of the code it explained (matched by the distinctive code line / enclosing def). Sentences already covered by an existing HEAD comment were deduped; the issue number always survives. Insert-only: no code lines changed.
