`hermes setup` launched by install.ps1 on PowerShell 5.1/conhost printed
`[35m`/`[0m` literally. hermes_cli.colors and the skins emit raw SGR codes
whenever stdout is a TTY, but no Hermes entry point ever set
ENABLE_VIRTUAL_TERMINAL_PROCESSING (origin/main didn't either; only
pm/cli.py did, for its own progress line), and shells hand native children
a console with VT off.
hermes_bootstrap now opts the stdout/stderr console in at import, before
anything prints and before the venv relaunch (the mode lives on the console
buffer, so the child inherits it). Non-console handles are left alone; a
console that refuses VT gets NO_COLOR so it shows plain text instead of
escape garbage. ctypes only: colorama is merely transitive.
Conflicts:
- hermes_bootstrap.py: main calls install_never_free_environ() in the
apply-on-import block right after the console fixes, where pm-clean runs its
PM block (activation, relaunch, environ writes) instead of
activate_durable_lazy_target(). It now runs ahead of the whole PM block,
main's order, so the glibc < 2.41 guard is in before anything writes
os.environ.
- tests/test_hermes_bootstrap.py: pm-clean dissolved the entry-point class and
dropped its source-reading test; main's new
test_library_imports_of_dual_use_entry_modules_stay_side_effect_free lands
as a module-level function.
ctypes.addressof(fresh) was the one audited call inside the retry loop, so a
hook writing os.environ on every ctypes event bumped the generation each pass
and the loop never exited (hung past 20 s on glibc 2.39 with a full array).
ctypes.cast(...).value reads the same address unaudited, leaving no audited call
in the loop. Stress on glibc 2.39 (8 threads x 50k writes, 2 native getenv
readers, 2,100 nested hook writes), twice: 0 crashes, 0 lost, C == os.environ.
Every ctypes call in the wrapper is audited, and a hook may write os.environ
re-entrantly under the RLock. The ctypes.addressof reads after the live array
was walked let such a write land, and the outer publish then overwrote it
(200/200 nested names lost on glibc 2.39). Store each array's address with it,
and redo the read when a publish generation changed underneath.
Review of the never-free environ wrapper found two defects on glibc < 2.41:
- Lost updates: two threads adding new names each copied the live array and the
later publish dropped the other's names (8x300 names left 411/2400 visible to
C getenv and to children; a concurrent replacement could be undone). One
RLock now covers the read-build-publish of os.putenv and glibc's in-place
shift in os.unsetenv; a fork hook keeps a child from inheriting it held.
- Unbounded leak: every set/del cycle of the same name built a fresh array and
entry string (~1.2 KB/cycle; kanban ticks, the spinner pause and
_restore_env churn names forever). Like glibc 2.41, entry strings are now
cached per NAME=value and new names are appended in place to an array with
spare room; only a full array is replaced (doubling), never freed.
Replacements are also done in place by the wrapper (found via getenv's pointer,
no string reads), so os.putenv raises exactly one os.putenv audit event again,
matching the real function. Comments cover the publish ordering (TSO vs
aarch64) and the residual native-setenv case.
The tui_gateway intermittently died with SIGSEGV (rc=-11) on CI with
session.create (rpc id=1) in flight. Root cause: glibc < 2.41 reallocs the
environ array and frees the old one when setenv adds a NEW name. session.create
turns on gateway prompts (os.environ.update of three new HERMES_* names) while
the picker-prewarm thread is inside getaddrinfo/OpenSSL with the GIL released,
walking environ in C getenv; the freed slots hold mangled tcache pointers, so
the walk segfaults the whole process. The agent build (gateway.run import-time
env, HERMES_SESSION_ID) and the title thread hit the same window.
Reproduced in ubuntu:24.04 (glibc 2.39, the CI runner's) with the real gateway
loop: faulthandler names the picker-cache-prewarm thread in socket.getaddrinfo
as the faulting thread, matching the CI dump thread-for-thread. glibc 2.41+
never frees an environ array, which is why it never reproduced on newer hosts.
hermes_bootstrap (imported first by every entry point) now publishes new names
in a fresh array with one pointer store and never frees an array, the same fix
glibc 2.41 made. Replacing or removing an existing name was already in place.
Inactive on glibc >= 2.41, musl, macOS and Windows.
An install editable-built from main maps only the top-level packages it
saw then (setuptools' flat-layout finder: no `pm`) and never puts the
checkout on sys.path. After moving that checkout to this tree, `hermes`
died in hermes_cli/__init__.py: `__version__` was evaluated at import
through pm.paths, before anything had put the root on sys.path.
Two layers of the same break:
- hermes_cli.__version__ is served lazily (PEP 562). Shipped updaters
still get it from `from hermes_cli import __version__`; importing the
package no longer needs pm or reads the stamp.
- hermes_bootstrap hardens the import path before its first Hermes
import, not only on the legacy post-swap branch. Without that, its
`from pm.environments import ...` raised ModuleNotFoundError, which
hermes_cli.main swallows, so the launch silently skipped
prepare_launch: no blessed-checkout adoption and no PM sync, and the
tree ran on main's dependencies until it hit ruamel.
Reproduced with a real main editable venv swapped to this tree: exact
user traceback before; after, a stamp-less blessed checkout adopts, PM
syncs, relaunches, and the command answers. The new test drives both
imports through a pre-PM style finder and is red with either half
reverted.
prepare_launch treated "dependencies current" as "update finished", but the
sync commits the generation before the product builds and the maintenance
run. A crash between the two left a current install that never built anything
and never would. The tail is now owed by a pending marker written before the
sync and cleared after the tail succeeds, so a repeat launch finishes it.
That tail imports the application, whose entry point is this same function:
inside the launching process's update-lock claim (holder pid is our ancestor)
prepare_launch is a no-op, otherwise the marker recursed forever. The claim
also keeps `hermes update` from racing the launch-time completion.
Completion output goes to stderr: it runs in front of whatever the user
typed, which may be emitting machine-readable stdout. Metadata queries
(--version, -V, --help) skip completion entirely; they read no dependencies.
A completion that fails offline no longer exits 1 from hermes_bootstrap: the
previous generation is still selected (a failed sync commits nothing), so
warn, point at `hermes update`, and launch.
Resolved toward the branch: PM provisions uv/python (main's install.ps1 uv-shim
salvage + its test and workflow steps dropped), the shim re-exec stays retired,
package.json carries no electron-builder block (afterExtract identity stamp wired
into electron-builder.config.cjs instead; after-pack.mjs keeps signing only),
Desktop workspace-deps helpers stay retired. Main's scratch-dir bootstrap
(export_scratch_tmp_env) is taken and re-run after profile resolution.
Hermes and everything it launches (browser profiles, PTY probes, skill scripts,
tempfile defaults in delegated code) wrote to the system temp dir, which is a
RAM-backed tmpfs on most Linux hosts and containers and fills under agent load.
- hermes_constants.get_scratch_dir(): HERMES_HOME/cache/scratch (0700), entries
older than 72h pruned once per process / once per hour across processes.
- apply_scratch_tmp_env(env) / export_scratch_tmp_env(): TMPDIR/TMP/TEMP point at
the scratch dir when the user or OS has not set them; a value Hermes itself
exported (== HERMES_SCRATCH_DIR) is re-derived for a re-homed process or a
child served under another profile, so profiles never share scratch.
- hermes_bootstrap runs the export on import (every entry point); hermes_cli.main
re-runs it after --profile resolution; the subprocess HOME contract
(apply_subprocess_home_env) and the routed-home rewrites in code_execution_env
and served_profile_child_env apply it to child envs.
- The runtime-environment prompt block names the scratch dir so the model stops
reaching for the system temp dir by reflex; hermes doctor reports the dir, its
size and whether a user TMPDIR overrides it.
hermes_cli.runtime_paths (venv generations, selection, activation) moves to
pm.environments, and gains venv_bin_dir / venv_python / project_python. Every
in-tree caller asks pm for an interpreter now; pm no longer reaches back into
hermes_cli for its own environment layout (pm.packages, pm.extras, pm.ensure,
pm.paths imported hermes_cli.runtime_paths). The three open-coded
"Scripts/python.exe or bin/python" ladders in pm collapse onto venv_python.
hermes_constants.venv_python_path / venv_bin_dir and hermes_cli.runtime_paths
stay as frozen-updater-surface shims only (tests/compat/old_updater_surface.json).
To keep the boot path light, pm/__init__ resolves its facade lazily (PEP 562)
and pm.registry loads the built-in package definitions on first read instead of
at import: `import hermes_bootstrap` now loads pm + pm.environments only (25ms,
was 37ms with the eager facade dragging in the downloader). The stripped-payload
fixtures that ship only pre-import files keep working for the same reason.
Also restores two frozen-surface re-exports the F401 sweep dropped
(banner._github_compare_behind, cua_backend.resolve_cua_driver_cmd).
Both installed racers caught any non-OSError from the Happy Eyeballs core and
re-ran the stock serial ``create_connection``. That branch was untested and
its only effect was to hide a bug in the racer by silently reintroducing the
exact IPv6-first stall the racer exists to remove. OSError (every candidate
failed) still propagates unchanged, identical to the serial original; anything
else now raises. Invariant test drives both racers with a raising core.
Wire the racer #114277 added once, at the seam every Hermes process already
crosses first: ``hermes_bootstrap`` (imported before anything else by
``hermes``, ``hermes-agent``, ``hermes-acp``, ``gateway.run``, ``batch_runner``,
``tui_gateway.entry`` and the slash worker). Installing it from
``init_agent`` was too late for the startup path the report is about — the TUI
gateway starts MCP discovery and the model-catalog prewarm before any AIAgent
exists, and ``hermes model`` / picker prewarm never build one.
- Move the stdlib-only racer core (``_happy_eyeballs_create_connection``,
``_interleave_addrinfos``) and the installer into ``hermes_bootstrap`` and
apply it on import; ``agent.process_bootstrap`` keeps only the httpcore
backend and imports the racer from there. The bootstrap must stay stdlib-only
because entry points call ``harden_import_path()`` after importing it.
- Patch urllib3's own serial connect walker lazily through a one-shot
``sys.meta_path`` hook instead of importing urllib3 eagerly: ``hermes`` and
the TUI gateway never load urllib3 at start, and importing it costs ~50 ms.
- Idempotence by a marker on the installed function, so a re-import of the
bootstrap (tests, ``importlib.reload``) never wraps the racer twice.
- Drop the ``init_agent`` call site (redundant: ``process_bootstrap`` imports
the bootstrap) and trim the five added tests to two invariants in the
mirroring ``tests/test_hermes_bootstrap.py``; the racer unit test moves its
monkeypatch seams to the new module.
- Docs: ``network.force_ipv4`` now describes the default racing behaviour.
Live A/B (stub resolver: blackholed 100::1 first, local IPv4 second, driving
the real clients after ``import hermes_cli.main``): catalog fetch via requests
5.05s -> 0.39s, shared keepalive httpx client 15.24s -> 0.32s, inline
httpx.Client 5.01s -> 0.26s; IPv4-only control 0.01s both sides. Refused ports
still fail instantly with the same exception types; a blackhole-only host still
raises ConnectTimeout at the configured connect timeout.
Fixes#114265
Old updaters keep running after the checkout changes. Returning None
from their removed uv helpers enables a pip fallback against the new tree.
Keep the historical imports as inert shims and stop dependency entrypoints
with a relaunch message instead. Do not call PM or write recovery markers
from that mixed-version process.
Self-managed source launches use PM's successful input stamp to decide
when dependencies need a sync. Restart on the managed interpreter before
activating the selected generation. Preserve launcher forms and options,
and do not sync while a live updater owns the installation.
Targeted runtime batch: 225 passed, 8 platform skips. Real PM worker tests
build and publish disposable dependency generations, retain prior state on
failure, and exercise fresh-process relaunch before dependency activation.
The final launch guard test also passes. Full suite and native Windows
execution were not run locally.
Pin uv and uvx to the PM interpreter instead of ambient Python discovery.
A matching dependency stamp cannot prove that installed files still exist.
Repair now rebuilds the recorded workspace and lock in a fresh generation,
checks startup imports, and publishes the selection only after success.
Run startup recovery before dependency activation. Keep manual PM repair
reachable when the selected environment is damaged. Preserve plugin
selection, retry ownership, and the previous generation on failure.
Remove the separate pip, ensurepip, per-extra, and install-time quarantine
ladders. Keep orphan launcher restoration.
Verification: 717 targeted tests passed on native Windows ARM64, with
56 skipped. Ruff, diff checks, and the source-scoped compat check passed.
A disposable real Hermes install recovered deleted YAML and dotenv files,
then printed CLI help with exit 0. Its lock and stamp stayed unchanged.
The full suite and a release build were not run for this change.
Prepare dependency generations before selecting them. Keep shipped tool
bytes separate from writable additions, and store facts beside their entries.
Validate proposed plugin sets before config publication. Restore the previous
config if the facts write fails.
Consolidate duplicate updater, backup, setup, and voice helpers. Repair
launcher selection, dependency consumers, download ownership, update feeds,
and native Windows process and file handling.
Verification: 206 changed/prior-failing Python files reported 4630 passed,
one failed, and 330 skipped. Fix the remaining Hindsight fixture boundary.
The final targeted rerun reported 234 passed and two skipped. The store
review regression batch reported 83 passed and one skipped. Desktop
TypeScript checks, 56 selected Electron tests, 24 release tests, and the
removed-import/compatibility guards passed.
This is an integration checkpoint, not full audit acceptance. The complete
Python suite has not run on this fixed tree. Crash-atomic plugin publication,
generation cleanup, receipt correlation, and packaged lifecycle acceptance
remain open in docs/pm-audit-status.md.
Introduce the pm store: a unified, hash-verified package store that
replaces lazy_deps and the old installer's ad-hoc tool downloads.
Store tools are provisioned on PATH (ffmpeg, node/npm via pinned uv),
with a resumable 8-way downloader, verify() returning failure reasons,
and adopt() made EPERM-safe. chromium ships in the payload for every
target. The 3600-line install.sh is replaced by a staged bootstrapper
(heavy deps are pm's job after this); setup-hermes.sh, Dockerfile and
nix pin tables are rewired onto the store. Old install-script tests,
lazy_deps/managed_uv/build_info, and the ps1/bash installer test
batteries are removed with the machinery they tested.
Rebuilt from ethie/pm onto upstream/main (ac6c8028e0) after the
utf-8-sig sweep. 16 hot files (main also churned them) hand-merged:
platform adapters, main.py, electron/main.ts, tui_gateway/server.py,
cua_backend, installer-tests workflow, install.sh (full rewrite),
setup-hermes.sh, plugins doc.
Two gaps found auditing the decode-crash cluster:
1. suppress_platform_ver_console() only ran in hermes_cli.main processes;
slash workers, tui_gateway/entry, run_agent, batch_runner, and cli.py
import only hermes_bootstrap and were exposed to both the console
flash and (on Python 3.11.0/3.11.1, which lack CPython's
encoding='locale' fix) a UnicodeDecodeError inside platform.win32_ver()
under PEP 540 — the crash #69413 reported. Move the stub into
hermes_bootstrap so every entry point gets it; the _subprocess_compat
copy stays for non-bootstrap callers.
2. The desktop Electron spawn built the backend env without PYTHONUTF8,
so anything the Python child emitted before hermes_bootstrap ran
(interpreter startup errors, pre-bootstrap tracebacks) decoded with
the locale default. Re-port of PR #56499's env half (echoriver89) to
backend-env.ts (original targeted the deleted backend-env.cjs);
explicit user setting wins.
* fix(windows): stop terminal-window popups from background spawns
Native-Windows desktop/gateway users saw cmd/conhost windows flash on
gateway restart, image paste, the dashboard Projects tree, voice notes,
and ~5 min after closing the app (detached cron). Two root causes:
- Console-subsystem exes (taskkill, schtasks, wmic, netstat, tasklist,
agent-browser, git, ffmpeg, powershell, git-bash) spawned via raw
subprocess allocate a fresh console when the launching process has
none (pythonw desktop backend / detached gateway) - even with output
captured.
- uv venv pythonw shims re-exec console python.exe, so Python children
get a console regardless of how they're launched.
Fixes:
- Single hidden-spawn primitive (_subprocess_compat.run/.popen) that ORs
CREATE_NO_WINDOW on Windows, no-op on POSIX. Route every Hermes-owned
console-exe spawn through it.
- FreeConsole() catch-all in hermes_bootstrap: any Python child that
exclusively owns an auto-allocated console detaches it at startup
(GetConsoleProcessList()==1 gate leaves shared interactive consoles
untouched).
- Replace PowerShell/wmic gateway PID scans with in-process psutil.
- Skip schtasks queries on non-interactive desktop restarts.
- Prefer native agent-browser .exe over .cmd shims.
- Guard test bans raw subprocess spawns of the Windows-only console
tools repo-wide so the popup class can't regress.
* fix(windows): scope FreeConsole to background entry points; fix merge fallout
Console detach review (per #53810 feedback): GetConsoleProcessList()==1 can't
tell a uv pythonw->python phantom console apart from a user opening the
interactive CLI/TUI in its own fresh console (double-click, shortcut, ConPTY) —
both report a single attached process with a tty. Running FreeConsole() in the
import-time bootstrap therefore risked detaching a legitimately-interactive
terminal.
- Extract FreeConsole into explicit hermes_bootstrap.detach_orphan_console();
remove it from apply_windows_utf8_bootstrap() (import side effect).
- Call it only from known background mains: gateway run, dashboard backend
(start_server, what the desktop spawns), cron standalone, tui_gateway entry,
slash worker. Interactive CLI/TUI never calls it.
- Behavior-contract tests: frees only when solo owner, leaves shared console,
no-op without console / on POSIX, and asserts it's not an import side effect.
Merge fallout from origin/main (#53791):
- local.py: 3-way merge left a dangling **_popen_kwargs (NameError crashing
every terminal init). _subprocess_compat.popen already hides the window, so
drop it.
- discord adapter: merge stacked an undefined windows_hide_flags() onto the
primitive call; drop the redundant arg.
- test_gateway: scan now goes psutil-first (zero spawn); rewrite the
case-variant test to drive that production path.
* test(claw): mock _subprocess_compat.run seam for Windows process scan
claw.py's Windows tasklist/powershell scan routes through the hidden-spawn
primitive; the tests still patched claw_mod.subprocess, so on win32 the mock
was never hit and real spawns returned nothing. Patch the actual seam.
The published Docker image seals the agent venv (root-owned, read-only
/opt/hermes) and sets HERMES_DISABLE_LAZY_INSTALLS=1 so a runtime install
can't mutate and brick the core. But opt-in backends (Firecrawl web search,
Exa, Feishu, ...) deliberately keep their SDKs in tools/lazy_deps.py and out
of [all] (pyproject policy 2026-05-12: one quarantined release must not break
every install). The two policies collided: the SDK isn't baked in AND can't
lazy-install, so the default Firecrawl web_search/web_extract fail out of the
box in Docker (#51136), as do Exa (#49445) and Feishu (#50205).
Fix the whole class instead of baking in one backend: when
HERMES_LAZY_INSTALL_TARGET is set, lazy installs are redirected to a writable
dir on the durable /opt/data volume via `pip/uv install --target`, and that
dir is APPENDED to the end of sys.path. Because the core venv always wins
name collisions, a package installed this way can only ADD new modules — it
can never shadow, downgrade, or break a module the core ships. The worst a
bad/incompatible backend package can do is fail to import and report itself
unavailable; the agent core stays healthy. That structural guarantee is what
made it safe to seal the venv, and it is preserved here even with installs
re-enabled.
- tools/lazy_deps.py: durable-target mode — `--target` install + core-pinned
`--constraint` file (shared deps resolve to core's versions, conflicts fail
loudly at install time), append-only sys.path activation, ABI/Python-version
stamp that wipes the store if an image rebuild bumps the interpreter, and a
reworked gate so HERMES_DISABLE_LAZY_INSTALLS=1 redirects (rather than hard-
blocks) when a target is set. security.allow_lazy_installs=false still
disables installs in every mode.
- hermes_bootstrap.py: activate the durable target on sys.path at first import
(before any backend imports its SDK) so packages installed on a previous run
are importable on this run.
- Dockerfile: set HERMES_LAZY_INSTALL_TARGET=/opt/data/lazy-packages.
- docker/stage2-hook.sh: seed + chown the dir on the data volume.
- tests: real-install E2E proving installs land in the target, import cleanly,
don't leak into the sealed venv, and that a core package is never shadowed;
ABI-stamp wipe/preserve; gate matrix; Dockerfile/stage2 contract test.
Fixes#51136
Launching Hermes from a directory that ships its own top-level package with a
Hermes-internal name (utils/, proxy/, ui/) crashed the gateway/TUI child with
an ImportError (exit 1, crash loop): from utils import atomic_replace resolved
to the user's package.
tui_gateway/entry.py already stripped the relative cwd forms ('' / '.'), but
the launch dir also reaches sys.path as its own ABSOLUTE path (venv activation
or a project that adds itself to PYTHONPATH), which the strip missed and which
sat ahead of the Hermes root.
Centralize a hardened guard in hermes_bootstrap.harden_import_path(): drop the
relative forms AND force the Hermes source root to the front even when an
absolute cwd entry is present. Wire it into tui_gateway/entry.py and
acp_adapter/entry.py (both spawn into arbitrary cwds); hermes_cli/main.py and
gateway/run.py already insert the root at front. gatewayClient.ts now also
exports HERMES_PYTHON_SRC_ROOT for defense in depth.
Codebase-wide fix for Python-on-Windows UTF-8 footguns, complementing
the earlier execute_code sandbox fixes (which remain load-bearing for
when the sandbox explicitly scrubs child env).
Problem: Python on Windows has two long-standing text-encoding pitfalls:
1. sys.stdout/stderr are bound to the console code page (cp1252 on
US-locale installs) — print('café') crashes with UnicodeEncodeError.
2. Subprocess children don't know to use UTF-8 unless PYTHONUTF8 and/or
PYTHONIOENCODING are set in their env — so any Python we spawn
(linters, sandbox children, delegation workers) hits the same bug.
Solution: A tiny bootstrap module (hermes_bootstrap.py) imported as the
first statement of every Hermes entry point:
- hermes_cli/main.py (hermes / hermes-agent console_script)
- run_agent.py (hermes-agent direct)
- acp_adapter/entry.py (hermes-acp)
- gateway/run.py (messaging gateway)
- batch_runner.py (parallel batch mode)
- cli.py (legacy direct-launch CLI)
On Windows, the bootstrap:
- os.environ.setdefault('PYTHONUTF8', '1') (PEP 540 UTF-8 mode)
- os.environ.setdefault('PYTHONIOENCODING', 'utf-8')
- sys.stdout/stderr/stdin.reconfigure(encoding='utf-8', errors='replace')
Children inherit the env vars → they run in UTF-8 mode.
Current process's stdio is reconfigured → print('café') works now.
On POSIX (Linux/macOS), the bootstrap is a complete no-op. We don't
touch LANG, LC_*, or anything else — users who have intentionally
configured a non-UTF-8 locale aren't affected. POSIX systems are
already UTF-8 by default in 99% of modern setups, so there's nothing
to fix.
setdefault() (not overwrite) means users who explicitly set PYTHONUTF8=0
or PYTHONIOENCODING=cp1252 in their environment are respected.
What this does NOT fix: bare open(path, 'w') calls in the *parent*
process still default to locale encoding because PYTHONUTF8 is only
read at interpreter init. A ruff PLW1514 sweep (separate follow-up)
will add explicit encoding='utf-8' at those ~219 call sites for
belt-and-suspenders.
Tests (17): 16 passed, 1 skipped on Windows.
- Windows: env vars set, stdio reconfigured, child inherits UTF-8 mode
- POSIX: complete no-op (verified on fake POSIX + skipped on real
POSIX since we don't have a Linux box in this session)
- Idempotence: multiple calls safe
- Graceful degradation: non-reconfigurable streams don't crash
- User opt-out: explicit PYTHONUTF8=0 is respected
- Load order: every entry point's FIRST top-level import is
hermes_bootstrap, enforced by an AST-level parametrized test
pyproject.toml: added hermes_bootstrap to py-modules so it ships with
pip installs.
