The dashboard build (Vite 8 + Rolldown + Tailwind v4) saturates every CPU
on small hosts (200%+ top readings, OOMs and frozen SSH sessions on 2–4
vCPU VPS boxes). Rolldown exposes no JS-level worker knob, but its native
binding parallelizes with rayon, which honours RAYON_NUM_THREADS.
Cap both sides of the build, everywhere it runs:
- scripts/build/web.mjs (the npm run build / packaging entry): limit the
native bundler's rayon pool to half the available cores before vite is
imported, and re-exec once with a --max-old-space-size cap (sized from
the cgroup memory limit, 75%, floored at 1 GB, ceiling 4 GB) unless
the user already set one.
- hermes_cli/web_build_limits.py + build_source_web(): the same caps for
the install/update/dashboard-triggered source build path.
User env always wins (existing NODE_OPTIONS heap flag or
RAYON_NUM_THREADS is preserved); HERMES_WEB_BUILD_MAX_OLD_SPACE_SIZE /
HERMES_WEB_BUILD_THREADS tailor the caps, and HERMES_WEB_BUILD_LIGHT=1
tightens them (1 thread, 1 GB heap) for hosts that cannot spare full CPU
during the build — the light-build idea from the closed PR #78131.
The reuse receipt check in node-deps.mjs accepted a tree where every locked
directory existed but package contents had been stripped (partial cleanup or
snapshot restore). npm trusts the hidden lockfile and never repairs such
entries, so every 'hermes update' skipped npm ci, the TUI build died on the
missing esbuild manifest, and the stale receipt kept the loop closed (#128935).
Probe each locked entry's package.json — the same file product builders
resolve tools through — so a gutted tree falls back to a real npm ci.
publishDirectory moved a finished product into place with one renameSync.
On Windows an antivirus scanner opens every freshly written file, so the
rename onto build/native-deps (right after hud-modifier-monitor.exe is
compiled) or hermes_cli/web_dist failed EPERM and threw away a build that
took minutes.
retryHeld retries EPERM/EACCES/EBUSY for about 3s and raises every other
code at once. All three publication renames use it, as does the HUD
helper's staging rename. The previous product's backup now sits beside
the output instead of inside withProduct's scratch tree, so a rollback
that also fails no longer has its only copy deleted by scratch cleanup;
the error names where it is.
Co-authored-by: dskwe <6286305+dskwe@users.noreply.github.com>
Co-authored-by: juliorobertoalvarez <39749815+juliorobertoalvarez@users.noreply.github.com>
Co-authored-by: bighamx <10262931+bighamx@users.noreply.github.com>
Co-authored-by: lzmuhioin <1814826+lzmuhioin@users.noreply.github.com>
WIP for the freshness-guard-os-metadata unit: the UpdateLock claim in
complete_source_checkout plus its tests are complete but unrun; the
Thumbs.db/Desktop.ini skip predicate is in but its test rows are not yet
extended, and the rmTree ENOTEMPTY test has not been written yet.
Opening the checkout in Finder mid-build lets .DS_Store, AppleDouble ._*
sidecars or .localized land inside a hashed input dir (apps/desktop, apps/shared,
assets, scripts/build). treeHash then sees a new input and recordProduct throws
"Build inputs changed during compilation; retry the build", so `hermes update`
rebuilds the desktop app, fails closed, and never prints Update complete.
Finder metadata is never a build input, so skip it as a baseline in treeHash —
covering source, output and prepared trees uniformly rather than only the source
predicate. A real source change is still detected and still names nothing new to
hash, so clean checkouts (CI) hash byte-identically.
macOS's Finder/Spotlight can drop a fresh .DS_Store (or xattr) into a
directory while we're removing it, making rmSync throw ENOTEMPTY on the
first pass. That file is cosmetic garbage — retry the removal so the
race can't abort an otherwise-clean product build.
Salvaged from #123078 (its freshness.mjs hunk is subsumed by the broader
metadata skip; the unrelated CI-workflow deletions in that PR are not
included).
Co-Authored-By: dlshumway <85191357+dlshumway@users.noreply.github.com>
write-build-stamp.mjs rewrites the stamp's `builtAt` clock on every build, so the
whole-file hash moved under a receipt that had just recorded it. Two builds
overlapping on one tree failed with "Build inputs changed during compilation" for
a difference the build machinery itself made, and the caller turned that into
SystemExit(1).
Hash the stamp's provenance identity instead, key-sorted so a cosmetic
re-serialisation is not a change. The clock is excluded from that comparison but
recorded as `stampClock` — the value the bake actually embedded, captured from
the same read that produces __HERMES_INSTALL_STAMP__ — because packaging copies
the live stamp into the bundle after the receipt is written. Re-reading the stamp
at record time would certify a clock the output never contained, and
productCurrent would then report a bundle current whose packaged clock disagrees
with its baked main, which is exactly what detectBundleSwap must never do.
recordProduct still compares the prepared inputs, so a clock-only restamp cannot
throw. productCurrent additionally requires the recorded clock to match, so a
restamp between bake and packaging is a rebuild rather than a silent divergence.
A receipt predating the field compares by hash alone until the next build
rewrites it; the release packaging path never consults productCurrent.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
write-build-stamp.mjs rewrites `builtAt` on every build, and build.mjs runs it
before compilation, so the desktop build rewrote a file its own receipt hashed
as a prepared input. Two concurrent `hermes desktop` builds therefore made
recordProduct() fail with "Build inputs changed during compilation" on a change
the build itself caused, and the caller turned that into SystemExit(1).
Hash the stamp's provenance identity instead of its raw bytes. builtAt is the
only field that moves between two stamps (the tree documents it as
"differs on every rebuild"), and it is a build clock, not provenance: the baked
output changes when commit/payload/tag change, never when time passes. A
non-JSON stamp still falls back to the whole-file hash, so content keeps
deciding freshness and only the clock is ignored.
Unlike b3618bda35, the stamp cannot simply leave the hash set:
bundleElectronMain bakes its bytes into electron-main.mjs, and
bundle-swap.test.ts pins "same commit rebuilt (builtAt moved)" as a real swap,
so freezing the clock would break a committed test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Nix agent derivation builds scripts/build/*.py from a fileset that
does not include pm/, so importing pm.store.ALL_TARGETS there failed
nix flake check with ModuleNotFoundError. Extend the local target
regex with linux-(x64|arm64)-musl instead.
npm_execpath can point through a symlink, and the manifest sits beside
the resolved CLI, never beside the link: resolve the realpath before
looking for package.json. Layouts without a readable manifest now fall
back to the pre-fix child probe instead of aborting with ENOENT.
Move the regression test to tests-js/node-deps.test.mjs (the module's
own suite) and cover the symlinked-execpath and fallback lanes there.
The npm version probe spawned node-under-node before the reuse
short-circuit, so on Windows a Job-Object EBUSY spawn failure aborted
the whole dependency preparation even when the install was already
complete — leaving the pending-completion marker behind and turning
every launch into the same doomed completion pass (#123933).
npm's own package manifest states its version without any process
creation, so the probe lane can no longer fail: the install-receipt
key keeps its exact value, engine checks still run, and completed
installs reuse with zero spawns.
The workspace npm ci printed nothing until 'added N packages': builders
set CI=1, which turns npm's progress off, and the stage name only went
to the desktop UI's status file.
Print a line before npm ci, and pass --progress=true so a terminal gets
npm's spinner back (npm still shows it only on a TTY, so piped output
such as the desktop app's log stays clean). The flag stays out of the
receipt-keyed args, so existing installs are still reused.
cryptography ships no win_arm64 wheel, so every Windows ARM64 venv sync
compiles it from the sdist and needs MSVC, Clang, Rust and static OpenSSL.
Only setup-hermes.ps1 (and so activate.ps1) prepared that environment,
between a `pm install --tools-only` and the real sync. install.ps1,
`hermes update` and repair ran the same sync without it and failed in
openssl-sys.
PM owns the sync, so PM prepares it. pm/native_build.py holds the adapter
(moved from scripts/build/windows_deps.py) plus source_build_environment(),
which prepares only on win32-arm64 when the synced project carries the
provider script. A payload has prebuilt dependencies and needs no compiler.
VenvPackage.apply and build_environment pass the result to uv children
only. It carries the bridged pip index settings, which managed_environment
applies only to the ambient environment. The state root stays the store
parent, so existing vcpkg/OpenSSL builds are reused.
setup-hermes.ps1 collapses to one `pm install`: the tools-only split existed
only for this preparation, and pm install already puts its tools on PATH
before the venv sync (pm/cli.py activate check).
Not yet verified live on Windows ARM64.
Install/update completion rewrites the root install-stamp.json (fresh
builtAt) after products are built, and the stamp was still a shared
web/desktop source input, so every install left its own web_dist
"missing, stale, or damaged" and the post-install probe failed on every
installer E2E leg. Nothing in either build reads the root stamp;
desktop's baked stamp is already tracked as a prepared input.
Merge origin/main at 8e806ae1b2. Keep native helper compilation in
prepareDesktopNativeDependencies and keep bundling/beforePack consume-only.
Bind helper sources and headers into preparation identities and cache keys;
copy admitted executable resources beside node_modules and preserve signing
semantics in product freshness checks.
Verified desktop typecheck, focused native/packaging/UI and gateway/cache
tests, and the real Linux preparation/copy/Xvfb execution path. Incoming
upstream anti-slop findings remain unchanged; no baseline was raised.
First launch of a bundled payload paid a cold-compile stall: the launcher
redirects bytecode writes to a user-level cache (signature-breaking on
macOS, read-only mount on AppImage/MSIX), so every import compiled from
source. Now staging bakes the cache into the payload:
- compileall with the payload's OWN staged 3.14 interpreter, unchecked-
hash pycs: repack mtimes cannot invalidate them, a stale source can
never trigger a rewrite, and read-only pycs mean the macOS signature
never observes a change. Dirs stay writable — in-place rebuilds
rmtree the tree; asserted coverage plus unchecked-hash means no
cache-miss write can target them.
- coverage is the perf contract: the bake FAILS if any parseable module
lacks a pyc (empirically 0 unparseable files ship, so compileall is
strict). Probe suite: py_compile/cache_from_source, PEP 552 flags,
multi-root read, stale-source no-rewrite, read-only cache-dir import.
- launcher: the baked marker makes configure() leave sys.pycache_prefix
UNSET — the prefix relocates reads too and would hide the baked pycs.
Payload modules read their source-adjacent cache (Python's default
multi-root lookup); plugin/user modules keep caching beside their own
sources under HERMES_HOME. Unmarked payloads keep the old redirect.
- snapshot(): the sealed payload ships without tests/website/evals/
.github/nix/docker/tests-js (~69MB, 46% of tracked bytes) and without
apps/ui-tui/web/scripts — CI prebuilds those products, and
is_bundled_payload routes sealed updates to the channel updater, so
the rebuild graph never runs in a bundle (linux_desktop_entry degrades
to the themed icon). Frontend product staging keeps the full tree.
- test_bundle_native now stages the FULL relocatable toolchain (a bare
interpreter ELF falls back to its compile-time /install prefix and
cannot create a venv), and runs on the real 3.14 for the first time
this campaign — the whole battery had been running 3.12 against the
3.14-pinned lock.
Dependency acquisition during packaging left native wheels and packager
inputs outside the pre-build cache save. Compose PM and existing providers
into a preparation phase, then require builds to consume admitted inputs.
Share native preparation with PM Bundle. Keep path-bound environments and
signing outputs separate from reusable caches. Use read-only cache tokens
for commit builds and preserve the one-command local build path.
Verify pinned tools through PM, probe PTYs under the prepared Electron,
and supply dmgbuild through a build-only PM package. Resolve bundled tool
stores from their payload manifest so relocation preserves discovery.
Validation: focused Python and JS tests, checkJs, Ruff, Windows checks,
anti-slop, cache relocation, and network-denied Linux AppImage builds.
Relocated runtime smoke passed with NixOS host libraries supplied.
Native Windows/macOS signing and live GitHub cache behavior remain untested.
A source child can resolve outside the checkout. The output guard must
protect its canonical path before the builder checks prepared inputs.
Exclude generated dist/build trees so in-tree products can still rebuild.
Keep explicit prepared inputs protected even inside generated trees.
The public buildDesktop regression test first failed with a missing-icon
error instead of an overlap error. All 7 desktop-builder tests now pass,
including real cold/warm builds under apps/desktop/build/products.
Node syntax checks and git diff --check pass. No full suite or native
packaging ran.
Run historical updater completion in a fresh interpreter so cached imports
cannot revive retired dependency installers. Share Git and ZIP completion,
carry receipt and recovery state, and preserve child exit status.
Route plugin admission, binary acquisition, desktop launch and build paths
through PM. Replace redundant helpers and tests with real worker, package,
publication and launch checks. Keep the shipped compatibility surface fixed.
Targeted Python and desktop checks pass. Native update journeys and fresh
production image qualification remain pending. This is a checkpoint before
those acceptance runs.
Competing installers and checkout-local venv assumptions bypassed PM
selection, install consent, and generation lifetimes. Route consumers
through PM and installation-bound launchers. Refresh source launchers
before obsolete Python entries can be collected.
Remove Node, browser, and CUA acquisition engines, obsolete venv-holder
handling, detached sync, and unused PM APIs. Keep historical updater
exports inert and preserve external tool ownership and native integration.
Share product freshness and prepared inputs across builders. Align plugin
admission, Docker provisioning, setup instructions, and behavioral tests.
Verified targeted Python and JavaScript tests, desktop and web typechecks,
scoped lint, real product builds, and the Docker frontend smoke test.
The missed post-setup test cleanup is included and verified.
Native Windows/macOS execution, full Rust compilation, and the complete
repository suite remain unverified. Historical compatibility requirements
were preserved and extended, not fully rescanned.
The minted launchers wired venv site-packages onto sys.path with a raw
insert (win32 wrapper) / PYTHONPATH (posix), neither of which runs .pth
files. pywin32.pth is load-bearing on Windows: it puts win32\lib on
sys.path, which is what makes 'import pywintypes' resolve — without it
portalocker's Win32Locker dies and concurrent-log-handler silently drops
every file-log record on Windows bundles.
* launcher_wrapper.py: site.addsitedir() for the site entry (repo first,
site directly after, .pth dirs last)
* launchers.py posix: same via HERMES_SITE env in the -c bootstrap
* pm/environment.py: prune_site_pth() drops _virtualenv.pth and the
__editable__ pointer (build-machine path) that must never run in a
sealed payload
* python_env.py: run the prune after every environment build
Merge ethie/shared-product-builders with the CI dependency cache and native Windows setup work. Preserve UTF-8 diagnostics in the shared Python environment runner. Pass a persistent cache through isolated native staging and PM-runtime construction. Reuse one Windows prerequisite installer from source setup, native adapters, and CI, preserving Rust homes across HOME isolation.
Verified 85 targeted Python tests (5 host skips), 18 JavaScript tests, workflow validation, and scoped lint/typecheck. On native Windows ARM64, five prerequisite contracts passed and the actual shared provider reused OpenSSL, compiled its header with MSVC, and retained Rust under isolated HOME. Full signed distribution builds and live Actions cache transfer remain CI verification.
Build TUI, web, desktop UI and runnable agent products from explicit
prepared inputs. Keep dependency preparation separate from distribution
packaging, with PM and native builds sharing uv environment construction.
Docker copies compiled frontend products instead of build dependencies.
Nix retains uv2nix environments and consumes shared assembly through store
references. Native desktop and Termux use the same launcher and frontend
contracts. Preserve the independent PM runtime and source imports from
arbitrary working directories.
Keep failed frontend builds from replacing the previous product, reject
source/output overlap, and bound dependency-process output draining.
Include hermes_wisdom in the Nix wheel: real CLI smoke tests exposed its
missing package declaration on the base revision too.
Verified focused Python and JavaScript suites, Docker build/runtime checks,
Nix desktop and CLI/ACP checks, standalone TUI and packaged Electron PTY,
and real full-Chromium interaction. Native signed installers, Android device
installation and the full repository suite remain CI verification.
