rc.10-v0.21.7 failed 13 release jobs because the release-only E2E still
assumed the old update behavior:
- Install/update E2E (Linux, macOS, Windows; hermes-update and desktop-app
routes). A fresh install of the official repo now defaults to the stable
channel. Installed at HEAD it is newer than v0.21.6 and correctly waits
("Already up to date"), so the HEAD -> NEXT legs never moved ("checkout
landed on NEXT", "no successful update completion receipt"). The drivers
now record `hermes update --set-channel main` before the user-state
snapshot, the same way the pytest sandboxes already do (3a4b4b6f83).
Releases older than --set-channel follow main already and are skipped by a
help probe.
- Network E2E (test_offline_and_flaky, test_proxy_only_egress). They asserted
that `main` reads and retries releases/channels/main.json through the
proxy. #136429 removed that read on purpose (regional WAF 403s, #128295).
Rewritten to the new contract: `main` reads no release record, survives a
release-host outage, and a cut tunnel to github.com still fails fast and
changes nothing.
The desktop unit test with the same assumption was already fixed on main
(0aa764fedd).
NSRunningApplication.terminated changes only when the run loop runs.
JXA delay() does not run it, so the check stayed false after the app
quit and the leg failed when the setup app had launched Hermes. Spin
the run loop while waiting. The quit stays the normal one.
The smoke's CLI inherited the driver's HERMES_HOME / HERMES_RUNTIME_DIR /
HERMES_PYTHON, so PM built the plugin's environment on the driver's own
Python under C:\a\_temp\setup-pm and never touched the packaged
interpreter: the bug could not show. Clear HERMES_*/UV_*/PYTHON* and the
driver's tool dirs from PATH in the basic-user batch, and fail if the
install log still names the driver's PM home.
The installed-package smoke only chatted, so nothing exercised PM building
a dependency generation inside the package. After the chat it now installs
a local plugin that declares a Python dependency through the package's CLI
execution alias with --enable --yes-deps and requires `plugins list` to show
it enabled.
It runs under a Safer NORMALUSER token at medium integrity: GitHub runners
are administrators, and Administrators can start files in a package
directory without package identity, which a normal user cannot. A probe
that the token is refused the packaged python.exe keeps the check honest.
Pull requests and pushes to main no longer run any E2E suite: ci.yaml's detect gate
forces every e2e* lane off unless the run is a stable-release run (inputs.release) or a
manual dispatch, whatever the diff, the update-path classifier or the run-e2e label
selects. That covers the Python e2e and e2e-upgrade jobs, the Windows E2E and Windows
install + update journey in tests-os, and the Desktop core/update suites.
install-e2e.yml drops its pull_request (path-filtered `pr` route) and 12-hourly
schedule triggers; it runs from the stable-release gate (workflow_call) or by dispatch.
live-providers drops its nightly schedule and keeps the release-tag push + dispatch.
Why: a one-row change to left_core_migration.py held a PR for 40 minutes on the
Windows install + update E2E; the suites cost most of the larger-runner minutes and
the release gate already runs all of them strictly.
* fix(install-e2e): tolerate the update-result consume race and held lock files
Two HEAD -> NEXT legs of the stable-release matrix went red for harness reasons,
not product ones.
macOS (app-driven update): source-update-observer listed
.hermes-update-result.json and then read it. The relaunched app consumes
(unlinks) that file on boot, so the read could land after the unlink and throw
ENOENT, which launch-from-spec reported as a failed update although the update
receipt said success. A vanished file is now "not there", the same as before it
was listed.
Windows (user-state verify): verify-user-state hashed every judged file. The
gateway's cron ticker holds an msvcrt byte-range lock on cron/.tick.lock, so
open() raised PermissionError and the upgrade was reported as changing user
state. Proven on a real Windows host: base raises Errno 13, the fix records the
lock as a sentinel. A lock file is held by a live process and is never user
content, so it is recorded without being read. Any other unreadable entry now
surfaces as a ScanError that names the path instead of a bare errno.
* test(install-e2e): an unreadable user file still fails the scan with its path
Skipping lock files must not let a user file the verifier cannot read pass
silently. The test makes a real mode-000 file under memories/ and asserts
snapshot_home raises ScanError naming the path, with the PermissionError kept
as the cause. It goes red when the OSError -> ScanError wrapper is removed.
Every tagged stable Windows candidate failed "Smoke Windows <arch> MSIX" with
"Package version disagrees with stamped semver". Since c4d2bb5575 a tagged
build's MSIX version is the native quad nativeQuad() derives from the claim
epoch (year.hourOfYear.secondOfHour.0, e.g. 2026.6724.738.0 for v0.21.6), while
the smoke still required it to start with the semver ("0.21.6.").
The quad is already pinned twice: the smoke compares the installed package to
the input manifest, and stable admission (stable_windows_version) recomputes it
from the claim epoch. Drop the stale semver-prefix rule for tagged builds; commit
builds keep their semver-with-zero-revision rule and channel builds their request
version.
The Desktop update check asks api.github.com for main's tip before it asks
Git. The windows GUI update driver only redirected Git transport, and the
app's probe Git runs with GIT_CONFIG_GLOBAL=/dev/null, so it never saw the
redirect. When real GitHub main equalled the installed OLD commit, the app
reported no update and the leg waited 3 minutes for an Update now button
that could not appear.
Install the staged-main probe the macOS and Linux source legs already use.
It wraps Git with the staged insteadOf rewrite and selects --branch main,
so the real checker, UI and updater still run against serve.git.
The smoke driver created .desktop-smoke-home inside the predicted Hermes
home before checking that the home was empty. Bundles with stamped env
that retained the explicit HERMES_HOME failed before Electron launched.
Resolve and admit the predicted home before any sandbox writes. Keep the
existing-profile refusal and provider seeding intact. Extend the seeding
contract for onboarding defaults and forced home clears.
Verified the old failure and fixed launch boundary on native Windows ARM64
with real driver imports and filesystem writes. Full package install/chat
acceptance still needs a fresh hosted smoke run.
Moves the source-root and predicted-home identity assertions verbatim
into a helper called at the same point with the same inputs; behaviour
unchanged. Brings runInstalledDesktopSmoke's CC back under its main cap
(27 -> 21).
A host-ledger backend is not an Electron descendant, so the app-reported
root no longer stands in for its tree when process evidence is unreadable
(Windows). Ledger records must match the live process's start time
(create_time, 2 s like the product), and the ledger is read at the machine
root the app resolves for the home, where the product writes it.
launch-from-spec.mjs waits --timeout-ms (default 10 min) for the update
receipt after clicking "Update now". A full app-driven update (git pull,
dependency sync, npm ci, TUI/web/desktop builds) takes 7.3-7.8 min on a
normal macOS runner, so a runner ~1.4x slower misses the window while the
update is still progressing: install-e2e run 37243656153 job 111557350181
gave up at 9:52 with all three dependency syncs done (receipts outcome=ok)
and the update about a minute from finishing. The job limit is 60 min and
the leg used 18, so 15 min costs nothing on a normal runner.
#125053 fixed the exe's receipt writer, but the installer .exe is not being
rebuilt, so every Desktop-installer machine keeps the old receipt
(completedAtUnix, and a null pinnedCommit without git on PATH) until
`hermes update` rewrites it. No reader depends on completedAt or
pinnedCommit: Desktop's launch gate is runtime usability, and every other
reader only checks that the file exists. So the stamp phase stops
presenting this as a gate that lifts when a fixed exe ships and asserts the
contract that holds with the released exe: only those two FAIL lines, an
integer completedAtUnix, and the checkout at the expected commit. Any
other FAIL line stays red.
Refs #124949, #125053.
With one PM store per leg the Desktop app no longer re-bootstraps on first
launch, so the receipt Hermes-Setup.exe writes survives to verify-stamp:
pinnedCommit null on a git-less machine and completedAtUnix instead of
completedAt, replacing install.ps1's correct receipt. Gate only that
writer's exact two FAIL lines, and then require the checkout itself to be
at the expected commit. Any other FAIL line stays red.
With one store per leg, HEAD's install.ps1 never finished on Windows: the
source completion relaunched its store Python into itself every ~19 s, 35
levels deep after 12 minutes (job 108573522351). The workflow's workroot is
<workspace>\..\hermes-desktop-gui-e2e, so the store path carried the "..";
Windows reports sys.executable normalized, and prepare_launch()'s
Path.absolute() comparison never matched (#122513, fix PR #122518). The
job's own store (setup-pm) had no "..", which hid this until now. Normalize
the workroot, as a user's HERMES_HOME is.
install.ps1 now runs under the same watchdog as `hermes update` (40 min),
and the watchdog adds a py-spy stack for every Python process of the leg
before it stops them. The workflow installs py-spy beside pywinpty.
`windows: {installer-script, installer-script+desktop, desktop-installer@latest}
-> hermes-desktop-app-update (v2026.7.1 -> HEAD)` failed with "Program
'git.exe' failed to run: The filename or extension is too long". A v2026.7.1
clone leaves hundreds of CRLF-churned paths, and Clear-HistoricalInstallerChurn
passed every one to `git checkout --` on the command line. Pass them through
a NUL-separated --pathspec-from-file with --literal-pathspecs instead.
`windows: desktop-installer@latest -> hermes-update (HEAD -> NEXT)` and
`installer-script+desktop -> hermes-update (HEAD -> NEXT)` failed with
"source launcher publication failed". install.ps1 and `hermes update` ran
with setup-pm's HERMES_RUNTIME_DIR, while the desktop smoke launched the app
without it (as a user's app runs) and settled the checkout onto a second
store at <HERMES_HOME>\tools. The update then re-pointed its own running,
locked hermes.exe.
windows-e2e.ps1 now drops HERMES_RUNTIME_DIR, HERMES_PYTHON and VIRTUAL_ENV
on entry, so every product step of a leg (installer, app, update, chat)
resolves the one store a user's machine has. The driver keeps setup-pm only
through $DriverPython (and PATH, which older installers need for uv and
ripgrep).
`hermes update` also runs under a 45-minute watchdog. Past it, the driver
writes every process (pid, parent, start time, command line) to
logs\update-hang-processes.txt, stops this leg's processes and fails with
that table, instead of being cancelled blind at the job cap.
This reverts commit c7883282c91. On real Windows (runs 36293787001 and
36293789061, c7883282 + #124702), dropping HERMES_RUNTIME_DIR for the update
turned a 40-second red ("source launcher publication failed") into a silent
40-minute hang: the update finished its code, Desktop rebuild and skills
steps, printed the curator notice, and never exited until the job was
cancelled. The store it then used (<HERMES_HOME>\tools) was created by the
desktop smoke and never received the install's PM tools, so the update's
post-update default-tool install ran against a half-populated second store.
Both one-sided variants (keep the variable in the smoke; drop it for the
update) fail. The real fix is one store per leg, and it touches every
Windows leg, so it goes in on its own with a full Windows run.
`windows: desktop-installer@latest -> hermes-update (HEAD -> NEXT)` and
`installer-script+desktop -> hermes-update (HEAD -> NEXT)` failed with
"source launcher publication failed". The desktop smoke runs the app
without the job's HERMES_RUNTIME_DIR, the way a user's app runs, so it
settles the checkout onto <HERMES_HOME>\tools and republishes .hermes\bin
for that store's Python. The driver's `hermes update` then ran from that
hermes.exe with the job's store (setup-pm\tools) back in scope. The update
re-pointed both launchers, and Windows refused to replace the running
hermes.exe.
Invoke-HermesUpdate now drops HERMES_RUNTIME_DIR while <HERMES_HOME>\tools
exists, so the update stays on the one store the install actually uses.
Keeping HERMES_RUNTIME_DIR in the smoke instead fixed these two legs but
broke eight others whose app never reached its backend (run 36288957697),
so that variant was reverted.
Once launch capture works, two Windows hermes-desktop-app-update failures
show up (run 36286580917):
- v2026.9.24 -> HEAD (installer-script, installer-script+desktop,
desktop-installer@latest): the app's update is correct but slow. From
v2026.9.24 it runs the historical venv->PM takeover and a full Desktop
rebuild (the `hermes update` alone took 9m43s). launch-from-spec's 10 min
default gave up 9m55s after the Update now click while the updater window
showed "Updating code and dependencies 9m 22s elapsed". windows-e2e.ps1
now passes --timeout-ms 1800000, in line with open-app-update's 35 min
wait. The driver's self-deadline is now measured after the update wait
instead of being a flat 20 min inside it.
- HEAD -> NEXT: Desktop offered real GitHub main (0f4a98f8) instead of the
staged NEXT, so assertStagedBranch refused to click. HEAD's Electron main
is one ESM bundle, and checkout-source.ts binds promisify(execFile) at
load, before installSourceBranchProbe patches execFile in the packaged
app. POSIX avoids this by wrapping the launcher script, which Windows
skips. The probe now also hooks ChildProcess.prototype.spawn, which every
child passes through, and reuses branchProbeArgs, which already knows both
the --run-module and cmd.exe .cmd shapes. Verified locally: a promisify
captured before the hook now gets `--git <real> --branch main`, and other
commands are untouched.
`windows: * -> hermes-desktop-app-update (HEAD -> NEXT)` failed with
"a launch was actually captured (exit 0 without a launch must not pass)":
`hermes desktop` built the app and launched the real Hermes.exe, and the
driver's hook never saw it. There were two gaps, and either one alone is
enough to miss the launch:
- HEAD installs publish the PM launcher (.hermes\bin\hermes.exe), which
runs its interpreter with -I and drops PYTHONPATH, so sitecustomize never
loads. windows-e2e.ps1 now does what installer-script-e2e.sh already does:
run launch-capture/pm-launch.py for a PM launcher, and keep PYTHONPATH
for pre-PM venv console scripts.
- Since 3ddf82fe24 the Windows packaged launch is a detached
subprocess.Popen, not subprocess.run. sitecustomize now also intercepts a
launch-shaped Popen: it records the same spec and stands in for a child
that already exited with 0. Every other Popen, including the one inside
subprocess.run, spawns for real.
Three `-> hermes-desktop-app-update (v2026.9.24 -> HEAD)` legs failed with
"installed source has only installer-generated changes (other: 1 .M N...
<same blob> <same blob> website/i18n/zh-Hans/.../user-stories.mdx)".
v2026.9.24's install.ps1 clones under Git for Windows' system
core.autocrlf=true and pins false only afterwards, so the ~1700 text files
.gitattributes does not pin to LF (*.mdx, LICENSE, ...) sit on disk as CRLF
over LF blobs. The stat cache hides them, except the last few the clone
wrote inside the index's racy timestamp window, which read as modified.
Which files show up varies (1 or 3 per leg); main's installer no longer
pins autocrlf.
Clear-HistoricalInstallerChurn now also restores a `1 .M` entry whose
index and worktree modes match and that has no `diff --numstat
--ignore-cr-at-eol` record (a line-ending-only difference). A real content
edit, an edit inside a CRLF file, a mode change, or an untracked file still
fails the assertion, listed.
A red scheduled run blocked nothing and told nobody. install-e2e-red.yml
runs after every scheduled "Install & Update E2E" run: red opens one issue
labelled install-e2e-red (or rewrites the open one's body in place) with the
red legs grouped by failure class and linked; the first green run closes
it. No per-run comment, never a second issue.
It is its own workflow_run workflow because install-e2e.yml is also called
by stable-release.yml with read-only permissions, and a nested job asking
for issues: write would fail that call at startup. workflow_dispatch with a
dry-run default previews the change for any run id.
Every Windows leg of every scheduled run since 2026-09-25 was cancelled at
the 60-minute job timeout in its Install step. The user-state phase runs
`hermes chat -q "..."` under pty-run.py (a ConPTY, so the CLI sees a real
TTY). Since a5c7eed (shipped in v2026.9.21) `-q` on a TTY seeds an
interactive session and only answers-and-exits with --oneshot; the turn
answered ("Hello from the mock inference server!") and then sat at the
prompt. pty-run.py's --timeout 900 never fired either: it checked the
deadline only between blocking PtyProcess.read() calls, and an idle prompt
never returns from read().
- windows-e2e.ps1 passes --oneshot when `chat --help` advertises it (older
tags have no flag and exit after -q on their own), lowers the turn budget
to 300s, and names a 124 as "never exited" instead of a generic failure.
- pty-run.py drains the pty on a daemon thread and waits on the queue with
the deadline, so --timeout holds whatever the child does, and writes a
TIMEOUT line into the captured log before killing it.
The driver passed --skip-browser whenever the installer's help listed it.
That was for historical installers' own Playwright step; on a PM installer
the flag is now a real opt-out and would hide the default users get.
The GUI driver refuses a dirty source tree. On Windows, v2026.7.1's
installer ran `npm install`, which rewrites package-lock.json (later
installers use `npm ci`), so its Desktop reported dirty and the upgrade
arm failed before clicking Update. Mirror the Linux harness: print the
status, restore only package-lock.json churn and exclude a generated
.install_method; any other change still fails with the list.
The v2026.7.1 app launched by the latest DMG was still starting its backend
and did not finish a normal Quit within 30s. Wait up to 120s; the quit must
still be the normal one (no force-kill).
The desktop feed base came from config.yaml's
updates.desktop_feed_base_url and then an undocumented env var. Config
already wins and is the documented override; a second env source only
lets a stray variable redirect where updates are fetched from. Remove
the fallback from resolveFeedBaseUrl, its caller, its test cases and
the Windows bundled E2E that blanked it.
PM launchers run python -I, which ignores PYTHONPATH, so the macOS
hermes-desktop-app-update route never loaded the sitecustomize capture hook.
Route PM launchers through pm-launch.py as the Linux driver already does, and
cover the packaged .app shape the macOS route launches.
The current source checker reports updateAvailable with behind null when GitHub
compare cannot count staged commits; the app-update predicate demanded an integer
behind and refused every HEAD->NEXT leg. Require behind > 0 only for the historical
shape without updateAvailable.
v2026.6.19's DMG bootstrap runs the same install.sh that writes .install_method,
so its macOS leg saw a dirty tree. Share the Linux driver's guarded exclude through
source-driver.sh and apply it before every app-driven macOS update.
The desktop-installer legs intermittently leave Hermes-Setup.exe on
LAUNCHING after the Launch click: no app window, no desktop.log, and the
installer never exits. Its tracing log is buffered and dies with the job,
so the stall left no trace of where it blocks.
On a driver failure, while the installer is still alive, record the
Hermes/webview/python/uv/git/node process table (was Hermes.exe ever
started, and by whom), the installer's thread states, and a full memory
dump of the installer into proof/driver-failure/. Capture failures are
logged and never replace the driver's own assertion.
Packaged Electron rejects NODE_OPTIONS --require, so the preload never reached the Desktop source check. Wrap only the E2E installation launcher and route its source-check call to the real staged Git main with an explicit branch; other launcher calls remain unchanged. Keep production channel resolution fail-closed.
