Both dashboard-auth JWT verification paths (self-hosted OIDC ID token and the
Nous Portal access token, via the shared verify_jwt) called PyJWT jwt.decode
with the default leeway=0 while requiring iat. Any host whose clock lags the
issuer's — unsynced containers/VMs, NTP jitter — failed login with
ImmatureSignatureError, and the failure was folded into ProviderError, i.e.
HTTP 503 'provider unreachable', discarding already-rotated refresh tokens
and forcing unrecoverable re-logins.
- verify_jwt gains leeway (default 60s, RFC 7519 §4.1.4-4.1.6) and classifies
ImmatureSignatureError as InvalidCodeError (validation failure -> 401 /
refresh) instead of ProviderError/503.
- New parse_leeway helper: fail-closed parsing (unparseable / negative / NaN /
inf -> 0, never widens the window); 0 honours strict mode.
- Self-hosted OIDC: id_token_leeway kwarg + dashboard.oauth.self_hosted.id_token_leeway.
- Nous: token_leeway kwarg + dashboard.oauth.token_leeway.
- No new env vars (config.yaml-only knobs).
Fixes#47815
audit_log() appended with a plain open(path, 'a') and no size cap, so
dashboard-auth.log grew unbounded for the life of the install — the one
log in the auth path outside hermes_logging's rotation.
Add self-contained size-check-and-rename rotation in audit.py matching
the agent.log convention (maxBytes=5MB, backupCount=3, stdlib
.1/.2/.3 naming). The pending record's UTF-8 bytes count toward the
threshold, so a crossing write rotates before appending. Implemented
locally (stdlib os.replace only) to keep the module import-light for
early-loading middleware; reusing _ManagedRotatingFileHandler would
pull in hermes_constants and break that guarantee.
Fixes#57749
The smoke driver created .desktop-smoke-home inside the predicted Hermes
home before checking that the home was empty. Bundles with stamped env
that retained the explicit HERMES_HOME failed before Electron launched.
Resolve and admit the predicted home before any sandbox writes. Keep the
existing-profile refusal and provider seeding intact. Extend the seeding
contract for onboarding defaults and forced home clears.
Verified the old failure and fixed launch boundary on native Windows ARM64
with real driver imports and filesystem writes. Full package install/chat
acceptance still needs a fresh hosted smoke run.
Select the expected archive and embedded app version from the admitted
channel request, or the release tag for tagged builds. Channel builds
have no tag, so indexing it crashed before macOS packages could stage.
Keep filename, identity and provenance validation intact.
Regression coverage reproduces the crash on both architectures and checks
that wrong filenames, embedded versions and commits remain rejected.
Session ids are only unique within a profile's store, and the keep-alive
registry is process-wide, so matching stranded PTYs on the key's resume
segment alone would let a resume in one profile close a detached PTY on the
same session id in another profile — somebody's reconnect window, with
whatever turn it had in flight. The key already carries the profile
segment; the resume arm now requires both to match the requested chat.
The lease-holder arm is unchanged: it reads the lease from the chat's own
home, so it already names the right process.
Found by Copilot's review.
Two gaps kept the targeted release from firing on the reported repro:
- The lease pid is never the bridge's pid. ptyprocess forks the leader
(`node entry.js`); the `tui_gateway` CHILD of that leader is what claims
the single-writer lease, so an equality match on `bridge.pid` matched
nothing in a real deployment (reporter's table: node 840, lease pid 850).
`hosts_pid` now matches the leader itself or any PTY whose leader is an
ancestor of the holder (psutil). A holder with no PTY ancestor — a plain
`hermes` terminal — still closes nothing, so that refusal stands.
- Leases live in the home the chat runs in. A profile-scoped chat's child
claims under the profile's HERMES_HOME, so reading the dashboard's launch
home found no holder. The lookup now takes the child's HERMES_HOME.
The matched sessions are popped under the registry lock and closed outside
it (a close can wait out the helper SIGHUP grace and the lock serializes
every new chat), still before the spawn so the lease is released first —
the close discipline main's teardown series adopted, carried from #131179.
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
New chat rotates the browser's attach token, so the chat it left behind
stays registered under the previous token while its TUI child keeps the
single-writer lease on that session. close_other_sessions() is keyed on
the incoming token and never sees it; returning to the chat spawns a
second child that single-writer ownership refuses.
PtySessionRegistry.close_orphaned_sessions(resume, keep_key, holder_pid)
closes the stranded keep-alive PTY before the spawn: match by the key's
resume segment, or by the process holding the session's lease (a fresh
chat's key carries no resume target). A PTY with a live viewer is never
a candidate.
Carried from #131181 (squashed PR delta; original author preserved).
Address @antifragileer's review suggestions on #44498:
1. test_partial_write_does_not_mutate_existing_list: pins that a failed
write with mixed valid/invalid items leaves the prior list untouched
(acceptance criterion #3 from #44496).
2. test_merge_new_item_rejects_whitespace_content: verifies whitespace-only
content is rejected for new items in merge mode, guarding against a
future code change that special-cases merge.
The sidecar now refills its redial budget only after a connection has
stayed open for the grace window; the events feed on the same component
still reset `attempt = 0` on every `open`. A /api/events socket that opens
and dies within a second therefore never reached
EVENTS_MAX_RECONNECT_ATTEMPTS — the same unbounded churn, minus the
agent boot. One shared HEALTHY_OPEN_GRACE_MS now gates both ladders; the
banner still clears on open, since a flap re-surfaces it on the next drop.
Tests: a brief open keeps the ladder on its next rung (2s, not 1s); forty
open-then-close cycles still hit the cap and surface the gave-up banner;
the existing "resets after reconnect" case now waits out the grace window.
Stacked on #129397. Clearing the pending redialTimer on open prevents the backoff timer from closing a just-established connection. The 3 regression cases align with the 10s grace period from #129397. Tests: ChatSidebar 24/24 + ChatPage 15/15.
(cherry picked from commit ee043461a307a68ed0cbe3a5a5ffde7adfb10e7b)
The same #132386 interaction as the ZIP completion tests: this test
(from main) replaces update_cmd._prepare_git_command with a no-argument
lambda inside its child process, and this branch calls it with
checkout_move=..., so the child died with TypeError and the test read it
as a failed update. The fake takes **_ like every other fake of it.
On main, #132365 moved container-exec routing out of main.py, which
lowered main.py's code-health line cap to its new size (3643). This
branch's paused-gateway recovery added four lines to main() and pushed
the file to 3647, so the code-health check refused it.
The owed-fleet-restart warning and the paused-gateway recovery move
together into main_install_repair._recover_update_debts_on_startup, next
to the Windows quarantine sweep that already lives there; main() calls it
at the same point, so startup order and behaviour are unchanged. Both
steps stay late-imported, so tests patching
update_cmd_fleet._warn_pending_fleet_restart_on_startup or
update_pause_record.recover still intercept them. The fleet warning's
silent `except Exception: pass` now logs at debug with the traceback,
which is what the health check asks of a catch-all boundary in new code;
a failure there still never blocks the launch.
#132386 added three ZIP tests that replace update_cmd._prepare_git_command
with a no-argument lambda. This branch calls it as
_prepare_git_command(checkout_move=...), so after the rebase onto main each
fake died with "unexpected keyword argument 'checkout_move'". They now take
**_ like every other fake of that function in the tree; the assertions are
unchanged.
The post-update relaunch verifier now polls every target in one loop through
`gateway_windows._live_gateway_pids` (d243ecf107f); these Windows-only tests
still stubbed only `_wait_for_gateway_ready`, so the relaunched `beta` was
never seen and resume raised "relaunch ... was not verified alive: beta".
The same probe also answers "is this profile already running?" before a cold
start, so the stub follows the lifecycle: a home answers (with a fresh PID,
through the verifier's own PID filter) only once its profile was relaunched via
`launch_detached_profile_gateway_restart`. A profile that has not started yet
still reads as down and is cold-started, and one profile never vouches for
another. Assertions are unchanged.
consume_planned_stop_marker_for_self took .hermes-update-paused-gateways.lock
on every platform for every stop (up to 2 s when busy), creating the lock
file in the root home, and on a busy lock fell back to keeping the request
and writing a .accepted receipt that nothing ever removed. The lock only
protects the checkpoint into a Windows update pause record; with no record
or claim of this checkout on disk (every non-Windows host, or a pause
recovery/the updater already retired) there is nothing to checkpoint.
The consumer now holds the lock only when a pause is on disk. The
checkpoint hook re-checks after the request was read, so a pause that
appeared after the first look (an update writes its record before its
request) is still checkpointed under the lock. With no pause on disk, the
consumer removes the .accepted receipt and an expired stamped request an
earlier drain left: their debt is settled. While any pause is on disk both
stay (never earlier).
test_a_busy_pause_mutex_never_turns_a_planned_stop_into_an_unplanned_one
now puts a pause record on disk first: without one, keeping the request on
a busy lock was the leftover this fixes; with one, its assertions (planned,
request retained, no block) are unchanged.
When the gateway's home refuses a new .accepted receipt, mark_stop_accepted
stamps the kept request itself with accepted=true; that stamp is then the
only durable trace that this draining incarnation accepted the update's
stop. _read_live_pid_marker unlinked every expired marker, so a second
signal past the request TTL (another Ctrl+C / SIGTERM, which calls the
consumer again) or a planned-stop watcher probe in the same home deleted
it, and recovery then retired the restart debt of a gateway that was still
draining.
An expired stamped request still matches nobody (no PID-reuse grief), but
the TTL cleanup no longer unlinks it.
Round 6 bound the branch switch, the upstream syncs, the pull and the rollback.
Other steps that run while the Windows pause holds the gateways still rewrote
tracked files unbound:
- `git stash push` and its cleanup `reset --hard`;
- `git stash apply` and the reset after a conflict;
- the rejected-restore `reset --hard` + `clean`;
- the lockfile-churn `checkout -- <locks>` and the line-ending `checkout`.
If one of them was interrupted mid-write, the gate judged the torn file only
against whatever refs a fetch had left. A restore after the pull had no
baseline at the moved HEAD at all. Either way the paused set was admitted.
Each step now runs inside `_checkout_move`. `paths=` binds the files it
rewrites in place, and `revert=True` binds every tracked change it finds,
because it puts them back to HEAD. A restore binds its stash commit. The gate
counts those paths as writable. A restore that applied its tracked changes in
full reports `whole`, so the tree it changed by design retires its baseline
instead of holding the set.
Removed the "already up to date" switch-back `checkout <current_branch>`. It
only ran when no checkout had left current_branch: in-place and release
updates never switch, and a parked switch is excluded. It was a no-op that
nothing gated. Helpers take a `checkout_move` binder (`_moves_for(token)`);
test doubles of `_prepare_git_command` accept its new keyword.
`git checkout <existing branch>` can exit 0 and move HEAD while leaving a file
unwritten ("unable to unlink old ..."): HEAD and the index name the new commit,
the file keeps the old bytes. `tree_is_whole` only judged baselines at an
unmoved HEAD, so that torn tree was admitted and the paused gateways started
on it.
`_checkout_move` now judges every move the moment git returns: the paths
between the commit it started from and the one HEAD reached that are changed
and were not already there byte for byte (`update_pause_record.torn_by_move`)
are recorded on the baseline as `landed[<head>]`. The gate holds while any of
those paths is still changed. Judging at return time, not at resume, keeps a
later dependency sync or stash restore that rewrites an upstream-changed file
(package-lock.json) from holding the set. A recorded move the updater died
before judging is judged at resume time against what the pause saw.
A step's own baseline is retired only when its move landed whole (it was
retired on any HEAD move before). The moved-HEAD gate reuses tracked_changes
and the dirty digests the record already carries.
Only the principal pull recorded its target (mark_move) before git wrote the
tree. The first branch switch in _prepare_checkout_for_update (and its
`checkout -B` fallback), a fork's early and late upstream sync, and the
post-pull syntax rollback wrote the tree unrecorded: a torn switch or sync at
an unmoved HEAD was judged against whatever refs a later fetch left, and a
failed second move (upstream sync after a successful pull) had no baseline at
its starting HEAD at all, so the gate admitted the torn tree.
All of them now run inside _checkout_move(token, *targets). Before git runs it
persists the resolved target(s): on the pause's own baseline when the step
starts from the pause HEAD (it knows the edits already there), otherwise on a
new baseline taken at the step's starting HEAD with its dirty digests. A
record that cannot be written raises, so the step does not run. Once git
returns, a per-step baseline is retired if HEAD moved or the tree is exactly as
the step found it, so a clean refusal never holds the set. The upstream sync
takes the context as `checkout_move`; the pull's move is extracted to
_move_checkout_to so _pull_updates does not nest deeper.
Retiring each paused runtime on its own readiness made the wait serial: every
profile and unmapped entry ran its own 2 s confirmation window (plus a 6 s
per-target floor past the deadline), so 3 profiles took 6 s and 5 took 10 s
where the fleet-wide wait had taken one window. All targets now share one poll:
one fleet process scan per tick, each target credited only on its own hits for
a whole confirmation window, and a PID one target vouched for is never credited
to another. Measured with readiness stubbed to succeed: N=1/3/5 went from
2.0/6.0/10.0 s to 2.0 s flat (N=10 also 2.0 s).
The reconciliation tests' liveness stub moves to the probe the verifier now
polls (_live_gateway_pids); their assertions are unchanged.
update_pause_record.recover() resumes paused gateways ahead of every hermes
command and prints its own notices to stderr, but the Windows resume path it
calls printed deferral/restart notices to stdout, corrupting the output of
unrelated commands (--json ones included). A recovering resume now routes its
stdout to stderr.
_pause_windows_gateways_for_update decided whether to abandon the durable pause
record by searching the error text for "rollback failures"; rewording that
message would delete the record while gateways are still stopped. The service
pause now raises ServicePauseFailed carrying rollback_failures, and the record
is abandoned only when that list is empty. Any other exception keeps the record.
Discovery owns an unmapped gateway whose environment is unreadable when the
exact-birth spawn ledger names its home, but the pause snapshot re-read only
the environment and wrote `home: null`. The readiness predicate read that as a
pre-home record and let another home's gateway with the same selectorless argv
retire the debt.
The snapshot, discovery and readiness now share one resolver
(update_fleet_scope.gateway_pid_home: environment, else ledger witness). A home
that no longer proves refuses the pause before any stop, and only an entry with
no `home` key (an older updater's token) keeps argv-only matching; a recorded
empty home is never matched.
update_lock caches this process's creation time (_OWN_CT). Warmed by an earlier
test through psutil, it differed from the stdlib probe identity() uses with
psutil hidden by that probe's resolution (macOS `ps -o lstart`: 1 s), so the
test failed in a full-file run and passed alone. It now starts from an empty
cache, so both sides use the same probe.
recover() runs before unrelated commands (anything printing --json, the
serve/dashboard processes Desktop reads over a pipe). Its own notices already go
to stderr, but the Windows resume helpers it calls print with bare print(),
which landed in that command's stdout. The resume now runs with sys.stdout
pointed at stderr. The record tests' resume stand-ins report on the real stdout
(sys.__stdout__) accordingly; no assertion changed.
update_pause_record.recover() runs before every command but `update` and
`gateway run` (chat, doctor, config, the serve/dashboard processes Desktop
spawns). Its Windows resume waits out the relaunch readiness budget for each
profile/unmapped gateway; one that never became ready was handed back unowned
"so the next launch retries at once", so every following command stalled
~30 s again (measured: 31 s on each of three consecutive launches).
Profile and unmapped entries now get the stored backoff SCM services already
have (service_retry): an attempted entry the resume left owed gets
relaunch_retry {attempts, next_at} (60 s doubling, capped at 1 h); a launch
inside the window holds it owed beside the draining entries without relaunching
it. The debt itself is never dropped, a tree-gate deferral does not count as
an attempt, and `hermes update` (merge_into does not carry the backoff) still
retries at once.
The pause record's second evidence reader (_names_request) turned a refused or
malformed planned-stop marker / accepted receipt into False, i.e. "this update
never asked it to stop". Past the request's TTL recovery then dropped the live,
still-draining gateway (drop_never_stopped) and durably retired its debt.
Now only an absent file or a valid one naming something else is "not asked";
refused/malformed is unknown: the entry stays owed (held as draining while its
process lives), and unknown is never resolved into stop_sent, so the next pass
judges it again.
The consumer's receipt (mark_stop_accepted) swallowed its own write failure, so
an accepted stop could leave no durable trace at all. When the home refuses the
new receipt file it now stamps the kept request in place ("accepted", opened r+,
never created); a torn stamp reads malformed, hence unknown, hence owed. An
expired request nobody accepted still expires.
The unmoved-HEAD tree gate decided which paths the failed move could have
written from the CURRENT FETCH_HEAD and remote refs, so a later fetch whose
target no longer touched a half-written file turned the gate from hold to
resume on unchanged partial bytes. And a path dirty at pause was exempt by name
alone, so an autostashed edit let different half-written bytes through.
_pull_updates now records the resolved move target on the pause's own baseline
before git writes (mark_move; a failure aborts the move), and the gate diffs
HEAD against that recorded target. The pause also stores each dirty path's
digest, so only the same bytes count as "seen at pause". A pause with no
recorded move, or a record from before these fields, keeps the fetched-refs
and pathname behaviour.
When both the producer's mark_stop_sent and the consumer's checkpoint into the
pause record were refused (busy mutex, Windows replace refusal), the consumer
still accepted the stop and left only the 60-second request marker. An updater
that died meanwhile left recovery to drop the still-draining gateway as never
asked once that TTL passed, losing its restart.
The consumer's fallback now writes a receipt beside the marker (the gateway's
own home, not the record) naming the accepting incarnation; recovery treats it
as stop evidence without a TTL. Unconsumed requests still expire, and a user's
stop (another stopper) is still no evidence.
read() returned None for ENOENT, a refused read and malformed bytes alike, so
the carrier survey dropped an unreadable newer claim (its older alpha+beta copy
then executed again, replaying completed alpha) and write() replaced an
unreadable orphaned record (losing its restart obligation).
Only a missing file is absent now; a refused or malformed one raises
RecordUnknown (an OSError), so selection, claiming and the new pause's write
fail closed and leave every file for a later, readable retry.
The pause saved an unmapped Windows gateway (no profile PID file) as pid + argv
only, and readiness accepted any owned-fleet gateway with the same argv tail.
Two homes can run identical selectorless argv, so home B's gateway (already
restored, or service-ready in the same pass) retired home A's debt while A
stayed down; the replay also ran A under the updater's HERMES_HOME.
Capture the runtime's home (dashboard_procs._hermes_home_for_pid) and birth at
pause, replay on that home, and require readiness from a gateway process on the
same home born after the one it replaces. Entries from before home capture keep
the argv-only match.
update_pause_record._git_executable reads the paused gateways' checkout outside the updater, where
install.ps1's PM-staged git is not on PATH, so it falls back to that staged git after PATH. Same
shape and reason as hermes_cli._early_recovery._git_executable (#132361), which it becomes once
both land; the managed-runtime ratchet needs the row until then.
Decision for review 5411136378's "same HEAD" item. At the pre-update HEAD the
gate counted every tracked change the pause had not seen as git's half-written
checkout. A no-op update (already at the target) still runs its dependency and
web builds; one that rewrites a tracked file (a lockfile, a generated asset)
made the end-of-update resume defer, and every later launch and every later
no-op `hermes update` (adoption keeps the old baseline) deferred again, so the
gateways stayed stopped until someone cleaned the file. A user editing a tracked
file had the same effect.
Rule chosen (a committed update's gateways never wait on a follow-up that cannot
succeed, and never start on a torn tree): a fast-forward, reset or merge only
writes paths that differ between HEAD and its target, and the update fetches
that target first. An unseen change now counts only on a path that differs
between HEAD and a commit in FETCH_HEAD or refs/remotes. When git cannot say or
nothing was fetched, every unseen change still counts (the previous behaviour),
so the torn-tree cells keep holding.
Decision for review 5411136378's "moved HEAD" item. tree_is_whole required
pm.venv_is_current at a moved HEAD and counted any exception as "not current".
The updater's early exits after the pull (wrong branch, channel sha mismatch)
reach the resume before the dependency sync, so they deferred; on a checkout
whose launches never sync dependencies (developer checkout, no install stamp,
lazy installs disabled, ZIP) or whose probe raises, the deferral never ended
and gateways stayed stopped where main restarted them.
Rule chosen (gateways never stay stopped after a committed update without an
owed follow-up that can succeed): defer only when the install is self-managed
(updateMechanism "self", the conditions venv_sync.prepare_launch syncs under),
because every later launch syncs dependencies before startup recovery runs, so
the owed record resumes on the next launch. Everywhere else, and when currency
cannot be established, resume now; the restarted gateway's own launch syncs or
prints the remedy.
Changed assertions: test_resume_waits_for_a_whole_tree pinned "dependencies"
deferral on a stamp-less checkout, which is the stranding this item reports; it
now expects a resume. test_an_adopted_baseline_never_replaces_this_runs_own
patches the renamed helper (_venv_is_current -> _deps_hold_resume).
update_pause_record._git ran a bare ["git", "-C", root] through subprocess.run.
On a Windows install whose only git is the copy install.ps1 staged in PM's store
(expose_pm_git puts it on PATH only inside the updater), every gate read died
with WinError 2: head_sha returned None and tree_is_whole refused on every
launch, so paused gateways stayed stopped. Reads now resolve git like the
updater does without installing anything (PATH, then PM's recorded entry, then
the lockfile's staged entry) and run through update_custody.run_git, so inside
an update they are job-bound on Windows and carry the custody config.
The resolver mirrors _early_recovery._git_executable from #132361 minus the
marker-recorded path; this lane does not carry that module's version.
Review 5411136378 (#132338) fix 3.
identity() took its creation time from process_identity._process_create_time
(psutil only), while identity_is_live judges with update_lock.incarnation_live,
whose producer update_lock.process_create_time falls back to the stdlib probe.
Without psutil every record was written with ct=None, which the reader can only
call unprovable (live), so a killed updater's pause was never orphaned on such a
host. identity() now calls update_lock.process_create_time, the same producer
the marker uses.
Review 5411136378 (#132338) fix 2.
Three hygiene gaps in the Windows paused-gateway record:
- _atomic_write left `<name>.<pid>.tmp` beside the record when os.replace
was refused (a Windows reader without FILE_SHARE_DELETE) or the write
failed: the temp is now unlinked and the error still propagates.
- _mutex counted re-entry process-wide, so a second thread of the same
process (gateway consumer, watcher) entered while the first held the
kernel lock: the depth is now threading.local and each thread's first
entry takes its own descriptor + flock/msvcrt lock, so threads exclude
each other like processes.
- A `.retired` list that reads but does not parse raised RetiredUnknown
from adopt_orphans -> every Windows update aborted, forever. With no
record or claim of this checkout left the list guards nothing: it is
renamed to `.corrupt` (warning on stderr) and the update goes on.
While a record/claim exists the list may be the only proof it was
completed, so it stays unknown (R9 fail-closed is unchanged); a list
that cannot be read at all (sharing violation) is unknown as before.
Review W3.
