Source activation could not build cryptography because the setup shell
could not discover the installed OpenSSL development libraries.
Configure Visual Studio ARM64, Clang, Rust, and static OpenSSL before PM
runs. Reuse installed tools and install missing prerequisites. Select a
classic vcpkg with a ports tree and use an explicit installation root.
Report damaged shared libraries without deleting the shared installation.
Verified native PowerShell activation and deactivation on Promise, then
warm activation with no new dependency generation. Cryptography imported
with static OpenSSL. Real vcpkg checks covered installation into a path
with spaces, warm reuse, manifest mode, and damaged-package rejection.
The canonical Windows runner passed the helper and output-encoding tests.
Fresh Visual Studio and Rust installation were not exercised because
Promise already had those toolchains installed.
Windows defaulted captured uv output to CP1252. A UnicodeDecodeError in
the pipe reader hid the OpenSSL build failure and left an empty diagnostic.
Decode uv and npm output as UTF-8, replacing malformed bytes while keeping
the exit status and build error.
Real subprocess tests cover stdout, stderr, legacy locale defaults, and
malformed output. The encoding tests passed on native Windows ARM64 and
Linux through scripts/run_tests.sh.
Activation reaches plugin discovery before the application dependencies
exist. Give PM its own locked Python project and runtime so it can install
or repair the application without importing that dependency tree.
Keep PM outside the application workspace. A shared uv workspace resolves
the application graph and cannot provide this isolation. Route mutations
through an isolated worker and preserve transaction callbacks, cancellation,
custom package registrations, and correlated receipts.
Use the same runtime builder for source installs and packaged payloads.
Keep offline wheelhouse support in that builder. Nix builds the independent
PM lock as a separate derivation. Refuse lazy-disabled bootstrap before
installing tools or dependencies.
Move first-party YAML readers and writers to ruamel. Keep the application
lock's transitive PyYAML requirements for third-party packages.
Verification:
- Focused canonical Python suite: 177 passed, 1 host-gated skip.
- Electron backend probes: 12 passed. Electron typecheck passed.
- Both uv locks, scoped lint, Bash syntax, and whitespace checks passed.
- Cold activation, corrupt-app repair, offline staging, and relocation ran.
- Built and exercised the Nix PM runtime and standalone YAML merge script.
Six broader caller test files retain the same 24 failing test IDs as an
archive of HEAD. The existing real-home guard blocks those tests before
they can exercise the affected paths. No full-suite pass is claimed.
Native Windows signing and full Bionic package execution remain unverified.
The merge combined old callers with newer lifecycle and update modules.
It also dropped native handlers while keeping their preload methods.
Type declarations alone could not repair those runtime failures.
Restore bounded backend teardown and retain failed-stop ownership.
Use API-only passive checkout checks with a daily disk cache, and pass
manual refresh requests through the updater strategy. Keep the shared
About UI and restore onboarding, feature flags, and notification wiring.
Verified all workspace typechecks, lint on the changed desktop files,
focused UI and Electron tests, and the development bundle. A headless
Electron smoke test exercised the real main process, preload, and native
IPC. A separate test exercised update checks with a linked git worktree,
loopback HTTP, and the disk cache. The full repository suite was not run.
The _sql_json_extract wrapper removed the `COALESCE(model_config` text the
alias string-replace keyed on, so the deferred-backfill SELECT joined an
unqualified `model_config`. Build the predicate from the alias directly and
derive the unaliased constant from it, so the two can never disagree.
Sibling widening of the #101726 salvage: FTS_TRIGRAM_SESSION_SQL (trigram view/triggers/backfill),
the v16 delegate-tagging data migration and reopen_session's legacy reset-child stamp still called
json_extract() on the raw model_config cell, so one malformed JSON row could still abort FTS
maintenance, a schema migration or /resume of a reset child. Zero raw model_config json_extract
reads remain in hermes_state_*.py.
Three invariants over real SQLite fixtures (TEXT and 8.4e252 timestamps written straight into the
REAL columns; 1200 ids under a 999-variable ceiling via setlimit): list/export/insights complete
and name the corrupt session in a WARNING; writers never persist an out-of-window timestamp;
prune/delete_sessions succeed with zero orphaned messages. All three fail on origin/main.
prune_sessions(), delete_empty_sessions() and prune_empty_ghost_sessions()
built one IN (?, ..., ?) clause containing every selected session id (the
parent-orphaning UPDATE), so cleaning more than SQLITE_MAX_VARIABLE_NUMBER
sessions failed atomically with "too many SQL variables". The per-row
DELETE loops that followed are folded into the same 900-id batches.
Same single _execute_write() transaction; only the binding is split.
Hand-ported from PR #100658 (targeted the pre-decomposition hermes_state.py
god file; the methods now live in hermes_state_maintenance.py /
hermes_state_sessions.py). The one-pass transcript-directory sweep from
that PR is not ported (out of scope for the variable-limit bug). Authored
by @Mi55ed; ported under --author.
`hermes sessions prune --source cron --older-than 14` on a store with ~60K
cron sessions (~50K matches) died with sqlite3.OperationalError: too many
SQL variables. SessionDB.delete_sessions, _collect_delegate_child_ids and
_delete_delegate_children each bound the full id list into a single
IN (?,?,...). SQLite caps bound parameters at SQLITE_MAX_VARIABLE_NUMBER
(999 on < 3.32, 32766 after), so any bulk delete above that failed outright.
Chunk every IN list (`_id_chunks` / `_SQL_IN_CHUNK` in hermes_state_common,
900 ids; the delegate walk binds each id twice so it chunks at half). Same
transaction, same cascade/orphan contract; only the parameter binding is
split.
Hand-ported from PR #102679 (targeted the pre-decomposition hermes_state.py
god file; the functions now live in hermes_state_sessions.py). Authored by
@mssteuer; ported under --author.
SQLite dynamic typing lets a TEXT cell ('not-a-timestamp'), inf/nan or a
garbage double (8.4e252 salvaged from a damaged page) sit in a REAL
timestamp column. Every reader called datetime.fromtimestamp()/float
arithmetic on the raw cell, so ONE bad row raised TypeError/OverflowError
out of the row loop and took down the whole `hermes sessions list`/browse
table (#102399), all three exporters — JSONL/MD, QMD, HTML (#102352) —
and `hermes insights` (#99959).
Fix the class with ONE helper, hermes_cli.timefmt.coerce_epoch(): a
stored cell becomes float epoch seconds inside a sane 1970..2103 window
or None after a WARNING that names the session id. Every reader routes
through it — relative_time (list/browse/resume picker), format_epoch
(prune/candidates tables), the three exporters' timestamp formatters,
insights' _get_sessions/_day/period range — so a bad row renders as
'?'/'N/A'/raw text for that one cell and the command completes.
Write side: hermes_state_messages._coerce_timestamp (append_message,
append_messages_batch, import) and the import path's started_at now use
the same window, so a new out-of-range timestamp falls back to now()
instead of being persisted — new bad rows cannot be written by Hermes.
Reported-by: #102399, #102352, #99959 reporters; kokhlo's insights
analysis pointed at every reporting site, not just line 860.
The schema initializer now replays SCHEMA_SQL through executescript (the
single-authority path from #94701's follow-up), which bypasses the
execute()-level DDL failure injection — the regression stopped raising.
Bind the same simulated failure onto the executescript path so the
connect-close-on-init-failure contract stays pinned for both replay
mechanisms.
Review follow-up on #94701: the delegation tool's _initialize_schema
still carried its own CREATE TABLE + ALTER column list for
async_delegations, leaving a second durable-shape authority even with
the column declared in SCHEMA_SQL. Its legacy ALTER added
origin_session_id as bare TEXT (nullable, no default); reconciliation
repairs missing column names only, so a database first opened through
the tool kept a non-canonical shape forever (#94691).
Remove the private DDL entirely. The tool's initializer now calls a new
reconcile_state_schema() in hermes_state_schema, which replays the
canonical SCHEMA_SQL (idempotent CREATE IF NOT EXISTS for every table,
canonical indexes included) and reuses SessionDB's declarative
_reconcile_columns for missing-column backfill — one reconciliation
implementation, one authority. Because _parse_schema_columns
reconstructs each column's full constraint expression (type, NOT NULL,
DEFAULT), the tool-first legacy path now adds origin_session_id as
TEXT NOT NULL DEFAULT '' — the canonical shape — and SQLite backfills
existing rows with the '' default.
Opening-order regressions compare FULL PRAGMA table_info metadata
(type, notnull, dflt_value, pk) plus the canonical index set across
fresh SessionDB→tool, legacy→SessionDB, and legacy→tool→SessionDB,
each preserving a pre-existing legacy delegation row.
The delegation tool carries its own CREATE TABLE for async_delegations
(tools/async_delegation.py _initialize_schema) plus a lazy ALTER TABLE
ADD COLUMN for the tables it finds already existing. Its column list
had drifted ahead of the canonical SCHEMA_SQL: origin_session_id
(raw api_server session id of the originating request, the wake
self-post target) existed only through the tool's lazy path, so two
databases at the same schema_version had different
async_delegations shapes depending solely on whether the delegation
tool had ever run. Rebuild/replay pipelines that reconstruct state.db
from the canonical schema then hit the column with no version gate to
explain it (#94691).
Declare the column in SCHEMA_SQL with the same TEXT NOT NULL DEFAULT ''
shape the tool uses. Fresh installs now carry it canonically; the
declarative _reconcile_columns backfills it into legacy databases on
the next writable open (same pattern as earlier additive columns); the
tool's lazy ALTER keeps serving pre-reconciliation databases. The two
schema authorities now agree, pinned by a test that runs the tool's
initializer over a canonical database and asserts the shape is
unchanged.
Fixes#94691
BtbN publishes new FFmpeg builds without changing the version number.
The shared-minor comparison therefore reported stale artifacts as current.
Compare advertised artifact URLs during resolution and hash changed URLs
when applying the update. Keep dry-run checks metadata-only and preserve
pins for targets without an update source, including Termux.
Verification: 41 focused tests passed. The regression exercises real
archive downloads, installation, retained target pins, and a second
update that performs no writes. Native ARM64 FFmpeg also passed a real
16 kHz audio encode after installation.
The corruption explainer filled `{db_path}` from `_default_db_path()`, the
process default. A Desktop `serve` backend launched on the root home hosts
named-profile sessions whose SessionDB is `profiles/<name>/state.db`, so the
operator was told to inspect/repair a different profile's database. Pass the
agent's own `_session_db.db_path` from the turn finalizer; the process
default remains the fallback for agents without a bound store.
Reported in #105887.
Reading a session by id that missed the caller's store fell through to
_locate_session_db(), which opened every profile's state.db read-only and
returned the first owner's full transcript — no opt-in, no profile named, and
the miss path even fired after an explicit non-matching profile= read. Any
caller holding an id (ids appear in logs and tool output) could read a
foreign profile's conversation. Profiles are isolated islands by design.
A miss now stays a miss, with a hint to name the owning profile
(profile=<name> / @session:<profile>/<id>), which remains the sanctioned,
explicit cross-profile read. The schema eval runner no longer needs to fake
the scan.
Reported by the #106761 filer; reproduced by @kokhlo. Refs #87779.
The salvaged #102534 test patched hermes_state.get_shared_session_db, a seam
main dropped (server._get_db now calls hermes_state_registry.acquire), so the
fixture errored at setup and the file reported 0 passed. Assert on the real
handle's db_path and on the foreign home staying untouched instead of on a
fake factory; red with the pin reverted, green with it.
The lazy _get_db() singleton followed get_hermes_home(), so a first touch
inside the multiplex cron ticker's per-profile override window permanently
bound the default backend to another profile's state.db (#102526).
close_all_under returned after the last release dropped the generation
and before the physical close finished, so rmtree still saw the open
handle. Wait directory-matching teardown barriers the same way close_all
does.
delete_profile already force-closes holographic memory_store.db in this
process, but the shared SessionDB registry kept state.db open. Recreate
then failed with a replaced/locked database. Close every shared handle
under the doomed directory, same contract as MemoryStore.release_all_under.
Co-authored-by: Cursor <cursoragent@cursor.com>
Since 0.21.0 reads go through mode=ro pooled connections. A read-only OPEN already
rides out the millisecond WAL transition window (checkpoint / WAL reset / frame flush
by a sibling process; the ro reader cannot rewrite the -shm index) with a bounded retry
(#100436), but a WARM pooled reader hitting the same window while its SELECT executes
propagated `disk I/O error` straight out of get_session(): 37 identical tracebacks on a
multi-process WSL2 ext4-on-vhdx install, each followed by "compression session recovery
failed", with quick_check=ok (#100871). The reporter's A/B shows the operator
workaround (journal_mode=delete) collapses read throughput ~30000x, so the flake has to
be absorbed on the read path.
_read_one/_read_all now replay the idempotent statement within the existing read-only
IOERR budget (3 x 50 ms) on the SAME connection -- close+reopen would cancel this
process's POSIX locks for every sibling connection -- and a persistent IOERR still
propagates. No quarantine: EIO on a read is busy, not broken. Every SELECT in the
SessionDB siblings (63 call sites) reaches the pool through these two helpers, so the
class is covered without a wrapper type.
Same-connection retry per #100882's analysis (@fangliquanflq); #100883
(@Sahilvishnaliya) diagnosed the missing recovery in the 0.21.0 read pool.
Fixes#100871.
Co-authored-by: fangliquanflq <fangliquan@qq.com>
Co-authored-by: Sahilvishnaliya <222165401+Sahilvishnaliya@users.noreply.github.com>
apply_wal_with_fallback() reports "wal" in two indeterminate cases -- the vulnerable-
SQLite gate (_apply_delete_for_wal_reset_bug) and the non-vulnerable probe-unknown path
(a7f2a593d1) -- meaning "touched nothing, the connection inherits the header's mode".
SessionDB turned that assumption into `_wal_active=True`, which enables the mode=ro read
pool that skips `self._lock`. On a file that is really in rollback-journal mode those
readers race the writer with a 5s busy timeout and no retry: random SQLITE_BUSY read
failures for the instance's lifetime (#86515).
Confirm the header on the freshly opened connection before enabling the pool. When the
probe is still blocked, reads queue on the writer connection under the lock -- slower,
never wrong. Every other apply_wal_with_fallback caller ignores the return value, so the
consumer is the right place to gate; changing the return contract to Optional across
15 call sites (#87044's shape) is not needed.
Live repro: DELETE-mode file, sibling holding BEGIN EXCLUSIVE during open ->
before: _wal_active=True and _checkout_read_conn() hands out a pooled mode=ro conn;
after: _wal_active=False, reads take the locked writer path.
Fixes#86515. Based on the analysis in #87044.
Co-authored-by: QDung210 <dqdung205@gmail.com>
_send_session_db_warning_notifications() broadcast the error recorded at startup
without asking whether it was still true. A startup `database is locked` routinely
clears while the adapters are still connecting (another profile's open, a `hermes
sessions` one-shot, a slow SMB lock release), so the home channels were told the
store was unavailable when it had already healed — and a warning that is wrong once
is ignored the next time it is right.
The RecoverableHandleCache opener already clears `_session_db_init_error` on
recovery; the broadcast now drives one open attempt through it first and only warns
when the error is still standing. A store that is genuinely still down warns exactly
as before.
Fixes#108031.
main replaced the file-local _make_db/_require_wal/_unlink_sidecars helpers with
tests/hermes_state/_wal_generation_harness (make_db/require_wal/lose_sidecars) after
PR #107411 branched; the cherry-picked tests referenced the old names and failed at
collection with NameError. Fixture-only change; the assertions are unchanged.
Classify deleted WAL generations separately from main-file replacement and point operators at the captured-generation manifest and mode-aware recovery path.
Co-authored-by: crazyief <8566250+crazyief@users.noreply.github.com>
A lock-free _raise_if_db_replaced() probe at the top of the _execute_write
retry loop raced a concurrent close(). close() runs under the same _lock and
ends the WAL generation: it checkpoints, closes the connection (SQLite
unlinks the -wal/-shm sidecars), nulls _conn and clears
_db_sidecar_identity. The probe could observe the mid-teardown state —
sidecars already unlinked while _db_sidecar_identity was not yet cleared —
and misclassify this process's OWN clean close as an externally deleted WAL
generation, raising a sticky DeletedWalGenerationError that permanently
refused every later write on that handle (#105567).
Move the live probe inside the lock, ahead of the close-race reopen
decision, so it only ever observes the stable post-close state (identity
cleared -> the existing adopt/reopen path). The corrupt flag check stays on
the lock-free fast path; external file/generation replacement detection is
unchanged, just serialized with teardown.
Synthetic repro (100 rounds x 40 writes, direct SessionDB handles): before
~9 failing rounds / ~360 DeletedWalGenerationError; after 0 failures,
4000/4000 writes persisted across repeated runs. tests/state (181) plus the
generation/replaced/corrupt guard suites (55) pass.
Fixes#105567
st_nlink == 0 alone cannot distinguish a genuine orphan from one that
still has a surviving hard link (e.g. a backup) after the watched
sidecar path itself was removed or replaced — that left st_nlink >= 1
on a truly orphaned generation, letting a new opener through while a
live writer still owned the old one. Compare (st_dev, st_ino) between
the fd and the current watched sidecar path instead: only an exact
match means they're the same live file, so any mismatch or unstattable
watched path still fails closed.
iter_deleted_sqlite_sidecar_holders() and SessionDB._wal_generation_was_lost()
both treated a `` (deleted)`` suffix on a /proc/<pid>/fd/* target as proof that
state.db-wal or state.db-shm was unlinked. On OpenZFS that suffix is not proof:
a live, still-linked file whose dentry was unhashed is reported the same way,
with st_nlink still 1 and the same (dev, ino) as the path. The guard then fires
permanently and the gateway falls back to JSONL forever, because the WAL was
never actually deleted.
Add _fd_is_truly_unlinked(), which confirms via os.stat(fd_path).st_nlink == 0
before a target counts as an orphaned generation. An unstattable descriptor
still counts as deleted, so the guard keeps failing closed. _iter_proc_fd_targets()
and _proc_fd_targets() now also yield the /proc fd path itself so both call
sites (open-path and the sticky write-path probe) can run the check.
Drop two PrimaryProfilePin cases that only restate the constructor
defaults and blank-string normalisation, and the wiring-routing test that
froze POOL_LIMITS_SETTINGS_ROUTE to a literal string — a snapshot of the
constant, not a behaviour contract. The two kept pin tests cover the bug
(a live primary keeps answering for its booted profile after the stored
preference moves; teardown releases the pin), and the notifications tests
cover the toast action end-to-end.
`primaryProfileKey()` re-read active-profile.json on every call. The rail's
live workspace switch rewrites that file via `hermes:profile:remember`
WITHOUT re-homing the primary, so after a switch the routing table disagreed
with the running process: a request for the profile the primary actually
booted as (e.g. "default") no longer matched `primaryProfile` in
`resolveProfileBackendRoute`, fell through to the pool, and spawned a second
backend for the same HERMES_HOME.
The duplicate was keepalive-fresh so LRU eviction spared it, it burned a pool
slot, and with the default cap of 3 every further profile queued and failed
with `Local backend start for "<profile>" timed out while waiting for a free
slot` (repro in desktop.log: "default" spawned as a pool backend while the
primary "default" was still running; coder/qwen then timed out for 20+ min).
Snapshot the launch profile in `startHermes()` (PrimaryProfilePin.pin) and
release it in `resetHermesConnection()` so the next start follows the stored
preference again. The pin is a tiny pure module with tests; main.ts only owns
the file read and the two call sites.
Switching the pooled dispatch probe to /api/health (salvaged from #97914)
would 404 on every dispatch against a remote older than 0.19, retire the
tunnel and reconnect forever - the same storm #107997 describes, moved to
old backends. Fall back to /api/status on an explicit 404 exactly the way
the boot readiness probe already does (backend-health.ts). The legacy
fallback idea and its test are taken from #101976 (@edosulai); the rest of
that PR (timeout-tolerance streak, ServerAlive SSH options) is not adopted.
Co-authored-by: Edo Sulaiman <edosulai@icloud.com>
Cold /api/status through a Windows no-mux SSH forward routinely exceeds
the 2.5s dispatch budget, so Desktop retires a live tunnel and respawns.
Use the cheap /api/health route (5s, same as DEFAULT_HEALTH_PROBE_TIMEOUT_MS).
Background liveness still probes /api/status at 10s.
A 2500ms dispatch probe is shorter than quiet-box hermes serve cold-start (~6-8s), so a just-woken pooled backend always fails and reconnects. Reuse REMOTE_LIVENESS_TIMEOUT_MS (10s) for that probe.
Co-authored-by: Cursor <cursoragent@cursor.com>
Keep the two invariant tests from #105428: a genuine second instance
whose argv is scoped to another HERMES_HOME is not a holder of our
state.db; argv that names our state.db stays flagged. The helper-shape
tests were change-detectors on `_argv_scoped_to_other_home` internals.
The salvaged commit is re-authored to TaoMasterCoder's GitHub noreply
identity: the original `devops@77hub.com` is a shared org address
(misconfigured local git, not malice).
Refs #107440
Field evidence (2026-09-07, production host): the host runs two independent
Hermes instances - a main gateway (user ubuntu, HERMES_HOME=/home/ubuntu/.hermes)
and a demo gateway (user demo, HERMES_HOME=/home/demo/.hermes). Because the
demo process is owned by another user, its /proc/<pid>/fd table is unreadable
and foreign_state_db_holders() falls back to cmdline + _looks_like_hermes().
The demo argv matches Hermes patterns exactly, so it was flagged as an
uninspectable holder of the MAIN instance's state.db even though lsof proved
0 open handles on it. Result: _recover_stale_fts was deferred 42 times across
6 gateway restarts, the fts_stale breadcrumb never cleared, and FTS
self-repair stayed permanently disabled.
#92419 removed substring false positives (journalctl/grep mentioning hermes);
a genuine second instance with a DIFFERENT HERMES_HOME was still misjudged.
Add _argv_scoped_to_other_home(): when the argv of an uninspectable Hermes
process proves it lives under a different /.hermes home (or a state.db
sidecar under a different parent) AND no token references our state.db,
sidecars, or home, do not count it as our holder. Applied to all three
uninspectable branches (holder + two descriptor paths). Ambiguous argv
without absolute-path tokens remains fail-closed, preserving the
conservative intent.
References #92401
Invariant test for the #103489 salvage: a profile gateway and the root
gateway resolve the same hosted-room coordination file at the install
root, and that file is never the root SessionDB (`state.db`). Every
profile gateway starts the hosted-room worker unconditionally, so the
old mapping made each profile process a long-lived writer on the master
session store — the simultaneous-restart corruption in #102120 and the
main offender in the #103339 / #103490 fleet reports.
Also: WAL-fallback labels say `shared-state.db`, and test fakes take the
store path from `default_db_path()` instead of hardcoding `state.db`.
Reported-by: #102120#103339 (@RChina) #103490
Refs #102120#103339#103490
`hermes profile delete` leaves `profiles/.deleted/<name>` behind.
`HostedRoomService.local_profiles()` fed every subdirectory name to the
roster, so `.deleted` failed `validate_roster`'s identifier check and
`plan_next_task` raised on every cycle for every room until the
directory was removed by hand (#106847, bug 2). Skip dot-dirs and
tombstoned profiles, using the same `named_profile_is_deleted` predicate
`hermes_cli.profiles` uses to list live profiles.
Refs #106847
The dashboard's `_eager_reconcile_own_session_db` did an unconditional
writable `acquire()` at every startup. When the gateway shares that
state.db the dashboard became a second long-lived writable SessionDB
owner: a close-time WAL checkpoint plus a possible FTS rebuild in
`_init_fts`, the two-writer vector behind the corruption reports in
#107688 and #100896 ("5 live SessionDB handles" precursor, gateway +
dashboard both holding the WAL).
Route the startup reconcile through `_open_session_db_at_path(...,
read_only=True)`, which already bootstraps a missing store and heals a
stale/malformed schema through exactly ONE writable open before
reopening read-only. A healthy store now gets zero writable opens from
the dashboard while the #79531/#80037 "bring schema current before the
first poll" contract is kept (existing heal test unchanged).
Live repro (healthy store, count writable SessionDB.__init__ calls from
the startup worker): before=1 after=0.
Reported-by: #107688, #100896 (@kokhlo diagnosis)
Refs #107688#100896
