Commit Graph

266 Commits

Author SHA1 Message Date
ethernet
9a42d60f25 fix(build): handle large release uploads and trim packaging waste 2026-09-13 11:04:03 -04:00
ethernet
5e4a2a3d24 refactor(pm): remove legacy dependency and launch managers
Competing installers and checkout-local venv assumptions bypassed PM
selection, install consent, and generation lifetimes. Route consumers
through PM and installation-bound launchers. Refresh source launchers
before obsolete Python entries can be collected.

Remove Node, browser, and CUA acquisition engines, obsolete venv-holder
handling, detached sync, and unused PM APIs. Keep historical updater
exports inert and preserve external tool ownership and native integration.

Share product freshness and prepared inputs across builders. Align plugin
admission, Docker provisioning, setup instructions, and behavioral tests.

Verified targeted Python and JavaScript tests, desktop and web typechecks,
scoped lint, real product builds, and the Docker frontend smoke test.
The missed post-setup test cleanup is included and verified.

Native Windows/macOS execution, full Rust compilation, and the complete
repository suite remain unverified. Historical compatibility requirements
were preserved and extended, not fully rescanned.
2026-09-12 14:57:38 -04:00
ethernet
26c4e8b160 refactor(update): use PM and shared source builders
PM owns Python dependency generations. Shared frontend builders own Node
preparation and compilation. Route source updates and launchers through
these owners instead of separate repair ladders.

Remove obsolete live-venv holder gates and soft build-failure plumbing.
Preserve source validation, staged publication, fleet outcomes, and
historical relaunch hooks.

Verification: 956 tests passed in the combined focused run, with 43 skips.
After the final ZIP exit fix, 583 focused tests passed. Shared JavaScript
builder tests, Ruff, and diff checks passed. Native Windows/macOS and full
packaged-app builds were not run.
2026-09-12 13:45:08 -04:00
ethernet
162aa8a902 docs(release): describe partial build diagnostic links 2026-09-11 20:16:56 -04:00
ethernet
2a1cb9c62e docs(release): name commit build pages before dispatch 2026-09-11 20:00:46 -04:00
ethernet
995a4617ac docs(release): describe nonstable package identity and branding 2026-09-11 19:43:54 -04:00
ethernet
8e49e5e517 docs: define side-by-side builds and source release channels 2026-09-11 19:42:28 -04:00
ethernet
547b5de29d docs(pm): route dependency setup through public operations
Direct package installs bypass PM's dependency selection and do not survive
new generations. Document explicit runtime extras, recorded repair, fresh
build outputs, and lock generation through PM instead.

Keep the prepared-interpreter prerequisite and explicit removal requirement
for disposable test environments. Preserve Nix and external-project package
manager ownership. Correct platform and Python-marker claims where the
manifest contradicts the installation hints.

Checked the public CLI help, literal PM calls against public signatures and
extra declarations, fenced blocks, and whitespace. No dependency build or
site build ran. The docs toolchain is not installed in this checkout.
2026-09-11 18:12:49 -04:00
ethernet
fea2858c99 merge: unify shared product builders, caches, and Windows prerequisites
Merge ethie/shared-product-builders with the CI dependency cache and native Windows setup work. Preserve UTF-8 diagnostics in the shared Python environment runner. Pass a persistent cache through isolated native staging and PM-runtime construction. Reuse one Windows prerequisite installer from source setup, native adapters, and CI, preserving Rust homes across HOME isolation.

Verified 85 targeted Python tests (5 host skips), 18 JavaScript tests, workflow validation, and scoped lint/typecheck. On native Windows ARM64, five prerequisite contracts passed and the actual shared provider reused OpenSSL, compiled its header with MSVC, and retained Rust under isolated HOME. Full signed distribution builds and live Actions cache transfer remain CI verification.
2026-09-11 13:45:05 -04:00
ethernet
1bf588234c refactor(build): share product recipes across distributions
Build TUI, web, desktop UI and runnable agent products from explicit
prepared inputs. Keep dependency preparation separate from distribution
packaging, with PM and native builds sharing uv environment construction.

Docker copies compiled frontend products instead of build dependencies.
Nix retains uv2nix environments and consumes shared assembly through store
references. Native desktop and Termux use the same launcher and frontend
contracts. Preserve the independent PM runtime and source imports from
arbitrary working directories.

Keep failed frontend builds from replacing the previous product, reject
source/output overlap, and bound dependency-process output draining.
Include hermes_wisdom in the Nix wheel: real CLI smoke tests exposed its
missing package declaration on the base revision too.

Verified focused Python and JavaScript suites, Docker build/runtime checks,
Nix desktop and CLI/ACP checks, standalone TUI and packaged Electron PTY,
and real full-Chromium interaction. Native signed installers, Android device
installation and the full repository suite remain CI verification.
2026-09-11 13:16:55 -04:00
ethernet
c8a9682505 fix(pm): preserve pinned binary inputs in R2
Termux removes old package files, so a pinned URL and hash do not keep
build inputs available. Preserve the exact bytes without changing pins.

Archive every PM HTTP artifact and the Termux runtime inputs by SHA256.
CI reads R2 first. Only a missing object permits an upstream download,
hash verification, immutable upload, and verified readback. Seed the
actual toolchain and payload stores before their consumers run.

Use the public archive as a pinned fallback in PM, bootstrap installers,
and Nix fetchers. Keep network retries bounded and report attempted URLs.
Keep publication credentials in protected CI jobs, not installed clients.

Verification:
- 283 targeted tests passed; five POSIX tests skipped on Windows.
- All 87 preserved Termux packages passed local archive miss/hit checks.
- Native ARM64 ripgrep installed through the mirror and ran successfully.
- Wheel import, workflow lint, Python lint, shell syntax, and pins passed.

Live R2 publication, POSIX tests, and Nix builds remain for native CI.
The real-byte archive checks used loopback HTTP, not the live bucket.
2026-09-10 18:17:08 -04:00
ethernet
90482e16b9 Merge branch 'ethie/pm-clean' of github.com:NousResearch/hermes-agent into ethie/pm-clean 2026-09-10 16:09:14 -04:00
ethernet
949a508308 fix(pm): remove download archives after package publication
Retain archives through extraction, verification and publication so failed
or paused installs can retry. After success, delete only the package's
archives while holding the store lock. Normal installs commit facts first.
Cross-target staging follows the same cleanup rule.

Keep unrelated archives and resumable partials. Docker uses this PM behavior
without a separate cleanup command. Later repairs may need a new download.

Verified 64 focused PM tests and 9 Docker tests. Failure-injection tests
cover extraction, verification, publication and facts writes. The rebuilt
linux/amd64 image contains no archives in any layer and remains 1.073 GB
compressed. Real Chromium navigation, clicks and screenshots pass.
2026-09-10 16:08:03 -04:00
ethernet
6756d11b5f fix(pm): ship full chromium without headless shell
Full Chromium serves both headed and headless sessions. The separate
shell duplicates the browser payload and is not needed for either mode.

Remove the shell from PM and Docker. Select the managed Chromium
executable for agent-browser and the full Chromium channel for direct
Playwright callers. Route setup through PM and remove retired packages
from cached bundle stores without changing the user's tool store.

Update signing, architecture checks, launch probes and install guidance.
Leave llama packages and Docker archive cleanup unchanged.

Verification:
- Real agent-browser navigation, clicks, DOM reads and screenshots pass
  in headed and headless modes with the same Chromium executable.
- The direct Playwright doctor probe passes.
- Focused Python and desktop packaging tests pass, as do six Docker
  checks and both real-browser task-scroll tests.
- The built linux/amd64 image is 1.393 GB compressed, 223.6 MB smaller.
- The broader PM suite and two unrelated setup tests still fail.
  Those failures reproduce on unchanged HEAD.
- Five updated eval scripts parse; their full scenarios were not run.
2026-09-10 16:08:03 -04:00
ethernet
f5981cbd14 fix(desktop): preserve MSIX runtime and add Store updates
Use a verified writable copy of pinned Python for uv builds. Keep the
signed launchers and bundled Python as the MSIX runtime so plugin
rebuilds do not remove package access.

Identify Microsoft Store builds from the artifact stamp. Use StoreContext
for update checks, downloads, and installation requests inside Hermes.
Keep relaunch and recovery ownership across the update. Do not report
unknown checks or no-op requests as successful installations.

Open the build commit link in the system browser.

Verification: 168 Python tests, 107 Electron tests, and 121 renderer tests
passed. Typechecks, lint, lock validation, and desktop builds passed.
A real packaged process completed plugin admission, import, and repair.

Store-acquired download, installation, and relaunch remain unverified.
This host has only the sideloaded MSIX. Its real Store API call returned
an error, which the app reports as unknown. macOS updates are unchanged.
2026-09-10 15:17:18 -04:00
ethernet
d233b6d7a9 feat(release): publish downloads pages to the release bucket
The builds table only ever existed inside a GitHub release body. Emit the
same rows as a tiny standalone page in R2, so a build is readable straight
from the download origin:

  releases/<channel>/index.html     latest stable / canary builds, every
                                    variant, replaced by each tag run
  releases/commit/<sha>/index.html  every expected binary of one commit
                                    build, built or not

scripts/render-builds-table.py keeps ONE row set per mode and renders it
into two sinks (release body markdown, page HTML), so the page can never
list different artifacts than the release. A channel page is a mutable
pointer written from a per-tag job, so it records its release tag and the
writer compares that against scripts/releases/semver.py before replacing:
re-running an older tag cannot regress a newer channel page.

Pages need two registrations to be usable: `.html` maps to
text/html; charset=utf-8 in release-content-types.json (unregistered, R2
serves the object as an octet-stream download) and to no-store in
r2.cache_control_for (the page is a pointer, not an artifact). Page keys
and public URLs come from new r2 layout helpers, shared with
`release.py --build-commit`, which now prints the commit page URL before
dispatching. No workflow change: the existing renderer jobs already carry
the R2 credentials.

Verified: 76 tests over the renderer/release/transport files, including a
loopback R2 PUT proving the page object lands as text/html with no-store.
2026-09-10 11:15:36 -04:00
ethernet
5874b11ec6 fix(release): move artifact handoffs to r2
Release jobs duplicate package transfers through GitHub artifacts and R2.
Use immutable R2 tag archives for build and stable candidate handoffs.
Keep stable feeds behind the existing acceptance and publication gates.

Publish tag and commit receipts after all files upload. Verify file sizes
and SHA256 digests during streamed downloads. Refresh request signatures
on retries. Remove candidate TAR copies and duplicate package uploads.
Select the previous Termux package by its published release tag.

Enable the existing uv built-wheel cache for Windows and macOS builds.

Verification: 76 focused tests pass through scripts/run_tests.sh.
Ruff, actionlint and scoped diff checks pass. No live release was run.
2026-09-08 21:56:39 -04:00
ethernet
7d2b3b767d merge: integrate upstream/main into ethie/pm-clean
Merge upstream b1f003e186 while preserving PM runtime ownership and
Python 3.14 worker startup, Windows signing, and macOS wait recovery.

Keep retired runtime modules deleted. Port upstream updater preflight
checks into the checkout strategy and preserve live build logging.
Carry checkpoint filename handling and process recovery into the current
module layout. Regenerate locks and adapt incoming platform test markers.

Focused Python and JavaScript tests, desktop and root-test typechecks,
conflict-path lint checks, lock validation, and retired-import checks pass.
The full test suite and packaged release builds were not run.
2026-09-08 19:17:39 -04:00
ethernet
712734436e fix(pm): make bootstrap and bundle ownership explicit
Finish bootstrap uv before PM replaces its store entry. Keep failure
receipts stdlib-only and align the cryptography requirement and override
with the locked version.

Let bundle builders declare launch paths and update ownership. Remove
payload discovery, Store probing, and the unused develop command.
Derive Nix Python from the PM lock and share its provenance stamp.

Document setup, activation, optional dependencies, and distribution
ownership. Targeted Windows tests, relocated runtime launches, Electron
bundling, and bilingual docs builds pass. Native Nix and signed-package
acceptance remain CI gates.
2026-09-08 00:24:51 -04:00
ethernet
8c2e88aa4d fix(release): bind stable package versions and manifest origins
The Windows package version ignored an explicit stable tag.
Use the tag for sideload version derivation and reject candidate metadata
that does not match. Keep the separate Store version policy unchanged.

Restrict baseline manifests to the configured release origin and reject
cross-origin manifest responses. Cover the checks with real loopback HTTPS
and extend the version and empty-gate regression tests.

Document retries that reuse the original artifacts. Remove an unreachable
manifest check and an unused test import.

Targeted Python and JavaScript tests, Ruff, ESLint and the shared MSIX
module typecheck passed. Full signed native release acceptance was not run.
2026-09-07 14:59:29 -04:00
ethernet
b0ab0162b0 feat(release): gate stable promotion through the full release pipeline
Run the entire CI workflow before Docker build and tests. Require Nix,
native payload smoke tests, install/update E2E and signed-package upgrade
acceptance before publishing. Keep Desktop Playwright E2E deferred.

Archive tested Docker images and signed bundle candidates with provenance
and hashes. Publishers consume those exact artifacts without rebuilding.
Advance stable channels only after all required publications succeed.
Keep canaries on their separate path and reject direct stable-builder
publication that bypasses the gate.

Move shared release transport, manifests and gates to Python. Keep native
Electron adapters in JS and share feed/MIME facts as JSON. Replace the
R2/feed JS implementation and move its protocol tests to Python.

Verified targeted Python and JS tests, real loopback transport and CLI
execution, temporary Git admission, workflow graph lint, and typechecks.
No live stable release was run. Native signing, package upgrades and real
registry/Store promotion still need their release-run receipts. Separate
services cannot promote atomically. A promotion failure keeps the run red.
2026-09-07 14:40:10 -04:00
Teknium
9c53128cc8 test(sessions): verify gateway status after timer removal 2026-09-07 06:10:54 -07:00
Teknium
34e512ae58 fix(sessions): remove remaining timer migration and guidance 2026-09-07 06:10:54 -07:00
ethernet
1a09c42414 refactor(bundle): share Python payload assembly across desktop and Termux 2026-09-06 22:21:06 -04:00
ethernet
da236308fd feat(desktop): wire macOS bundle updates and guarded feed publication
Route packaged macOS bundles and Light through the updater strategy.
Use electron-updater 6.8.9 and wait for native signature acceptance before
backend teardown. Keep checkout and Store ownership separate.

Share Darwin feed paths between packaging, runtime and publication.
Validate both native feeds, verify streamed artifact hashes, prevent
same-tag artifact replacement, and conditionally update the channel
pointer. Protect live feed references during canary retention.

Use one notarization owner. Require publishing credentials and validate
the stapled app. Keep Windows, Linux and Termux jobs unchanged.

Verified with updater/feed unit and transport tests, release-helper tests,
desktop typechecks, the desktop JS build, and workflow lint. No E2E,
native macOS install, release dispatch or public publication was run.
2026-09-06 21:27:58 -04:00
ethernet
495ebb83d3 docs(pm): refresh non-E2E closeout evidence 2026-09-06 17:06:11 -04:00
ethernet
ec59854d2f docs(pm): record verified closeout and remaining acceptance 2026-09-06 15:36:54 -04:00
ethernet
3c08d16ba7 fix(pm): close runtime publication and updater audit gaps
Dependency publication now recovers interrupted config/facts changes before
activation and leases live generations during collection. Receipts retain
update correlation and failed steps across nested command boundaries.
Doctor and desktop surfaces report those failures through shared owners.

Move checkout updates out of the desktop facade. Stage a detached Windows
relaunch waiter before shutdown, with bounded handshake and process-birth
checks. Keep packaged lifecycle tests isolated from the installed app.

Native verification exposed two production races: cron maintenance imported
the interactive CLI and rewrote TERMINAL_CWD, and install-ID reads collided
with first publication. Use the existing owners and locks. Plugin checks
now run at startup and each due-gated housekeeping tick, not after 60 ticks.

Share updater-test mutation boundaries and remove collection-root fixtures.
Separate cold MCP startup from command latency and give the real HTTP drip
test enough time to reach body handling.

Root npm check passed, including packaging. The fixed-tree Windows Python
run reported 44557 passed, one failed, and 1404 skipped, plus one retry-only
HTTP test. Those final failures now pass in a 35-test bounded batch. A real
isolated gateway wrote startup and periodic plugin-check receipts.

Full final-tree CI, bundled Sandbox deployment, and actual App Installer
relaunch remain unverified. docs/pm-audit-status.md records these limits.
2026-09-06 11:45:41 -04:00
ethernet
92686159d1 fix(pm): integrate audited runtime and lifecycle repairs
Prepare dependency generations before selecting them. Keep shipped tool
bytes separate from writable additions, and store facts beside their entries.
Validate proposed plugin sets before config publication. Restore the previous
config if the facts write fails.

Consolidate duplicate updater, backup, setup, and voice helpers. Repair
launcher selection, dependency consumers, download ownership, update feeds,
and native Windows process and file handling.

Verification: 206 changed/prior-failing Python files reported 4630 passed,
one failed, and 330 skipped. Fix the remaining Hindsight fixture boundary.
The final targeted rerun reported 234 passed and two skipped. The store
review regression batch reported 83 passed and one skipped. Desktop
TypeScript checks, 56 selected Electron tests, 24 release tests, and the
removed-import/compatibility guards passed.

This is an integration checkpoint, not full audit acceptance. The complete
Python suite has not run on this fixed tree. Crash-atomic plugin publication,
generation cleanup, receipt correlation, and packaged lifecycle acceptance
remain open in docs/pm-audit-status.md.
2026-09-05 22:36:48 -04:00
Teknium
66d478c6d3 docs: sync root docs, CONTRIBUTING tree, docs/*.md,
windows-quirks with the facade/siblings layout (#102117)
2026-09-04 00:14:22 -07:00
Teknium
eeb7671e69 simplify(compat): hermes_cli small facades — drop 7 re-exports/aliases (+relay_runtime alias module), repoint 12 callers/tests 2026-09-03 13:05:57 -07:00
Jeffrey Quesnelle
48c0c3a873 Merge pull request #96633 from afourniernv/chore/relay-0.8.0
fix(relay): upgrade to 0.8.3
2026-09-02 22:03:27 -04:00
Alex Fournier
4a339ad373 fix(relay): upgrade to 0.8.3
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-09-02 08:35:16 -07:00
pierrenode
3245264668 fix(relay): carry routed profile through the passthrough-plane forward
The relay text lane stamps `SessionSource.profile` from the wire frame
(#60586), but `PassthroughForward` had no profile field, so a relayed
Discord slash-command/button/modal always landed in the default profile's
agent:main namespace even when the connector resolved a specific profile.

Add an optional `profile` to `PassthroughForward` (read off the wire in
`_passthrough_from_wire`) and stamp it on the interaction's SessionSource.
Absent on the wire → None → legacy routing, byte-identical for
single-profile gateways. Contract doc updated.

Salvaged from #61012 (pierrenode); two context conflicts resolved
(delivered_via_upstream_relay / _platform_by_chat landed on main).
2026-09-02 06:47:45 -07:00
Teknium
b6f0106602 docs(design): note loader-boundary dotenv guard, scoped ${VAR} expansion and scoped .env publish in multiplexing design doc
Accuracy pass on the #89950 doc for the fixes landing alongside it (#77562, #84079, #88441).
2026-09-02 06:19:03 -07:00
Eva
011a60b7cc docs(design): add the multiplexing-gateway design doc referenced by secret_scope
agent/secret_scope.py has pointed at docs/design/multiplexing-gateway.md
("Workstream A") since the fail-closed secret scope landed, but the file was
never added. This writes the missing doc from the code as it stands today:
the mode flag, scope composition (_profile_runtime_scope seams), the
context-local secret scope and HERMES_HOME override, routing/serving/
persistence/session-lane isolation, the control-plane RPCs, failure modes,
and an honest table of what is still process-global (per-profile MCP
registries are tracked in #67605).

Doc-only change; no code touched. Style follows docs/profile-routing.md.
2026-09-02 06:19:03 -07:00
kshitijk4poor
e9160625dc fix(state): also disable SQLite's internal close-time checkpoint on quarantine (py3.12+)
Skipping the explicit PRAGMA wal_checkpoint(PASSIVE) in close() left
sqlite3.Connection.close() running SQLite's own last-connection PASSIVE
checkpoint, which still checkpoints the WAL and unlinks -wal/-shm on a
structurally corrupt file (E2E: the -wal vanished on close despite the
quarantine). Python 3.12+ exposes SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE via
Connection.setconfig(); arm it in _halt_db_corrupt so the WAL image
survives close() for forensics/recovery. On 3.11 the switch does not
exist; the docstring and docs now say so instead of claiming sqlite3
cannot reach it at all.

Follow-up to #101095; flagged by JoaoMarcos44 on #101093.
2026-09-02 16:57:21 +05:30
leomcamilo
bcc2e65818 fix(state): quarantine SessionDB handle after structural corruption
A bare SQLITE_CORRUPT/NOTADB on a live write (not FTS-scoped, not a
replaced file) now sets a sticky per-instance flag: later writes fail
fast with StateDbCorruptError, the handle never reopens after close(),
and close() skips its explicit PASSIVE WAL checkpoint. Gateway and agent
flush paths divert pending transcripts to JSONL/spool like the replaced
case instead of retrying forever.

Field evidence: a handle that kept writing for ~50 minutes after the
first structural error checkpointed 15 pages under the wrong page
numbers on shutdown (page 1 <- messages_fts_trigram_data leaf), turning
"malformed" into "file is not a database".

Refs #90837, #90950, #97940, #89332, #45383

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01CNX8rNYHqA5pT4tAGSzXtb
2026-09-02 16:57:21 +05:30
Ben Barclay
180291162f feat(telemetry): opt-in shared-metrics exporter (#95278)
feat(telemetry): opt-in shared-metrics exporter
2026-09-02 08:35:36 +10:00
Teknium
33797073bb fix: harden startup route salvage — aggregator-slug guard, alias credential ownership, oneshot dedup
Follow-ups on top of #87210 (@liuhao1024) and #87246 (@JoaoMarcos44):
- resolve_startup_model_route: aggregator-native slugs stay on the current
  routing aggregator (bare vendor slugs resolve WITHIN the aggregator first);
  URL-bearing aliases resolve via direct_alias_runtime_request so a foreign
  provider label never carries the vendor token to the alias host (#28660);
  route carries the alias's own api_key.
- cli.py: pass current_provider; explicit --api-key wins over alias key.
- Drop #87246's oneshot double-handling (main's oneshot alias+detection path
  already covers it once #87210's detection fix is in) and the PR-body SVG.
- Rewrote/extended startup-route tests for the hardened semantics.
2026-09-01 12:07:35 -07:00
joaomarcos
4c870951e2 fix(cli): resolve startup model routes before provider defaults
Resolve configured aliases and provider/model inputs before HermesCLI attaches the configured default provider. Keep aggregator namespaces intact, cover oneshot startup, and document the supported CLI forms.
2026-09-01 12:07:35 -07:00
Alex Fournier
e4123f1a70 docs(relay): align version references with 0.8.2
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-31 19:06:50 -07:00
the3asic
18ac3c4fb6 fix(state): defer corrupt FTS rebuilds past live operations 2026-08-31 12:08:30 -07:00
joe102084
b028fe632e feat: add cron doctor health check 2026-08-31 10:00:29 -07:00
Alex Fournier
31e7237adb fix(relay): upgrade to 0.8.1
Signed-off-by: Alex Fournier <afournier@nvidia.com>
2026-08-28 08:28:21 -04:00
Ben Barclay
a69a9c351d feat(telemetry): transmit the stable install_id as-is
Product-owner decision, 2026-08-27: the analytical need is stable
cross-window identity (retention curves, longitudinal install
behaviour), which the rotating pseudonym destroyed by design. The
feature has not shipped - zero consented users, zero production
transmissions - so identity semantics can change without breaking any
promise made to a user; existing (dev-only) consent windows carry
forward unchanged.

Removed in full rather than weakened in place:
- shared_metrics_identity.py (salt generation/rotation, HMAC-SHA256
  derivation, payload substitution) and its 19-test file.
- The sender's derivation step. _freeze_identity keeps its validation
  role (unreadable/non-object/id-less payloads still reject rather than
  block the queue) and now records the raw install_id in
  sent_install_id; _body rewrites the payload's install_id from that
  frozen column, keeping byte-identical resends anchored to one
  recorded value.

Consent surface updated in the same change: the setup wizard now states
plainly that packages carry the stable profile-scoped install ID (a
random UUID, no personal information, reset by deleting the
shared-metrics directory). No consent was ever collected under the old
wording in any shipped build.

Docs A.2/A.3 rewritten as decision records rather than silently
edited: A.2 records what is transmitted now and states the
consequences plainly (indefinite cross-package correlation is the
designed behaviour); A.3 records why rotation existed and why its
removal was accepted. The main-body "must not reuse the persistent
local identifier by default" escape hatch is exercised, not deleted:
that paragraph required exactly this product decision, which has now
been made. A.6's deletion note updated: install_id is now itself the
lookup key, so a future delete-on-request needs only a service-side
API, not a mapping.

Tests: the two privacy assertions invert deliberately
(test_the_stable_install_id_is_transmitted_as_is and the e2e wire
variant); freezing/byte-identical-retry coverage unchanged. Staging
E2E script now asserts transmitted == install_id.

258 targeted tests pass; ruff + footguns clean; both staging E2E
harnesses green with the raw id observed on the wire (202s).
2026-08-28 15:22:16 +10:00
Ben Barclay
151632c333 Merge branch 'main' into feat/telemetry-exporter
387 commits from main; no conflicts (verified with merge-tree before
merging). Overlap limited to hermes_cli/config_defaults.py and
hermes_cli/setup.py, both auto-merged; all shared-metrics surfaces
untouched by main.
2026-08-27 14:42:41 +10:00
Ben Barclay
67d152bc7e fix(telemetry): bound forward-clock damage to the consent horizon
Sixth review - the first against the interval architecture - verdict:
the architecture holds (idempotence, order-independence, 4-process
concurrent-writer safety, rollback immunity, format consistency, and a
120-permutation order sweep all verified), with ONE high finding, which
I had independently reproduced while the review ran: the FORWARD clock
adversary was unhandled, and unlike every other failure mode in this
subsystem it failed OPEN.

The 'obs' mark is a MAX-upsert - monotonic in the leak direction. One
glitched-forward sample (NTP flap reading 2099) while consented dragged
last_confirmed_at to 2099; a later revoke stamped closed_at = 2099; the
closed window then CONTAINED every refused period that followed. Both
the reviewer and I reproduced refused packages becoming gate-eligible.
The rollback twin was mutation-tested since round 5; nobody had asked
whether the mirror image existed.

Two clamps, each covering what the other cannot:
- The obs mark advances at most MAX_OBS_ADVANCE_SECONDS (30 days) per
  call. Honest heartbeats never bind it; a machine off for months
  catches up in a few hook fires (fail-closed latency only); one insane
  sample moves the horizon by a bounded step that real time overtakes.
- A close is MIN(last_confirmed_at, closing observation's raw stamp).
  Confirmed-time keeps unobserved gaps out of windows (v1's leak); the
  raw stamp lets an honest clock at revoke time pull a poisoned horizon
  back to the true revoke moment. A rolled-back clock at close time
  only closes earlier - fail-closed.

Also from the review:
- D2: the data-mark advance in the REAL package writer had no coverage
  (the harness re-implemented the insert; deleting the production line
  survived 314 tests). Now driven through create_and_export_package_if_due.
- D3: the "don't create ~/.hermes/telemetry for fully-disabled users"
  skip was dead code - the store constructor creates the directory
  before the exists() check ran. The probe now checks the default path
  without constructing; verified empirically on a fresh HERMES_HOME.
- Upgrade note in A.4: pre-interval backlog is never transmitted after
  upgrade (fail-closed; deliberate).

New harness scenarios: forward-poison-then-revoke (the leak), and
forward-poison-cannot-wedge (the cap). Mutation check: unclamping the
close, removing the cap, and removing the real writer's data-mark
advance each fail the suite.

273 tests pass; ruff and windows-footguns clean; staging E2E 202.
2026-08-27 13:30:25 +10:00
Ben Barclay
5e380d95ba refactor(telemetry): replace consent day-stamp with explicit intervals
Structural fix after five review rounds put four blockers in the same
subsystem. The root cause was representational: consent history is a
sequence of on/off intervals, but it was stored as ONE moving day-stamp
plus a revoked flag. Every fix had to mutate that scalar at exactly the
right moment from exactly the right place, and each round the mutation
was missing from some reachable path (write-once stamp in R3; recorded
inside a loop that never runs when sending is off in R4; dead code
whenever collection was off in R5).

Consent is now recorded as explicit intervals (send_consent_windows) and
eligibility is a pure derivation: a package is sent only when its whole
period falls inside a recorded window. One writer -
reconcile_send_consent - derives window state from an observation of
(config, now). It is idempotent and order-independent, so the wizard,
the relay, and the mid-pass check all call the same function and cannot
disagree; there are no edges to detect and no ordering between writers
to get wrong. The relay reconciles once per process BEFORE the
collection gate, which fixes round-5 D1 (enabled:false made the only
idle-path observer unreachable). The claim reads the table and never
writes it, removing the read-path mutation (D2's rewrite vector).

Timestamp discipline, each rule load-bearing and mutation-tested:
- 'obs' high-water mark: monotonic, advanced only by observations;
  confirms an open window forward (last_confirmed_at).
- 'data' high-water mark: advanced only by stored package period_end;
  clamps window OPENS so a rolled-back clock cannot slide a window
  under refused packages already on disk (round-5 D2).
- A close stamps last_confirmed_at, never "now": consent is asserted
  only for observed time, so a hand-edited config with no process
  running for 90 days fails closed (round-5 D1 strongest form).
- The gate requires period containment, not period_start >=, so an
  intra-day revoke/re-enable holds back the day package (round-5 D3).
- Unlike the day-stamp, a revoke/re-enable cycle no longer destroys the
  undelivered backlog from the earlier consented window (round-5 D4).

The redesign was validated BEFORE implementation against all 13
reproduced defect scenarios on a real store; the first two drafts each
failed scenarios in that harness (v1 leaked the unobserved-gap case by
closing at "now"; v2 leaked refused windows by letting data stamps
confirm consent). The harness ships as
tests/hermes_cli/test_shared_metrics_consent_windows.py.

Deleted: OPT_IN_PERIOD_KEY, SEND_REVOKED_KEY, LAST_SEEN_SEND_KEY,
opt_in_period(), record_revoked(), the relay edge detector body, and the
setup wizard's key bookkeeping (~170 lines of transition machinery).
Schema: two additive tables, version deliberately unchanged; verified
against a copy of the real production DB (13 rows intact, reopen no-op).

Also kills round-5's M8 survivor: the seen-exclusion mutation now fails
the suite. New mutation sweep: 8/8 killed, including one vacuous test of
my own this round (obs-mark monotonicity was covered only by
coincidence of the data mark; now pinned directly).

Documented cost: a fresh package waits at most one process start after
its period completes before release (fail-closed direction).

270 tests pass; ruff and windows-footguns clean. Staging E2E re-run
through the interval gate: both packages 202.
2026-08-27 10:42:31 +10:00
Ben Barclay
613849c190 fix(telemetry): close the consent window on the config transition
Fourth independent review. Two more consent leaks, both reproduced through
the real relay entry point before and after the fix. Both are failures of
my own round-3 fix, which recorded revocation in the wrong place.

BLOCKER 1 - revoking while idle recorded nothing. _record_revocation lived
inside send_pending's loop, but _send_exported_packages returns early when
send is false, before a sender is ever constructed. The dominant case is a
user turning sending off while no pass is running, so the loop that was
meant to observe the revocation could never run. Reproduced: 6 periods
collected during a refused window were transmitted on re-enable.

The window now closes on the observed config EDGE, before the early return.
Last-seen send state is persisted because each hook fires in a fresh
process, so a true->false transition is only visible by comparison. The
rising edge also opens the window explicitly: the sender only runs when
there is something to send, so a user who opts in and out before any
package exists would otherwise have no window for record_revoked to close.

BLOCKER 2 - turning COLLECTION off never recorded revocation. The
not-enabled branch in setup.py force-set send=false and returned without
calling _record_send_consent_change, so `hermes tools` -> disable shared
metrics silently dropped consent while leaving the window open. Same
retroactive release on re-enable. Both consent surfaces now record, and
setup keeps the relay's edge detector in step.

Also, from the same review's mutation sweep:
- the scheme check is now pinned as an allowlist. Replacing the http test
  with `if True` survived the entire suite, because every non-http case
  targeted a REMOTE host where the loopback branch rejects anyway. Only a
  non-http scheme on loopback distinguishes the two. Shipped behaviour was
  already correct; nothing guarded it.
- A.3 no longer claims rotation bounds long-term linkability outright.
  Measured against 11 real packages: resource is a stable low-entropy
  tuple and periods are contiguous across a rotation, so for a RARE
  configuration those can bridge windows. The honest claim is that
  rotation raises the cost, not that it makes correlation impossible.

Two mutants are documented as unkillable rather than papered over with
tests that only appear to cover them: the _defer clamp is unreachable from
any current caller, and widening the falling-edge check to an
unconditional else is behaviourally equivalent because record_revoked is
idempotent and no-ops without an open window.

An earlier version of the anti-spurious-revocation test could not fail
either - it used a never-consented store, where record_revoked no-ops
regardless. Rewritten to opt in, revoke, re-enable, and then assert that a
steady enabled state does not re-close the reopened window.

259 tests pass. Staging E2E re-run: both packages 202.
2026-08-27 09:47:47 +10:00