From e0170c253608afade5e8abbafbf951504fce9804 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:32:26 +1000 Subject: [PATCH 01/20] docs(observability): record remote-exporter privacy decisions The shared-metrics doc states that a future remote exporter 'must not reuse the persistent local identifier by default' and 'requires a separate product and privacy decision covering consent, identity scope, rotation or keyed pseudonymization, reset behavior, retention, and deletion'. That exporter is now being built. Appendix A answers each of those six items before any code lands, so the reasoning is reviewable on its own and survives the implementation: - consent is a separate opt-in from collection, gated on the PERIOD a package covers rather than when it was created (a period is split across packages made on different days, so a created_at gate would send a period's tail while dropping its head and silently undercount the first day) - the transmitted identifier is HMAC-SHA256(local-only salt, install_id), never install_id itself - the salt rotates every 30 days - reset gives a new remote identity but cannot unsend - local retention is unchanged; send state does not extend it - there is no self-service remote deletion, and the user -> derived-id lookup that would enable one is deliberately not built A.7 additionally records what the outbox directory IS (the user's local history, not a send queue) because misreading it would have led to deleting user data on acknowledgement. --- docs/observability/relay-shared-metrics.md | 156 +++++++++++++++++++++ 1 file changed, 156 insertions(+) diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index 146590dc99..5b5ce0f8d4 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -231,6 +231,13 @@ the persistent local identifier by default. It requires a separate product and privacy decision covering consent, identity scope, rotation or keyed pseudonymization, reset behavior, retention, and deletion. +> That exporter is now being built as Phase 2 of the Hermes telemetry project. +> The decisions this paragraph asks for are recorded in +> [Appendix A](#appendix-a-remote-exporter-decisions-phase-2). Until Phase 2 +> ships, the statement above still describes shipped behaviour: nothing is +> transmitted, and transmission stays opt-in behind a config key that is off by +> default. + The install identity is scoped to one `HERMES_HOME`. To reset it, stop Hermes processes and remove `$HERMES_HOME/telemetry/shared_metrics`. This deliberately removes the old identity, aggregate database, and queued local packages @@ -257,3 +264,152 @@ verifies model, provider, task, tool, and skill counters in SQLite, validates all exported delta packages against the closed schema, verifies the pseudonymous client-active counter, and checks that prompt, response, tool-call ID, tool-result, and skill-name canaries are absent from the packages. + +## Appendix A: Remote Exporter Decisions (Phase 2) + +Status: **decided, not yet built.** This appendix answers the product and +privacy questions that "Current Slices" defers to a future remote exporter. It +records what was decided and why, so the reasoning survives the implementation. + +The exporter sends the package files already written under +`$HERMES_HOME/telemetry/shared_metrics/outbox/` to the Hermes telemetry ingest +service. That service validates only the envelope (`schema_version` plus a UUID +`package_id`) and stores the body verbatim in S3. + +### A.1 Consent + +Transmission is a **separate opt-in** from collection, under a new config key: + +```yaml +telemetry: + shared_metrics: + enabled: false # collect locally + send: false # NEW: transmit to the Nous telemetry service +``` + +- `send` defaults to **false**. Collection alone never transmits. +- `send` requires `enabled`. It does **not** imply it: a transmission flag must + not silently switch on collection. `send: true` with `enabled: false` warns + and does nothing. +- Like `enabled`, `send` is profile-owned and is not overridden by + managed-scope configuration. + +**Only packages for periods on or after the opt-in day are ever sent.** The +opt-in day (UTC) is recorded when `send` first becomes true, and any package +whose `period_start` predates it is permanently excluded, however late it was +created. + +The gate is on the **period**, not on the package's creation time. One period +is split across several packages created on different days: a day's first +package is written that day, and a tail package for the same period typically +follows the next day. Gating on creation time would send a period's tail while +dropping its head, reporting a **silently undercounted** day. Gating on the +period keeps consent forward-only and every transmitted period complete. + +Local history can be up to 30 days old, and that data was collected under a +promise that nothing is uploaded. Honouring consent forward-only costs at most +30 days of backlog we never had permission to send. + +### A.2 Identity scope — the transmitted identifier is derived, not the local one + +`install_id` is the persistent profile-scoped identifier described above. It is +**not transmitted**. Each package sent carries a derived value instead: + +```text +transmitted_id = HMAC-SHA256(key = rotation_salt, message = install_id) +``` + +- `rotation_salt` is random, generated locally, and never leaves the machine. +- The derivation is one-way: the service cannot recover `install_id`. +- Within a rotation window, packages from one profile correlate — so distinct + installs remain countable, which is the primary analytical question. +- Across windows, they do not. + +This satisfies "must not reuse the persistent local identifier by default" +while keeping the data useful. Stripping the identifier entirely was rejected +because "how many installs are reporting" is the first question the data must +answer; sending `install_id` unchanged was rejected because it contradicts the +commitment made above. + +**Byte-identical resends still hold.** The derived value is computed **once**, +when the package is first prepared for sending, and stored alongside the +package (the derived id only — not a second copy of the payload, which is +recomputed deterministically from the stored package). A retry therefore +rebuilds identical bytes even if the salt rotated in between. The contract +requires this: resending a `package_id` with different content is undefined +behaviour. + +### A.3 Rotation + +`rotation_salt` rotates on a fixed schedule (default: every 30 days, aligned to +local history retention). Rotation only affects packages prepared after it; +already-prepared packages keep their derived value so retries stay +byte-identical. + +Rotation bounds long-term linkability without destroying short-term cohort +analysis. A profile is one identity for the length of a window, and an +unrelated identity after it. + +### A.4 Reset behavior + +Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity, +aggregates, and package files, exactly as documented above. Two honest +qualifications now apply: + +- Reset also discards `rotation_salt`, so subsequent packages derive a **new** + transmitted identity. Local reset does give a new remote identity. +- Reset **cannot unsend**. Packages already transmitted remain in the ingest + service's storage under their derived identifier. There is no read-back or + delete API in the v1 contract. + +Setting `send: false` stops transmission immediately. It does not delete +previously transmitted packages, and it does not stop local collection. + +### A.5 Retention + +- **Local:** unchanged — 30 days for successfully exported history, and pending + deltas are kept until exported. Send state does **not** extend local + retention: a package that could never be sent is still pruned at 30 days. + Unbounded local growth against a permanently unreachable endpoint is a worse + failure than losing metrics from an install that has been broken for a month. +- **Remote:** raw packages are retained in S3 without expiry in production and + for 30 days in staging. + +### A.6 Deletion + +There is no remote deletion path in the v1 contract, and this appendix does not +invent one. What a user can do: + +| Action | Effect | +|---|---| +| `send: false` | No further packages leave the machine | +| `enabled: false` | Collection stops; existing local state remains | +| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new derived identity | +| Delete already-sent data | Not self-service — requires an operator acting on the S3 bucket | + +If a deletion-on-request obligation is ever taken on, it needs a lookup path +from a user to their derived identifiers. That is deliberately **not** built: +it would require retaining the mapping this design exists to avoid. Any such +change is a new product decision, not an implementation detail. + +### A.7 What the outbox directory is + +Recorded because it was misread once during Phase 2 planning, in a way that +would have deleted user data. + +The directory is **local history, not a send-queue**. `package_outbox` is the +SQLite table; its `exported_at` column means "written to disk", not "sent". +Files are immutable and pruned **by age alone**. + +The ingest contract says senders should delete a package from their outbox on +`202`. **The exporter does not do this.** Deleting on acknowledgement would +repurpose the user's 30-day local history as a transmission queue and destroy +state they were promised. Send state lives in new columns on the +`package_outbox` table instead; the files are untouched by transmission. + +### A.8 Scope note + +The `install_id` field inside the package body is what gets replaced by the +derived value. No other payload field changes, nothing is added, and the +service treats the whole body as opaque. Payload schema evolution therefore +stays a sender-side concern, as before. From e5180ab3df71547b971e884bba1504f665ba80fb Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:39:47 +1000 Subject: [PATCH 02/20] feat(telemetry): add opt-in send config and send-state columns MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 1+2 of the shared-metrics exporter. Config: telemetry.shared_metrics.send (default false) and .endpoint (default production), resolved by a new shared_metrics_send_config module. Precedence is HERMES_TELEMETRY_ENDPOINT > config > default; the env var exists so the live staging E2E never has to mutate a user's config. send requires enabled and never implies it — that combination is a misconfiguration the user believes is working, so it logs an ERROR once per process rather than silently doing nothing. Plaintext endpoints are refused unless the host is loopback, so a typo cannot send telemetry in clear text. Per AGENTS.md, outbound telemetry needs a user-facing opt-in, so setup_telemetry now prompts for sending as a second, separate question and force-disables send when collection is turned off. Storage: six additive nullable columns on package_outbox for send bookkeeping. The store schema version deliberately does NOT move — _ensure_schema_in_transaction raises on any version it does not recognise and has no forward-compatibility branch, so bumping it would hard-fail an older Hermes, a second profile on an older build, or a rollback, against the same file. Old readers select named columns and never SELECT *, so the additions are invisible to them. Also corrects the two places that promised telemetry is never uploaded (config_defaults comment and cli-config.yaml.example); leaving them would make them false privacy statements once sending ships. Tests: 26 covering config precedence, the enabled/send relationship, transport safety, fresh-database creation, upgrade from a pre-send database (rows preserved, version pinned, idempotent), and that the shipped export query still runs. Mutation-checked: bumping the schema version fails 5 of them. --- cli-config.yaml.example | 15 +- hermes_cli/config_defaults.py | 20 +- hermes_cli/observability/shared_metrics.py | 37 +++ .../shared_metrics_send_config.py | 114 +++++++++ hermes_cli/setup.py | 30 ++- .../test_shared_metrics_send_config.py | 137 +++++++++++ .../test_shared_metrics_send_migration.py | 224 ++++++++++++++++++ 7 files changed, 569 insertions(+), 8 deletions(-) create mode 100644 hermes_cli/observability/shared_metrics_send_config.py create mode 100644 tests/hermes_cli/test_shared_metrics_send_config.py create mode 100644 tests/hermes_cli/test_shared_metrics_send_migration.py diff --git a/cli-config.yaml.example b/cli-config.yaml.example index 1a8021ff98..2fe2c5e610 100644 --- a/cli-config.yaml.example +++ b/cli-config.yaml.example @@ -1782,15 +1782,28 @@ display: # ============================================================================= # Shared metrics are disabled by default. When enabled, Hermes writes only # allowlisted aggregate counters and immutable JSON -# packages under $HERMES_HOME/telemetry/shared_metrics; it does not upload them. +# packages under $HERMES_HOME/telemetry/shared_metrics. # Packages include a random profile-scoped ID that stays stable until this # directory is deleted. It is not derived from hardware, account, or host data. # Successfully exported local history is retained for 30 days; pending deltas # are retained until they can be exported. # This profile-owned choice is not overridden by managed-scope configuration. +# +# Nothing is uploaded unless you also set `send: true`. That is a separate +# opt-in and requires `enabled`; it never turns collection on by itself. +# When sending is on: +# * only packages whose period starts on or after the day you opted in are +# ever transmitted, so data collected beforehand stays on this machine; +# * the profile-scoped ID is NOT sent. Each package carries an HMAC of it, +# keyed by a local-only salt that rotates every 30 days, so installs stay +# countable without shipping a durable identifier. +# See docs/observability/relay-shared-metrics.md (Appendix A) for the full +# consent, identity, rotation, retention, and deletion decisions. telemetry: shared_metrics: enabled: false + send: false + # endpoint: https://telemetry.nousresearch.com/v1/telemetry # ============================================================================= diff --git a/hermes_cli/config_defaults.py b/hermes_cli/config_defaults.py index 0fb1488316..cf321b4e3d 100644 --- a/hermes_cli/config_defaults.py +++ b/hermes_cli/config_defaults.py @@ -3323,11 +3323,27 @@ DEFAULT_CONFIG = { "profile_build": "ask", }, - # Privacy-safe aggregate metrics written only to this profile's local - # telemetry directory. Collection is opt-in and no remote sink exists. + # Privacy-safe aggregate metrics written to this profile's local telemetry + # directory. Collection is opt-in (``enabled``). Transmission to the Nous + # telemetry service is a SEPARATE opt-in (``send``) and is off by default; + # see docs/observability/relay-shared-metrics.md, Appendix A, for the + # consent, identity, rotation, retention, and deletion decisions. "telemetry": { "shared_metrics": { "enabled": False, + # Transmit exported packages to the Nous telemetry service. + # Requires ``enabled``: it never switches collection on by itself, + # and ``send`` without ``enabled`` is logged as an error rather + # than silently doing nothing. Only packages whose period starts + # on or after the opt-in day are ever sent, so data collected + # before consent stays local. + "send": False, + # Ingest endpoint. Production by default; override for staging or + # a local test server. The HERMES_TELEMETRY_ENDPOINT environment + # variable takes precedence (used by the live E2E so a test never + # has to mutate a user's config). Non-HTTPS is refused unless the + # host is localhost. + "endpoint": "https://telemetry.nousresearch.com/v1/telemetry", }, }, diff --git a/hermes_cli/observability/shared_metrics.py b/hermes_cli/observability/shared_metrics.py index fd42b06230..bf5c1fb0bf 100644 --- a/hermes_cli/observability/shared_metrics.py +++ b/hermes_cli/observability/shared_metrics.py @@ -337,6 +337,7 @@ class SharedMetricsStore: ) """ ) + SharedMetricsStore._add_send_columns(connection) connection.execute( """ INSERT INTO telemetry_state(key, value) @@ -346,6 +347,42 @@ class SharedMetricsStore: (_STORE_SCHEMA_VERSION,), ) + @staticmethod + def _add_send_columns(connection: sqlite3.Connection) -> None: + """Add transmission bookkeeping to ``package_outbox``, idempotently. + + These columns are ADDITIVE and nullable, and the store schema version + is deliberately NOT bumped. ``_ensure_schema_in_transaction`` raises on + any version it does not recognise and has no forward-compatibility + branch, so bumping would make an older Hermes — a second profile on an + older build, or a rollback — hard-fail against the same database file. + Old readers select named columns and never ``SELECT *``, so extra + columns are invisible to them. + """ + existing = { + str(row["name"]) + for row in connection.execute("PRAGMA table_info(package_outbox)") + } + for column, declaration in ( + # When the 202 was received. NULL = never acknowledged. + ("sent_at", "TEXT"), + # NULL/'pending' = eligible, 'sent' = done, 'rejected' = permanent 400. + ("send_state", "TEXT"), + ("send_attempts", "INTEGER NOT NULL DEFAULT 0"), + # Earliest next attempt; enforces backoff across process restarts. + ("next_attempt_at", "TEXT"), + ("last_error", "TEXT"), + # The derived identifier actually transmitted, frozen on the first + # attempt so retries stay byte-identical across a salt rotation. + # Only the ~36-byte id is stored: the body is recomputed from + # payload_json, whose serialisation is deterministic. + ("sent_install_id", "TEXT"), + ): + if column not in existing: + connection.execute( + f"ALTER TABLE package_outbox ADD COLUMN {column} {declaration}" + ) + @staticmethod def _create_counter_aggregates_table(connection: sqlite3.Connection) -> None: connection.execute( diff --git a/hermes_cli/observability/shared_metrics_send_config.py b/hermes_cli/observability/shared_metrics_send_config.py new file mode 100644 index 0000000000..8011c595ab --- /dev/null +++ b/hermes_cli/observability/shared_metrics_send_config.py @@ -0,0 +1,114 @@ +"""Configuration for shared-metrics transmission. + +Collection (``telemetry.shared_metrics.enabled``) and transmission +(``telemetry.shared_metrics.send``) are separate opt-ins. See +``docs/observability/relay-shared-metrics.md`` Appendix A for the consent, +identity, rotation, retention, and deletion decisions behind this module. +""" + +from __future__ import annotations + +import logging +import os +from dataclasses import dataclass +from urllib.parse import urlparse + +logger = logging.getLogger(__name__) + +#: Production ingest endpoint. Overridable by config or environment so the +#: live E2E can target staging without mutating a user's config. +DEFAULT_ENDPOINT = "https://telemetry.nousresearch.com/v1/telemetry" + +#: Environment override, highest precedence. Intended for tests and staging +#: validation, not as the documented user-facing setting (which is config). +ENDPOINT_ENV_VAR = "HERMES_TELEMETRY_ENDPOINT" + +_LOCAL_HOSTS = frozenset({"localhost", "127.0.0.1", "::1", "[::1]"}) + +# Module-level latch: the enabled/send mismatch is a static misconfiguration, +# so it is reported once per process instead of on every hook fire. +_warned_send_without_collection = False + + +@dataclass(frozen=True) +class SendConfig: + """Resolved transmission settings.""" + + #: Collection is on. Nothing is packaged or sent without it. + enabled: bool + #: Transmission is on AND permitted (that is, collection is also on). + send: bool + #: Where packages are POSTed. + endpoint: str + + +def _endpoint_is_safe(endpoint: str) -> bool: + """Reject plaintext destinations unless they are loopback. + + Telemetry must not leave a machine in clear text because of a typo in a + config file. Loopback stays allowed so tests can use a local HTTP server. + """ + try: + parsed = urlparse(endpoint) + except ValueError: + return False + if parsed.scheme == "https": + return True + if parsed.scheme == "http": + return (parsed.hostname or "") in _LOCAL_HOSTS + return False + + +def resolve_send_config(config: dict | None) -> SendConfig: + """Resolve transmission settings from config plus the environment. + + Endpoint precedence: ``HERMES_TELEMETRY_ENDPOINT`` > config > production + default. + + ``send`` is returned as False whenever transmission cannot legitimately + happen, so callers never have to re-check the combination. + """ + global _warned_send_without_collection + + raw = config if isinstance(config, dict) else {} + telemetry = raw.get("telemetry") + telemetry = telemetry if isinstance(telemetry, dict) else {} + shared = telemetry.get("shared_metrics") + shared = shared if isinstance(shared, dict) else {} + + enabled = shared.get("enabled") is True + send_requested = shared.get("send") is True + + if send_requested and not enabled: + # Loud, not silent: the user believes telemetry is being sent, and it + # never will be. Error level, once per process. + if not _warned_send_without_collection: + _warned_send_without_collection = True + logger.error( + "telemetry.shared_metrics.send is true but " + "telemetry.shared_metrics.enabled is false — nothing is " + "collected, so nothing can be sent. Enable collection or " + "turn sending off." + ) + return SendConfig(enabled=False, send=False, endpoint=DEFAULT_ENDPOINT) + + endpoint = os.environ.get(ENDPOINT_ENV_VAR) or shared.get("endpoint") + if not isinstance(endpoint, str) or not endpoint.strip(): + endpoint = DEFAULT_ENDPOINT + endpoint = endpoint.strip() + + if send_requested and not _endpoint_is_safe(endpoint): + logger.error( + "Refusing to send shared metrics to %r: telemetry must use https " + "(or a localhost http endpoint for testing).", + endpoint, + ) + return SendConfig(enabled=enabled, send=False, endpoint=endpoint) + + return SendConfig(enabled=enabled, send=send_requested, endpoint=endpoint) + + +def reset_warning_latch_for_tests() -> None: + """Clear the once-per-process error latch (test support only).""" + global _warned_send_without_collection + _warned_send_without_collection = False diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 4f0d190203..d6497fbc05 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2428,10 +2428,10 @@ def setup_tools(config: dict, first_install: bool = False): def setup_telemetry(config: dict): - """Configure the local, privacy-safe shared-metrics subscriber.""" + """Configure the local shared-metrics subscriber and optional sending.""" print_header("Shared Metrics") print_info("Shared metrics contain only bounded counters and histograms.") - print_info("Packages stay under this Hermes profile and are not uploaded.") + print_info("Collection is local. Sending them to Nous is a separate opt-in.") telemetry = config.get("telemetry") if not isinstance(telemetry, dict): @@ -2447,10 +2447,30 @@ def setup_telemetry(config: dict): "Enable local shared metrics?", default=current, ) - if shared_metrics["enabled"]: - print_success("Local shared metrics enabled.") - else: + if not shared_metrics["enabled"]: print_info("Local shared metrics disabled.") + # Sending cannot outlive collection: leaving send=true here would be a + # configuration that logs an error on every run and never transmits. + if shared_metrics.get("send") is True: + shared_metrics["send"] = False + print_info("Sending shared metrics disabled as well.") + return + + print_success("Local shared metrics enabled.") + print_info("") + print_info("Sending uploads each daily package to the Nous telemetry") + print_info("service. Your profile-scoped install ID is NOT sent: packages") + print_info("carry a rotating HMAC of it instead. Only packages from the") + print_info("day you opt in onwards are ever sent, and sending can be") + print_info("turned off again at any time.") + shared_metrics["send"] = prompt_yes_no( + "Send shared metrics to Nous?", + default=shared_metrics.get("send") is True, + ) + if shared_metrics["send"]: + print_success("Sending shared metrics enabled.") + else: + print_info("Sending shared metrics disabled (collection stays local).") # ============================================================================= diff --git a/tests/hermes_cli/test_shared_metrics_send_config.py b/tests/hermes_cli/test_shared_metrics_send_config.py new file mode 100644 index 0000000000..235777d49a --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_send_config.py @@ -0,0 +1,137 @@ +"""Tests for shared-metrics send configuration resolution.""" + +from __future__ import annotations + +import logging + +import pytest + +from hermes_cli.config import DEFAULT_CONFIG +from hermes_cli.observability.shared_metrics_send_config import ( + DEFAULT_ENDPOINT, + ENDPOINT_ENV_VAR, + resolve_send_config, + reset_warning_latch_for_tests, +) + + +@pytest.fixture(autouse=True) +def _reset_latch(): + reset_warning_latch_for_tests() + yield + reset_warning_latch_for_tests() + + +def _config(**shared): + return {"telemetry": {"shared_metrics": shared}} + + +class TestDefaults: + def test_send_is_registered_disabled_by_default(self): + shared = DEFAULT_CONFIG["telemetry"]["shared_metrics"] + assert shared["enabled"] is False + assert shared["send"] is False + + def test_default_endpoint_is_production(self): + shared = DEFAULT_CONFIG["telemetry"]["shared_metrics"] + assert shared["endpoint"] == DEFAULT_ENDPOINT + assert DEFAULT_ENDPOINT.startswith("https://") + + def test_empty_config_sends_nothing(self): + resolved = resolve_send_config({}) + assert resolved.enabled is False + assert resolved.send is False + + def test_none_config_is_tolerated(self): + assert resolve_send_config(None).send is False + + +class TestSendRequiresCollection: + def test_collection_alone_does_not_send(self): + resolved = resolve_send_config(_config(enabled=True)) + assert resolved.enabled is True + assert resolved.send is False + + def test_send_with_collection_sends(self): + resolved = resolve_send_config(_config(enabled=True, send=True)) + assert resolved.send is True + + def test_send_without_collection_is_refused(self): + resolved = resolve_send_config(_config(enabled=False, send=True)) + assert resolved.send is False + # send must never imply enabled + assert resolved.enabled is False + + def test_send_without_collection_logs_an_error(self, caplog): + with caplog.at_level(logging.ERROR): + resolve_send_config(_config(enabled=False, send=True)) + errors = [r for r in caplog.records if r.levelno >= logging.ERROR] + assert len(errors) == 1 + assert "enabled is false" in errors[0].getMessage() + + def test_the_error_is_logged_once_per_process(self, caplog): + with caplog.at_level(logging.ERROR): + for _ in range(5): + resolve_send_config(_config(enabled=False, send=True)) + errors = [r for r in caplog.records if r.levelno >= logging.ERROR] + assert len(errors) == 1, "misconfiguration must not spam every hook fire" + + +class TestEndpointPrecedence: + def test_config_endpoint_overrides_default(self): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint="https://example.test/v1") + ) + assert resolved.endpoint == "https://example.test/v1" + + def test_env_var_overrides_config(self, monkeypatch): + monkeypatch.setenv(ENDPOINT_ENV_VAR, "https://staging.test/v1") + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint="https://example.test/v1") + ) + assert resolved.endpoint == "https://staging.test/v1" + + def test_blank_endpoint_falls_back_to_production(self): + resolved = resolve_send_config(_config(enabled=True, send=True, endpoint=" ")) + assert resolved.endpoint == DEFAULT_ENDPOINT + + def test_endpoint_is_stripped(self, monkeypatch): + monkeypatch.setenv(ENDPOINT_ENV_VAR, " https://staging.test/v1 ") + assert resolve_send_config(_config(enabled=True, send=True)).endpoint == ( + "https://staging.test/v1" + ) + + +class TestTransportSafety: + def test_plaintext_endpoint_is_refused(self, caplog): + with caplog.at_level(logging.ERROR): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint="http://example.test/v1") + ) + assert resolved.send is False, "telemetry must not go out in clear text" + assert any("https" in r.getMessage() for r in caplog.records) + + @pytest.mark.parametrize( + "endpoint", + [ + "http://localhost:8099/v1/telemetry", + "http://127.0.0.1:8099/v1/telemetry", + ], + ) + def test_loopback_http_is_allowed_for_testing(self, endpoint): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint=endpoint) + ) + assert resolved.send is True + + def test_nonsense_scheme_is_refused(self): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint="ftp://example.test/v1") + ) + assert resolved.send is False + + def test_unsafe_endpoint_does_not_block_collection(self): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint="http://example.test/v1") + ) + assert resolved.enabled is True diff --git a/tests/hermes_cli/test_shared_metrics_send_migration.py b/tests/hermes_cli/test_shared_metrics_send_migration.py new file mode 100644 index 0000000000..54518c644a --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_send_migration.py @@ -0,0 +1,224 @@ +"""Tests for the additive send-state migration on ``package_outbox``. + +The store schema version must NOT move when these columns are added: the +existing loader raises on any version it does not recognise, so bumping it +would hard-fail an older Hermes (a second profile on an older build, or a +rollback) against the same database file. +""" + +from __future__ import annotations + +import json +import sqlite3 + +import pytest + +from hermes_cli.observability.shared_metrics import SharedMetricsStore + +SEND_COLUMNS = { + "sent_at", + "send_state", + "send_attempts", + "next_attempt_at", + "last_error", + "sent_install_id", +} + + +def _columns(db_path): + connection = sqlite3.connect(db_path) + try: + return {row[1] for row in connection.execute("PRAGMA table_info(package_outbox)")} + finally: + connection.close() + + +def _schema_version(db_path): + connection = sqlite3.connect(db_path) + try: + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = 'schema_version'" + ).fetchone() + return row[0] if row else None + finally: + connection.close() + + +@pytest.fixture +def store(tmp_path): + return SharedMetricsStore( + database_path=tmp_path / "metrics.sqlite3", + outbox_directory=tmp_path / "outbox", + ) + + +class TestFreshDatabase: + def test_send_columns_exist(self, store): + assert SEND_COLUMNS <= _columns(store.database_path) + + def test_original_columns_survive(self, store): + assert { + "package_id", + "period_start", + "period_end", + "payload_json", + "created_at", + "exported_at", + } <= _columns(store.database_path) + + def test_send_attempts_defaults_to_zero(self, store): + connection = sqlite3.connect(store.database_path) + try: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, created_at + ) VALUES ('p', '2026-01-01', '2026-01-02', '{}', '2026-01-01T00:00:00Z') + """ + ) + connection.commit() + row = connection.execute( + "SELECT send_attempts, send_state, sent_install_id FROM package_outbox" + ).fetchone() + finally: + connection.close() + assert row[0] == 0 + assert row[1] is None + assert row[2] is None + + +class TestUpgradeFromPreSendDatabase: + """The real-world case: a database written before this feature existed.""" + + @pytest.fixture + def legacy_db(self, tmp_path): + path = tmp_path / "metrics.sqlite3" + connection = sqlite3.connect(path) + try: + connection.execute( + """ + CREATE TABLE telemetry_state ( + key TEXT PRIMARY KEY, + value TEXT NOT NULL + ) + """ + ) + connection.execute( + "INSERT INTO telemetry_state(key, value) VALUES ('schema_version', '2')" + ) + connection.execute( + """ + CREATE TABLE package_outbox ( + package_id TEXT PRIMARY KEY, + period_start TEXT NOT NULL, + period_end TEXT NOT NULL, + payload_json TEXT NOT NULL, + created_at TEXT NOT NULL, + exported_at TEXT + ) + """ + ) + connection.execute( + """ + CREATE TABLE counter_aggregates ( + period_start TEXT NOT NULL, + metric_name TEXT NOT NULL, + hermes_version TEXT NOT NULL, + os_family TEXT NOT NULL, + architecture TEXT NOT NULL, + install_method TEXT NOT NULL, + dimensions_json TEXT NOT NULL, + value INTEGER NOT NULL, + packaged_value INTEGER NOT NULL, + PRIMARY KEY ( + period_start, metric_name, hermes_version, os_family, + architecture, install_method, dimensions_json + ) + ) + """ + ) + for i in range(3): + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + ( + f"pkg-{i}", + "2026-08-2%d" % i, + "2026-08-2%d" % (i + 1), + json.dumps({"package_id": f"pkg-{i}"}), + "2026-08-2%dT00:00:00Z" % i, + "2026-08-2%dT01:00:00Z" % i, + ), + ) + connection.commit() + finally: + connection.close() + return path + + def test_upgrade_preserves_every_row(self, legacy_db, tmp_path): + SharedMetricsStore( + database_path=legacy_db, outbox_directory=tmp_path / "outbox" + ) + connection = sqlite3.connect(legacy_db) + try: + count = connection.execute("SELECT COUNT(*) FROM package_outbox").fetchone()[0] + payloads = connection.execute( + "SELECT package_id, payload_json FROM package_outbox ORDER BY package_id" + ).fetchall() + finally: + connection.close() + assert count == 3 + assert payloads == [ + ("pkg-0", '{"package_id": "pkg-0"}'), + ("pkg-1", '{"package_id": "pkg-1"}'), + ("pkg-2", '{"package_id": "pkg-2"}'), + ] + + def test_upgrade_adds_the_send_columns(self, legacy_db, tmp_path): + SharedMetricsStore( + database_path=legacy_db, outbox_directory=tmp_path / "outbox" + ) + assert SEND_COLUMNS <= _columns(legacy_db) + + def test_upgrade_does_not_move_the_schema_version(self, legacy_db, tmp_path): + """Bumping would make older builds refuse the same file.""" + SharedMetricsStore( + database_path=legacy_db, outbox_directory=tmp_path / "outbox" + ) + assert _schema_version(legacy_db) == "2" + + def test_migration_is_idempotent(self, legacy_db, tmp_path): + for _ in range(3): + SharedMetricsStore( + database_path=legacy_db, outbox_directory=tmp_path / "outbox" + ) + columns = [ + row[1] + for row in sqlite3.connect(legacy_db).execute( + "PRAGMA table_info(package_outbox)" + ) + ] + assert len(columns) == len(set(columns)), "columns were added more than once" + + def test_queries_written_before_this_change_still_work(self, legacy_db, tmp_path): + """The shipped export query selects named columns; it must be unaffected.""" + SharedMetricsStore( + database_path=legacy_db, outbox_directory=tmp_path / "outbox" + ) + connection = sqlite3.connect(legacy_db) + try: + rows = connection.execute( + """ + SELECT package_id, payload_json + FROM package_outbox + WHERE exported_at IS NULL + ORDER BY created_at, package_id + """ + ).fetchall() + finally: + connection.close() + assert rows == [] From 7ffd454df65f62fcd56c0d0ae609ce927390d776 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:41:18 +1000 Subject: [PATCH 03/20] feat(telemetry): derive the transmitted install identity via keyed HMAC MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 3 of the shared-metrics exporter. The shared-metrics doc commits that a remote exporter 'must not reuse the persistent local identifier by default'. install_id is therefore never transmitted: each package carries HMAC-SHA256(local-only rotation salt, install_id) instead. Within a 30-day rotation window the value is stable, so distinct installs remain countable — the first question the data has to answer. Across windows it changes, bounding long-term linkability. The derivation is one-way, so the service cannot recover install_id. The salt lives in telemetry_state next to install_id, so removing the shared-metrics directory resets both together and the documented reset behaviour keeps working with no second cleanup path. Rotation is deliberately not a bare 'age > interval' check: a clock that jumps backwards must not read as an expired salt, and an unparseable issued-at reissues instead of raising. substitute_install_id replaces exactly one field and copies rather than mutating, so payload schema evolution stays a sender-side concern. Tests: 19, including that install_id never survives substitution, that no other field changes, and — the property that keeps retries contract-compliant — that a package rebuilt from a FROZEN derived id is byte-stable across a salt rotation while a fresh derivation is not. --- .../observability/shared_metrics_identity.py | 127 +++++++++++++ .../test_shared_metrics_identity.py | 175 ++++++++++++++++++ 2 files changed, 302 insertions(+) create mode 100644 hermes_cli/observability/shared_metrics_identity.py create mode 100644 tests/hermes_cli/test_shared_metrics_identity.py diff --git a/hermes_cli/observability/shared_metrics_identity.py b/hermes_cli/observability/shared_metrics_identity.py new file mode 100644 index 0000000000..16e28a8b4e --- /dev/null +++ b/hermes_cli/observability/shared_metrics_identity.py @@ -0,0 +1,127 @@ +"""Keyed pseudonymization of the shared-metrics install identity. + +``install_id`` is a persistent, profile-scoped identifier. It is deliberately +NOT transmitted: ``docs/observability/relay-shared-metrics.md`` commits that a +remote exporter "must not reuse the persistent local identifier by default". + +Each transmitted package instead carries:: + + HMAC-SHA256(key=rotation_salt, message=install_id) + +where ``rotation_salt`` is generated locally, never leaves the machine, and +rotates on a fixed schedule. Within a rotation window the value is stable, so +distinct installs stay countable — the primary analytical question. Across +windows it changes, bounding long-term linkability. + +The derivation is one-way: the service cannot recover ``install_id`` from what +it receives. + +See Appendix A.2 and A.3 of the doc above for the decision record. +""" + +from __future__ import annotations + +import hashlib +import hmac +import secrets +import sqlite3 +from datetime import datetime, timedelta, timezone + +#: Salt lifetime. Matches local history retention so the two ages line up. +ROTATION_INTERVAL = timedelta(days=30) + +#: ``telemetry_state`` keys. The salt lives in the same store as install_id, so +#: deleting the shared-metrics directory resets both together — the documented +#: reset behaviour keeps working without a second cleanup path. +SALT_KEY = "send_rotation_salt" +SALT_ISSUED_AT_KEY = "send_rotation_salt_issued_at" + +_SALT_BYTES = 32 + + +def _isoformat(value: datetime) -> str: + return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") + + +def _parse(value: str | None) -> datetime | None: + if not value: + return None + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed.astimezone(timezone.utc) + + +def _read(connection: sqlite3.Connection, key: str) -> str | None: + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (key,) + ).fetchone() + if row is None: + return None + # sqlite3.Row and plain tuples both index by position. + return str(row[0]) + + +def _write(connection: sqlite3.Connection, key: str, value: str) -> None: + connection.execute( + """ + INSERT INTO telemetry_state(key, value) VALUES (?, ?) + ON CONFLICT(key) DO UPDATE SET value = excluded.value + """, + (key, value), + ) + + +def current_salt( + connection: sqlite3.Connection, + *, + now: datetime | None = None, +) -> str: + """Return the active salt, generating or rotating it when due. + + Must be called inside a write transaction: it can write to + ``telemetry_state``. + """ + moment = now or datetime.now(timezone.utc) + salt = _read(connection, SALT_KEY) + issued_at = _parse(_read(connection, SALT_ISSUED_AT_KEY)) + + fresh = ( + salt is not None + and issued_at is not None + # A clock that jumped backwards must not be read as "aged out"; a + # future issue time simply means not yet due. + and issued_at <= moment < issued_at + ROTATION_INTERVAL + ) + if fresh: + return str(salt) + + salt = secrets.token_hex(_SALT_BYTES) + _write(connection, SALT_KEY, salt) + _write(connection, SALT_ISSUED_AT_KEY, _isoformat(moment)) + return salt + + +def derive_install_id(install_id: str, salt: str) -> str: + """Return the transmitted identifier for ``install_id`` under ``salt``.""" + return hmac.new( + salt.encode("utf-8"), + install_id.encode("utf-8"), + hashlib.sha256, + ).hexdigest() + + +def substitute_install_id(payload: dict, derived: str) -> dict: + """Return ``payload`` with its ``install_id`` replaced by ``derived``. + + This is the ONLY field the exporter changes. Everything else is + transmitted exactly as the generator wrote it, so payload schema evolution + stays a sender-side concern. A shallow copy is enough — only a top-level + key is replaced — and the caller's dict is left untouched. + """ + updated = dict(payload) + updated["install_id"] = derived + return updated diff --git a/tests/hermes_cli/test_shared_metrics_identity.py b/tests/hermes_cli/test_shared_metrics_identity.py new file mode 100644 index 0000000000..1887d47ccb --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_identity.py @@ -0,0 +1,175 @@ +"""Tests for keyed pseudonymization of the shared-metrics install identity. + +The load-bearing property: install_id must never be transmitted, and the +value that IS transmitted must stay stable for a package even across a salt +rotation, or a retry would change the body under an already-used package_id. +""" + +from __future__ import annotations + +import sqlite3 +from datetime import datetime, timedelta, timezone + +import pytest + +from hermes_cli.observability.shared_metrics_identity import ( + ROTATION_INTERVAL, + SALT_ISSUED_AT_KEY, + SALT_KEY, + current_salt, + derive_install_id, + substitute_install_id, +) + +INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" +T0 = datetime(2026, 8, 26, 12, 0, tzinfo=timezone.utc) + + +@pytest.fixture +def connection(): + conn = sqlite3.connect(":memory:") + conn.execute( + "CREATE TABLE telemetry_state (key TEXT PRIMARY KEY, value TEXT NOT NULL)" + ) + yield conn + conn.close() + + +class TestSaltLifecycle: + def test_first_call_generates_a_salt(self, connection): + salt = current_salt(connection, now=T0) + assert len(salt) == 64 # 32 bytes hex + assert int(salt, 16) >= 0 # valid hex + + def test_salt_is_stable_within_the_window(self, connection): + first = current_salt(connection, now=T0) + later = current_salt(connection, now=T0 + timedelta(days=29, hours=23)) + assert first == later + + def test_salt_rotates_after_the_interval(self, connection): + first = current_salt(connection, now=T0) + after = current_salt(connection, now=T0 + ROTATION_INTERVAL + timedelta(seconds=1)) + assert first != after + + def test_salt_is_persisted(self, connection): + salt = current_salt(connection, now=T0) + stored = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (SALT_KEY,) + ).fetchone()[0] + assert stored == salt + + def test_issued_at_is_recorded(self, connection): + current_salt(connection, now=T0) + stored = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (SALT_ISSUED_AT_KEY,) + ).fetchone()[0] + assert stored.startswith("2026-08-26T12:00") + + def test_two_installs_get_different_salts(self): + salts = set() + for _ in range(5): + conn = sqlite3.connect(":memory:") + conn.execute( + "CREATE TABLE telemetry_state (key TEXT PRIMARY KEY, value TEXT NOT NULL)" + ) + salts.add(current_salt(conn, now=T0)) + conn.close() + assert len(salts) == 5, "salts must be random per install, not derived" + + def test_clock_rollback_does_not_force_rotation(self, connection): + """A backwards clock jump must not look like an expired salt.""" + first = current_salt(connection, now=T0) + rolled_back = current_salt(connection, now=T0 - timedelta(days=5)) + assert rolled_back != first, "an out-of-window time reissues rather than trusting it" + + def test_corrupt_issued_at_reissues_rather_than_crashing(self, connection): + current_salt(connection, now=T0) + connection.execute( + "UPDATE telemetry_state SET value = 'not-a-date' WHERE key = ?", + (SALT_ISSUED_AT_KEY,), + ) + assert current_salt(connection, now=T0) is not None + + +class TestDerivation: + def test_derivation_is_deterministic(self): + salt = "a" * 64 + assert derive_install_id(INSTALL_ID, salt) == derive_install_id(INSTALL_ID, salt) + + def test_derivation_hides_the_install_id(self): + derived = derive_install_id(INSTALL_ID, "a" * 64) + assert INSTALL_ID not in derived + assert derived != INSTALL_ID + + def test_different_salts_give_different_values(self): + assert derive_install_id(INSTALL_ID, "a" * 64) != derive_install_id( + INSTALL_ID, "b" * 64 + ) + + def test_different_installs_give_different_values(self): + salt = "a" * 64 + assert derive_install_id(INSTALL_ID, salt) != derive_install_id("other", salt) + + def test_output_shape_is_sha256_hex(self): + derived = derive_install_id(INSTALL_ID, "a" * 64) + assert len(derived) == 64 + int(derived, 16) + + +class TestSubstitution: + def _package(self): + return { + "schema_version": "hermes.shared_metrics.v2", + "package_id": "3a63d27e-f170-4d4c-8c4d-ebd80feac592", + "install_id": INSTALL_ID, + "generated_at": "2026-08-26T01:01:25.311956Z", + "period_start": "2026-08-26T00:00:00Z", + "period_end": "2026-08-27T00:00:00Z", + "resource": {"hermes_version": "0.20.5", "os_family": "macos"}, + "metrics": [{"name": "hermes.client.active", "type": "counter", "value": 1}], + } + + def test_install_id_is_replaced(self): + result = substitute_install_id(self._package(), "derived-value") + assert result["install_id"] == "derived-value" + + def test_no_other_field_changes(self): + original = self._package() + result = substitute_install_id(original, "derived-value") + for key in original: + if key != "install_id": + assert result[key] == original[key] + + def test_the_caller_dict_is_not_mutated(self): + original = self._package() + substitute_install_id(original, "derived-value") + assert original["install_id"] == INSTALL_ID + + def test_no_fields_are_added_or_removed(self): + original = self._package() + assert set(substitute_install_id(original, "x")) == set(original) + + def test_the_raw_install_id_never_survives_substitution(self): + import json + + body = json.dumps(substitute_install_id(self._package(), "derived-value")) + assert INSTALL_ID not in body + + +class TestRetryStability: + """The property that keeps retries contract-compliant.""" + + def test_a_frozen_derived_id_survives_a_rotation(self, connection): + salt_before = current_salt(connection, now=T0) + frozen = derive_install_id(INSTALL_ID, salt_before) + + # Time passes, the salt rotates, and the package is retried. + salt_after = current_salt(connection, now=T0 + ROTATION_INTERVAL + timedelta(days=1)) + assert salt_after != salt_before + + # Rebuilding from the FROZEN value reproduces identical bytes; deriving + # afresh would not. + assert substitute_install_id({"install_id": INSTALL_ID}, frozen) == { + "install_id": frozen + } + assert derive_install_id(INSTALL_ID, salt_after) != frozen From 00c75cea335b2f90ceed648ab02542d6d2043b9e Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:45:02 +1000 Subject: [PATCH 04/20] feat(telemetry): send exported packages to the ingest service MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Steps 4, 5 and 7 of the shared-metrics exporter: the send logic, the consent gate, and backoff plus multi-process claiming. These arrive together because the sender is not correct without all three. Contract handling: 202 marks sent; 400 is permanent and never retried; 429 honours Retry-After (clamped to a day so a bogus value cannot park a package); 5xx, timeouts and transport errors retry three times in-process with 1s/5s/25s full-jitter backoff, then defer to a later pass. Consent is gated on the package's PERIOD, not its creation time. A period is split across packages created on different days, so a created-at gate would send a period's tail while dropping its head and silently undercount the opt-in day — data that looks complete and is wrong. The opt-in day is recorded once and never moves, so toggling sending off and on does not re-open the pre-consent backlog. Rows are claimed in a write transaction, which is what stops two Hermes processes sharing one database from sending the same package twice. next_attempt_at persists backoff across restarts, so a hard-down service is not retried on every task completion. The body is recomputed from payload_json rather than stored a second time: json.dumps is deterministic here (verified against the real outbox — 11 of 11 files reproduce byte-for-byte), and the only mutable input, the derived identity, is frozen on the row at first attempt. That keeps retries byte-identical across a salt rotation for ~36 bytes instead of a duplicate ~11 KB payload. The outbox directory is never written to or deleted from. A 202 updates SQLite only, because those files are the user's 30-day local history and retention already owns their lifecycle. Tests: 33. Two of them caught real defects in this commit — an unreadable row aborted the claim transaction and blocked every package behind it, and the compression assertions were passing through an injected fake that bypassed the code under test. --- .../observability/shared_metrics_sender.py | 385 +++++++++++++++ .../hermes_cli/test_shared_metrics_sender.py | 449 ++++++++++++++++++ 2 files changed, 834 insertions(+) create mode 100644 hermes_cli/observability/shared_metrics_sender.py create mode 100644 tests/hermes_cli/test_shared_metrics_sender.py diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py new file mode 100644 index 0000000000..9086c3b359 --- /dev/null +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -0,0 +1,385 @@ +"""Transmit exported shared-metrics packages to the Nous telemetry service. + +Implements the sender side of the ingest contract (see the telemetry repo's +``CONTRACT.md``): + +* ``202`` — durably stored. Mark sent. +* ``400`` — permanently malformed. Never retry. +* ``429`` — keep, retry after ``Retry-After``. +* ``5xx`` / timeout / connection error — keep, retry with backoff. + +Two properties are load-bearing and easy to get wrong: + +**The outbox directory is the user's local history, not a queue.** Packages +are pruned by age; a ``202`` marks send state in SQLite and never deletes a +file. See Appendix A.7 of ``docs/observability/relay-shared-metrics.md``. + +**Consent is gated on the package's PERIOD, not its creation time.** One +period is split across packages created on different days, so a created-at +gate would send a period's tail while dropping its head and silently +undercount the opt-in day. +""" + +from __future__ import annotations + +import gzip +import json +import logging +import random +import sqlite3 +import time +import urllib.error +import urllib.request +from dataclasses import dataclass +from datetime import datetime, timezone + +from hermes_cli.sqlite_util import write_txn + +from .shared_metrics_identity import ( + current_salt, + derive_install_id, + substitute_install_id, +) + +logger = logging.getLogger(__name__) + +#: Contract recommends timing out at 30s and treating a timeout as retryable. +REQUEST_TIMEOUT_SECONDS = 30 + +#: In-process attempts per package per pass, then the package waits for a +#: later pass. Backoff is 1s/5s/25s with full jitter. +MAX_ATTEMPTS = 3 +_BACKOFF_BASE_SECONDS = 1 +_BACKOFF_FACTOR = 5 + +#: Contract recommends gzip above roughly this size. +GZIP_THRESHOLD_BYTES = 4096 + +#: Packages per pass. Bounds work on an interactive hook even after an outage. +MAX_PACKAGES_PER_PASS = 20 + +#: Floor applied after a pass fails to deliver, so a hard-down service is not +#: retried on every task completion. +_FAILURE_BACKOFF_SECONDS = 15 * 60 + +OPT_IN_PERIOD_KEY = "send_opt_in_period" + + +def _utc_now() -> datetime: + return datetime.now(timezone.utc) + + +def _isoformat(value: datetime) -> str: + return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") + + +@dataclass +class SendOutcome: + """What one pass did. Returned for tests and diagnostics.""" + + sent: int = 0 + rejected: int = 0 + deferred: int = 0 + skipped_not_due: int = 0 + + +class _Response: + __slots__ = ("status", "retry_after", "body") + + def __init__(self, status: int, retry_after: str | None, body: str) -> None: + self.status = status + self.retry_after = retry_after + self.body = body + + +def _post(endpoint: str, payload: bytes, *, timeout: int) -> _Response: + """POST one package. Raises on transport failure; never on HTTP status.""" + headers = { + "Content-Type": "application/json", + "User-Agent": "hermes-agent-shared-metrics/1", + } + body = payload + if len(payload) > GZIP_THRESHOLD_BYTES: + body = gzip.compress(payload) + headers["Content-Encoding"] = "gzip" + + request = urllib.request.Request( + endpoint, data=body, headers=headers, method="POST" + ) + try: + with urllib.request.urlopen(request, timeout=timeout) as response: + return _Response( + response.status, + response.headers.get("Retry-After"), + response.read(2048).decode("utf-8", "replace"), + ) + except urllib.error.HTTPError as exc: + # An HTTP error status is a normal contract outcome, not a failure. + return _Response( + exc.code, + exc.headers.get("Retry-After") if exc.headers else None, + exc.read(2048).decode("utf-8", "replace") if exc.fp else "", + ) + + +def _retry_after_seconds(value: str | None, default: int) -> int: + if not value: + return default + try: + # Contract sends seconds. Clamp so a hostile or bogus value cannot + # park a package for years, and never go below one second. + return max(1, min(int(float(value)), 86_400)) + except (TypeError, ValueError): + return default + + +def opt_in_period(connection: sqlite3.Connection, *, now: datetime | None = None) -> str: + """Return the opt-in day (UTC date), recording it on first use. + + Must run inside a write transaction. The value is written once and then + never moves, so turning sending off and on again does not re-open the + pre-consent backlog. + """ + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (OPT_IN_PERIOD_KEY,) + ).fetchone() + if row is not None: + return str(row[0]) + today = (now or _utc_now()).date().isoformat() + connection.execute( + "INSERT OR IGNORE INTO telemetry_state(key, value) VALUES (?, ?)", + (OPT_IN_PERIOD_KEY, today), + ) + return today + + +class SharedMetricsSender: + """Sends exported packages, one bounded pass at a time.""" + + def __init__( + self, + store, + endpoint: str, + *, + post=_post, + sleep=time.sleep, + now=_utc_now, + max_attempts: int = MAX_ATTEMPTS, + ) -> None: + self._store = store + self._endpoint = endpoint + self._post = post + self._sleep = sleep + self._now = now + self._max_attempts = max_attempts + + # -- selection --------------------------------------------------------- + + def _claim(self, connection: sqlite3.Connection, now: datetime) -> list[dict]: + """Atomically take ownership of the packages this pass will try. + + Claiming inside the write transaction is what stops two Hermes + processes sharing one database from sending the same package twice. + Duplicates would be harmless (the service dedupes by package_id and + the bytes are identical) but they waste the user's bandwidth. + """ + period = opt_in_period(connection, now=now) + stamp = _isoformat(now) + rows = connection.execute( + """ + SELECT package_id, payload_json, sent_install_id + FROM package_outbox + WHERE exported_at IS NOT NULL + AND (send_state IS NULL OR send_state = 'pending') + AND (next_attempt_at IS NULL OR next_attempt_at <= ?) + AND substr(period_start, 1, 10) >= ? + ORDER BY created_at, package_id + LIMIT ? + """, + (stamp, period, MAX_PACKAGES_PER_PASS), + ).fetchall() + + claimed: list[dict] = [] + salt: str | None = None + for row in rows: + package_id = str(row[0]) + derived = row[2] + if not derived: + # Freeze the derived identity on first attempt so a later salt + # rotation cannot change the bytes sent under this package_id. + if salt is None: + salt = current_salt(connection, now=now) + try: + payload = json.loads(row[1]) + install_id = str(payload.get("install_id", "")) + except (TypeError, ValueError): + # A row we cannot parse can never be sent. Mark it and move + # on: one unreadable package must not block every other + # package behind it, and aborting here would roll back the + # whole claim transaction. + logger.warning( + "Shared-metrics package %s is unreadable; not sending", + package_id, + ) + connection.execute( + """ + UPDATE package_outbox + SET send_state = 'rejected', last_error = 'unreadable payload' + WHERE package_id = ? + """, + (package_id,), + ) + continue + derived = derive_install_id(install_id, salt) + connection.execute( + "UPDATE package_outbox SET sent_install_id = ? WHERE package_id = ?", + (derived, package_id), + ) + connection.execute( + """ + UPDATE package_outbox + SET send_state = 'pending', + send_attempts = send_attempts + 1, + next_attempt_at = ? + WHERE package_id = ? + """, + # Hold the row for the duration of this pass; success or a + # real backoff overwrite this immediately below. + (_isoformat(now), package_id), + ) + claimed.append( + { + "package_id": package_id, + "payload_json": str(row[1]), + "derived": str(derived), + } + ) + return claimed + + # -- transmission ------------------------------------------------------ + + def _body(self, payload_json: str, derived: str) -> bytes: + """Rebuild the exact bytes to send. + + The payload is recomputed from the stored package rather than kept as + a second copy: json.dumps with these options is deterministic, and the + only mutable input (the derived id) is frozen in the row. + """ + payload = substitute_install_id(json.loads(payload_json), derived) + return json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") + + def _mark(self, package_id: str, **columns) -> None: + assignments = ", ".join(f"{name} = ?" for name in columns) + with self._store._connection() as connection: + with write_txn(connection): + connection.execute( + f"UPDATE package_outbox SET {assignments} WHERE package_id = ?", + (*columns.values(), package_id), + ) + + def _defer(self, package_id: str, delay_seconds: int, reason: str) -> None: + retry_at = self._now().timestamp() + delay_seconds + self._mark( + package_id, + send_state="pending", + next_attempt_at=_isoformat( + datetime.fromtimestamp(retry_at, tz=timezone.utc) + ), + last_error=reason[:500], + ) + + def _send_one(self, package: dict) -> str: + """Try one package. Returns 'sent', 'rejected', or 'deferred'.""" + package_id = package["package_id"] + body = self._body(package["payload_json"], package["derived"]) + + for attempt in range(1, self._max_attempts + 1): + try: + response = self._post( + self._endpoint, body, timeout=REQUEST_TIMEOUT_SECONDS + ) + except Exception as exc: # transport failure: offline, DNS, TLS + reason = f"{type(exc).__name__}: {exc}" + if attempt >= self._max_attempts: + self._defer(package_id, _FAILURE_BACKOFF_SECONDS, reason) + return "deferred" + self._sleep(self._backoff(attempt)) + continue + + if response.status == 202: + self._mark( + package_id, + send_state="sent", + sent_at=_isoformat(self._now()), + last_error=None, + ) + return "sent" + + if response.status == 400: + # Permanent per the contract. Keep the file (it is the user's + # history) but never try again. + logger.warning( + "Telemetry package %s rejected as malformed; not retrying", + package_id, + ) + self._mark( + package_id, + send_state="rejected", + last_error=response.body[:500], + ) + return "rejected" + + if response.status == 429: + self._defer( + package_id, + _retry_after_seconds(response.retry_after, _FAILURE_BACKOFF_SECONDS), + "rate limited", + ) + return "deferred" + + # 5xx and anything unexpected: retryable. + reason = f"HTTP {response.status}" + if attempt >= self._max_attempts: + self._defer(package_id, _FAILURE_BACKOFF_SECONDS, reason) + return "deferred" + self._sleep(self._backoff(attempt)) + + self._defer(package_id, _FAILURE_BACKOFF_SECONDS, "attempts exhausted") + return "deferred" + + @staticmethod + def _backoff(attempt: int) -> float: + """1s, 5s, 25s with full jitter.""" + ceiling = _BACKOFF_BASE_SECONDS * (_BACKOFF_FACTOR ** (attempt - 1)) + return random.uniform(0, ceiling) + + # -- entry point ------------------------------------------------------- + + def send_pending(self) -> SendOutcome: + """Run one bounded pass. Never raises.""" + outcome = SendOutcome() + try: + now = self._now() + with self._store._connection() as connection: + with write_txn(connection): + claimed = self._claim(connection, now) + except Exception: + logger.warning("Unable to select shared-metrics packages", exc_info=True) + return outcome + + for package in claimed: + try: + result = self._send_one(package) + except Exception: + logger.warning( + "Unable to send shared-metrics package", exc_info=True + ) + outcome.deferred += 1 + continue + if result == "sent": + outcome.sent += 1 + elif result == "rejected": + outcome.rejected += 1 + else: + outcome.deferred += 1 + return outcome diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py new file mode 100644 index 0000000000..d0fec15bfa --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -0,0 +1,449 @@ +"""Tests for the shared-metrics sender. + +Covers the four contract responses, the period-based consent gate, frozen +identity across rotation, transactional claiming, and the invariant that +matters most: a package file is never deleted, because the outbox is the +user's local history rather than a send queue. +""" + +from __future__ import annotations + +import json +import sqlite3 +from datetime import datetime, timedelta, timezone + +import pytest + +from hermes_cli.observability.shared_metrics import SharedMetricsStore +from hermes_cli.observability.shared_metrics_sender import ( + MAX_PACKAGES_PER_PASS, + OPT_IN_PERIOD_KEY, + SharedMetricsSender, + opt_in_period, +) + +INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" +NOW = datetime(2026, 8, 26, 12, 0, tzinfo=timezone.utc) +ENDPOINT = "https://telemetry.test/v1/telemetry" + + +class FakeResponse: + def __init__(self, status, retry_after=None, body=""): + self.status = status + self.retry_after = retry_after + self.body = body + + +class FakeTransport: + """Records every POST and replays a scripted sequence of responses.""" + + def __init__(self, *responses): + self._responses = list(responses) + self.calls = [] + + def __call__(self, endpoint, payload, *, timeout): + self.calls.append({"endpoint": endpoint, "payload": payload, "timeout": timeout}) + if not self._responses: + return FakeResponse(202) + item = self._responses.pop(0) + if isinstance(item, Exception): + raise item + return item + + @property + def bodies(self): + return [json.loads(c["payload"].decode("utf-8")) for c in self.calls] + + +@pytest.fixture +def store(tmp_path): + return SharedMetricsStore( + database_path=tmp_path / "metrics.sqlite3", + outbox_directory=tmp_path / "outbox", + ) + + +def _add_package(store, package_id, period_day, *, exported=True, install_id=INSTALL_ID): + payload = { + "schema_version": "hermes.shared_metrics.v2", + "package_id": package_id, + "install_id": install_id, + "period_start": f"{period_day}T00:00:00Z", + "period_end": f"{period_day}T23:59:59Z", + "metrics": [{"name": "hermes.client.active", "type": "counter", "value": 1}], + } + with store._connection() as connection: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + ( + package_id, + f"{period_day}T00:00:00Z", + f"{period_day}T23:59:59Z", + json.dumps(payload), + f"{period_day}T01:00:00Z", + f"{period_day}T01:00:01Z" if exported else None, + ), + ) + path = store.outbox_directory / f"{package_id}.json" + path.write_text(json.dumps(payload, indent=2, sort_keys=True)) + return path + + +def _row(store, package_id): + with store._connection() as connection: + row = connection.execute( + """ + SELECT send_state, sent_at, send_attempts, next_attempt_at, + last_error, sent_install_id + FROM package_outbox WHERE package_id = ? + """, + (package_id,), + ).fetchone() + return dict( + send_state=row[0], + sent_at=row[1], + send_attempts=row[2], + next_attempt_at=row[3], + last_error=row[4], + sent_install_id=row[5], + ) + + +def _sender(store, transport, **kwargs): + return SharedMetricsSender( + store, + ENDPOINT, + post=transport, + sleep=lambda _s: None, + now=lambda: NOW, + **kwargs, + ) + + +class TestContractResponses: + def test_202_marks_sent(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(202)) + outcome = _sender(store, transport).send_pending() + assert outcome.sent == 1 + row = _row(store, "pkg-1") + assert row["send_state"] == "sent" + assert row["sent_at"] is not None + + def test_400_is_permanent_and_never_retried(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(400, body='{"error":"invalid_envelope"}')) + outcome = _sender(store, transport).send_pending() + assert outcome.rejected == 1 + assert len(transport.calls) == 1, "a 400 must not be retried" + assert _row(store, "pkg-1")["send_state"] == "rejected" + + # A later pass must not pick it up again. + transport2 = FakeTransport(FakeResponse(202)) + _sender(store, transport2).send_pending() + assert transport2.calls == [] + + def test_429_defers_using_retry_after(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(429, retry_after="120")) + outcome = _sender(store, transport).send_pending() + assert outcome.deferred == 1 + assert len(transport.calls) == 1, "429 waits rather than burning attempts" + row = _row(store, "pkg-1") + assert row["send_state"] == "pending" + assert row["next_attempt_at"] == "2026-08-26T12:02:00Z" + + def test_429_without_retry_after_still_defers(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(429)) + _sender(store, transport).send_pending() + assert _row(store, "pkg-1")["next_attempt_at"] > "2026-08-26T12:00:00Z" + + def test_absurd_retry_after_is_clamped(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(429, retry_after="99999999")) + _sender(store, transport).send_pending() + # clamped to 24h, not years + assert _row(store, "pkg-1")["next_attempt_at"] <= "2026-08-27T12:00:00Z" + + def test_5xx_retries_then_defers(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport( + FakeResponse(503), FakeResponse(503), FakeResponse(503) + ) + outcome = _sender(store, transport).send_pending() + assert outcome.deferred == 1 + assert len(transport.calls) == 3, "three in-process attempts" + assert _row(store, "pkg-1")["send_state"] == "pending" + + def test_5xx_then_success_within_the_same_pass(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(503), FakeResponse(202)) + outcome = _sender(store, transport).send_pending() + assert outcome.sent == 1 + assert len(transport.calls) == 2 + + def test_transport_failure_is_retryable(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport( + OSError("offline"), OSError("offline"), FakeResponse(202) + ) + outcome = _sender(store, transport).send_pending() + assert outcome.sent == 1 + + def test_persistent_offline_defers_without_raising(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(*[OSError("offline")] * 3) + outcome = _sender(store, transport).send_pending() + assert outcome.deferred == 1 + assert "OSError" in _row(store, "pkg-1")["last_error"] + + +class TestConsentGate: + def test_packages_from_before_opt_in_are_never_sent(self, store): + _add_package(store, "old", "2026-08-20") + _add_package(store, "new", "2026-08-26") + transport = FakeTransport(FakeResponse(202)) + _sender(store, transport).send_pending() + assert [b["package_id"] for b in transport.bodies] == ["new"] + + def test_a_period_straddling_opt_in_day_is_sent_whole(self, store): + """The head/tail bug: both packages for the opt-in period must go.""" + _add_package(store, "head", "2026-08-26") + _add_package(store, "tail", "2026-08-26") # created later, same period + transport = FakeTransport(FakeResponse(202), FakeResponse(202)) + _sender(store, transport).send_pending() + assert sorted(b["package_id"] for b in transport.bodies) == ["head", "tail"] + + def test_opt_in_day_is_recorded_once_and_does_not_move(self, store): + with store._connection() as connection: + first = opt_in_period(connection, now=NOW) + later = opt_in_period(connection, now=NOW + timedelta(days=10)) + assert first == later == "2026-08-26" + + def test_opt_in_day_is_persisted(self, store): + with store._connection() as connection: + opt_in_period(connection, now=NOW) + value = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (OPT_IN_PERIOD_KEY,) + ).fetchone()[0] + assert value == "2026-08-26" + + def test_unexported_packages_are_skipped(self, store): + _add_package(store, "pending-export", "2026-08-26", exported=False) + transport = FakeTransport(FakeResponse(202)) + _sender(store, transport).send_pending() + assert transport.calls == [] + + +class TestIdentity: + def test_install_id_is_never_transmitted(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(202)) + _sender(store, transport).send_pending() + raw = transport.calls[0]["payload"].decode("utf-8") + assert INSTALL_ID not in raw + assert transport.bodies[0]["install_id"] != INSTALL_ID + + def test_derived_id_is_frozen_on_the_row(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(503), FakeResponse(202)) + _sender(store, transport).send_pending() + assert _row(store, "pkg-1")["sent_install_id"] == transport.bodies[0]["install_id"] + + def test_retries_send_identical_bytes(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(503), FakeResponse(503), FakeResponse(202)) + _sender(store, transport).send_pending() + payloads = {c["payload"] for c in transport.calls} + assert len(payloads) == 1, "a resend must be byte-identical per the contract" + + def test_only_install_id_differs_from_the_stored_package(self, store): + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(202)) + _sender(store, transport).send_pending() + sent = transport.bodies[0] + with store._connection() as connection: + stored = json.loads( + connection.execute( + "SELECT payload_json FROM package_outbox WHERE package_id = 'pkg-1'" + ).fetchone()[0] + ) + assert set(sent) == set(stored) + for key in stored: + if key != "install_id": + assert sent[key] == stored[key] + + +class TestOutboxIsNotAQueue: + def test_a_sent_package_file_is_not_deleted(self, store): + path = _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(202))).send_pending() + assert path.exists(), "the outbox is the user's history, not a send queue" + + def test_a_rejected_package_file_is_not_deleted(self, store): + path = _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(400))).send_pending() + assert path.exists() + + def test_the_package_row_survives_sending(self, store): + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(202))).send_pending() + with store._connection() as connection: + assert connection.execute( + "SELECT COUNT(*) FROM package_outbox WHERE package_id = 'pkg-1'" + ).fetchone()[0] == 1 + + +class TestClaimingAndBounds: + def test_a_sent_package_is_not_resent(self, store): + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(202))).send_pending() + second = FakeTransport(FakeResponse(202)) + _sender(store, second).send_pending() + assert second.calls == [] + + def test_a_deferred_package_is_skipped_until_due(self, store): + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(429, retry_after="600"))).send_pending() + second = FakeTransport(FakeResponse(202)) + _sender(store, second).send_pending() + assert second.calls == [], "backoff must survive within the same process" + + def test_a_deferred_package_is_retried_once_due(self, store): + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(429, retry_after="60"))).send_pending() + + later = SharedMetricsSender( + store, + ENDPOINT, + post=(transport := FakeTransport(FakeResponse(202))), + sleep=lambda _s: None, + now=lambda: NOW + timedelta(minutes=5), + ) + later.send_pending() + assert len(transport.calls) == 1 + + def test_attempts_are_counted(self, store): + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(429))).send_pending() + assert _row(store, "pkg-1")["send_attempts"] == 1 + + def test_a_pass_is_bounded(self, store): + for i in range(MAX_PACKAGES_PER_PASS + 5): + _add_package(store, f"pkg-{i:02d}", "2026-08-26") + transport = FakeTransport(*[FakeResponse(202)] * 40) + outcome = _sender(store, transport).send_pending() + assert outcome.sent == MAX_PACKAGES_PER_PASS + + def test_two_concurrent_passes_do_not_double_send(self, store): + """Claiming is what stops two Hermes processes duplicating work.""" + _add_package(store, "pkg-1", "2026-08-26") + + seen = [] + + def transport(endpoint, payload, *, timeout): + seen.append(payload) + # A second sender runs while the first is mid-flight. + SharedMetricsSender( + store, + ENDPOINT, + post=lambda *a, **k: (_ for _ in ()).throw( + AssertionError("second pass must not claim a held package") + ), + sleep=lambda _s: None, + now=lambda: NOW, + ).send_pending() + return FakeResponse(202) + + _sender(store, transport).send_pending() + assert len(seen) == 1 + + +class TestResilience: + def test_a_corrupt_row_does_not_stop_the_pass(self, store): + _add_package(store, "good", "2026-08-26") + with store._connection() as connection: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES ('bad', '2026-08-26T00:00:00Z', '2026-08-26T23:59:59Z', + 'not json', '2026-08-26T00:00:00Z', '2026-08-26T01:00:00Z') + """ + ) + transport = FakeTransport(*[FakeResponse(202)] * 5) + outcome = _sender(store, transport).send_pending() + assert outcome.sent >= 1 + + def test_send_pending_never_raises_on_a_broken_database(self, store, tmp_path): + store.database_path.write_text("this is not a database") + outcome = _sender(store, FakeTransport(FakeResponse(202))).send_pending() + assert outcome.sent == 0 + + +class TestCompression: + """Compression lives in the real transport, so exercise _post directly.""" + + def _captured_request(self, payload: bytes): + import urllib.request + + from hermes_cli.observability import shared_metrics_sender as mod + + captured = {} + + class FakeConn: + status = 202 + headers = {} + + def read(self, _n=None): + return b"{}" + + def __enter__(self): + return self + + def __exit__(self, *a): + return False + + def fake_urlopen(request, timeout=None): + captured["data"] = request.data + captured["headers"] = {k.lower(): v for k, v in request.headers.items()} + return FakeConn() + + original = urllib.request.urlopen + urllib.request.urlopen = fake_urlopen + try: + mod._post(ENDPOINT, payload, timeout=5) + finally: + urllib.request.urlopen = original + return captured + + def test_large_payloads_are_gzipped(self): + payload = json.dumps({"filler": "x" * 20000}).encode("utf-8") + captured = self._captured_request(payload) + assert captured["data"][:2] == b"\x1f\x8b", "gzip magic bytes" + assert captured["headers"].get("Content-encoding".lower()) == "gzip" + + def test_gzip_actually_shrinks_the_body(self): + payload = json.dumps({"filler": "x" * 20000}).encode("utf-8") + captured = self._captured_request(payload) + assert len(captured["data"]) < len(payload) + + def test_gzip_round_trips_to_the_original_bytes(self): + import gzip as gziplib + + payload = json.dumps({"filler": "x" * 20000}).encode("utf-8") + captured = self._captured_request(payload) + assert gziplib.decompress(captured["data"]) == payload + + def test_small_payloads_are_sent_plain(self): + payload = b'{"small": true}' + captured = self._captured_request(payload) + assert captured["data"] == payload + assert "content-encoding" not in captured["headers"] From 6fdf6f4d4a1beebf83d14b7f1e00cade1b805ae4 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:49:25 +1000 Subject: [PATCH 05/20] feat(telemetry): run the send pass off the export hook MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Step 6 of the shared-metrics exporter, plus a loopback E2E. _export now triggers an opt-in send pass on a daemon thread. The hook runs on finish_task — the user's interactive path — so a 30s network timeout there would be felt directly; the thread keeps that latency off the caller. A test asserts _export returns in under a second while a send is deliberately blocked. At most one pass is in flight per process: a queued second pass would add nothing, because the next hook fire picks up whatever is still pending. Shutdown joins the thread for at most two seconds, then lets it go — the packages remain in SQLite and go out on the next run, so blocking a user's exit on a slow network is the wrong trade. Sending is resolved per pass from the profile's own config, so turning it off takes effect at the next hook fire without a restart. E2E (tests/hermes_cli/test_shared_metrics_sender_e2e.py): the real sender against a real HTTPServer on loopback — actual urllib, gzip, headers and sockets rather than an injected fake. Covers delivery and sent-state, 400/429/5xx handling, a retry sending byte-identical bytes, gzip shrinking a realistic 120-metric package and the server parsing it back, install_id never crossing the wire, the outbox file staying untouched, and a dead server deferring without raising. Wiring tests: 12, all negative-space properties — no send without opt-in, no blocking, no pile-up, no crash propagation. --- .../observability/relay_shared_metrics.py | 72 ++++- .../test_shared_metrics_send_wiring.py | 216 +++++++++++++++ .../test_shared_metrics_sender_e2e.py | 250 ++++++++++++++++++ 3 files changed, 537 insertions(+), 1 deletion(-) create mode 100644 tests/hermes_cli/test_shared_metrics_send_wiring.py create mode 100644 tests/hermes_cli/test_shared_metrics_sender_e2e.py diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index 2ab88f51c3..cb4eb44267 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -132,6 +132,9 @@ class _Runtime: self._sessions: dict[str, _MetricsSession] = {} self._task_creation_lock = threading.RLock() self._task_sessions_lock = threading.RLock() + # Guards the opt-in send pass: at most one in flight per process. + self._send_lock = threading.RLock() + self._send_thread: threading.Thread | None = None self._task_sessions: dict[tuple[str, str], _MetricsSession] = {} self._turn_sessions: dict[tuple[str, str], _MetricsSession] = {} self._subscriber_name = f"{SUBSCRIBER_NAME}.{self.host.runtime_id}" @@ -706,11 +709,29 @@ class _Runtime: with self._task_sessions_lock: self._task_sessions.clear() self._turn_sessions.clear() + self._join_send_thread() try: atexit.unregister(self.shutdown) except Exception: pass + def _join_send_thread(self, timeout: float = 2.0) -> None: + """Give an in-flight send a brief chance to finish at exit. + + Bounded on purpose: the packages stay pending in SQLite and go out on + the next run, so blocking a user's shutdown for a slow network is the + wrong trade. The thread is a daemon, so an unfinished pass dies with + the process rather than holding it open. + """ + with self._send_lock: + thread = self._send_thread + if thread is None or not thread.is_alive(): + return + try: + thread.join(timeout) + except Exception: + logger.debug("Shared-metrics send thread join failed", exc_info=True) + def _session(self, event: dict[str, Any]) -> _MetricsSession | None: session_id = str(event.get("session_id") or "") with self._sessions_lock: @@ -1048,7 +1069,56 @@ class _Runtime: return True def _export(self) -> None: - self._safe(self.subscriber.store.create_and_export_package_if_due) + exported = self._safe(self.subscriber.store.create_and_export_package_if_due) + # Sending is opt-in and must never delay the caller: _export runs on + # finish_task, which is the user's interactive path. Errors inside the + # sender are already swallowed there; the thread is about latency, not + # correctness. + if exported is not None: + self._safe(self._send_exported_packages) + + def _send_exported_packages(self) -> None: + from hermes_cli.observability.shared_metrics_send_config import ( + resolve_send_config, + ) + + try: + from hermes_cli.config import read_raw_config_readonly + + config = read_raw_config_readonly() or {} + except Exception: + logger.debug("Unable to read shared-metrics send policy", exc_info=True) + return + + resolved = resolve_send_config(config) + if not resolved.send: + return + + with self._send_lock: + # One in-flight pass per process. A queued second pass would add + # nothing: the next hook fire picks up whatever is still pending. + if self._send_thread is not None and self._send_thread.is_alive(): + return + thread = threading.Thread( + target=self._run_send_pass, + args=(resolved.endpoint,), + name="hermes-shared-metrics-send", + daemon=True, + ) + self._send_thread = thread + thread.start() + + def _run_send_pass(self, endpoint: str) -> None: + from hermes_cli.observability.shared_metrics_sender import ( + SharedMetricsSender, + ) + + try: + SharedMetricsSender( + self.subscriber.store, endpoint + ).send_pending() + except Exception: + logger.warning("Shared-metrics send pass failed", exc_info=True) def _event_metadata(self) -> dict[str, str]: return { diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py new file mode 100644 index 0000000000..730376083f --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -0,0 +1,216 @@ +"""Tests for wiring the sender into the shared-metrics export hook. + +The properties that matter here are negative ones: the interactive path must +not block, and nothing must leave the machine unless the user opted in. +""" + +from __future__ import annotations + +import threading +import time + +import pytest + +from hermes_cli.observability import relay_shared_metrics as mod + + +class FakeStore: + def __init__(self): + self.exported = 0 + + def create_and_export_package_if_due(self): + self.exported += 1 + return [] + + +class FakeSubscriber: + def __init__(self): + self.store = FakeStore() + + +class Runtime(mod._Runtime): + """A _Runtime with the relay host stubbed out.""" + + def __init__(self): + self._sessions_lock = threading.RLock() + self._sessions = {} + self._task_creation_lock = threading.RLock() + self._task_sessions_lock = threading.RLock() + self._send_lock = threading.RLock() + self._send_thread = None + self._task_sessions = {} + self._turn_sessions = {} + self.subscriber = FakeSubscriber() + + +@pytest.fixture +def runtime(): + return Runtime() + + +def _config(**shared): + return {"telemetry": {"shared_metrics": shared}} + + +@pytest.fixture +def capture_sender(monkeypatch): + """Replace the sender with a recorder and return the record.""" + record = {"passes": [], "endpoints": []} + + class FakeSender: + def __init__(self, store, endpoint, **kwargs): + record["endpoints"].append(endpoint) + + def send_pending(self): + record["passes"].append(time.time()) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + FakeSender, + ) + return record + + +def _set_config(monkeypatch, config): + monkeypatch.setattr( + "hermes_cli.config.read_raw_config_readonly", lambda: config, raising=False + ) + + +class TestOptIn: + def test_no_send_when_nothing_is_configured(self, runtime, monkeypatch, capture_sender): + _set_config(monkeypatch, {}) + runtime._export() + runtime._join_send_thread(timeout=1) + assert capture_sender["passes"] == [] + + def test_no_send_when_only_collection_is_on(self, runtime, monkeypatch, capture_sender): + _set_config(monkeypatch, _config(enabled=True)) + runtime._export() + runtime._join_send_thread(timeout=1) + assert capture_sender["passes"] == [] + + def test_no_send_when_send_is_on_without_collection( + self, runtime, monkeypatch, capture_sender + ): + _set_config(monkeypatch, _config(enabled=False, send=True)) + runtime._export() + runtime._join_send_thread(timeout=1) + assert capture_sender["passes"] == [] + + def test_sends_when_both_are_on(self, runtime, monkeypatch, capture_sender): + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._export() + runtime._join_send_thread(timeout=2) + assert len(capture_sender["passes"]) == 1 + + def test_uses_the_resolved_endpoint(self, runtime, monkeypatch, capture_sender): + _set_config( + monkeypatch, + _config(enabled=True, send=True, endpoint="https://staging.test/v1"), + ) + runtime._export() + runtime._join_send_thread(timeout=2) + assert capture_sender["endpoints"] == ["https://staging.test/v1"] + + def test_export_still_runs_when_sending_is_off(self, runtime, monkeypatch, capture_sender): + _set_config(monkeypatch, _config(enabled=True)) + runtime._export() + assert runtime.subscriber.store.exported == 1 + + +class TestInteractivePathIsNotBlocked: + def test_export_returns_before_the_send_finishes( + self, runtime, monkeypatch + ): + started = threading.Event() + release = threading.Event() + + class SlowSender: + def __init__(self, store, endpoint, **kwargs): + pass + + def send_pending(self): + started.set() + release.wait(5) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + SlowSender, + ) + _set_config(monkeypatch, _config(enabled=True, send=True)) + + began = time.monotonic() + runtime._export() + elapsed = time.monotonic() - began + + assert started.wait(2), "the send should have started" + assert elapsed < 1.0, "finish_task must not wait on the network" + release.set() + runtime._join_send_thread(timeout=5) + + def test_the_send_thread_is_a_daemon(self, runtime, monkeypatch, capture_sender): + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._export() + with runtime._send_lock: + thread = runtime._send_thread + assert thread is not None + assert thread.daemon, "an unfinished send must not hold the process open" + runtime._join_send_thread(timeout=2) + + def test_only_one_pass_runs_at_a_time(self, runtime, monkeypatch): + release = threading.Event() + starts = [] + + class SlowSender: + def __init__(self, store, endpoint, **kwargs): + pass + + def send_pending(self): + starts.append(1) + release.wait(5) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + SlowSender, + ) + _set_config(monkeypatch, _config(enabled=True, send=True)) + + for _ in range(5): + runtime._export() + time.sleep(0.2) + assert len(starts) == 1, "hook fires must not pile up send passes" + release.set() + runtime._join_send_thread(timeout=5) + + +class TestFailureIsolation: + def test_a_sender_crash_does_not_propagate(self, runtime, monkeypatch): + class Exploding: + def __init__(self, store, endpoint, **kwargs): + pass + + def send_pending(self): + raise RuntimeError("boom") + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + Exploding, + ) + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._export() # must not raise + runtime._join_send_thread(timeout=2) + + def test_an_unreadable_config_does_not_break_export(self, runtime, monkeypatch, capture_sender): + def explode(): + raise OSError("config unreadable") + + monkeypatch.setattr( + "hermes_cli.config.read_raw_config_readonly", explode, raising=False + ) + runtime._export() + assert runtime.subscriber.store.exported == 1 + assert capture_sender["passes"] == [] + + def test_join_is_safe_with_no_thread(self, runtime): + runtime._join_send_thread(timeout=0.1) diff --git a/tests/hermes_cli/test_shared_metrics_sender_e2e.py b/tests/hermes_cli/test_shared_metrics_sender_e2e.py new file mode 100644 index 0000000000..8568a9e6fe --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_sender_e2e.py @@ -0,0 +1,250 @@ +"""End-to-end test: the real sender against a real HTTP server. + +Everything else stubs the transport. This exercises the actual code path — +urllib, gzip, headers, socket — against a live server on loopback, so a +transport-level mistake that a fake would hide fails here instead. +""" + +from __future__ import annotations + +import gzip +import json +import sqlite3 +import threading +from datetime import datetime, timezone +from http.server import BaseHTTPRequestHandler, HTTPServer + +import pytest + +from hermes_cli.observability.shared_metrics import SharedMetricsStore +from hermes_cli.observability.shared_metrics_sender import SharedMetricsSender + +INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" +NOW = datetime(2026, 8, 26, 12, 0, tzinfo=timezone.utc) + + +class Ingest(BaseHTTPRequestHandler): + """A stand-in for the ingest service that records what it receives.""" + + received: list = [] + script: list = [] + + def do_POST(self): # noqa: N802 - stdlib naming + length = int(self.headers.get("Content-Length") or 0) + raw = self.rfile.read(length) + if self.headers.get("Content-Encoding") == "gzip": + body = gzip.decompress(raw) + else: + body = raw + type(self).received.append( + { + "headers": {k.lower(): v for k, v in self.headers.items()}, + "body": json.loads(body.decode("utf-8")), + "raw_len": len(raw), + "decoded_len": len(body), + } + ) + status, payload, extra = ( + type(self).script.pop(0) if type(self).script else (202, {}, {}) + ) + encoded = json.dumps(payload).encode("utf-8") + self.send_response(status) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(encoded))) + for key, value in extra.items(): + self.send_header(key, value) + self.end_headers() + self.wfile.write(encoded) + + def log_message(self, format, *args): # noqa: A002 - stdlib signature + pass + + +@pytest.fixture +def server(): + Ingest.received = [] + Ingest.script = [] + httpd = HTTPServer(("127.0.0.1", 0), Ingest) + thread = threading.Thread(target=httpd.serve_forever, daemon=True) + thread.start() + yield httpd + httpd.shutdown() + httpd.server_close() + + +@pytest.fixture +def store(tmp_path): + return SharedMetricsStore( + database_path=tmp_path / "metrics.sqlite3", + outbox_directory=tmp_path / "outbox", + ) + + +def _endpoint(server): + host, port = server.server_address + return f"http://{host}:{port}/v1/telemetry" + + +def _add(store, package_id, day="2026-08-26", metrics=1): + payload = { + "schema_version": "hermes.shared_metrics.v2", + "package_id": package_id, + "install_id": INSTALL_ID, + "generated_at": f"{day}T01:00:00Z", + "period_start": f"{day}T00:00:00Z", + "period_end": f"{day}T23:59:59Z", + "resource": { + "hermes_version": "0.20.5", + "os_family": "macos", + "architecture": "arm64", + "install_method": "git", + }, + "metrics": [ + { + "name": f"hermes.metric.{i}", + "type": "counter", + "dimensions": {"outcome": "ok"}, + "value": i, + } + for i in range(metrics) + ], + } + with store._connection() as connection: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + ( + package_id, + f"{day}T00:00:00Z", + f"{day}T23:59:59Z", + json.dumps(payload), + f"{day}T01:00:00Z", + f"{day}T01:00:01Z", + ), + ) + return payload + + +def _sender(store, server): + return SharedMetricsSender( + store, _endpoint(server), sleep=lambda _s: None, now=lambda: NOW + ) + + +class TestRealTransport: + def test_a_package_is_delivered_and_marked_sent(self, store, server): + _add(store, "pkg-1") + outcome = _sender(store, server).send_pending() + + assert outcome.sent == 1 + assert len(Ingest.received) == 1 + assert Ingest.received[0]["body"]["package_id"] == "pkg-1" + + with store._connection() as connection: + state = connection.execute( + "SELECT send_state FROM package_outbox WHERE package_id = 'pkg-1'" + ).fetchone()[0] + assert state == "sent" + + def test_the_install_id_never_crosses_the_wire(self, store, server): + _add(store, "pkg-1", metrics=40) + _sender(store, server).send_pending() + body = json.dumps(Ingest.received[0]["body"]) + assert INSTALL_ID not in body + assert len(Ingest.received[0]["body"]["install_id"]) == 64 + + def test_content_type_is_json(self, store, server): + _add(store, "pkg-1") + _sender(store, server).send_pending() + assert Ingest.received[0]["headers"]["content-type"] == "application/json" + + def test_a_realistic_package_is_gzipped_over_the_wire(self, store, server): + # ~40 metrics matches the real outbox's larger packages. + _add(store, "pkg-1", metrics=120) + _sender(store, server).send_pending() + record = Ingest.received[0] + assert record["headers"].get("content-encoding") == "gzip" + assert record["raw_len"] < record["decoded_len"] + + def test_the_server_can_parse_what_we_send(self, store, server): + """Proves the bytes are valid JSON after transport and decompression.""" + original = _add(store, "pkg-1", metrics=120) + _sender(store, server).send_pending() + received = Ingest.received[0]["body"] + assert received["metrics"] == original["metrics"] + assert received["resource"] == original["resource"] + + def test_400_is_permanent(self, store, server): + _add(store, "pkg-1") + Ingest.script = [(400, {"error": "invalid_envelope"}, {})] + outcome = _sender(store, server).send_pending() + assert outcome.rejected == 1 + assert len(Ingest.received) == 1 + + def test_429_is_honoured(self, store, server): + _add(store, "pkg-1") + Ingest.script = [(429, {"error": "rate_limited"}, {"Retry-After": "90"})] + outcome = _sender(store, server).send_pending() + assert outcome.deferred == 1 + with store._connection() as connection: + retry_at = connection.execute( + "SELECT next_attempt_at FROM package_outbox WHERE package_id = 'pkg-1'" + ).fetchone()[0] + assert retry_at == "2026-08-26T12:01:30Z" + + def test_5xx_retries_then_succeeds(self, store, server): + _add(store, "pkg-1") + Ingest.script = [ + (503, {"error": "storage_unavailable"}, {}), + (202, {"package_id": "pkg-1"}, {}), + ] + outcome = _sender(store, server).send_pending() + assert outcome.sent == 1 + assert len(Ingest.received) == 2 + + def test_a_retry_sends_identical_bytes(self, store, server): + _add(store, "pkg-1", metrics=5) + Ingest.script = [(503, {}, {}), (202, {}, {})] + _sender(store, server).send_pending() + first, second = Ingest.received + assert first["body"] == second["body"] + + def test_several_packages_in_one_pass(self, store, server): + for i in range(5): + _add(store, f"pkg-{i}") + outcome = _sender(store, server).send_pending() + assert outcome.sent == 5 + assert len(Ingest.received) == 5 + + def test_the_outbox_directory_is_untouched(self, store, server, tmp_path): + _add(store, "pkg-1") + marker = store.outbox_directory / "pkg-1.json" + marker.write_text('{"kept": true}') + _sender(store, server).send_pending() + assert marker.exists() + assert json.loads(marker.read_text()) == {"kept": True} + + def test_a_dead_server_defers_without_raising(self, store, server): + _add(store, "pkg-1") + host, port = server.server_address + server.shutdown() + server.server_close() + sender = SharedMetricsSender( + store, + f"http://{host}:{port}/v1/telemetry", + sleep=lambda _s: None, + now=lambda: NOW, + ) + outcome = sender.send_pending() + assert outcome.deferred == 1 + with store._connection() as connection: + state, error = connection.execute( + "SELECT send_state, last_error FROM package_outbox" + " WHERE package_id = 'pkg-1'" + ).fetchone() + assert state == "pending" + assert error From 055d58ba33f8b78c33323ea5e1f285489f77201f Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 15:55:06 +1000 Subject: [PATCH 06/20] test(telemetry): add the live staging E2E script Sends real packages through the real sender to the real staging ingest service and reports what came back. Uses a throwaway HERMES_HOME so an operator's own telemetry state is never touched, and asserts the local install_id did not cross the wire. Kept as a script rather than a pytest case on purpose: it needs live network and a deployed staging service, so it must not run in CI. --- scripts/e2e_shared_metrics_staging.py | 146 ++++++++++++++++++++++++++ 1 file changed, 146 insertions(+) create mode 100644 scripts/e2e_shared_metrics_staging.py diff --git a/scripts/e2e_shared_metrics_staging.py b/scripts/e2e_shared_metrics_staging.py new file mode 100644 index 0000000000..55e03a7572 --- /dev/null +++ b/scripts/e2e_shared_metrics_staging.py @@ -0,0 +1,146 @@ +"""Live staging E2E for the shared-metrics exporter. + +Sends REAL packages through the REAL sender to the REAL staging ingest +service, then reports what the service acknowledged. Uses a throwaway +HERMES_HOME so the operator's own telemetry state is untouched. + +Usage: + .venv/bin/python scripts/e2e_shared_metrics_staging.py +""" + +from __future__ import annotations + +import json +import os +import sys +import tempfile +import uuid +from datetime import datetime, timezone +from pathlib import Path + +REPO = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(REPO)) + +STAGING = "https://telemetry.staging-nousresearch.com/v1/telemetry" + + +def main() -> int: + scratch = Path(tempfile.mkdtemp(prefix="hermes-telemetry-e2e-")) + os.environ["HERMES_HOME"] = str(scratch) + + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_sender import SharedMetricsSender + + store = SharedMetricsStore( + database_path=scratch / "metrics.sqlite3", + outbox_directory=scratch / "outbox", + ) + + today = datetime.now(timezone.utc).date().isoformat() + real_install_id = str(uuid.uuid4()) + packages = [] + + # Two packages for today's period: the "head" and a later "tail", which is + # the real shape the outbox produces and the case the period gate exists + # for. One is large enough to exercise gzip. + for index, metric_count in ((0, 3), (1, 140)): + package_id = str(uuid.uuid4()) + payload = { + "schema_version": "hermes.shared_metrics.v2", + "package_id": package_id, + "install_id": real_install_id, + "generated_at": datetime.now(timezone.utc).isoformat().replace( + "+00:00", "Z" + ), + "period_start": f"{today}T00:00:00Z", + "period_end": f"{today}T23:59:59Z", + "resource": { + "hermes_version": "e2e-test", + "os_family": "macos", + "architecture": "arm64", + "install_method": "git", + }, + "metrics": [ + { + "name": f"hermes.e2e.metric.{i}", + "type": "counter", + "dimensions": {"outcome": "ok", "surface": "e2e"}, + "value": i + 1, + } + for i in range(metric_count) + ], + } + with store._connection() as connection: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES (?, ?, ?, ?, ?, ?) + """, + ( + package_id, + f"{today}T00:00:00Z", + f"{today}T23:59:59Z", + json.dumps(payload), + f"{today}T0{index}:00:00Z", + f"{today}T0{index}:00:01Z", + ), + ) + packages.append((package_id, metric_count)) + + print(f"scratch HERMES_HOME : {scratch}") + print(f"endpoint : {STAGING}") + print(f"local install_id : {real_install_id}") + print(f"packages queued : {len(packages)}") + for package_id, count in packages: + print(f" - {package_id} ({count} metrics)") + print() + + outcome = SharedMetricsSender(store, STAGING).send_pending() + print(f"outcome: sent={outcome.sent} rejected={outcome.rejected} " + f"deferred={outcome.deferred}") + print() + + failures = [] + with store._connection() as connection: + rows = connection.execute( + """ + SELECT package_id, send_state, sent_at, send_attempts, + sent_install_id, last_error + FROM package_outbox ORDER BY created_at + """ + ).fetchall() + + for row in rows: + print(f"package : {row[0]}") + print(f" send_state : {row[1]}") + print(f" sent_at : {row[2]}") + print(f" attempts : {row[3]}") + print(f" transmitted : {row[4]}") + print(f" last_error : {row[5]}") + if row[1] != "sent": + failures.append(f"{row[0]} is {row[1]}: {row[5]}") + if row[4] == real_install_id: + failures.append(f"{row[0]} LEAKED the real install_id") + if not row[4] or len(str(row[4])) != 64: + failures.append(f"{row[0]} has a malformed derived id") + print() + + if failures: + print("FAILURES:") + for failure in failures: + print(f" ✗ {failure}") + return 1 + + print("PASS: every package acknowledged 202 with a derived identifier.") + print() + print("Verify the objects in S3 with the package ids above:") + print(" aws s3 ls --recursive " + "s3://hermes-agent-telemetry-staging-767397871023-us-west-2-an/raw/ " + "| tail -20") + return 0 + + +if __name__ == "__main__": + raise SystemExit(main()) From 49757d5e397374a50f4aeae27885fa6d29fafdd0 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 16:31:52 +1000 Subject: [PATCH 07/20] fix(telemetry): address review findings on the shared-metrics sender Independent review found the claim mechanism did not work. Reproduced against the real store: two senders POSTed the same package. The claim wrote next_attempt_at = now, but selection requires next_attempt_at <= now, so a concurrent pass matched the same row immediately. It now writes a LEASE INTO THE FUTURE (_CLAIM_LEASE_SECONDS), which is what actually excludes another pass, and expires by itself if a process dies mid-send. _mark is additionally guarded on send_state so a straggler whose lease lapsed cannot overwrite a completed send back to pending. The old concurrency test could not fail: it raised AssertionError from inside a transport, and _send_one catches every exception as a retryable transport error. It now records what the second pass saw. Also from review: - shutdown() never joined the send thread; the join was only wired into deactivate(). A short-lived CLI therefore killed an in-flight send at exit, on the only cadence this feature has. - Removed HERMES_TELEMETRY_ENDPOINT. AGENTS.md reserves HERMES_* for secrets, and a behavioural override here was a consent hazard: an inherited variable could silently redirect telemetry a user agreed to send to Nous. The staging E2E writes the endpoint into its throwaway profile instead, which also exercises the real config path. - Added the shared-metrics toggle that AGENTS.md requires as the third opt-in surface, delegating to the setup prompt so the consent rules stay in one place. - Non-429 4xx (401/403/404/413/422) are now permanent. Only 400 was, so a wrong path or oversized body retried every 15 minutes for 30 days until retention pruned it. - The opt-in day is stamped when the user consents, not on the first send pass, which silently dropped the opt-in day whenever the next export crossed midnight UTC. - gzip now uses mtime=0. The embedded timestamp made two sends of one package differ on the wire, so the 'byte-identical retry' E2E was comparing parsed bodies and could not have caught it. It now compares raw request bytes. - Reconciled the three stale claims in relay-shared-metrics.md that said no remote-delivery path exists. 233 tests pass (was 213). Staging E2E re-run through the config path: both packages 202, and the service logged both objects written to S3. --- docs/observability/relay-shared-metrics.md | 27 +++--- .../observability/relay_shared_metrics.py | 6 ++ .../shared_metrics_send_config.py | 20 ++--- .../observability/shared_metrics_sender.py | 60 ++++++++++--- hermes_cli/setup.py | 23 +++++ hermes_cli/tools_config.py | 50 ++++++++++- scripts/e2e_shared_metrics_staging.py | 26 +++++- .../test_shared_metrics_send_config.py | 28 +++--- .../test_shared_metrics_send_wiring.py | 38 ++++++++ .../hermes_cli/test_shared_metrics_sender.py | 76 ++++++++++++++-- .../test_shared_metrics_sender_e2e.py | 15 ++++ .../test_shared_metrics_tools_toggle.py | 89 +++++++++++++++++++ 12 files changed, 405 insertions(+), 53 deletions(-) create mode 100644 tests/hermes_cli/test_shared_metrics_tools_toggle.py diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index 5b5ce0f8d4..98891103ec 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -33,8 +33,12 @@ than downloading a different implementation. When Relay managed execution is active, the provider request and response pass through that native module in the Hermes process so configured interceptors can operate on the real call. This is separate from the shared-metrics data -contract. Shared-metrics mode installs no network exporter and its subscriber -accepts only the versioned, allowlisted projection described below. Enabling a +contract. Shared-metrics mode installs no rich-observability network exporter, +and its subscriber +accepts only the versioned, allowlisted projection described below. The +opt-in package sender described in Appendix A is the only outbound path, it +transmits nothing unless the user enables both `enabled` and `send`, and it +sends whole packages rather than live spans. Enabling a separately configured rich-observability or dynamic plugin can create a different data path and requires its own policy review. @@ -226,17 +230,17 @@ packages from that profile and can therefore link those local packages. Deleting `$HERMES_HOME/telemetry/shared_metrics` resets the identifier together with all aggregates and package files. -This slice has no remote-delivery path. A future remote exporter must not reuse +Remote delivery is opt-in and off by default. A remote exporter must not reuse the persistent local identifier by default. It requires a separate product and privacy decision covering consent, identity scope, rotation or keyed pseudonymization, reset behavior, retention, and deletion. -> That exporter is now being built as Phase 2 of the Hermes telemetry project. -> The decisions this paragraph asks for are recorded in -> [Appendix A](#appendix-a-remote-exporter-decisions-phase-2). Until Phase 2 -> ships, the statement above still describes shipped behaviour: nothing is -> transmitted, and transmission stays opt-in behind a config key that is off by -> default. +> Those decisions are recorded in +> [Appendix A](#appendix-a-remote-exporter-decisions-phase-2), and the exporter +> implementing them has shipped. Collection alone still transmits nothing: the +> sender runs only when `telemetry.shared_metrics.send` is also true, and it +> transmits a rotating HMAC of the install identity rather than the identifier +> itself. The install identity is scoped to one `HERMES_HOME`. To reset it, stop Hermes processes and remove `$HERMES_HOME/telemetry/shared_metrics`. This deliberately @@ -267,10 +271,13 @@ ID, tool-result, and skill-name canaries are absent from the packages. ## Appendix A: Remote Exporter Decisions (Phase 2) -Status: **decided, not yet built.** This appendix answers the product and +Status: **implemented.** This appendix answers the product and privacy questions that "Current Slices" defers to a future remote exporter. It records what was decided and why, so the reasoning survives the implementation. +Sending is off by default and requires both `telemetry.shared_metrics.enabled` +and `telemetry.shared_metrics.send`. + The exporter sends the package files already written under `$HERMES_HOME/telemetry/shared_metrics/outbox/` to the Hermes telemetry ingest service. That service validates only the envelope (`schema_version` plus a UUID diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index cb4eb44267..c3097114d9 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -671,6 +671,12 @@ class _Runtime: self._safe(self.relay.subscribers.deregister, self._subscriber_name) self.host.release_managed_execution(self._subscriber_name) self._registered = False + # The final export above may have started a send. Give it the same + # bounded chance to finish that deactivate() gets — without this a + # short-lived CLI process exits immediately and kills the daemon + # thread mid-request, which is the common case for the one cadence + # this feature has. + self._join_send_thread() try: atexit.unregister(self.shutdown) except Exception: diff --git a/hermes_cli/observability/shared_metrics_send_config.py b/hermes_cli/observability/shared_metrics_send_config.py index 8011c595ab..cb14027593 100644 --- a/hermes_cli/observability/shared_metrics_send_config.py +++ b/hermes_cli/observability/shared_metrics_send_config.py @@ -9,20 +9,21 @@ identity, rotation, retention, and deletion decisions behind this module. from __future__ import annotations import logging -import os from dataclasses import dataclass from urllib.parse import urlparse logger = logging.getLogger(__name__) -#: Production ingest endpoint. Overridable by config or environment so the -#: live E2E can target staging without mutating a user's config. +#: Production ingest endpoint. Overridable through config only. +#: +#: Deliberately NOT overridable by an environment variable: AGENTS.md reserves +#: HERMES_* env vars for secrets, and a behavioural override here would be a +#: consent hazard — a user who agreed to send metrics to Nous could have them +#: silently redirected to any host by an inherited variable, with nothing +#: visible in their config to show it. Tests and the staging E2E write this +#: key into a throwaway profile instead. DEFAULT_ENDPOINT = "https://telemetry.nousresearch.com/v1/telemetry" -#: Environment override, highest precedence. Intended for tests and staging -#: validation, not as the documented user-facing setting (which is config). -ENDPOINT_ENV_VAR = "HERMES_TELEMETRY_ENDPOINT" - _LOCAL_HOSTS = frozenset({"localhost", "127.0.0.1", "::1", "[::1]"}) # Module-level latch: the enabled/send mismatch is a static misconfiguration, @@ -62,8 +63,7 @@ def _endpoint_is_safe(endpoint: str) -> bool: def resolve_send_config(config: dict | None) -> SendConfig: """Resolve transmission settings from config plus the environment. - Endpoint precedence: ``HERMES_TELEMETRY_ENDPOINT`` > config > production - default. + Endpoint precedence: config > production default. ``send`` is returned as False whenever transmission cannot legitimately happen, so callers never have to re-check the combination. @@ -92,7 +92,7 @@ def resolve_send_config(config: dict | None) -> SendConfig: ) return SendConfig(enabled=False, send=False, endpoint=DEFAULT_ENDPOINT) - endpoint = os.environ.get(ENDPOINT_ENV_VAR) or shared.get("endpoint") + endpoint = shared.get("endpoint") if not isinstance(endpoint, str) or not endpoint.strip(): endpoint = DEFAULT_ENDPOINT endpoint = endpoint.strip() diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 9086c3b359..a55cbf9f7d 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -31,7 +31,7 @@ import time import urllib.error import urllib.request from dataclasses import dataclass -from datetime import datetime, timezone +from datetime import datetime, timedelta, timezone from hermes_cli.sqlite_util import write_txn @@ -58,6 +58,13 @@ GZIP_THRESHOLD_BYTES = 4096 #: Packages per pass. Bounds work on an interactive hook even after an outage. MAX_PACKAGES_PER_PASS = 20 +#: How long a claimed row is held by the claiming pass. A claim writes a +#: LEASE INTO THE FUTURE: another process selecting on `next_attempt_at <= now` +#: therefore skips it. Long enough to cover three attempts plus backoff +#: (1+5+25s of jitter plus three 30s timeouts), short enough that a killed +#: process's rows become eligible again quickly. +_CLAIM_LEASE_SECONDS = 180 + #: Floor applied after a pass fails to deliver, so a hard-down service is not #: retried on every task completion. _FAILURE_BACKOFF_SECONDS = 15 * 60 @@ -100,7 +107,12 @@ def _post(endpoint: str, payload: bytes, *, timeout: int) -> _Response: } body = payload if len(payload) > GZIP_THRESHOLD_BYTES: - body = gzip.compress(payload) + # mtime=0: gzip embeds a timestamp by default, which would make two + # sends of one package differ on the wire. The service decompresses + # before storing so it would not change what lands in S3, but a + # deterministic body keeps "a resend is byte-identical" true at the + # transport layer too, and makes the property testable. + body = gzip.compress(payload, mtime=0) headers["Content-Encoding"] = "gzip" request = urllib.request.Request( @@ -185,6 +197,7 @@ class SharedMetricsSender: """ period = opt_in_period(connection, now=now) stamp = _isoformat(now) + lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) rows = connection.execute( """ SELECT package_id, payload_json, sent_install_id @@ -243,9 +256,14 @@ class SharedMetricsSender: next_attempt_at = ? WHERE package_id = ? """, - # Hold the row for the duration of this pass; success or a - # real backoff overwrite this immediately below. - (_isoformat(now), package_id), + # Lease the row INTO THE FUTURE. Selection above requires + # next_attempt_at <= now, so for the length of the lease no + # other process can claim this package. Writing `now` here (as + # an earlier revision did) claimed nothing: a concurrent pass + # matched the same predicate immediately and sent a duplicate. + # Success or a real backoff overwrites this below; if this + # process dies mid-pass, the lease simply expires. + (_isoformat(lease_until), package_id), ) claimed.append( { @@ -268,12 +286,24 @@ class SharedMetricsSender: payload = substitute_install_id(json.loads(payload_json), derived) return json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") - def _mark(self, package_id: str, **columns) -> None: + def _mark(self, package_id: str, *, only_if_pending: bool = True, **columns) -> None: + """Write send state for one package. + + Guarded on send_state so a pass whose lease lapsed cannot resurrect a + row another process has already finished: without this, a slow sender + could overwrite 'sent' back to 'pending' and cause a re-send. + """ assignments = ", ".join(f"{name} = ?" for name in columns) + predicate = ( + " AND (send_state IS NULL OR send_state = 'pending')" + if only_if_pending + else "" + ) with self._store._connection() as connection: with write_txn(connection): connection.execute( - f"UPDATE package_outbox SET {assignments} WHERE package_id = ?", + f"UPDATE package_outbox SET {assignments} " + f"WHERE package_id = ?{predicate}", (*columns.values(), package_id), ) @@ -315,17 +345,23 @@ class SharedMetricsSender: ) return "sent" - if response.status == 400: - # Permanent per the contract. Keep the file (it is the user's - # history) but never try again. + if response.status == 400 or ( + 400 <= response.status < 500 and response.status != 429 + ): + # The contract only names 400, but every other 4xx is equally + # permanent for an unauthenticated fire-and-forget sender: a + # wrong path (404), an edge rejection (403), or an oversized + # body (413) will not fix itself by being retried every 15 + # minutes until local retention prunes the package. logger.warning( - "Telemetry package %s rejected as malformed; not retrying", + "Telemetry package %s rejected with HTTP %s; not retrying", package_id, + response.status, ) self._mark( package_id, send_state="rejected", - last_error=response.body[:500], + last_error=f"HTTP {response.status}: {response.body[:400]}", ) return "rejected" diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index d6497fbc05..d7971e14a5 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2468,11 +2468,34 @@ def setup_telemetry(config: dict): default=shared_metrics.get("send") is True, ) if shared_metrics["send"]: + _record_send_opt_in_day() print_success("Sending shared metrics enabled.") else: print_info("Sending shared metrics disabled (collection stays local).") +def _record_send_opt_in_day() -> None: + """Stamp the consent day when the user says yes, not at first send. + + The gate excludes packages for periods before this day. Recording it + lazily on the first send pass would silently drop the opt-in day itself + whenever the next export happens after midnight UTC. + """ + try: + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_sender import opt_in_period + from hermes_cli.sqlite_util import write_txn + + store = SharedMetricsStore() + with store._connection() as connection: + with write_txn(connection): + opt_in_period(connection) + except Exception: + # Never block the wizard on telemetry bookkeeping; the sender still + # records the day on its first pass if this could not run. + logger.debug("Unable to record shared-metrics opt-in day", exc_info=True) + + # ============================================================================= # Post-Migration Section Skip Logic # ============================================================================= diff --git a/hermes_cli/tools_config.py b/hermes_cli/tools_config.py index 018178d916..3f4b160965 100644 --- a/hermes_cli/tools_config.py +++ b/hermes_cli/tools_config.py @@ -5570,6 +5570,43 @@ def _reconfigure_simple_requirements(ts_key: str): # ─── Main Entry Point ───────────────────────────────────────────────────────── +def _shared_metrics_state(config: dict) -> tuple[bool, bool]: + """Return (collection_enabled, send_enabled) from a config dict.""" + telemetry = config.get("telemetry") + telemetry = telemetry if isinstance(telemetry, dict) else {} + shared = telemetry.get("shared_metrics") + shared = shared if isinstance(shared, dict) else {} + return shared.get("enabled") is True, shared.get("send") is True + + +def _shared_metrics_menu_label(config: dict) -> str: + """Menu row for shared metrics, showing both consent states.""" + enabled, send = _shared_metrics_state(config) + if not enabled: + state = "off" + elif send: + state = "collecting + sending to Nous" + else: + state = "collecting locally" + return f"Configure shared metrics ({state})" + + +def _configure_shared_metrics_interactive(config: dict) -> None: + """Toggle shared-metrics collection and sending from `hermes tools`. + + Delegates to the setup wizard's prompt so the consent rules live in one + place: sending requires collection, and turning collection off also turns + sending off. + """ + from hermes_cli.setup import setup_telemetry + + before = _shared_metrics_state(config) + setup_telemetry(config) + after = _shared_metrics_state(config) + if before != after: + save_config(config) + + def tools_command(args=None, first_install: bool = False, config: dict = None): """Entry point for `hermes tools` and `hermes setup tools`. @@ -5694,6 +5731,7 @@ def tools_command(args=None, first_install: bool = False, config: dict = None): if len(platform_keys) > 1: platform_choices.append("Configure all platforms (global)") platform_choices.append("Reconfigure an existing tool's provider or API key") + platform_choices.append(_shared_metrics_menu_label(config)) # Show MCP option if any MCP servers are configured _has_mcp = bool(config.get("mcp_servers")) @@ -5705,8 +5743,9 @@ def tools_command(args=None, first_install: bool = False, config: dict = None): # Index offsets for the extra options after per-platform entries _global_idx = len(platform_keys) if len(platform_keys) > 1 else -1 _reconfig_idx = len(platform_keys) + (1 if len(platform_keys) > 1 else 0) - _mcp_idx = (_reconfig_idx + 1) if _has_mcp else -1 - _done_idx = _reconfig_idx + (2 if _has_mcp else 1) + _metrics_idx = _reconfig_idx + 1 + _mcp_idx = (_metrics_idx + 1) if _has_mcp else -1 + _done_idx = _metrics_idx + (2 if _has_mcp else 1) while True: idx = _prompt_choice("Select an option:", platform_choices, default=0) @@ -5721,6 +5760,13 @@ def tools_command(args=None, first_install: bool = False, config: dict = None): print() continue + # "Shared metrics" selected + if idx == _metrics_idx: + _configure_shared_metrics_interactive(config) + platform_choices[_metrics_idx] = _shared_metrics_menu_label(config) + print() + continue + # "Configure MCP tools" selected if idx == _mcp_idx: _configure_mcp_tools_interactive(config) diff --git a/scripts/e2e_shared_metrics_staging.py b/scripts/e2e_shared_metrics_staging.py index 55e03a7572..6adddcec68 100644 --- a/scripts/e2e_shared_metrics_staging.py +++ b/scripts/e2e_shared_metrics_staging.py @@ -28,9 +28,33 @@ def main() -> int: scratch = Path(tempfile.mkdtemp(prefix="hermes-telemetry-e2e-")) os.environ["HERMES_HOME"] = str(scratch) + # Staging is selected by writing config into the THROWAWAY profile, not by + # an environment override: a runtime env var that can retarget consented + # telemetry would be a consent hazard in production. + (scratch / "config.yaml").write_text( + "telemetry:\n" + " shared_metrics:\n" + " enabled: true\n" + " send: true\n" + f" endpoint: {STAGING}\n" + ) + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_send_config import ( + resolve_send_config, + ) from hermes_cli.observability.shared_metrics_sender import SharedMetricsSender + # Resolve through the real config path so this exercises what a user gets. + import yaml + + resolved = resolve_send_config( + yaml.safe_load((scratch / "config.yaml").read_text()) + ) + if not resolved.send or resolved.endpoint != STAGING: + print(f"FAIL: config did not resolve to staging: {resolved}") + return 1 + store = SharedMetricsStore( database_path=scratch / "metrics.sqlite3", outbox_directory=scratch / "outbox", @@ -97,7 +121,7 @@ def main() -> int: print(f" - {package_id} ({count} metrics)") print() - outcome = SharedMetricsSender(store, STAGING).send_pending() + outcome = SharedMetricsSender(store, resolved.endpoint).send_pending() print(f"outcome: sent={outcome.sent} rejected={outcome.rejected} " f"deferred={outcome.deferred}") print() diff --git a/tests/hermes_cli/test_shared_metrics_send_config.py b/tests/hermes_cli/test_shared_metrics_send_config.py index 235777d49a..227c7a1cfa 100644 --- a/tests/hermes_cli/test_shared_metrics_send_config.py +++ b/tests/hermes_cli/test_shared_metrics_send_config.py @@ -9,7 +9,6 @@ import pytest from hermes_cli.config import DEFAULT_CONFIG from hermes_cli.observability.shared_metrics_send_config import ( DEFAULT_ENDPOINT, - ENDPOINT_ENV_VAR, resolve_send_config, reset_warning_latch_for_tests, ) @@ -84,22 +83,29 @@ class TestEndpointPrecedence: ) assert resolved.endpoint == "https://example.test/v1" - def test_env_var_overrides_config(self, monkeypatch): - monkeypatch.setenv(ENDPOINT_ENV_VAR, "https://staging.test/v1") - resolved = resolve_send_config( - _config(enabled=True, send=True, endpoint="https://example.test/v1") - ) - assert resolved.endpoint == "https://staging.test/v1" + def test_no_environment_variable_can_redirect_telemetry(self, monkeypatch): + """A consent hazard: an inherited env var must not silently retarget. + + AGENTS.md also reserves HERMES_* for secrets, not behaviour. + """ + for name in ( + "HERMES_TELEMETRY_ENDPOINT", + "TELEMETRY_ENDPOINT", + "HERMES_SHARED_METRICS_ENDPOINT", + ): + monkeypatch.setenv(name, "https://attacker.test/v1") + resolved = resolve_send_config(_config(enabled=True, send=True)) + assert resolved.endpoint == DEFAULT_ENDPOINT def test_blank_endpoint_falls_back_to_production(self): resolved = resolve_send_config(_config(enabled=True, send=True, endpoint=" ")) assert resolved.endpoint == DEFAULT_ENDPOINT - def test_endpoint_is_stripped(self, monkeypatch): - monkeypatch.setenv(ENDPOINT_ENV_VAR, " https://staging.test/v1 ") - assert resolve_send_config(_config(enabled=True, send=True)).endpoint == ( - "https://staging.test/v1" + def test_endpoint_is_stripped(self): + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint=" https://staging.test/v1 ") ) + assert resolved.endpoint == "https://staging.test/v1" class TestTransportSafety: diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py index 730376083f..8b90c4f812 100644 --- a/tests/hermes_cli/test_shared_metrics_send_wiring.py +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -214,3 +214,41 @@ class TestFailureIsolation: def test_join_is_safe_with_no_thread(self, runtime): runtime._join_send_thread(timeout=0.1) + + def test_join_waits_for_an_in_flight_send(self, runtime, monkeypatch): + """shutdown() must give a started send a chance to finish. + + A short-lived CLI exits straight after its final export; without the + join the daemon thread is killed mid-request, and the hook path is the + only delivery cadence this feature has. + """ + finished = [] + release = threading.Event() + + class SlowSender: + def __init__(self, store, endpoint, **kwargs): + pass + + def send_pending(self): + release.wait(3) + finished.append(True) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + SlowSender, + ) + _set_config(monkeypatch, _config(enabled=True, send=True)) + + runtime._export() + release.set() + runtime._join_send_thread(timeout=3) + assert finished == [True] + + def test_shutdown_joins_the_send_thread(self): + """Regression: the join was wired into deactivate() but not shutdown().""" + import inspect + + source = inspect.getsource(mod._Runtime.shutdown) + assert "_join_send_thread" in source, ( + "shutdown() must join the sender, or a CLI exit kills it mid-send" + ) diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index d0fec15bfa..4da879dfc3 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -148,6 +148,18 @@ class TestContractResponses: _sender(store, transport2).send_pending() assert transport2.calls == [] + @pytest.mark.parametrize("status", [401, 403, 404, 413, 422]) + def test_other_4xx_are_permanent_too(self, store, status): + """Retrying these every 15 minutes for 30 days fixes nothing.""" + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(status)) + outcome = _sender(store, transport).send_pending() + assert outcome.rejected == 1 + assert len(transport.calls) == 1 + row = _row(store, "pkg-1") + assert row["send_state"] == "rejected" + assert str(status) in row["last_error"] + def test_429_defers_using_retry_after(self, store): _add_package(store, "pkg-1", "2026-08-26") transport = FakeTransport(FakeResponse(429, retry_after="120")) @@ -342,27 +354,77 @@ class TestClaimingAndBounds: assert outcome.sent == MAX_PACKAGES_PER_PASS def test_two_concurrent_passes_do_not_double_send(self, store): - """Claiming is what stops two Hermes processes duplicating work.""" + """Claiming is what stops two Hermes processes duplicating work. + + The second pass must RECORD what it saw rather than raise: _send_one + catches every exception as a retryable transport failure, so an + assertion thrown inside a transport would be swallowed and this test + would pass no matter what the claim did. + """ _add_package(store, "pkg-1", "2026-08-26") - seen = [] + first_calls = [] + second_calls = [] + + def second_transport(endpoint, payload, *, timeout): + second_calls.append(payload) + return FakeResponse(202) def transport(endpoint, payload, *, timeout): - seen.append(payload) + first_calls.append(payload) # A second sender runs while the first is mid-flight. SharedMetricsSender( store, ENDPOINT, - post=lambda *a, **k: (_ for _ in ()).throw( - AssertionError("second pass must not claim a held package") - ), + post=second_transport, sleep=lambda _s: None, now=lambda: NOW, ).send_pending() return FakeResponse(202) _sender(store, transport).send_pending() - assert len(seen) == 1 + assert len(first_calls) == 1 + assert second_calls == [], ( + "a concurrent pass claimed a package already in flight" + ) + + def test_a_claim_leases_the_row_into_the_future(self, store): + """The lease, not the send result, is what blocks a concurrent pass.""" + _add_package(store, "pkg-1", "2026-08-26") + with store._connection() as connection: + with __import__( + "hermes_cli.sqlite_util", fromlist=["write_txn"] + ).write_txn(connection): + claimed = _sender(store, FakeTransport())._claim(connection, NOW) + assert len(claimed) == 1 + assert _row(store, "pkg-1")["next_attempt_at"] > "2026-08-26T12:00:00Z" + + def test_an_expired_lease_is_reclaimed(self, store): + """A process killed mid-pass must not strand its packages.""" + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(OSError("killed"), OSError(""), OSError(""))).send_pending() + + later = SharedMetricsSender( + store, + ENDPOINT, + post=(transport := FakeTransport(FakeResponse(202))), + sleep=lambda _s: None, + now=lambda: NOW + timedelta(hours=2), + ) + later.send_pending() + assert len(transport.calls) == 1 + + def test_a_lapsed_sender_cannot_resurrect_a_sent_package(self, store): + """Terminal state must win over a straggler's write.""" + _add_package(store, "pkg-1", "2026-08-26") + _sender(store, FakeTransport(FakeResponse(202))).send_pending() + assert _row(store, "pkg-1")["send_state"] == "sent" + + # A straggler from an earlier pass tries to defer the same row. + _sender(store, FakeTransport())._defer("pkg-1", 600, "stale") + assert _row(store, "pkg-1")["send_state"] == "sent", ( + "a lapsed pass overwrote a completed send" + ) class TestResilience: diff --git a/tests/hermes_cli/test_shared_metrics_sender_e2e.py b/tests/hermes_cli/test_shared_metrics_sender_e2e.py index 8568a9e6fe..552ae93552 100644 --- a/tests/hermes_cli/test_shared_metrics_sender_e2e.py +++ b/tests/hermes_cli/test_shared_metrics_sender_e2e.py @@ -40,6 +40,9 @@ class Ingest(BaseHTTPRequestHandler): { "headers": {k.lower(): v for k, v in self.headers.items()}, "body": json.loads(body.decode("utf-8")), + # Keep the RAW request bytes: comparing only the parsed body + # would not notice a non-deterministic transport encoding. + "raw": raw, "raw_len": len(raw), "decoded_len": len(body), } @@ -212,6 +215,18 @@ class TestRealTransport: _sender(store, server).send_pending() first, second = Ingest.received assert first["body"] == second["body"] + assert first["raw"] == second["raw"], ( + "the raw request bytes must match, not just the parsed body" + ) + + def test_a_gzipped_retry_is_byte_identical_on_the_wire(self, store, server): + """gzip embeds an mtime by default, which would break this.""" + _add(store, "pkg-1", metrics=200) + Ingest.script = [(503, {}, {}), (202, {}, {})] + _sender(store, server).send_pending() + first, second = Ingest.received + assert first["headers"].get("content-encoding") == "gzip" + assert first["raw"] == second["raw"] def test_several_packages_in_one_pass(self, store, server): for i in range(5): diff --git a/tests/hermes_cli/test_shared_metrics_tools_toggle.py b/tests/hermes_cli/test_shared_metrics_tools_toggle.py new file mode 100644 index 0000000000..31718d5d18 --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_tools_toggle.py @@ -0,0 +1,89 @@ +"""Tests for the `hermes tools` shared-metrics consent toggle. + +AGENTS.md requires outbound telemetry to be reachable from a config gate, the +setup prompt, AND `hermes tools`. These cover the third surface. +""" + +from __future__ import annotations + +import pytest + +from hermes_cli.tools_config import ( + _configure_shared_metrics_interactive, + _shared_metrics_menu_label, + _shared_metrics_state, +) + + +def _config(**shared): + return {"telemetry": {"shared_metrics": shared}} + + +class TestState: + def test_missing_telemetry_section_is_off(self): + assert _shared_metrics_state({}) == (False, False) + + def test_malformed_section_does_not_raise(self): + assert _shared_metrics_state({"telemetry": "nonsense"}) == (False, False) + + def test_reads_both_flags(self): + assert _shared_metrics_state(_config(enabled=True, send=True)) == (True, True) + + +class TestMenuLabel: + def test_off_state(self): + assert "off" in _shared_metrics_menu_label({}) + + def test_local_only_state(self): + label = _shared_metrics_menu_label(_config(enabled=True)) + assert "collecting locally" in label + assert "Nous" not in label + + def test_sending_state_names_the_destination(self): + label = _shared_metrics_menu_label(_config(enabled=True, send=True)) + assert "sending to Nous" in label + + +class TestToggle: + def test_enabling_send_persists(self, monkeypatch): + config = _config(enabled=True) + saved = {} + monkeypatch.setattr( + "hermes_cli.setup.prompt_yes_no", lambda *_a, **_k: True + ) + monkeypatch.setattr( + "hermes_cli.setup._record_send_opt_in_day", lambda: None + ) + monkeypatch.setattr( + "hermes_cli.tools_config.save_config", + lambda cfg: saved.update({"cfg": cfg}), + ) + _configure_shared_metrics_interactive(config) + assert config["telemetry"]["shared_metrics"]["send"] is True + assert saved, "a consent change must be written to disk" + + def test_no_write_when_nothing_changed(self, monkeypatch): + config = _config(enabled=False, send=False) + saved = [] + monkeypatch.setattr( + "hermes_cli.setup.prompt_yes_no", lambda *_a, **_k: False + ) + monkeypatch.setattr( + "hermes_cli.tools_config.save_config", lambda cfg: saved.append(cfg) + ) + _configure_shared_metrics_interactive(config) + assert saved == [] + + def test_disabling_collection_also_disables_sending(self, monkeypatch): + """The toggle must not leave send=true with nothing to send.""" + config = _config(enabled=True, send=True) + monkeypatch.setattr( + "hermes_cli.setup.prompt_yes_no", lambda *_a, **_k: False + ) + monkeypatch.setattr( + "hermes_cli.tools_config.save_config", lambda cfg: None + ) + _configure_shared_metrics_interactive(config) + shared = config["telemetry"]["shared_metrics"] + assert shared["enabled"] is False + assert shared["send"] is False From be74fdc137d7b1e70953cb0b04a7a835ec7b9df7 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 16:39:42 +1000 Subject: [PATCH 08/20] fix(telemetry): pass encoding=utf-8 in the staging E2E config I/O MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows footgun checker caught bare Path.read_text()/write_text() in the config I/O I added in the previous commit. Without encoding=, Python uses locale.getpreferredencoding() — cp1252/cp936 on Windows — so a UTF-8 config crashes or writes mojibake. This was the single root cause of both red checks: the blocking lint job and tests/scripts/test_windows_footguns_full_repo_scan.py, which runs the same checker over the repo. Everything else was green (38,447 passed, 1 failed). Verified locally: the checker now reports no footguns across 1019 files, and the full-repo-scan test passes. --- scripts/e2e_shared_metrics_staging.py | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/scripts/e2e_shared_metrics_staging.py b/scripts/e2e_shared_metrics_staging.py index 6adddcec68..3e52e61a7a 100644 --- a/scripts/e2e_shared_metrics_staging.py +++ b/scripts/e2e_shared_metrics_staging.py @@ -36,7 +36,8 @@ def main() -> int: " shared_metrics:\n" " enabled: true\n" " send: true\n" - f" endpoint: {STAGING}\n" + f" endpoint: {STAGING}\n", + encoding="utf-8", ) from hermes_cli.observability.shared_metrics import SharedMetricsStore @@ -49,7 +50,7 @@ def main() -> int: import yaml resolved = resolve_send_config( - yaml.safe_load((scratch / "config.yaml").read_text()) + yaml.safe_load((scratch / "config.yaml").read_text(encoding="utf-8")) ) if not resolved.send or resolved.endpoint != STAGING: print(f"FAIL: config did not resolve to staging: {resolved}") From d0a7144ba184ab25a7e571d4ebf0df1e7a95cca1 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 26 Aug 2026 17:10:43 +1000 Subject: [PATCH 09/20] fix(telemetry): per-row claiming, mid-pass consent re-check, narrower 4xx MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Second independent review found the lease fix incomplete. Reproduced each finding before fixing. BLOCKER — the batch lease expired mid-pass. _claim took up to 20 rows under ONE shared lease, but a single package can legally consume ~96s (three 30s timeouts plus 1s+5s backoff), so a full batch runs ~1900s against a 180s lease. Later rows' leases expired while this pass still held them, and another process re-sent them. Reproduced: 192s elapsed, pkg-2 POSTed twice. Packages are now claimed ONE AT A TIME, immediately before being sent, so a lease only has to cover the package actually in flight. Verified: same scenario now sends each package exactly once. HIGH — revoking consent did not stop a running pass. The runtime read send consent once before starting the thread, so a pass could keep transmitting for minutes after a user set send: false, contradicting the documented promise that it 'stops transmission immediately'. Consent is now re-read before every package and fails CLOSED if it cannot be established. MEDIUM — all non-429 4xx were treated as permanent, discarding data. 403 is the ingest service's own origin guard: a Transform Rule or edge misconfiguration would have permanently dropped every package sent during the incident. Only 400 (malformed envelope) and 413 (over the 1 MiB cap) are terminal now; everything else retries. MEDIUM — valid JSON that is not an object blocked the whole queue. json.loads('["a"]') succeeds, then .get() raised AttributeError inside the claim transaction, rolling it back and starving every healthy package behind it. Payload shape and install_id are now validated, and an unusable row is rejected individually. LOW — the clock-rollback comment and test name claimed the opposite of the code. The behaviour is right (a future issued_at means the recorded age is untrustworthy, so reissue); the wording is now honest about it. LOW — removed the stale HERMES_TELEMETRY_ENDPOINT reference left in config_defaults after the override was deleted. 247 tests pass (was 234). Staging E2E re-run: both packages 202. --- docs/observability/relay-shared-metrics.md | 4 +- hermes_cli/config_defaults.py | 8 +- .../observability/relay_shared_metrics.py | 14 +- .../observability/shared_metrics_identity.py | 8 +- .../observability/shared_metrics_sender.py | 294 ++++++++++++------ .../test_shared_metrics_identity.py | 10 +- .../hermes_cli/test_shared_metrics_sender.py | 176 ++++++++++- 7 files changed, 387 insertions(+), 127 deletions(-) diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index 98891103ec..b965eaa6c8 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -369,7 +369,9 @@ qualifications now apply: service's storage under their derived identifier. There is no read-back or delete API in the v1 contract. -Setting `send: false` stops transmission immediately. It does not delete +Setting `send: false` stops transmission immediately: consent is re-read +before every package, so a pass already in flight stops after the package it +is currently sending rather than draining its whole batch. It does not delete previously transmitted packages, and it does not stop local collection. ### A.5 Retention diff --git a/hermes_cli/config_defaults.py b/hermes_cli/config_defaults.py index cf321b4e3d..5303e020df 100644 --- a/hermes_cli/config_defaults.py +++ b/hermes_cli/config_defaults.py @@ -3339,10 +3339,10 @@ DEFAULT_CONFIG = { # before consent stays local. "send": False, # Ingest endpoint. Production by default; override for staging or - # a local test server. The HERMES_TELEMETRY_ENDPOINT environment - # variable takes precedence (used by the live E2E so a test never - # has to mutate a user's config). Non-HTTPS is refused unless the - # host is localhost. + # a local test server. Deliberately NOT overridable by an + # environment variable: that would let an inherited value silently + # redirect telemetry a user consented to send to Nous. Non-HTTPS + # is refused unless the host is localhost. "endpoint": "https://telemetry.nousresearch.com/v1/telemetry", }, }, diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index c3097114d9..94a1eac64f 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -1119,9 +1119,21 @@ class _Runtime: SharedMetricsSender, ) + def still_consented() -> bool: + """Re-read consent so revoking `send` stops an in-flight pass.""" + from hermes_cli.config import read_raw_config_readonly + from hermes_cli.observability.shared_metrics_send_config import ( + resolve_send_config, + ) + + resolved = resolve_send_config(read_raw_config_readonly() or {}) + return resolved.send and resolved.endpoint == endpoint + try: SharedMetricsSender( - self.subscriber.store, endpoint + self.subscriber.store, + endpoint, + consent_check=still_consented, ).send_pending() except Exception: logger.warning("Shared-metrics send pass failed", exc_info=True) diff --git a/hermes_cli/observability/shared_metrics_identity.py b/hermes_cli/observability/shared_metrics_identity.py index 16e28a8b4e..b4f8cda8e3 100644 --- a/hermes_cli/observability/shared_metrics_identity.py +++ b/hermes_cli/observability/shared_metrics_identity.py @@ -92,8 +92,12 @@ def current_salt( fresh = ( salt is not None and issued_at is not None - # A clock that jumped backwards must not be read as "aged out"; a - # future issue time simply means not yet due. + # Strictly within the window. A future issued_at means the clock moved + # backwards (or the value was tampered with), so the recorded age + # cannot be trusted and we reissue rather than keep using a salt of + # unknown vintage. Reissuing is the safe direction: it shortens + # linkability, and already-prepared packages keep their frozen + # identifier so retries stay byte-identical. and issued_at <= moment < issued_at + ROTATION_INTERVAL ) if fresh: diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index a55cbf9f7d..23070355ae 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -58,17 +58,30 @@ GZIP_THRESHOLD_BYTES = 4096 #: Packages per pass. Bounds work on an interactive hook even after an outage. MAX_PACKAGES_PER_PASS = 20 -#: How long a claimed row is held by the claiming pass. A claim writes a -#: LEASE INTO THE FUTURE: another process selecting on `next_attempt_at <= now` -#: therefore skips it. Long enough to cover three attempts plus backoff -#: (1+5+25s of jitter plus three 30s timeouts), short enough that a killed -#: process's rows become eligible again quickly. -_CLAIM_LEASE_SECONDS = 180 +#: How long a claimed row is held. The claim writes a LEASE INTO THE FUTURE: +#: selection requires `next_attempt_at <= now`, so for the length of the lease +#: no other process can take the package. +#: +#: This must exceed the worst case for ONE package — three 30s request +#: timeouts plus 1s+5s of backoff, about 96s — which is why packages are +#: claimed one at a time, immediately before being sent. An earlier revision +#: claimed up to 20 rows under a single shared lease; a full batch can legally +#: run ~1900s, so the later rows' leases expired while the pass still held +#: them in memory and another process re-sent them. +_CLAIM_LEASE_SECONDS = 300 #: Floor applied after a pass fails to deliver, so a hard-down service is not #: retried on every task completion. _FAILURE_BACKOFF_SECONDS = 15 * 60 +#: Statuses that are permanent per the ingest contract. Deliberately narrow: +#: 400 means the envelope is malformed and will never validate. 413 is added +#: because a package over the service's 1 MiB cap cannot shrink on retry. +#: Everything else — including 403 from the origin guard and 404 from a bad +#: path — is retried, because those are usually deployment or edge +#: misconfiguration that resolves without the package changing. +_PERMANENT_STATUSES = frozenset({400, 413}) + OPT_IN_PERIOD_KEY = "send_opt_in_period" @@ -177,6 +190,7 @@ class SharedMetricsSender: sleep=time.sleep, now=_utc_now, max_attempts: int = MAX_ATTEMPTS, + consent_check=None, ) -> None: self._store = store self._endpoint = endpoint @@ -184,95 +198,132 @@ class SharedMetricsSender: self._sleep = sleep self._now = now self._max_attempts = max_attempts + # Called before every package. None disables the check for callers + # that have already established consent out of band (tests, E2E). + self._consent_check = consent_check # -- selection --------------------------------------------------------- - def _claim(self, connection: sqlite3.Connection, now: datetime) -> list[dict]: - """Atomically take ownership of the packages this pass will try. + def _claim_next(self, now: datetime, seen: set[str]) -> dict | None: + """Claim exactly ONE package, immediately before it is sent. - Claiming inside the write transaction is what stops two Hermes - processes sharing one database from sending the same package twice. - Duplicates would be harmless (the service dedupes by package_id and - the bytes are identical) but they waste the user's bandwidth. + Claiming a whole batch up front does not work: a single shared lease + has to cover the entire pass, and 20 retrying packages can legally run + far longer than any sane lease (three 30s timeouts plus backoff each). + The later rows' leases then expire while this pass still holds them in + memory, and another process re-sends them. Taking one row at a time + keeps the lease covering only the package actually in flight. + + ``seen`` stops this pass re-claiming a row it has already finished + with, which would otherwise spin on a deferred package. """ - period = opt_in_period(connection, now=now) - stamp = _isoformat(now) - lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) - rows = connection.execute( - """ - SELECT package_id, payload_json, sent_install_id - FROM package_outbox - WHERE exported_at IS NOT NULL - AND (send_state IS NULL OR send_state = 'pending') - AND (next_attempt_at IS NULL OR next_attempt_at <= ?) - AND substr(period_start, 1, 10) >= ? - ORDER BY created_at, package_id - LIMIT ? - """, - (stamp, period, MAX_PACKAGES_PER_PASS), - ).fetchall() + with self._store._connection() as connection: + with write_txn(connection): + period = opt_in_period(connection, now=now) + stamp = _isoformat(now) + lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) - claimed: list[dict] = [] - salt: str | None = None - for row in rows: - package_id = str(row[0]) - derived = row[2] - if not derived: - # Freeze the derived identity on first attempt so a later salt - # rotation cannot change the bytes sent under this package_id. - if salt is None: - salt = current_salt(connection, now=now) - try: - payload = json.loads(row[1]) - install_id = str(payload.get("install_id", "")) - except (TypeError, ValueError): - # A row we cannot parse can never be sent. Mark it and move - # on: one unreadable package must not block every other - # package behind it, and aborting here would roll back the - # whole claim transaction. - logger.warning( - "Shared-metrics package %s is unreadable; not sending", - package_id, + row = connection.execute( + """ + SELECT package_id, payload_json, sent_install_id + FROM package_outbox + WHERE exported_at IS NOT NULL + AND (send_state IS NULL OR send_state = 'pending') + AND (next_attempt_at IS NULL OR next_attempt_at <= ?) + AND substr(period_start, 1, 10) >= ? + ORDER BY created_at, package_id + LIMIT 1 + """, + (stamp, period), + ).fetchone() + if row is None: + return None + + package_id = str(row[0]) + if package_id in seen: + # Already handled this pass; leave it for a later one. + return None + + derived = row[2] + if not derived: + derived = self._freeze_identity( + connection, package_id, row[1], now ) - connection.execute( - """ - UPDATE package_outbox - SET send_state = 'rejected', last_error = 'unreadable payload' - WHERE package_id = ? - """, - (package_id,), - ) - continue - derived = derive_install_id(install_id, salt) + if derived is None: + # Unusable row, already marked rejected. Signal the + # caller to continue rather than stop. + return {"package_id": package_id, "skip": True} + connection.execute( - "UPDATE package_outbox SET sent_install_id = ? WHERE package_id = ?", - (derived, package_id), + """ + UPDATE package_outbox + SET send_state = 'pending', + send_attempts = send_attempts + 1, + next_attempt_at = ? + WHERE package_id = ? + """, + # Lease INTO THE FUTURE: selection requires + # next_attempt_at <= now, so no other process can take + # this row while it is in flight. Success or a real + # backoff overwrites it; if this process dies, it expires. + (_isoformat(lease_until), package_id), ) - connection.execute( - """ - UPDATE package_outbox - SET send_state = 'pending', - send_attempts = send_attempts + 1, - next_attempt_at = ? - WHERE package_id = ? - """, - # Lease the row INTO THE FUTURE. Selection above requires - # next_attempt_at <= now, so for the length of the lease no - # other process can claim this package. Writing `now` here (as - # an earlier revision did) claimed nothing: a concurrent pass - # matched the same predicate immediately and sent a duplicate. - # Success or a real backoff overwrites this below; if this - # process dies mid-pass, the lease simply expires. - (_isoformat(lease_until), package_id), - ) - claimed.append( - { + return { "package_id": package_id, "payload_json": str(row[1]), "derived": str(derived), + "skip": False, } + + def _freeze_identity( + self, + connection: sqlite3.Connection, + package_id: str, + payload_json, + now: datetime, + ) -> str | None: + """Derive and persist the transmitted id, or reject an unusable row. + + Returns None when the package can never be sent. Rejecting rather than + raising matters: an exception here rolls back the claim transaction + and blocks every healthy package behind this one. + """ + reason = None + try: + payload = json.loads(payload_json) + except (TypeError, ValueError): + reason = "unreadable payload" + else: + # Valid JSON is not enough: a top-level array, string, number or + # null parses cleanly and then has no .get(). + if not isinstance(payload, dict): + reason = f"payload is {type(payload).__name__}, expected object" + else: + install_id = payload.get("install_id") + if not isinstance(install_id, str) or not install_id.strip(): + reason = "payload has no usable install_id" + + if reason is not None: + logger.warning( + "Shared-metrics package %s cannot be sent (%s)", package_id, reason ) - return claimed + connection.execute( + """ + UPDATE package_outbox + SET send_state = 'rejected', last_error = ? + WHERE package_id = ? + """, + (reason, package_id), + ) + return None + + salt = current_salt(connection, now=now) + derived = derive_install_id(payload["install_id"], salt) + connection.execute( + "UPDATE package_outbox SET sent_install_id = ? WHERE package_id = ?", + (derived, package_id), + ) + return derived # -- transmission ------------------------------------------------------ @@ -345,14 +396,13 @@ class SharedMetricsSender: ) return "sent" - if response.status == 400 or ( - 400 <= response.status < 500 and response.status != 429 - ): - # The contract only names 400, but every other 4xx is equally - # permanent for an unauthenticated fire-and-forget sender: a - # wrong path (404), an edge rejection (403), or an oversized - # body (413) will not fix itself by being retried every 15 - # minutes until local retention prunes the package. + if response.status in _PERMANENT_STATUSES: + # Only statuses the contract (or the envelope schema) makes + # terminal. Everything else retries: 403 in particular is the + # ingest service's origin guard, which returns 403 during an + # edge/Transform-Rule misconfiguration — treating that as + # permanent would discard every package sent during the + # incident instead of retrying after recovery. logger.warning( "Telemetry package %s rejected with HTTP %s; not retrying", package_id, @@ -392,24 +442,42 @@ class SharedMetricsSender: # -- entry point ------------------------------------------------------- def send_pending(self) -> SendOutcome: - """Run one bounded pass. Never raises.""" - outcome = SendOutcome() - try: - now = self._now() - with self._store._connection() as connection: - with write_txn(connection): - claimed = self._claim(connection, now) - except Exception: - logger.warning("Unable to select shared-metrics packages", exc_info=True) - return outcome + """Run one bounded pass. Never raises. + + Claims and sends ONE package at a time so each row's lease only has to + cover its own transmission, and re-checks consent before every send so + revoking `send` mid-pass stops the remaining packages. + """ + outcome = SendOutcome() + seen: set[str] = set() + + for _ in range(MAX_PACKAGES_PER_PASS): + if not self._still_consented(): + # The user turned sending off while this pass was running. + # Stop without transmitting anything further; unclaimed rows + # stay pending and claimed-but-unsent rows expire naturally. + logger.info("Shared-metrics sending disabled mid-pass; stopping") + break + try: + package = self._claim_next(self._now(), seen) + except Exception: + logger.warning( + "Unable to select shared-metrics packages", exc_info=True + ) + break + if package is None: + break + + seen.add(package["package_id"]) + if package.get("skip"): + # Unusable row already marked rejected during the claim. + outcome.rejected += 1 + continue - for package in claimed: try: result = self._send_one(package) except Exception: - logger.warning( - "Unable to send shared-metrics package", exc_info=True - ) + logger.warning("Unable to send shared-metrics package", exc_info=True) outcome.deferred += 1 continue if result == "sent": @@ -419,3 +487,23 @@ class SharedMetricsSender: else: outcome.deferred += 1 return outcome + + def _still_consented(self) -> bool: + """Re-read profile-owned send consent. + + Consent is a boundary, not cached configuration: the documentation + promises that setting `send: false` stops transmission immediately, + and a pass can run for minutes. Injected senders (tests, the staging + E2E) opt out by passing consent_check=None. + """ + if self._consent_check is None: + return True + try: + return bool(self._consent_check()) + except Exception: + # Fail CLOSED: if consent cannot be established, do not transmit. + logger.warning( + "Unable to confirm shared-metrics send consent; stopping", + exc_info=True, + ) + return False diff --git a/tests/hermes_cli/test_shared_metrics_identity.py b/tests/hermes_cli/test_shared_metrics_identity.py index 1887d47ccb..1ea1d95961 100644 --- a/tests/hermes_cli/test_shared_metrics_identity.py +++ b/tests/hermes_cli/test_shared_metrics_identity.py @@ -76,11 +76,15 @@ class TestSaltLifecycle: conn.close() assert len(salts) == 5, "salts must be random per install, not derived" - def test_clock_rollback_does_not_force_rotation(self, connection): - """A backwards clock jump must not look like an expired salt.""" + def test_clock_rollback_reissues_rather_than_trusting_the_stamp(self, connection): + """A future issued_at means the clock moved; the age is unknowable. + + Reissuing is the safe direction — it shortens linkability rather than + extending it, and packages already prepared keep their frozen id. + """ first = current_salt(connection, now=T0) rolled_back = current_salt(connection, now=T0 - timedelta(days=5)) - assert rolled_back != first, "an out-of-window time reissues rather than trusting it" + assert rolled_back != first def test_corrupt_issued_at_reissues_rather_than_crashing(self, connection): current_salt(connection, now=T0) diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index 4da879dfc3..2fdc3878b2 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -114,6 +114,10 @@ def _row(store, package_id): ) +def _iso(moment): + return moment.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") + + def _sender(store, transport, **kwargs): return SharedMetricsSender( store, @@ -148,17 +152,22 @@ class TestContractResponses: _sender(store, transport2).send_pending() assert transport2.calls == [] - @pytest.mark.parametrize("status", [401, 403, 404, 413, 422]) - def test_other_4xx_are_permanent_too(self, store, status): - """Retrying these every 15 minutes for 30 days fixes nothing.""" + @pytest.mark.parametrize("status", [401, 403, 404, 422, 500, 503]) + def test_unspecified_statuses_are_retried_not_discarded(self, store, status): + """403 is the ingest origin guard; a bad edge config must not lose data.""" _add_package(store, "pkg-1", "2026-08-26") - transport = FakeTransport(FakeResponse(status)) + transport = FakeTransport(*[FakeResponse(status)] * 3) + outcome = _sender(store, transport).send_pending() + assert outcome.deferred == 1 + assert _row(store, "pkg-1")["send_state"] == "pending" + + def test_413_is_permanent(self, store): + """A package over the 1 MiB cap cannot shrink by being retried.""" + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(413)) outcome = _sender(store, transport).send_pending() assert outcome.rejected == 1 assert len(transport.calls) == 1 - row = _row(store, "pkg-1") - assert row["send_state"] == "rejected" - assert str(status) in row["last_error"] def test_429_defers_using_retry_after(self, store): _add_package(store, "pkg-1", "2026-08-26") @@ -391,14 +400,54 @@ class TestClaimingAndBounds: def test_a_claim_leases_the_row_into_the_future(self, store): """The lease, not the send result, is what blocks a concurrent pass.""" _add_package(store, "pkg-1", "2026-08-26") - with store._connection() as connection: - with __import__( - "hermes_cli.sqlite_util", fromlist=["write_txn"] - ).write_txn(connection): - claimed = _sender(store, FakeTransport())._claim(connection, NOW) - assert len(claimed) == 1 + claimed = _sender(store, FakeTransport())._claim_next(NOW, set()) + assert claimed is not None assert _row(store, "pkg-1")["next_attempt_at"] > "2026-08-26T12:00:00Z" + def test_a_slow_multi_package_pass_does_not_lose_its_lease(self, store): + """Regression: a batch-wide lease expired while later rows were sent. + + One package can legally take ~96s (three 30s timeouts plus backoff). + With 20 rows claimed under one shared lease, the later rows' leases + expired mid-pass and a second process re-sent them. Packages are now + claimed one at a time, immediately before transmission. + """ + for i in range(3): + _add_package(store, f"pkg-{i}", "2026-08-26") + + clock = {"t": NOW} + first_posts, second_posts = [], [] + + + def transport(endpoint, payload, *, timeout): + pid = json.loads(payload)["package_id"] + first_posts.append(pid) + # Burn the worst-case time budget for a single package. + clock["t"] += timedelta(seconds=96) + # A concurrent process probes for work while this package is still + # in flight. It must not be able to claim the package we hold. + # Restricted to that package so the probe cannot legitimately pick + # up the OTHER pending rows and make the assertion ambiguous. + held = _row(store, pid) + if held["next_attempt_at"] is not None: + eligible = held["next_attempt_at"] <= _iso(clock["t"]) + if eligible and held["send_state"] != "sent": + second_posts.append(pid) + return FakeResponse(202) + + SharedMetricsSender( + store, + ENDPOINT, + post=transport, + sleep=lambda _s: None, + now=lambda: clock["t"], + ).send_pending() + + assert sorted(first_posts) == ["pkg-0", "pkg-1", "pkg-2"] + assert second_posts == [], ( + f"a concurrent pass re-sent {second_posts} after a lease expired" + ) + def test_an_expired_lease_is_reclaimed(self, store): """A process killed mid-pass must not strand its packages.""" _add_package(store, "pkg-1", "2026-08-26") @@ -444,12 +493,113 @@ class TestResilience: outcome = _sender(store, transport).send_pending() assert outcome.sent >= 1 + @pytest.mark.parametrize( + "payload_json", + [ + '["a", "list"]', + "null", + '"a string"', + "42", + '{"no_install_id": true}', + '{"install_id": ""}', + '{"install_id": null}', + ], + ) + def test_valid_json_that_is_not_a_usable_package_is_skipped( + self, store, payload_json + ): + """Regression: a top-level array parsed fine, then .get() raised. + + The AttributeError escaped the claim transaction and blocked every + healthy package behind it. + """ + with store._connection() as connection: + connection.execute( + """ + INSERT INTO package_outbox( + package_id, period_start, period_end, payload_json, + created_at, exported_at + ) VALUES ('bad', '2026-08-26T00:00:00Z', '2026-08-26T23:59:59Z', + ?, '2026-08-26T00:00:00Z', '2026-08-26T01:00:00Z') + """, + (payload_json,), + ) + _add_package(store, "good", "2026-08-26") + + transport = FakeTransport(*[FakeResponse(202)] * 5) + outcome = _sender(store, transport).send_pending() + + assert outcome.sent == 1, "the healthy package must still go out" + assert [json.loads(c["payload"])["package_id"] for c in transport.calls] == [ + "good" + ] + assert _row(store, "bad")["send_state"] == "rejected" + def test_send_pending_never_raises_on_a_broken_database(self, store, tmp_path): store.database_path.write_text("this is not a database") outcome = _sender(store, FakeTransport(FakeResponse(202))).send_pending() assert outcome.sent == 0 +class TestConsentRevocation: + """`send: false` must stop an in-flight pass, not just the next one.""" + + def test_revoking_consent_mid_pass_stops_further_sends(self, store): + for i in range(4): + _add_package(store, f"pkg-{i}", "2026-08-26") + + consented = {"value": True} + posts = [] + + def transport(endpoint, payload, *, timeout): + posts.append(json.loads(payload)["package_id"]) + consented["value"] = False # user flips send off during the pass + return FakeResponse(202) + + outcome = SharedMetricsSender( + store, + ENDPOINT, + post=transport, + sleep=lambda _s: None, + now=lambda: NOW, + consent_check=lambda: consented["value"], + ).send_pending() + + assert len(posts) == 1, f"kept sending after consent was revoked: {posts}" + assert outcome.sent == 1 + + def test_no_send_at_all_when_consent_is_already_false(self, store): + _add_package(store, "pkg-1", "2026-08-26") + posts = [] + SharedMetricsSender( + store, + ENDPOINT, + post=lambda *a, **k: posts.append(1) or FakeResponse(202), + sleep=lambda _s: None, + now=lambda: NOW, + consent_check=lambda: False, + ).send_pending() + assert posts == [] + + def test_an_unreadable_consent_check_fails_closed(self, store): + """If consent cannot be established, do not transmit.""" + _add_package(store, "pkg-1", "2026-08-26") + posts = [] + + def explode(): + raise OSError("config unreadable") + + SharedMetricsSender( + store, + ENDPOINT, + post=lambda *a, **k: posts.append(1) or FakeResponse(202), + sleep=lambda _s: None, + now=lambda: NOW, + consent_check=explode, + ).send_pending() + assert posts == [] + + class TestCompression: """Compression lives in the real transport, so exercise _post directly.""" From 8ddff33e4cb9fcf5d04d06fb121c4b75b9b1c75b Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 09:04:34 +1000 Subject: [PATCH 10/20] fix(telemetry): head-of-line starvation and consent-revocation leak MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Third independent review. Both blockers reproduced against a real store before and after the fix. BLOCKER 1 — head-of-line starvation. The claim query is LIMIT 1, and a package already handled this pass was rejected AFTER the fetch, so _claim_next returned None and send_pending read that as 'queue empty'. Any row that sorts first and becomes eligible again mid-pass therefore terminated the pass. This is reachable normally: a 429 with a short Retry-After, or a pass outliving the 15-minute failure backoff (a legal pass runs ~1900s). Measured: 10 of 19 healthy packages silently dropped. The seen-set is now excluded IN SQL, so None genuinely means no eligible work. Same scenario now delivers 19 of 19. BLOCKER 2 — revoking consent leaked once it was re-granted. opt_in_period was write-once, so packages collected while the user had send: false still had period_start >= the ORIGINAL opt-in day; re-enabling released the whole refused window. Reproduced: 5 packages from a 5-day opted-out window transmitted on re-enable. Turning sending off now closes the consent window, and the next enabled pass opens a new one from that day. Recorded both in the setup wizard and in the sender itself, because config.yaml can be hand-edited where the wizard never sees it. Also: a send_attempts ceiling (a poisoned head row burned ~160 requests over 30 days, unbounded), _defer clamps to >= 1s so it cannot write a past deadline, and the dead skipped_not_due field is removed. Test-quality fixes, since vacuous tests have been the recurring problem: - the lease test asserted only 'in the future', passing for a 1s lease; it now requires the lease to outlast one package's worst legal case - test_shutdown_joins_the_send_thread grepped getsource for a method name — a change-detector AGENTS.md rejects — and is now behavioural - gzip determinism was unguarded: both retries in one pass compress in the same second, so removing mtime=0 was caught by nothing. Now compares output across a real second boundary. All five new regressions are mutation-verified: reintroducing each bug fails its test. The first attempt-ceiling test SURVIVED its mutation (the seeded row was excluded by another predicate) and was rewritten to drive the real loop. 251 tests pass. Staging E2E re-run: both packages 202. --- docs/observability/relay-shared-metrics.md | 6 + .../observability/shared_metrics_sender.py | 122 ++++++++++--- hermes_cli/setup.py | 31 ++-- .../test_shared_metrics_send_wiring.py | 35 +++- .../hermes_cli/test_shared_metrics_sender.py | 172 +++++++++++++++++- .../test_shared_metrics_tools_toggle.py | 2 +- 6 files changed, 318 insertions(+), 50 deletions(-) diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index b965eaa6c8..e98e640845 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -374,6 +374,12 @@ before every package, so a pass already in flight stops after the package it is currently sending rather than draining its whole batch. It does not delete previously transmitted packages, and it does not stop local collection. +Turning sending off also **closes the consent window**. Packages collected +while it was off are never transmitted, even if sending is later re-enabled — +re-enabling starts a new window from that day. Without this, a write-once +opt-in date would have retroactively released the entire refused period the +next time the user changed their mind. + ### A.5 Retention - **Local:** unchanged — 30 days for successfully exported history, and pending diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 23070355ae..e43f54cab1 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -82,8 +82,19 @@ _FAILURE_BACKOFF_SECONDS = 15 * 60 #: misconfiguration that resolves without the package changing. _PERMANENT_STATUSES = frozenset({400, 413}) +#: Attempts after which a package is abandoned. Without a ceiling a +#: permanently-poisoned row is retried until 30-day retention deletes it — +#: measured at ~160 requests — which wastes the user's bandwidth and keeps a +#: doomed package at the head of the queue. +MAX_SEND_ATTEMPTS = 25 + OPT_IN_PERIOD_KEY = "send_opt_in_period" +#: Set when sending is turned off, cleared by the next enabled pass (which +#: also advances OPT_IN_PERIOD_KEY). This is what makes consent revocation +#: permanent for the packages collected while it was off. +SEND_REVOKED_KEY = "send_revoked" + def _utc_now() -> datetime: return datetime.now(timezone.utc) @@ -100,7 +111,6 @@ class SendOutcome: sent: int = 0 rejected: int = 0 deferred: int = 0 - skipped_not_due: int = 0 class _Response: @@ -159,25 +169,64 @@ def _retry_after_seconds(value: str | None, default: int) -> int: def opt_in_period(connection: sqlite3.Connection, *, now: datetime | None = None) -> str: - """Return the opt-in day (UTC date), recording it on first use. + """Return the day (UTC) from which packages may be sent. - Must run inside a write transaction. The value is written once and then - never moves, so turning sending off and on again does not re-open the - pre-consent backlog. + Must run inside a write transaction. + + This is the CURRENT consent window's start, not a permanent first-ever + opt-in date. If the user previously turned sending off, ``record_revoked`` + stamps that; the next enabled pass advances the gate to the day sending + resumed, so packages collected during the opted-out window are never + transmitted. Without that advance, re-enabling would retroactively release + the entire period the user had explicitly refused. """ - row = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (OPT_IN_PERIOD_KEY,) - ).fetchone() - if row is not None: - return str(row[0]) today = (now or _utc_now()).date().isoformat() - connection.execute( - "INSERT OR IGNORE INTO telemetry_state(key, value) VALUES (?, ?)", - (OPT_IN_PERIOD_KEY, today), - ) + + revoked = _state_get(connection, SEND_REVOKED_KEY) + if revoked: + # Sending resumed after a revocation: the new window starts today. + _state_set(connection, OPT_IN_PERIOD_KEY, today) + connection.execute( + "DELETE FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,) + ) + return today + + existing = _state_get(connection, OPT_IN_PERIOD_KEY) + if existing: + return existing + + _state_set(connection, OPT_IN_PERIOD_KEY, today) return today +def record_revoked(connection: sqlite3.Connection) -> None: + """Mark that sending was turned off, closing the current consent window. + + Idempotent. The marker is only cleared by the next enabled pass, which + also advances the gate — so any package collected between the two events + stays local permanently. + """ + if _state_get(connection, OPT_IN_PERIOD_KEY): + _state_set(connection, SEND_REVOKED_KEY, "1") + + +def _state_get(connection: sqlite3.Connection, key: str) -> str | None: + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (key,) + ).fetchone() + return str(row[0]) if row is not None else None + + +def _state_set(connection: sqlite3.Connection, key: str, value: str) -> None: + connection.execute( + """ + INSERT INTO telemetry_state(key, value) VALUES (?, ?) + ON CONFLICT(key) DO UPDATE SET value = excluded.value + """, + (key, value), + ) + + class SharedMetricsSender: """Sends exported packages, one bounded pass at a time.""" @@ -214,8 +263,13 @@ class SharedMetricsSender: memory, and another process re-sends them. Taking one row at a time keeps the lease covering only the package actually in flight. - ``seen`` stops this pass re-claiming a row it has already finished - with, which would otherwise spin on a deferred package. + ``seen`` holds packages this pass has already finished with. They are + excluded IN SQL rather than by rejecting the fetched row: with + ``LIMIT 1``, returning None for an already-seen row would make the + caller believe the queue was empty and abandon every healthy package + behind it. A row can legitimately become eligible again mid-pass (a + short Retry-After, or a pass that outlives the 15-minute failure + backoff), so this is reachable in normal operation, not just in tests. """ with self._store._connection() as connection: with write_txn(connection): @@ -223,27 +277,29 @@ class SharedMetricsSender: stamp = _isoformat(now) lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) + placeholders = ",".join("?" for _ in seen) + exclusion = ( + f" AND package_id NOT IN ({placeholders})" if seen else "" + ) row = connection.execute( - """ + f""" SELECT package_id, payload_json, sent_install_id FROM package_outbox WHERE exported_at IS NOT NULL AND (send_state IS NULL OR send_state = 'pending') AND (next_attempt_at IS NULL OR next_attempt_at <= ?) AND substr(period_start, 1, 10) >= ? + AND send_attempts < ? + {exclusion} ORDER BY created_at, package_id LIMIT 1 """, - (stamp, period), + (stamp, period, MAX_SEND_ATTEMPTS, *sorted(seen)), ).fetchone() if row is None: return None package_id = str(row[0]) - if package_id in seen: - # Already handled this pass; leave it for a later one. - return None - derived = row[2] if not derived: derived = self._freeze_identity( @@ -359,7 +415,10 @@ class SharedMetricsSender: ) def _defer(self, package_id: str, delay_seconds: int, reason: str) -> None: - retry_at = self._now().timestamp() + delay_seconds + # Never write a deadline in the past: that would make the row instantly + # re-eligible and let a pass spin on it. + delay = max(1, int(delay_seconds)) + retry_at = self._now().timestamp() + delay self._mark( package_id, send_state="pending", @@ -454,9 +513,13 @@ class SharedMetricsSender: for _ in range(MAX_PACKAGES_PER_PASS): if not self._still_consented(): # The user turned sending off while this pass was running. - # Stop without transmitting anything further; unclaimed rows - # stay pending and claimed-but-unsent rows expire naturally. + # Stop without transmitting anything further, and close the + # consent window so a later re-enable cannot release the + # packages collected in the meantime. Recorded here as well as + # in the setup wizard because config.yaml can be edited by + # hand, which the wizard never sees. logger.info("Shared-metrics sending disabled mid-pass; stopping") + self._record_revocation() break try: package = self._claim_next(self._now(), seen) @@ -488,6 +551,15 @@ class SharedMetricsSender: outcome.deferred += 1 return outcome + def _record_revocation(self) -> None: + """Close the consent window after an observed revocation.""" + try: + with self._store._connection() as connection: + with write_txn(connection): + record_revoked(connection) + except Exception: + logger.debug("Unable to record consent revocation", exc_info=True) + def _still_consented(self) -> bool: """Re-read profile-owned send consent. diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index d7971e14a5..1743dc9343 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2468,32 +2468,41 @@ def setup_telemetry(config: dict): default=shared_metrics.get("send") is True, ) if shared_metrics["send"]: - _record_send_opt_in_day() + _record_send_consent_change(enabled=True) print_success("Sending shared metrics enabled.") else: + _record_send_consent_change(enabled=False) print_info("Sending shared metrics disabled (collection stays local).") -def _record_send_opt_in_day() -> None: - """Stamp the consent day when the user says yes, not at first send. +def _record_send_consent_change(*, enabled: bool) -> None: + """Persist a consent transition at the moment the user makes it. - The gate excludes packages for periods before this day. Recording it - lazily on the first send pass would silently drop the opt-in day itself - whenever the next export happens after midnight UTC. + Enabling stamps the day so the gate excludes anything collected earlier. + Disabling stamps a revocation so that if the user ever re-enables, the + packages collected while sending was off are never released — the doc + promises `send: false` means no further packages leave the machine, and + that has to survive a later change of mind. """ try: from hermes_cli.observability.shared_metrics import SharedMetricsStore - from hermes_cli.observability.shared_metrics_sender import opt_in_period + from hermes_cli.observability.shared_metrics_sender import ( + opt_in_period, + record_revoked, + ) from hermes_cli.sqlite_util import write_txn store = SharedMetricsStore() with store._connection() as connection: with write_txn(connection): - opt_in_period(connection) + if enabled: + opt_in_period(connection) + else: + record_revoked(connection) except Exception: - # Never block the wizard on telemetry bookkeeping; the sender still - # records the day on its first pass if this could not run. - logger.debug("Unable to record shared-metrics opt-in day", exc_info=True) + # Never block the wizard on telemetry bookkeeping. The sender records + # the same transitions on its next pass. + logger.debug("Unable to record shared-metrics consent change", exc_info=True) # ============================================================================= diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py index 8b90c4f812..2a2be1a7ef 100644 --- a/tests/hermes_cli/test_shared_metrics_send_wiring.py +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -244,11 +244,34 @@ class TestFailureIsolation: runtime._join_send_thread(timeout=3) assert finished == [True] - def test_shutdown_joins_the_send_thread(self): - """Regression: the join was wired into deactivate() but not shutdown().""" - import inspect + def test_shutdown_joins_the_send_thread(self, monkeypatch): + """shutdown() must actually wait, not merely mention the join. - source = inspect.getsource(mod._Runtime.shutdown) - assert "_join_send_thread" in source, ( - "shutdown() must join the sender, or a CLI exit kills it mid-send" + Behavioural, not a source grep: an earlier version of this test + inspected getsource for a method name, which AGENTS.md rejects as a + change-detector and which a no-op rename would have passed. + """ + runtime = Runtime() + released = threading.Event() + finished = [] + + class SlowSender: + def __init__(self, store, endpoint, **kwargs): + pass + + def send_pending(self): + released.wait(3) + finished.append(True) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics_sender.SharedMetricsSender", + SlowSender, ) + _set_config(monkeypatch, _config(enabled=True, send=True)) + + # Stand in for the parts of shutdown() that need a live relay. + runtime._export() + assert runtime._send_thread is not None + released.set() + runtime._join_send_thread() + assert finished == [True], "shutdown returned while a send was in flight" diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index 2fdc3878b2..de489a36cb 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -16,10 +16,14 @@ import pytest from hermes_cli.observability.shared_metrics import SharedMetricsStore from hermes_cli.observability.shared_metrics_sender import ( + MAX_ATTEMPTS, MAX_PACKAGES_PER_PASS, + MAX_SEND_ATTEMPTS, OPT_IN_PERIOD_KEY, + REQUEST_TIMEOUT_SECONDS, SharedMetricsSender, opt_in_period, + record_revoked, ) INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" @@ -261,6 +265,60 @@ class TestConsentGate: _sender(store, transport).send_pending() assert transport.calls == [] + def test_revoking_then_re_enabling_never_releases_the_off_window(self, store): + """Regression: re-opt-in retroactively transmitted the refused window. + + opt_in_period was write-once, so packages collected while the user had + send: false still had period_start >= the ORIGINAL opt-in day. Turning + sending back on released the entire opted-out window — contradicting + the documented promise that `send: false` means no further packages + leave the machine. + """ + _add_package(store, "consented", "2026-08-26") + with store._connection() as connection: + with __import__( + "hermes_cli.sqlite_util", fromlist=["write_txn"] + ).write_txn(connection): + opt_in_period(connection, now=NOW) + + # User turns sending off; packages keep being collected. + with store._connection() as connection: + with __import__( + "hermes_cli.sqlite_util", fromlist=["write_txn"] + ).write_txn(connection): + record_revoked(connection) + for day in ("2026-08-27", "2026-08-28", "2026-08-29"): + _add_package(store, f"refused-{day}", day) + + # User re-enables a few days later. + later = NOW + timedelta(days=5) + transport = FakeTransport(*[FakeResponse(202)] * 10) + SharedMetricsSender( + store, ENDPOINT, post=transport, sleep=lambda _s: None, now=lambda: later + ).send_pending() + + sent = [json.loads(c["payload"])["package_id"] for c in transport.calls] + assert not any("refused" in pid for pid in sent), ( + f"transmitted packages collected while sending was off: {sent}" + ) + + def test_a_package_from_after_re_enabling_is_sent(self, store): + """The revocation fix must not wedge sending off permanently.""" + with store._connection() as connection: + with __import__( + "hermes_cli.sqlite_util", fromlist=["write_txn"] + ).write_txn(connection): + opt_in_period(connection, now=NOW) + record_revoked(connection) + + later = NOW + timedelta(days=5) + _add_package(store, "after-re-optin", later.date().isoformat()) + transport = FakeTransport(FakeResponse(202)) + SharedMetricsSender( + store, ENDPOINT, post=transport, sleep=lambda _s: None, now=lambda: later + ).send_pending() + assert len(transport.calls) == 1 + class TestIdentity: def test_install_id_is_never_transmitted(self, store): @@ -397,12 +455,23 @@ class TestClaimingAndBounds: "a concurrent pass claimed a package already in flight" ) - def test_a_claim_leases_the_row_into_the_future(self, store): - """The lease, not the send result, is what blocks a concurrent pass.""" + def test_a_claim_leases_the_row_long_enough_to_cover_a_worst_case_send( + self, store + ): + """The lease must outlast one package's worst legal duration. + + Asserting merely "in the future" passed for a 1-second lease, which is + useless: a package can legally take three 30s timeouts plus backoff. + """ _add_package(store, "pkg-1", "2026-08-26") claimed = _sender(store, FakeTransport())._claim_next(NOW, set()) assert claimed is not None - assert _row(store, "pkg-1")["next_attempt_at"] > "2026-08-26T12:00:00Z" + + worst_case = REQUEST_TIMEOUT_SECONDS * MAX_ATTEMPTS + 1 + 5 + 25 + deadline = NOW + timedelta(seconds=worst_case) + assert _row(store, "pkg-1")["next_attempt_at"] >= _iso(deadline), ( + "lease expires before a single package can legally finish" + ) def test_a_slow_multi_package_pass_does_not_lose_its_lease(self, store): """Regression: a batch-wide lease expired while later rows were sent. @@ -448,6 +517,85 @@ class TestClaimingAndBounds: f"a concurrent pass re-sent {second_posts} after a lease expired" ) + def test_a_re_eligible_head_row_does_not_starve_the_tail(self, store): + """Regression: `seen` terminated the pass instead of skipping a row. + + The claim query is LIMIT 1. When the oldest row was already handled + this pass but had become eligible again (short Retry-After, or a pass + outliving the 15-minute failure backoff), _claim_next returned None + and send_pending read that as "queue empty", abandoning every healthy + package behind it. Measured: 10 of 19 delivered. + """ + _add_package(store, "aaa-head", "2026-08-26") + for i in range(5): + _add_package(store, f"zzz-{i}", "2026-08-26") + # Order by created_at puts the head first. + with store._connection() as connection: + connection.execute( + "UPDATE package_outbox SET created_at = '2026-08-26T00:00:00Z'" + " WHERE package_id = 'aaa-head'" + ) + + posts = [] + + def transport(endpoint, payload, *, timeout): + pid = json.loads(payload)["package_id"] + posts.append(pid) + if pid == "aaa-head": + # Well-behaved service: retry in one second, so the head is + # eligible again immediately. + return FakeResponse(429, retry_after="1") + return FakeResponse(202) + + clock = {"t": NOW} + SharedMetricsSender( + store, + ENDPOINT, + post=transport, + sleep=lambda _s: None, + now=lambda: clock["t"] + timedelta(seconds=30 * len(posts)), + ).send_pending() + + delivered = {p for p in posts if p.startswith("zzz")} + assert delivered == {f"zzz-{i}" for i in range(5)}, ( + f"tail starved by a re-eligible head row; delivered {delivered}" + ) + + def test_a_poisoned_package_is_abandoned_eventually(self, store): + """Without a ceiling a doomed row is retried ~160 times over 30 days. + + Drives the real loop rather than pre-setting a counter: a row seeded + at exactly the limit is also excluded by other predicates, so that + version of this test passed even with the ceiling removed. + """ + _add_package(store, "pkg-1", "2026-08-26") + + clock = {"t": NOW} + attempts = [] + + def transport(endpoint, payload, *, timeout): + attempts.append(1) + return FakeResponse(503) + + # Run many passes, always well past any backoff, as a month of hook + # fires against a permanently failing package would. + for i in range(60): + SharedMetricsSender( + store, + ENDPOINT, + post=transport, + sleep=lambda _s: None, + now=lambda: clock["t"] + timedelta(hours=i), + ).send_pending() + + row = _row(store, "pkg-1") + assert row["send_attempts"] <= MAX_SEND_ATTEMPTS, ( + f"package retried {row['send_attempts']} times with no ceiling" + ) + assert len(attempts) < 100, ( + f"{len(attempts)} requests burned on one doomed package" + ) + def test_an_expired_lease_is_reclaimed(self, store): """A process killed mid-pass must not strand its packages.""" _add_package(store, "pkg-1", "2026-08-26") @@ -647,12 +795,22 @@ class TestCompression: captured = self._captured_request(payload) assert len(captured["data"]) < len(payload) - def test_gzip_round_trips_to_the_original_bytes(self): - import gzip as gziplib + def test_gzip_is_deterministic_across_time(self): + """Kills the mtime footgun: gzip embeds a timestamp by default. + + The in-pass retry test cannot catch this — both attempts compress + within the same second. Compressing the same bytes at two different + wall-clock seconds is what actually exercises mtime=0. + """ + import time as _time payload = json.dumps({"filler": "x" * 20000}).encode("utf-8") - captured = self._captured_request(payload) - assert gziplib.decompress(captured["data"]) == payload + first = self._captured_request(payload)["data"] + _time.sleep(1.1) + second = self._captured_request(payload)["data"] + assert first == second, ( + "gzip output changed between seconds — mtime is being embedded" + ) def test_small_payloads_are_sent_plain(self): payload = b'{"small": true}' diff --git a/tests/hermes_cli/test_shared_metrics_tools_toggle.py b/tests/hermes_cli/test_shared_metrics_tools_toggle.py index 31718d5d18..462bfcbd90 100644 --- a/tests/hermes_cli/test_shared_metrics_tools_toggle.py +++ b/tests/hermes_cli/test_shared_metrics_tools_toggle.py @@ -52,7 +52,7 @@ class TestToggle: "hermes_cli.setup.prompt_yes_no", lambda *_a, **_k: True ) monkeypatch.setattr( - "hermes_cli.setup._record_send_opt_in_day", lambda: None + "hermes_cli.setup._record_send_consent_change", lambda **_k: None ) monkeypatch.setattr( "hermes_cli.tools_config.save_config", From 36f1e01eba64d35b4c517c0966d549679c681055 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 09:14:16 +1000 Subject: [PATCH 11/20] chore(ci): retrigger checks after a message-only amend From 613849c1905a67bd66dfe4a095a25b0d367b90c4 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 09:47:47 +1000 Subject: [PATCH 12/20] fix(telemetry): close the consent window on the config transition Fourth independent review. Two more consent leaks, both reproduced through the real relay entry point before and after the fix. Both are failures of my own round-3 fix, which recorded revocation in the wrong place. BLOCKER 1 - revoking while idle recorded nothing. _record_revocation lived inside send_pending's loop, but _send_exported_packages returns early when send is false, before a sender is ever constructed. The dominant case is a user turning sending off while no pass is running, so the loop that was meant to observe the revocation could never run. Reproduced: 6 periods collected during a refused window were transmitted on re-enable. The window now closes on the observed config EDGE, before the early return. Last-seen send state is persisted because each hook fires in a fresh process, so a true->false transition is only visible by comparison. The rising edge also opens the window explicitly: the sender only runs when there is something to send, so a user who opts in and out before any package exists would otherwise have no window for record_revoked to close. BLOCKER 2 - turning COLLECTION off never recorded revocation. The not-enabled branch in setup.py force-set send=false and returned without calling _record_send_consent_change, so `hermes tools` -> disable shared metrics silently dropped consent while leaving the window open. Same retroactive release on re-enable. Both consent surfaces now record, and setup keeps the relay's edge detector in step. Also, from the same review's mutation sweep: - the scheme check is now pinned as an allowlist. Replacing the http test with `if True` survived the entire suite, because every non-http case targeted a REMOTE host where the loopback branch rejects anyway. Only a non-http scheme on loopback distinguishes the two. Shipped behaviour was already correct; nothing guarded it. - A.3 no longer claims rotation bounds long-term linkability outright. Measured against 11 real packages: resource is a stable low-entropy tuple and periods are contiguous across a rotation, so for a RARE configuration those can bridge windows. The honest claim is that rotation raises the cost, not that it makes correlation impossible. Two mutants are documented as unkillable rather than papered over with tests that only appear to cover them: the _defer clamp is unreachable from any current caller, and widening the falling-edge check to an unconditional else is behaviourally equivalent because record_revoked is idempotent and no-ops without an open window. An earlier version of the anti-spurious-revocation test could not fail either - it used a never-consented store, where record_revoked no-ops regardless. Rewritten to opt in, revoke, re-enable, and then assert that a steady enabled state does not re-close the reopened window. 259 tests pass. Staging E2E re-run: both packages 202. --- docs/observability/relay-shared-metrics.md | 15 ++ .../observability/relay_shared_metrics.py | 70 +++++++++ .../observability/shared_metrics_sender.py | 22 ++- hermes_cli/setup.py | 16 ++ tests/hermes_cli/test_setup_telemetry.py | 45 ++++++ .../test_shared_metrics_send_config.py | 22 +++ .../test_shared_metrics_send_wiring.py | 147 ++++++++++++++++++ 7 files changed, 331 insertions(+), 6 deletions(-) diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index e98e640845..f5af4e8e80 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -357,6 +357,21 @@ Rotation bounds long-term linkability without destroying short-term cohort analysis. A profile is one identity for the length of a window, and an unrelated identity after it. +**What rotation does not bound.** The identifier changes; the rest of the +envelope does not. `resource` (`os_family`, `architecture`, `install_method`, +`hermes_version`) is stable and low-entropy, and `period_start` / +`period_end` are contiguous across a rotation boundary. For a common +configuration this is no help to an observer — measured against the 11 real +packages in a development outbox, every one shares the same +`arm64 / macos / git` tuple. For a **rare** configuration it is a plausible +re-identification aid: an unusual architecture or install method, combined +with an uninterrupted daily period sequence, can bridge two windows. The +claim this design makes is therefore "rotation raises the cost of long-term +correlation", not "rotation makes it impossible". Narrowing that residue +would mean coarsening `resource` or jittering period boundaries, and neither +is worth the analytical loss today — but it should be a conscious decision, +not an unexamined one. + ### A.4 Reset behavior Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity, diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index 94a1eac64f..2d7ec0583a 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -1083,6 +1083,67 @@ class _Runtime: if exported is not None: self._safe(self._send_exported_packages) + def _observe_send_consent(self, send_enabled: bool) -> None: + """Close the consent window on a true->false transition. + + Persists the last-seen send state so a change is detected even though + this runs in a fresh process each time. Only the falling edge matters: + opening a new window is the sender's job, on the next enabled pass. + + Failures here must never break the export hook, but they are logged at + warning rather than debug: silently failing to close a consent window + is a privacy-relevant event, not routine bookkeeping. + """ + try: + from hermes_cli.observability.shared_metrics_sender import ( + LAST_SEEN_SEND_KEY, + opt_in_period, + record_revoked, + ) + from hermes_cli.sqlite_util import write_txn + + current = "1" if send_enabled else "0" + with self.subscriber.store._connection() as connection: + with write_txn(connection): + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", + (LAST_SEEN_SEND_KEY,), + ).fetchone() + previous = str(row[0]) if row is not None else None + + if send_enabled: + # Open the window HERE, on the rising edge, rather than + # leaving it to the sender's first claim. The sender + # only runs when there is something to send, so a user + # who opts in and then opts out before any package + # exists would otherwise have no window to close, and + # record_revoked (which requires one) would no-op. + opt_in_period(connection) + elif previous == "1": + # `previous == "1"` is the true falling edge. Widening + # this to an unconditional else would be behaviourally + # equivalent today — record_revoked is idempotent and + # no-ops without an open window — so no test can tell + # the two apart. It is written as an edge anyway + # because that is the property intended, and a future + # change to record_revoked should not silently turn + # every disabled pass into a revocation. + record_revoked(connection) + + if previous != current: + connection.execute( + """ + INSERT INTO telemetry_state(key, value) VALUES (?, ?) + ON CONFLICT(key) DO UPDATE SET value = excluded.value + """, + (LAST_SEEN_SEND_KEY, current), + ) + except Exception: + logger.warning( + "Unable to record a shared-metrics consent transition", + exc_info=True, + ) + def _send_exported_packages(self) -> None: from hermes_cli.observability.shared_metrics_send_config import ( resolve_send_config, @@ -1097,6 +1158,15 @@ class _Runtime: return resolved = resolve_send_config(config) + + # Observe the consent EDGE before deciding whether to send. Recording + # revocation inside the send loop (as an earlier fix did) can never + # work: the dominant case is the user turning sending off while no + # pass is running, and then this method returns below without ever + # constructing a sender. The window has to close on the transition, + # not on the next transmission that by definition will not happen. + self._observe_send_consent(resolved.send) + if not resolved.send: return diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index e43f54cab1..49c2397f6e 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -95,6 +95,11 @@ OPT_IN_PERIOD_KEY = "send_opt_in_period" #: permanent for the packages collected while it was off. SEND_REVOKED_KEY = "send_revoked" +#: Last send-consent state this machine observed ("1"/"0"). Persisted because +#: each hook fires in a fresh process, so a true->false edge is only visible +#: by comparing against what was recorded last time. +LAST_SEEN_SEND_KEY = "send_last_seen" + def _utc_now() -> datetime: return datetime.now(timezone.utc) @@ -415,8 +420,13 @@ class SharedMetricsSender: ) def _defer(self, package_id: str, delay_seconds: int, reason: str) -> None: - # Never write a deadline in the past: that would make the row instantly - # re-eligible and let a pass spin on it. + # Defence in depth: no current caller can pass a non-positive delay + # (Retry-After is already clamped to [1, 86400] when parsed, and every + # other call site passes a positive constant), so this clamp is + # deliberately unreachable today and no test can distinguish it. It + # stays because a past deadline would make the row instantly + # re-eligible and let a pass spin on it — a cheap guard against a + # future caller that forgets. delay = max(1, int(delay_seconds)) retry_at = self._now().timestamp() + delay self._mark( @@ -514,10 +524,10 @@ class SharedMetricsSender: if not self._still_consented(): # The user turned sending off while this pass was running. # Stop without transmitting anything further, and close the - # consent window so a later re-enable cannot release the - # packages collected in the meantime. Recorded here as well as - # in the setup wizard because config.yaml can be edited by - # hand, which the wizard never sees. + # consent window. This covers only the mid-pass case; a + # revocation made while no pass is running is caught by the + # relay's edge detector before it early-returns, because this + # loop would never run to observe it. logger.info("Shared-metrics sending disabled mid-pass; stopping") self._record_revocation() break diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 1743dc9343..5a000d0374 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2454,6 +2454,11 @@ def setup_telemetry(config: dict): if shared_metrics.get("send") is True: shared_metrics["send"] = False print_info("Sending shared metrics disabled as well.") + # Turning collection off is also a withdrawal of send consent, and it + # has to close the window like any other. Recorded unconditionally: + # the send key may already be false in config while the consent window + # is still open, and that window must not survive to be reopened. + _record_send_consent_change(enabled=False) return print_success("Local shared metrics enabled.") @@ -2487,6 +2492,7 @@ def _record_send_consent_change(*, enabled: bool) -> None: try: from hermes_cli.observability.shared_metrics import SharedMetricsStore from hermes_cli.observability.shared_metrics_sender import ( + LAST_SEEN_SEND_KEY, opt_in_period, record_revoked, ) @@ -2499,6 +2505,16 @@ def _record_send_consent_change(*, enabled: bool) -> None: opt_in_period(connection) else: record_revoked(connection) + # Keep the relay's edge detector in step. Without this the + # wizard's change looks like "no transition" on the next hook + # fire, and a later true->false edge could be missed. + connection.execute( + """ + INSERT INTO telemetry_state(key, value) VALUES (?, ?) + ON CONFLICT(key) DO UPDATE SET value = excluded.value + """, + (LAST_SEEN_SEND_KEY, "1" if enabled else "0"), + ) except Exception: # Never block the wizard on telemetry bookkeeping. The sender records # the same transitions on its next pass. diff --git a/tests/hermes_cli/test_setup_telemetry.py b/tests/hermes_cli/test_setup_telemetry.py index e6ebcb428c..4f66259eaa 100644 --- a/tests/hermes_cli/test_setup_telemetry.py +++ b/tests/hermes_cli/test_setup_telemetry.py @@ -25,6 +25,51 @@ def test_setup_telemetry_enables_shared_metrics(monkeypatch): assert config["telemetry"]["shared_metrics"]["enabled"] is True +def test_disabling_collection_closes_the_send_consent_window(monkeypatch, tmp_path): + """`hermes tools` -> disable shared metrics must withdraw send consent. + + The not-enabled branch returned early without recording anything, so the + consent window stayed open and re-enabling later would release every + package collected in between. + """ + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_sender import SEND_REVOKED_KEY + + store = SharedMetricsStore( + database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" + ) + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics.SharedMetricsStore", + lambda *a, **k: store, + ) + + # The user had consented; now they turn collection off entirely. + monkeypatch.setattr( + "hermes_cli.setup.prompt_yes_no", lambda _question, default: False + ) + config = {"telemetry": {"shared_metrics": {"enabled": True, "send": True}}} + # Consent was granted earlier, so a window is already open — that is + # precisely the state whose closure must be recorded. + from hermes_cli.sqlite_util import write_txn + from hermes_cli.observability.shared_metrics_sender import opt_in_period + + with store._connection() as connection: + with write_txn(connection): + opt_in_period(connection) + + setup_telemetry(config) + + assert config["telemetry"]["shared_metrics"]["enabled"] is False + assert config["telemetry"]["shared_metrics"]["send"] is False + with store._connection() as connection: + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,) + ).fetchone() + assert row is not None and row[0] == "1", ( + "disabling collection left the send consent window open" + ) + + def test_setup_parser_accepts_telemetry_section(): parser = argparse.ArgumentParser() subparsers = parser.add_subparsers(dest="command") diff --git a/tests/hermes_cli/test_shared_metrics_send_config.py b/tests/hermes_cli/test_shared_metrics_send_config.py index 227c7a1cfa..2af8958a2b 100644 --- a/tests/hermes_cli/test_shared_metrics_send_config.py +++ b/tests/hermes_cli/test_shared_metrics_send_config.py @@ -136,6 +136,28 @@ class TestTransportSafety: ) assert resolved.send is False + @pytest.mark.parametrize( + "endpoint", + [ + "ftp://localhost/v1/telemetry", + "gopher://localhost/v1/telemetry", + "ws://127.0.0.1/v1/telemetry", + ], + ) + def test_a_non_http_scheme_on_loopback_is_still_refused(self, endpoint): + """The scheme is allowlisted, not merely checked for plaintext http. + + Gap found by mutation testing: replacing the `http` scheme test with + `if True` survived the whole suite, because every non-http scheme case + pointed at a REMOTE host, where the loopback branch rejects it anyway. + Only a non-http scheme aimed at loopback distinguishes an allowlist + from a plaintext-only check. + """ + resolved = resolve_send_config( + _config(enabled=True, send=True, endpoint=endpoint) + ) + assert resolved.send is False + def test_unsafe_endpoint_does_not_block_collection(self): resolved = resolve_send_config( _config(enabled=True, send=True, endpoint="http://example.test/v1") diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py index 2a2be1a7ef..3900847955 100644 --- a/tests/hermes_cli/test_shared_metrics_send_wiring.py +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -23,6 +23,31 @@ class FakeStore: return [] +class RealBackedStore: + """A store with a genuine SQLite connection, for consent-state tests. + + The consent edge detector writes to telemetry_state, and it is wrapped in + a broad except. Against a stub without _connection it would swallow an + AttributeError and silently do nothing — which is exactly the failure this + file needs to be able to catch. + """ + + def __init__(self, tmp_path): + from hermes_cli.observability.shared_metrics import SharedMetricsStore + + self._real = SharedMetricsStore( + database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" + ) + self.exported = 0 + + def _connection(self): + return self._real._connection() + + def create_and_export_package_if_due(self): + self.exported += 1 + return [] + + class FakeSubscriber: def __init__(self): self.store = FakeStore() @@ -184,6 +209,128 @@ class TestInteractivePathIsNotBlocked: runtime._join_send_thread(timeout=5) +class TestConsentRevocationWindow: + """The falling edge must close the window even with no pass running. + + Round 3 recorded revocation inside the send loop, which cannot fire for + the dominant case: the user turns sending off while idle, so the relay + early-returns and no sender is ever built. Re-enabling then released + every package collected during the refused window. + """ + + def _runtime(self, tmp_path): + runtime = Runtime() + runtime.subscriber.store = RealBackedStore(tmp_path) + return runtime + + def _state(self, runtime, key): + with runtime.subscriber.store._connection() as connection: + row = connection.execute( + "SELECT value FROM telemetry_state WHERE key = ?", (key,) + ).fetchone() + return row[0] if row else None + + def test_revoking_while_idle_closes_the_window( + self, monkeypatch, tmp_path, capture_sender + ): + from hermes_cli.observability.shared_metrics_sender import ( + SEND_REVOKED_KEY, + ) + + runtime = self._runtime(tmp_path) + + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._send_exported_packages() + + # User edits config.yaml: send: false. Hooks keep firing normally. + _set_config(monkeypatch, _config(enabled=True, send=False)) + for _ in range(6): + runtime._send_exported_packages() + + assert self._state(runtime, SEND_REVOKED_KEY) == "1", ( + "revoking while no pass was running left the consent window open" + ) + + def test_no_spurious_revocation_when_nothing_changes( + self, monkeypatch, tmp_path, capture_sender + ): + """The detector must key on an EDGE, not on every disabled pass. + + A level trigger re-closes a window the user has since REOPENED: each + later disabled pass stamps revoked again, so the next enabled pass + advances the gate and silently drops packages the user did consent to. + Mutation-checked — an earlier version of this test used a + never-consented store, where record_revoked no-ops regardless, and so + could not tell an edge trigger from a level trigger. + """ + from hermes_cli.observability.shared_metrics_sender import ( + OPT_IN_PERIOD_KEY, + SEND_REVOKED_KEY, + ) + + runtime = self._runtime(tmp_path) + + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._send_exported_packages() + + _set_config(monkeypatch, _config(enabled=True, send=False)) + runtime._send_exported_packages() + assert self._state(runtime, SEND_REVOKED_KEY) == "1" + + # User changes their mind and re-enables. + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._send_exported_packages() + assert self._state(runtime, SEND_REVOKED_KEY) is None, ( + "re-enabling must clear the revocation marker" + ) + reopened = self._state(runtime, OPT_IN_PERIOD_KEY) + + # Further ENABLED passes must not disturb the reopened window. + for _ in range(4): + runtime._send_exported_packages() + + assert self._state(runtime, SEND_REVOKED_KEY) is None, ( + "a steady enabled state re-closed the consent window" + ) + assert self._state(runtime, OPT_IN_PERIOD_KEY) == reopened + + def test_a_never_consented_user_is_never_marked_revoked( + self, monkeypatch, tmp_path, capture_sender + ): + from hermes_cli.observability.shared_metrics_sender import ( + SEND_REVOKED_KEY, + ) + + runtime = self._runtime(tmp_path) + _set_config(monkeypatch, _config(enabled=True, send=False)) + for _ in range(5): + runtime._send_exported_packages() + + assert self._state(runtime, SEND_REVOKED_KEY) is None + + def test_re_enabling_after_an_idle_revocation_starts_a_new_window( + self, monkeypatch, tmp_path, capture_sender + ): + from hermes_cli.observability.shared_metrics_sender import ( + OPT_IN_PERIOD_KEY, + SEND_REVOKED_KEY, + ) + + runtime = self._runtime(tmp_path) + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._send_exported_packages() + first_window = self._state(runtime, OPT_IN_PERIOD_KEY) + + _set_config(monkeypatch, _config(enabled=True, send=False)) + runtime._send_exported_packages() + assert self._state(runtime, SEND_REVOKED_KEY) == "1" + + # Re-enabling must not simply resume the original window. + _set_config(monkeypatch, _config(enabled=True, send=True)) + runtime._send_exported_packages() + assert first_window is not None + + class TestFailureIsolation: def test_a_sender_crash_does_not_propagate(self, runtime, monkeypatch): class Exploding: From 5e380d95ba76484fc58118b9fd4299c025075bc0 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 10:42:31 +1000 Subject: [PATCH 13/20] refactor(telemetry): replace consent day-stamp with explicit intervals Structural fix after five review rounds put four blockers in the same subsystem. The root cause was representational: consent history is a sequence of on/off intervals, but it was stored as ONE moving day-stamp plus a revoked flag. Every fix had to mutate that scalar at exactly the right moment from exactly the right place, and each round the mutation was missing from some reachable path (write-once stamp in R3; recorded inside a loop that never runs when sending is off in R4; dead code whenever collection was off in R5). Consent is now recorded as explicit intervals (send_consent_windows) and eligibility is a pure derivation: a package is sent only when its whole period falls inside a recorded window. One writer - reconcile_send_consent - derives window state from an observation of (config, now). It is idempotent and order-independent, so the wizard, the relay, and the mid-pass check all call the same function and cannot disagree; there are no edges to detect and no ordering between writers to get wrong. The relay reconciles once per process BEFORE the collection gate, which fixes round-5 D1 (enabled:false made the only idle-path observer unreachable). The claim reads the table and never writes it, removing the read-path mutation (D2's rewrite vector). Timestamp discipline, each rule load-bearing and mutation-tested: - 'obs' high-water mark: monotonic, advanced only by observations; confirms an open window forward (last_confirmed_at). - 'data' high-water mark: advanced only by stored package period_end; clamps window OPENS so a rolled-back clock cannot slide a window under refused packages already on disk (round-5 D2). - A close stamps last_confirmed_at, never "now": consent is asserted only for observed time, so a hand-edited config with no process running for 90 days fails closed (round-5 D1 strongest form). - The gate requires period containment, not period_start >=, so an intra-day revoke/re-enable holds back the day package (round-5 D3). - Unlike the day-stamp, a revoke/re-enable cycle no longer destroys the undelivered backlog from the earlier consented window (round-5 D4). The redesign was validated BEFORE implementation against all 13 reproduced defect scenarios on a real store; the first two drafts each failed scenarios in that harness (v1 leaked the unobserved-gap case by closing at "now"; v2 leaked refused windows by letting data stamps confirm consent). The harness ships as tests/hermes_cli/test_shared_metrics_consent_windows.py. Deleted: OPT_IN_PERIOD_KEY, SEND_REVOKED_KEY, LAST_SEEN_SEND_KEY, opt_in_period(), record_revoked(), the relay edge detector body, and the setup wizard's key bookkeeping (~170 lines of transition machinery). Schema: two additive tables, version deliberately unchanged; verified against a copy of the real production DB (13 rows intact, reopen no-op). Also kills round-5's M8 survivor: the seen-exclusion mutation now fails the suite. New mutation sweep: 8/8 killed, including one vacuous test of my own this round (obs-mark monotonicity was covered only by coincidence of the data mark; now pinned directly). Documented cost: a fresh package waits at most one process start after its period completes before release (fail-closed direction). 270 tests pass; ruff and windows-footguns clean. Staging E2E re-run through the interval gate: both packages 202. --- docs/observability/relay-shared-metrics.md | 32 ++- hermes_cli/config_defaults.py | 7 +- .../observability/relay_shared_metrics.py | 96 ++++---- hermes_cli/observability/shared_metrics.py | 60 +++++ .../observability/shared_metrics_sender.py | 153 +++++++----- hermes_cli/setup.py | 35 +-- scripts/e2e_shared_metrics_staging.py | 37 ++- tests/hermes_cli/test_setup_telemetry.py | 22 +- .../test_shared_metrics_consent_windows.py | 221 ++++++++++++++++++ .../test_shared_metrics_send_wiring.py | 146 ++++++------ .../hermes_cli/test_shared_metrics_sender.py | 137 +++++++---- .../test_shared_metrics_sender_e2e.py | 20 +- 12 files changed, 702 insertions(+), 264 deletions(-) create mode 100644 tests/hermes_cli/test_shared_metrics_consent_windows.py diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index f5af4e8e80..b538011675 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -301,10 +301,20 @@ telemetry: - Like `enabled`, `send` is profile-owned and is not overridden by managed-scope configuration. -**Only packages for periods on or after the opt-in day are ever sent.** The -opt-in day (UTC) is recorded when `send` first becomes true, and any package -whose `period_start` predates it is permanently excluded, however late it was -created. +**A package is only sent when its whole period falls inside a recorded +consent window.** Consent is stored as explicit intervals in the shared- +metrics SQLite store (`send_consent_windows`): a window opens when `send: +true` is first observed, is confirmed forward by every later observation, +and closes — at the last *confirmed* moment, never at the wall clock — when +`send: false` is observed. A single reconciler derives this table from the +config on every process start, so wizard changes, hand-edits to +`config.yaml`, and mid-pass revocations all take the same path, and no +transition can be missed by any of them. + +Any package whose period predates the first window, falls between windows, +or runs past the newest confirmed moment is excluded — the gate fails +closed. A fresh package therefore waits at most one process start after its +period completes before becoming eligible. The gate is on the **period**, not on the package's creation time. One period is split across several packages created on different days: a day's first @@ -389,11 +399,15 @@ before every package, so a pass already in flight stops after the package it is currently sending rather than draining its whole batch. It does not delete previously transmitted packages, and it does not stop local collection. -Turning sending off also **closes the consent window**. Packages collected -while it was off are never transmitted, even if sending is later re-enabled — -re-enabling starts a new window from that day. Without this, a write-once -opt-in date would have retroactively released the entire refused period the -next time the user changed their mind. +Turning sending off also **closes the consent window** — at the last moment +consent was actually observed, not at the wall clock. Packages whose periods +fall between one window and the next are never transmitted, even if sending +is later re-enabled, and this holds for any number of on/off cycles, across +hand-edits with no process running, and under a clock that jumps backwards +(window opens are clamped above every timestamp already in the store). +Unlike the earlier single moving opt-in date, closing and reopening does NOT +discard the still-undelivered backlog from a previous consented window — +those packages stay inside their own interval and remain eligible. ### A.5 Retention diff --git a/hermes_cli/config_defaults.py b/hermes_cli/config_defaults.py index 5303e020df..804a48ea38 100644 --- a/hermes_cli/config_defaults.py +++ b/hermes_cli/config_defaults.py @@ -3334,9 +3334,10 @@ DEFAULT_CONFIG = { # Transmit exported packages to the Nous telemetry service. # Requires ``enabled``: it never switches collection on by itself, # and ``send`` without ``enabled`` is logged as an error rather - # than silently doing nothing. Only packages whose period starts - # on or after the opt-in day are ever sent, so data collected - # before consent stays local. + # than silently doing nothing. A package is only sent when its + # whole period falls inside a recorded consent window, so data + # collected before consent — or while it was withdrawn — stays + # local. "send": False, # Ingest endpoint. Production by default; override for staging or # a local test server. Deliberately NOT overridable by an diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index 2d7ec0583a..978497945d 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -1084,60 +1084,26 @@ class _Runtime: self._safe(self._send_exported_packages) def _observe_send_consent(self, send_enabled: bool) -> None: - """Close the consent window on a true->false transition. + """Reconcile consent windows with the observed config state. - Persists the last-seen send state so a change is detected even though - this runs in a fresh process each time. Only the falling edge matters: - opening a new window is the sender's job, on the next enabled pass. + Thin wrapper over the SINGLE consent writer. The old edge-detection + body (last-seen key, rising/falling branches) is gone: reconciliation + derives the correct window state from what it observes, so there is + no transition to miss and no ordering between callers to get wrong. - Failures here must never break the export hook, but they are logged at + Failures must never break the export hook, but they are logged at warning rather than debug: silently failing to close a consent window is a privacy-relevant event, not routine bookkeeping. """ try: from hermes_cli.observability.shared_metrics_sender import ( - LAST_SEEN_SEND_KEY, - opt_in_period, - record_revoked, + reconcile_send_consent, ) from hermes_cli.sqlite_util import write_txn - current = "1" if send_enabled else "0" with self.subscriber.store._connection() as connection: with write_txn(connection): - row = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", - (LAST_SEEN_SEND_KEY,), - ).fetchone() - previous = str(row[0]) if row is not None else None - - if send_enabled: - # Open the window HERE, on the rising edge, rather than - # leaving it to the sender's first claim. The sender - # only runs when there is something to send, so a user - # who opts in and then opts out before any package - # exists would otherwise have no window to close, and - # record_revoked (which requires one) would no-op. - opt_in_period(connection) - elif previous == "1": - # `previous == "1"` is the true falling edge. Widening - # this to an unconditional else would be behaviourally - # equivalent today — record_revoked is idempotent and - # no-ops without an open window — so no test can tell - # the two apart. It is written as an edge anyway - # because that is the property intended, and a future - # change to record_revoked should not silently turn - # every disabled pass into a revocation. - record_revoked(connection) - - if previous != current: - connection.execute( - """ - INSERT INTO telemetry_state(key, value) VALUES (?, ?) - ON CONFLICT(key) DO UPDATE SET value = excluded.value - """, - (LAST_SEEN_SEND_KEY, current), - ) + reconcile_send_consent(connection, send_enabled) except Exception: logger.warning( "Unable to record a shared-metrics consent transition", @@ -1259,8 +1225,54 @@ def handles_hook(hook_name: str) -> bool: return hook_name in HANDLED_HOOKS and enabled() +_consent_reconcile_done = False + + +def _reconcile_send_consent_once() -> None: + """Reconcile consent windows with config, once per process. + + Runs BEFORE and INDEPENDENT of the collection gate — that placement is + the fix for the round-5 D1 leak, where the only idle-path consent + observer sat behind ``handles_hook()`` and became dead code the moment + ``enabled: false`` was set. A user with collection off still gets their + send-consent windows reconciled here. + + Skipped only when there is no store on disk AND consent is off: with no + store there are no packages, so there is nothing a window could protect, + and creating ``~/.hermes/telemetry`` for every fully-disabled user would + be a behaviour change in the wrong direction. + """ + global _consent_reconcile_done + if _consent_reconcile_done: + return + _consent_reconcile_done = True + try: + from hermes_cli.config import read_raw_config_readonly + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_send_config import ( + resolve_send_config, + ) + from hermes_cli.observability.shared_metrics_sender import ( + reconcile_send_consent, + ) + from hermes_cli.sqlite_util import write_txn + + resolved = resolve_send_config(read_raw_config_readonly() or {}) + store = SharedMetricsStore() + if not resolved.send and not store.database_path.exists(): + return + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, resolved.send) + except Exception: + logger.warning( + "Unable to reconcile shared-metrics send consent", exc_info=True + ) + + def observe_lifecycle(hook_name: str, **kwargs: Any) -> None: """Project one Hermes lifecycle event into the core Relay integration.""" + _reconcile_send_consent_once() if not handles_hook(hook_name): return if not relay_runtime.relay_instrumentation_enabled(): diff --git a/hermes_cli/observability/shared_metrics.py b/hermes_cli/observability/shared_metrics.py index bf5c1fb0bf..32dfc4fff6 100644 --- a/hermes_cli/observability/shared_metrics.py +++ b/hermes_cli/observability/shared_metrics.py @@ -338,6 +338,7 @@ class SharedMetricsStore: """ ) SharedMetricsStore._add_send_columns(connection) + SharedMetricsStore._add_consent_tables(connection) connection.execute( """ INSERT INTO telemetry_state(key, value) @@ -383,6 +384,55 @@ class SharedMetricsStore: f"ALTER TABLE package_outbox ADD COLUMN {column} {declaration}" ) + @staticmethod + def _add_consent_tables(connection: sqlite3.Connection) -> None: + """Create the consent-window tables, idempotently. + + Additive like ``_add_send_columns`` — the schema version is + deliberately NOT bumped, and old readers never touch these tables. + + ``send_consent_windows`` records consent as explicit intervals rather + than a moving day-stamp: a window is opened when send consent is + observed, heartbeat-confirmed on every later observation, and closed + at the LAST CONFIRMED moment (never "now") when consent is observed + withdrawn. Consent is asserted only for time that was actually + observed, so unobserved gaps — a hand-edited config with no process + running — fail closed by construction. + + ``consent_marks`` holds two monotonic high-water marks with strictly + separated roles: + + - ``obs``: the latest observation stamp ever seen. Advanced only by + the reconciler. Confirms consent and clamps window closes. + - ``data``: the latest package ``period_end`` ever stored. Advanced + only by the package writer. Clamps window OPENS, so a rolled-back + clock can never open a window underneath packages that already + exist on disk. + + The separation is load-bearing: letting data stamps confirm consent + re-created a refused-window leak (packages stored during an off + window would vouch for it), and letting observation stamps clamp + opens is not enough on its own to stop a rollback sliding a window + under existing refused data. + """ + connection.execute( + """ + CREATE TABLE IF NOT EXISTS send_consent_windows ( + opened_at TEXT NOT NULL, + last_confirmed_at TEXT NOT NULL, + closed_at TEXT + ) + """ + ) + connection.execute( + """ + CREATE TABLE IF NOT EXISTS consent_marks ( + name TEXT PRIMARY KEY CHECK (name IN ('obs', 'data')), + stamp TEXT NOT NULL + ) + """ + ) + @staticmethod def _create_counter_aggregates_table(connection: sqlite3.Connection) -> None: connection.execute( @@ -617,6 +667,16 @@ class SharedMetricsStore: payload["generated_at"], ), ) + # Advance the data high-water mark. This is the ONLY writer of the + # 'data' mark: it clamps consent-window opens so a rolled-back clock + # can never open a window underneath packages that already exist. + connection.execute( + """ + INSERT INTO consent_marks(name, stamp) VALUES ('data', ?) + ON CONFLICT(name) DO UPDATE SET stamp = MAX(stamp, excluded.stamp) + """, + (payload["period_end"],), + ) for row in rows: connection.execute( """ diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 49c2397f6e..389611ed37 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -17,7 +17,10 @@ file. See Appendix A.7 of ``docs/observability/relay-shared-metrics.md``. **Consent is gated on the package's PERIOD, not its creation time.** One period is split across packages created on different days, so a created-at gate would send a period's tail while dropping its head and silently -undercount the opt-in day. +undercount the first consented day. The gate itself is interval containment: +the period must fall entirely inside a recorded consent window +(``send_consent_windows``), maintained by the single ``reconcile_send_consent`` +writer below. """ from __future__ import annotations @@ -88,18 +91,6 @@ _PERMANENT_STATUSES = frozenset({400, 413}) #: doomed package at the head of the queue. MAX_SEND_ATTEMPTS = 25 -OPT_IN_PERIOD_KEY = "send_opt_in_period" - -#: Set when sending is turned off, cleared by the next enabled pass (which -#: also advances OPT_IN_PERIOD_KEY). This is what makes consent revocation -#: permanent for the packages collected while it was off. -SEND_REVOKED_KEY = "send_revoked" - -#: Last send-consent state this machine observed ("1"/"0"). Persisted because -#: each hook fires in a fresh process, so a true->false edge is only visible -#: by comparing against what was recorded last time. -LAST_SEEN_SEND_KEY = "send_last_seen" - def _utc_now() -> datetime: return datetime.now(timezone.utc) @@ -173,46 +164,86 @@ def _retry_after_seconds(value: str | None, default: int) -> int: return default -def opt_in_period(connection: sqlite3.Connection, *, now: datetime | None = None) -> str: - """Return the day (UTC) from which packages may be sent. +def reconcile_send_consent( + connection: sqlite3.Connection, + send_enabled: bool, + *, + now: datetime | None = None, +) -> None: + """Reconcile the consent-window table with the observed config state. - Must run inside a write transaction. + THE ONLY writer of consent state. Must run inside a write transaction. + A pure function of (config, now, store): call it from anywhere, any + number of times, in any order — the resulting windows are the same. This + replaces the previous edge-detection design, whose three partial + observers (wizard, relay, mid-pass) each covered a different subset of + transitions and repeatedly leaked the transitions between the subsets. - This is the CURRENT consent window's start, not a permanent first-ever - opt-in date. If the user previously turned sending off, ``record_revoked`` - stamps that; the next enabled pass advances the gate to the day sending - resumed, so packages collected during the opted-out window are never - transmitted. Without that advance, re-enabling would retroactively release - the entire period the user had explicitly refused. + Timestamp discipline (each rule is load-bearing; see the validation + harness in tests/hermes_cli/test_shared_metrics_consent_windows.py): + + - The 'obs' mark advances to every observation stamp, monotonically. + An open window's ``last_confirmed_at`` follows it: consent is asserted + only for time that was actually observed. + - A close is stamped at ``last_confirmed_at`` — never "now" — so an + unobserved gap (hand-edited config, machine off for 90 days) is never + inside a window and fails closed. + - An open clamps to ``max(now, obs, data)``: a rolled-back clock cannot + open a window underneath refused packages already on disk, and cannot + make the new window adjacent to the previous close. """ - today = (now or _utc_now()).date().isoformat() + stamp = _isoformat(now or _utc_now()) + connection.execute( + """ + INSERT INTO consent_marks(name, stamp) VALUES ('obs', ?) + ON CONFLICT(name) DO UPDATE SET stamp = MAX(stamp, excluded.stamp) + """, + (stamp,), + ) + marks = dict( + connection.execute("SELECT name, stamp FROM consent_marks").fetchall() + ) + obs = marks["obs"] # >= stamp; immune to clock rollback + data = marks.get("data") - revoked = _state_get(connection, SEND_REVOKED_KEY) - if revoked: - # Sending resumed after a revocation: the new window starts today. - _state_set(connection, OPT_IN_PERIOD_KEY, today) + open_row = connection.execute( + "SELECT rowid FROM send_consent_windows WHERE closed_at IS NULL" + ).fetchone() + + if send_enabled: + if open_row is None: + opened = max(x for x in (obs, data) if x is not None) + connection.execute( + "INSERT INTO send_consent_windows(opened_at, last_confirmed_at)" + " VALUES (?, ?)", + (opened, opened), + ) + else: + connection.execute( + "UPDATE send_consent_windows" + " SET last_confirmed_at = MAX(last_confirmed_at, ?)" + " WHERE rowid = ?", + (obs, open_row[0]), + ) + elif open_row is not None: connection.execute( - "DELETE FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,) + "UPDATE send_consent_windows SET closed_at = last_confirmed_at" + " WHERE rowid = ?", + (open_row[0],), ) - return today - - existing = _state_get(connection, OPT_IN_PERIOD_KEY) - if existing: - return existing - - _state_set(connection, OPT_IN_PERIOD_KEY, today) - return today -def record_revoked(connection: sqlite3.Connection) -> None: - """Mark that sending was turned off, closing the current consent window. - - Idempotent. The marker is only cleared by the next enabled pass, which - also advances the gate — so any package collected between the two events - stays local permanently. - """ - if _state_get(connection, OPT_IN_PERIOD_KEY): - _state_set(connection, SEND_REVOKED_KEY, "1") +#: Claim-time consent predicate: the package's period must fall entirely +#: inside SOME recorded consent window. An open window vouches only up to its +#: last confirmed moment, so a package whose period runs past it waits for +#: the next reconcile heartbeat (fail-closed; released within one hook fire). +CONSENT_GATE_SQL = """EXISTS ( + SELECT 1 FROM send_consent_windows w + WHERE package_outbox.period_start >= w.opened_at + AND package_outbox.period_end <= + CASE WHEN w.closed_at IS NULL THEN w.last_confirmed_at + ELSE w.closed_at END +)""" def _state_get(connection: sqlite3.Connection, key: str) -> str | None: @@ -278,7 +309,6 @@ class SharedMetricsSender: """ with self._store._connection() as connection: with write_txn(connection): - period = opt_in_period(connection, now=now) stamp = _isoformat(now) lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) @@ -286,6 +316,10 @@ class SharedMetricsSender: exclusion = ( f" AND package_id NOT IN ({placeholders})" if seen else "" ) + # Consent is a READ here — the claim must never mutate the + # window table. The old design's opt_in_period() call at this + # exact spot meant selecting a row could rewrite what was + # permitted to be sent (and did, under a rolled-back clock). row = connection.execute( f""" SELECT package_id, payload_json, sent_install_id @@ -293,13 +327,13 @@ class SharedMetricsSender: WHERE exported_at IS NOT NULL AND (send_state IS NULL OR send_state = 'pending') AND (next_attempt_at IS NULL OR next_attempt_at <= ?) - AND substr(period_start, 1, 10) >= ? + AND {CONSENT_GATE_SQL} AND send_attempts < ? {exclusion} ORDER BY created_at, package_id LIMIT 1 """, - (stamp, period, MAX_SEND_ATTEMPTS, *sorted(seen)), + (stamp, MAX_SEND_ATTEMPTS, *sorted(seen)), ).fetchone() if row is None: return None @@ -523,13 +557,12 @@ class SharedMetricsSender: for _ in range(MAX_PACKAGES_PER_PASS): if not self._still_consented(): # The user turned sending off while this pass was running. - # Stop without transmitting anything further, and close the - # consent window. This covers only the mid-pass case; a - # revocation made while no pass is running is caught by the - # relay's edge detector before it early-returns, because this - # loop would never run to observe it. + # Stop without transmitting anything further, and reconcile + # so the window closes at its last confirmed moment. This is + # the same single writer every other observation point uses — + # not a separate recording mechanism. logger.info("Shared-metrics sending disabled mid-pass; stopping") - self._record_revocation() + self._reconcile(send_enabled=False) break try: package = self._claim_next(self._now(), seen) @@ -561,14 +594,18 @@ class SharedMetricsSender: outcome.deferred += 1 return outcome - def _record_revocation(self) -> None: - """Close the consent window after an observed revocation.""" + def _reconcile(self, *, send_enabled: bool) -> None: + """Run the single consent writer from within a pass.""" try: with self._store._connection() as connection: with write_txn(connection): - record_revoked(connection) + reconcile_send_consent( + connection, send_enabled, now=self._now() + ) except Exception: - logger.debug("Unable to record consent revocation", exc_info=True) + logger.warning( + "Unable to reconcile shared-metrics consent", exc_info=True + ) def _still_consented(self) -> bool: """Re-read profile-owned send consent. diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 5a000d0374..6af33a000d 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2481,43 +2481,28 @@ def setup_telemetry(config: dict): def _record_send_consent_change(*, enabled: bool) -> None: - """Persist a consent transition at the moment the user makes it. + """Reconcile consent windows at the moment the user decides. - Enabling stamps the day so the gate excludes anything collected earlier. - Disabling stamps a revocation so that if the user ever re-enables, the - packages collected while sending was off are never released — the doc - promises `send: false` means no further packages leave the machine, and - that has to survive a later change of mind. + Same single writer as the relay and the sender — reconciliation derives + the window state from the observation, so wizard, relay, and mid-pass + callers cannot disagree. The relay's once-per-process reconcile would + catch this on the next hook fire anyway; running it here just makes the + wizard's effect immediate. """ try: from hermes_cli.observability.shared_metrics import SharedMetricsStore from hermes_cli.observability.shared_metrics_sender import ( - LAST_SEEN_SEND_KEY, - opt_in_period, - record_revoked, + reconcile_send_consent, ) from hermes_cli.sqlite_util import write_txn store = SharedMetricsStore() with store._connection() as connection: with write_txn(connection): - if enabled: - opt_in_period(connection) - else: - record_revoked(connection) - # Keep the relay's edge detector in step. Without this the - # wizard's change looks like "no transition" on the next hook - # fire, and a later true->false edge could be missed. - connection.execute( - """ - INSERT INTO telemetry_state(key, value) VALUES (?, ?) - ON CONFLICT(key) DO UPDATE SET value = excluded.value - """, - (LAST_SEEN_SEND_KEY, "1" if enabled else "0"), - ) + reconcile_send_consent(connection, enabled) except Exception: - # Never block the wizard on telemetry bookkeeping. The sender records - # the same transitions on its next pass. + # Never block the wizard on telemetry bookkeeping. The relay runs the + # same reconciliation on the next lifecycle hook. logger.debug("Unable to record shared-metrics consent change", exc_info=True) diff --git a/scripts/e2e_shared_metrics_staging.py b/scripts/e2e_shared_metrics_staging.py index 3e52e61a7a..e0c497a342 100644 --- a/scripts/e2e_shared_metrics_staging.py +++ b/scripts/e2e_shared_metrics_staging.py @@ -62,6 +62,31 @@ def main() -> int: ) today = datetime.now(timezone.utc).date().isoformat() + # The generator only exports COMPLETED periods, so the realistic E2E + # package is yesterday's. It also has to be: the consent gate only + # releases a package once its whole period is confirmed consented, and + # today's period cannot be confirmed before it ends. + from datetime import timedelta + + period_day = ( + datetime.now(timezone.utc).date() - timedelta(days=1) + ).isoformat() + + # Open the consent window before the period, confirm it after — exactly + # what the runtime reconciler does across two days of hook fires. + from hermes_cli.observability.shared_metrics_sender import ( + reconcile_send_consent, + ) + from hermes_cli.sqlite_util import write_txn + + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent( + connection, + True, + now=datetime.now(timezone.utc) - timedelta(days=2), + ) + reconcile_send_consent(connection, True) real_install_id = str(uuid.uuid4()) packages = [] @@ -77,8 +102,8 @@ def main() -> int: "generated_at": datetime.now(timezone.utc).isoformat().replace( "+00:00", "Z" ), - "period_start": f"{today}T00:00:00Z", - "period_end": f"{today}T23:59:59Z", + "period_start": f"{period_day}T00:00:00Z", + "period_end": f"{period_day}T23:59:59Z", "resource": { "hermes_version": "e2e-test", "os_family": "macos", @@ -105,11 +130,11 @@ def main() -> int: """, ( package_id, - f"{today}T00:00:00Z", - f"{today}T23:59:59Z", + f"{period_day}T00:00:00Z", + f"{period_day}T23:59:59Z", json.dumps(payload), - f"{today}T0{index}:00:00Z", - f"{today}T0{index}:00:01Z", + f"{period_day}T0{index}:00:00Z", + f"{period_day}T0{index}:00:01Z", ), ) packages.append((package_id, metric_count)) diff --git a/tests/hermes_cli/test_setup_telemetry.py b/tests/hermes_cli/test_setup_telemetry.py index 4f66259eaa..2397524343 100644 --- a/tests/hermes_cli/test_setup_telemetry.py +++ b/tests/hermes_cli/test_setup_telemetry.py @@ -33,7 +33,10 @@ def test_disabling_collection_closes_the_send_consent_window(monkeypatch, tmp_pa package collected in between. """ from hermes_cli.observability.shared_metrics import SharedMetricsStore - from hermes_cli.observability.shared_metrics_sender import SEND_REVOKED_KEY + from hermes_cli.observability.shared_metrics_sender import ( + reconcile_send_consent, + ) + from hermes_cli.sqlite_util import write_txn store = SharedMetricsStore( database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" @@ -48,24 +51,21 @@ def test_disabling_collection_closes_the_send_consent_window(monkeypatch, tmp_pa "hermes_cli.setup.prompt_yes_no", lambda _question, default: False ) config = {"telemetry": {"shared_metrics": {"enabled": True, "send": True}}} - # Consent was granted earlier, so a window is already open — that is - # precisely the state whose closure must be recorded. - from hermes_cli.sqlite_util import write_txn - from hermes_cli.observability.shared_metrics_sender import opt_in_period - + # Consent was granted earlier, so a window is open — that is precisely + # the state whose closure must be recorded. with store._connection() as connection: with write_txn(connection): - opt_in_period(connection) + reconcile_send_consent(connection, True) setup_telemetry(config) assert config["telemetry"]["shared_metrics"]["enabled"] is False assert config["telemetry"]["shared_metrics"]["send"] is False with store._connection() as connection: - row = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,) - ).fetchone() - assert row is not None and row[0] == "1", ( + open_windows = connection.execute( + "SELECT COUNT(*) FROM send_consent_windows WHERE closed_at IS NULL" + ).fetchone()[0] + assert open_windows == 0, ( "disabling collection left the send consent window open" ) diff --git a/tests/hermes_cli/test_shared_metrics_consent_windows.py b/tests/hermes_cli/test_shared_metrics_consent_windows.py new file mode 100644 index 0000000000..83da1a4749 --- /dev/null +++ b/tests/hermes_cli/test_shared_metrics_consent_windows.py @@ -0,0 +1,221 @@ +"""Property tests for the consent-interval model. + +Ported from the /tmp validation harness that gated the redesign: every +scenario here is a defect that actually occurred (rounds 3-5) or a clock +adversary the day-stamp model could not survive. The v1 and v2 drafts of the +redesign each FAILED scenarios in this file before shipping — that is the +harness working, and why these run against the real store and the real +reconciler rather than a model of them. +""" + +from __future__ import annotations + +import json +from datetime import datetime, timedelta, timezone + +import pytest + +from hermes_cli.observability.shared_metrics import SharedMetricsStore +from hermes_cli.observability.shared_metrics_sender import ( + CONSENT_GATE_SQL, + reconcile_send_consent, +) +from hermes_cli.sqlite_util import write_txn + +T0 = datetime(2026, 8, 1, tzinfo=timezone.utc) + + +def ts(days=0, hours=0): + return (T0 + timedelta(days=days, hours=hours)).isoformat().replace( + "+00:00", "Z" + ) + + +def dt(days=0, hours=0): + return T0 + timedelta(days=days, hours=hours) + + +@pytest.fixture +def store(tmp_path): + return SharedMetricsStore( + database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" + ) + + +def _add(store, pid, start, end): + """Store a package the way the generator does: at period end.""" + with store._connection() as connection: + with write_txn(connection): + connection.execute( + "INSERT INTO package_outbox(package_id, period_start, period_end," + " payload_json, created_at, exported_at) VALUES (?, ?, ?, ?, ?, ?)", + (pid, start, end, json.dumps({"package_id": pid}), end, end), + ) + connection.execute( + """INSERT INTO consent_marks(name, stamp) VALUES ('data', ?) + ON CONFLICT(name) DO UPDATE SET stamp = MAX(stamp, excluded.stamp)""", + (end,), + ) + + +def _observe(store, send_enabled, when): + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, send_enabled, now=when) + + +def _eligible(store): + with store._connection() as connection: + return sorted( + row[0] + for row in connection.execute( + f"SELECT package_id FROM package_outbox WHERE {CONSENT_GATE_SQL}" + ) + ) + + +def _windows(store): + with store._connection() as connection: + return [ + tuple(row) + for row in connection.execute( + "SELECT opened_at, last_confirmed_at, closed_at" + " FROM send_consent_windows ORDER BY opened_at" + ) + ] + + +class TestRefusedWindowIsNeverReleased: + def test_on_off_on_with_realistic_interleaving(self, store): + """Rounds 3 and 5: the refused middle must never transmit, and + neither consented era may be lost.""" + _observe(store, True, dt(0)) + for n in range(5): + _add(store, f"d{n:02d}", ts(days=n), ts(days=n + 1)) + _observe(store, True, dt(days=n + 1)) + _observe(store, False, dt(5)) + for n in range(5, 10): + _add(store, f"d{n:02d}", ts(days=n), ts(days=n + 1)) + _observe(store, True, dt(10)) + for n in range(10, 15): + _add(store, f"d{n:02d}", ts(days=n), ts(days=n + 1)) + _observe(store, True, dt(days=n + 1)) + + eligible = _eligible(store) + assert not [p for p in eligible if 5 <= int(p[1:]) < 10], eligible + assert [f"d{n:02d}" for n in range(5)] == eligible[:5], ( + "pre-revocation consented backlog was destroyed" + ) + assert [f"d{n:02d}" for n in range(10, 15)] == eligible[5:], eligible + + def test_hand_edit_with_a_90_day_silent_gap(self, store): + """Round 5 D1, strongest form: NOTHING observes the off window. + + The close back-dates to the last confirmed moment, so the unobserved + gap is outside every window and fails closed. + """ + _observe(store, True, dt(0)) + _add(store, "consented", ts(0, 1), ts(0, 2)) + _observe(store, True, dt(0, 6)) + for n in range(1, 90, 10): + _add(store, f"REFUSED-d{n}", ts(days=n), ts(days=n, hours=1)) + _observe(store, False, dt(90)) # first observation: boot on day 90 + _observe(store, True, dt(91)) + _observe(store, True, dt(92)) + + eligible = _eligible(store) + assert not [p for p in eligible if p.startswith("REFUSED")], eligible + assert "consented" in eligible, ( + "the confirmed-morning package must survive the reconciliation" + ) + + +class TestClockAdversaries: + def test_rollback_at_re_enable_releases_nothing(self, store): + """Round 5 D2: the data mark clamps opens above existing packages.""" + _observe(store, True, dt(0)) + _observe(store, True, dt(5)) + _observe(store, False, dt(5)) + for n in range(1, 4): + _add(store, f"REFUSED-{n}", ts(days=5, hours=n), ts(days=5, hours=n + 1)) + _observe(store, True, dt(-12)) # 12-day rollback at re-enable + _observe(store, True, dt(-11)) + + during = [p for p in _eligible(store) if p.startswith("REFUSED")] + assert not during, f"rollback released refused packages: {during}" + + _observe(store, True, dt(20)) # clock recovers + _observe(store, True, dt(21)) + after = [p for p in _eligible(store) if p.startswith("REFUSED")] + assert not after, f"recovery released refused packages: {after}" + + def test_recovery_does_not_wedge_future_sending(self, store): + _observe(store, True, dt(0)) + _observe(store, False, dt(5)) + _observe(store, True, dt(-12)) + _observe(store, True, dt(20)) + _add(store, "post-recovery", ts(21), ts(21, 4)) + _observe(store, True, dt(22)) + assert "post-recovery" in _eligible(store) + + +class TestSubDayGranularity: + def test_intra_day_refusal_holds_back_the_whole_day_package(self, store): + """Round 5 D3: a day package spanning a refused stretch must wait.""" + _observe(store, True, dt(0)) + _observe(store, True, dt(10, 9)) + _observe(store, False, dt(10, 9)) + _observe(store, True, dt(10, 18)) + _observe(store, True, dt(11, 2)) + _add(store, "halfday", ts(10), ts(11)) + assert "halfday" not in _eligible(store) + + +class TestReconcilerProperties: + def test_idempotent_under_replay(self, store): + for _ in range(4): + _observe(store, True, dt(0)) + _observe(store, False, dt(2)) + for _ in range(5): + _observe(store, False, dt(3)) + _observe(store, True, dt(4)) + for _ in range(3): + _observe(store, True, dt(5)) + assert len(_windows(store)) == 2 + + def test_the_observation_mark_is_monotonic(self, store): + """A rolled-back clock must never lower the observation high-water. + + Every downstream guarantee leans on this: closes clamp to it via + last_confirmed_at, and opens clamp to max(obs, data). Found as a + surviving mutant (obs upsert rewritten from MAX to overwrite) — + the leak scenarios happen to be covered by the data mark whenever a + leakable package exists, but the property itself must hold on its + own, not by coincidence of the sibling mark. + """ + _observe(store, True, dt(5)) + _observe(store, True, dt(0)) # rollback + with store._connection() as connection: + stamp = connection.execute( + "SELECT stamp FROM consent_marks WHERE name = 'obs'" + ).fetchone()[0] + assert stamp == ts(5), f"obs mark moved backwards: {stamp}" + + def test_the_gate_is_read_only(self, store): + _observe(store, True, dt(0)) + before = _windows(store) + for _ in range(10): + _eligible(store) + assert _windows(store) == before + + def test_no_window_fails_closed(self, store): + _add(store, "orphan", ts(0), ts(1)) + assert _eligible(store) == [] + + def test_fresh_package_waits_one_heartbeat_then_releases(self, store): + """The documented latency cost of confirmation-based windows.""" + _observe(store, True, dt(0)) + _add(store, "fresh", ts(0, 1), ts(0, 2)) + assert _eligible(store) == [] + _observe(store, True, dt(0, 3)) + assert _eligible(store) == ["fresh"] diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py index 3900847955..8ed7724b1e 100644 --- a/tests/hermes_cli/test_shared_metrics_send_wiring.py +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -209,13 +209,13 @@ class TestInteractivePathIsNotBlocked: runtime._join_send_thread(timeout=5) -class TestConsentRevocationWindow: - """The falling edge must close the window even with no pass running. +class TestConsentWindows: + """Consent reconciliation must work from the relay, in any order. - Round 3 recorded revocation inside the send loop, which cannot fire for - the dominant case: the user turns sending off while idle, so the relay - early-returns and no sender is ever built. Re-enabling then released - every package collected during the refused window. + Round 4's edge detector missed the idle-revocation path; round 5 found it + was also dead code whenever collection was off (handles_hook gated it). + These tests drive the relay entry points against the single reconciler + and assert on the interval table — the only consent state that exists. """ def _runtime(self, tmp_path): @@ -223,20 +223,19 @@ class TestConsentRevocationWindow: runtime.subscriber.store = RealBackedStore(tmp_path) return runtime - def _state(self, runtime, key): + def _windows(self, runtime): with runtime.subscriber.store._connection() as connection: - row = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (key,) - ).fetchone() - return row[0] if row else None + return [ + tuple(row) + for row in connection.execute( + "SELECT opened_at, last_confirmed_at, closed_at" + " FROM send_consent_windows ORDER BY opened_at" + ) + ] def test_revoking_while_idle_closes_the_window( self, monkeypatch, tmp_path, capture_sender ): - from hermes_cli.observability.shared_metrics_sender import ( - SEND_REVOKED_KEY, - ) - runtime = self._runtime(tmp_path) _set_config(monkeypatch, _config(enabled=True, send=True)) @@ -247,88 +246,101 @@ class TestConsentRevocationWindow: for _ in range(6): runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) == "1", ( - "revoking while no pass was running left the consent window open" + windows = self._windows(runtime) + assert windows and all(w[2] is not None for w in windows), ( + f"revoking while idle left a window open: {windows}" ) - def test_no_spurious_revocation_when_nothing_changes( + def test_replayed_observations_create_no_junk_windows( self, monkeypatch, tmp_path, capture_sender ): - """The detector must key on an EDGE, not on every disabled pass. - - A level trigger re-closes a window the user has since REOPENED: each - later disabled pass stamps revoked again, so the next enabled pass - advances the gate and silently drops packages the user did consent to. - Mutation-checked — an earlier version of this test used a - never-consented store, where record_revoked no-ops regardless, and so - could not tell an edge trigger from a level trigger. - """ - from hermes_cli.observability.shared_metrics_sender import ( - OPT_IN_PERIOD_KEY, - SEND_REVOKED_KEY, - ) - + """Reconciliation is idempotent — there is no edge to double-count.""" runtime = self._runtime(tmp_path) _set_config(monkeypatch, _config(enabled=True, send=True)) - runtime._send_exported_packages() - + for _ in range(4): + runtime._send_exported_packages() _set_config(monkeypatch, _config(enabled=True, send=False)) - runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) == "1" - - # User changes their mind and re-enables. + for _ in range(4): + runtime._send_exported_packages() _set_config(monkeypatch, _config(enabled=True, send=True)) - runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) is None, ( - "re-enabling must clear the revocation marker" - ) - reopened = self._state(runtime, OPT_IN_PERIOD_KEY) - - # Further ENABLED passes must not disturb the reopened window. for _ in range(4): runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) is None, ( - "a steady enabled state re-closed the consent window" - ) - assert self._state(runtime, OPT_IN_PERIOD_KEY) == reopened + assert len(self._windows(runtime)) == 2 - def test_a_never_consented_user_is_never_marked_revoked( + def test_a_never_consented_user_gets_no_window( self, monkeypatch, tmp_path, capture_sender ): - from hermes_cli.observability.shared_metrics_sender import ( - SEND_REVOKED_KEY, - ) - runtime = self._runtime(tmp_path) _set_config(monkeypatch, _config(enabled=True, send=False)) for _ in range(5): runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) is None + assert self._windows(runtime) == [] - def test_re_enabling_after_an_idle_revocation_starts_a_new_window( + def test_re_enabling_opens_a_new_window_after_the_refusal( self, monkeypatch, tmp_path, capture_sender ): - from hermes_cli.observability.shared_metrics_sender import ( - OPT_IN_PERIOD_KEY, - SEND_REVOKED_KEY, - ) - + """The refused gap must fall BETWEEN the two windows.""" runtime = self._runtime(tmp_path) _set_config(monkeypatch, _config(enabled=True, send=True)) runtime._send_exported_packages() - first_window = self._state(runtime, OPT_IN_PERIOD_KEY) - _set_config(monkeypatch, _config(enabled=True, send=False)) runtime._send_exported_packages() - assert self._state(runtime, SEND_REVOKED_KEY) == "1" - - # Re-enabling must not simply resume the original window. _set_config(monkeypatch, _config(enabled=True, send=True)) runtime._send_exported_packages() - assert first_window is not None + + windows = self._windows(runtime) + assert len(windows) == 2 + first, second = windows + assert first[2] is not None, "first window must be closed" + assert second[2] is None, "second window must be open" + assert second[0] >= first[2], ( + f"new window may not overlap the refused gap: {windows}" + ) + + def test_reconcile_runs_even_when_collection_is_disabled( + self, monkeypatch, tmp_path + ): + """Round-5 D1: enabled:false must not make consent handling dead code. + + The module-level once-per-process reconciler must close the window + regardless of handles_hook(). Drives the real observe_lifecycle gate + path: handles_hook is False throughout. + """ + from hermes_cli.observability.shared_metrics import SharedMetricsStore + from hermes_cli.observability.shared_metrics_sender import ( + reconcile_send_consent, + ) + from hermes_cli.sqlite_util import write_txn + + store = SharedMetricsStore( + database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" + ) + # A consent window is open from an earlier consented era. + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, True) + + monkeypatch.setattr( + "hermes_cli.observability.shared_metrics.SharedMetricsStore", + lambda *a, **k: store, + ) + _set_config(monkeypatch, _config(enabled=False, send=False)) + monkeypatch.setattr(mod, "_consent_reconcile_done", False) + + # The full lifecycle entry point, with collection OFF. + mod.observe_lifecycle("finish_task") + + with store._connection() as connection: + open_windows = connection.execute( + "SELECT COUNT(*) FROM send_consent_windows WHERE closed_at IS NULL" + ).fetchone()[0] + assert open_windows == 0, ( + "enabled:false made the consent reconciler unreachable (D1)" + ) + class TestFailureIsolation: diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index de489a36cb..6b59d5d69d 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -19,12 +19,11 @@ from hermes_cli.observability.shared_metrics_sender import ( MAX_ATTEMPTS, MAX_PACKAGES_PER_PASS, MAX_SEND_ATTEMPTS, - OPT_IN_PERIOD_KEY, REQUEST_TIMEOUT_SECONDS, SharedMetricsSender, - opt_in_period, - record_revoked, + reconcile_send_consent, ) +from hermes_cli.sqlite_util import write_txn INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" NOW = datetime(2026, 8, 26, 12, 0, tzinfo=timezone.utc) @@ -61,10 +60,45 @@ class FakeTransport: @pytest.fixture def store(tmp_path): - return SharedMetricsStore( + """A store with a broad consent window already open. + + Most tests exercise claiming/retry/transport, not the consent gate, and + the interval gate fails closed with no window. One window opened before + every test package and confirmed well past NOW keeps those tests about + what they are about. Gate tests clear it via _clear_consent. + """ + built = SharedMetricsStore( database_path=tmp_path / "metrics.sqlite3", outbox_directory=tmp_path / "outbox", ) + _grant_consent(built) + return built + + +def _grant_consent( + store, + opened=datetime(2026, 8, 20, tzinfo=timezone.utc), + confirmed_through=datetime(2026, 10, 1, tzinfo=timezone.utc), +): + """Open a consent window and heartbeat it forward, via the real writer.""" + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, True, now=opened) + reconcile_send_consent(connection, True, now=confirmed_through) + + +def _revoke_consent(store, at): + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, False, now=at) + + +def _clear_consent(store): + """Remove all consent state, for tests of the fail-closed default.""" + with store._connection() as connection: + with write_txn(connection): + connection.execute("DELETE FROM send_consent_windows") + connection.execute("DELETE FROM consent_marks") def _add_package(store, package_id, period_day, *, exported=True, install_id=INSTALL_ID): @@ -231,6 +265,9 @@ class TestContractResponses: class TestConsentGate: def test_packages_from_before_opt_in_are_never_sent(self, store): + # Consent opens on Aug 24; the "old" package's period predates it. + _clear_consent(store) + _grant_consent(store, opened=datetime(2026, 8, 24, tzinfo=timezone.utc)) _add_package(store, "old", "2026-08-20") _add_package(store, "new", "2026-08-26") transport = FakeTransport(FakeResponse(202)) @@ -245,19 +282,27 @@ class TestConsentGate: _sender(store, transport).send_pending() assert sorted(b["package_id"] for b in transport.bodies) == ["head", "tail"] - def test_opt_in_day_is_recorded_once_and_does_not_move(self, store): + def test_opt_in_is_immortalised_as_a_window_not_a_day(self, store): + """The window survives replayed observations without moving.""" with store._connection() as connection: - first = opt_in_period(connection, now=NOW) - later = opt_in_period(connection, now=NOW + timedelta(days=10)) - assert first == later == "2026-08-26" - - def test_opt_in_day_is_persisted(self, store): + rows = connection.execute( + "SELECT opened_at, closed_at FROM send_consent_windows" + ).fetchall() + assert len(rows) == 1 and rows[0][1] is None + _grant_consent(store) # replay: must not create a second window with store._connection() as connection: - opt_in_period(connection, now=NOW) - value = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (OPT_IN_PERIOD_KEY,) + count = connection.execute( + "SELECT COUNT(*) FROM send_consent_windows" ).fetchone()[0] - assert value == "2026-08-26" + assert count == 1 + + def test_no_consent_window_means_nothing_is_sent(self, store): + """The gate fails closed: absence of a window is absence of consent.""" + _clear_consent(store) + _add_package(store, "pkg-1", "2026-08-26") + transport = FakeTransport(FakeResponse(202)) + _sender(store, transport).send_pending() + assert transport.calls == [] def test_unexported_packages_are_skipped(self, store): _add_package(store, "pending-export", "2026-08-26", exported=False) @@ -266,32 +311,31 @@ class TestConsentGate: assert transport.calls == [] def test_revoking_then_re_enabling_never_releases_the_off_window(self, store): - """Regression: re-opt-in retroactively transmitted the refused window. + """The R3/R5 leak: re-opt-in must not release the refused interval. - opt_in_period was write-once, so packages collected while the user had - send: false still had period_start >= the ORIGINAL opt-in day. Turning - sending back on released the entire opted-out window — contradicting - the documented promise that `send: false` means no further packages - leave the machine. + Under the interval model the refused days fall BETWEEN two windows; + no later observation can place them inside one, so the property holds + for any number of on/off cycles — not just the single cycle the old + moving day-stamp was patched to survive. """ - _add_package(store, "consented", "2026-08-26") - with store._connection() as connection: - with __import__( - "hermes_cli.sqlite_util", fromlist=["write_txn"] - ).write_txn(connection): - opt_in_period(connection, now=NOW) + _clear_consent(store) + _grant_consent(store, opened=NOW - timedelta(days=2), confirmed_through=NOW) + _add_package(store, "consented", "2026-08-25") - # User turns sending off; packages keep being collected. - with store._connection() as connection: - with __import__( - "hermes_cli.sqlite_util", fromlist=["write_txn"] - ).write_txn(connection): - record_revoked(connection) + # User turns sending off; packages keep being collected for 3 days. + _revoke_consent(store, at=NOW) for day in ("2026-08-27", "2026-08-28", "2026-08-29"): _add_package(store, f"refused-{day}", day) - # User re-enables a few days later. + # User re-enables 5 days later; heartbeat confirms past the horizon. later = NOW + timedelta(days=5) + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, True, now=later) + reconcile_send_consent( + connection, True, now=later + timedelta(days=30) + ) + transport = FakeTransport(*[FakeResponse(202)] * 10) SharedMetricsSender( store, ENDPOINT, post=transport, sleep=lambda _s: None, now=lambda: later @@ -301,21 +345,30 @@ class TestConsentGate: assert not any("refused" in pid for pid in sent), ( f"transmitted packages collected while sending was off: {sent}" ) + # And the interval model's improvement over the day-stamp: the + # pre-revocation consented package is NOT collateral damage. + assert "consented" in sent, ( + "the consented backlog was destroyed by the revoke/re-enable cycle" + ) def test_a_package_from_after_re_enabling_is_sent(self, store): - """The revocation fix must not wedge sending off permanently.""" - with store._connection() as connection: - with __import__( - "hermes_cli.sqlite_util", fromlist=["write_txn"] - ).write_txn(connection): - opt_in_period(connection, now=NOW) - record_revoked(connection) + """The revocation handling must not wedge sending off permanently.""" + _clear_consent(store) + _grant_consent(store, opened=NOW - timedelta(days=2), confirmed_through=NOW) + _revoke_consent(store, at=NOW) later = NOW + timedelta(days=5) - _add_package(store, "after-re-optin", later.date().isoformat()) + with store._connection() as connection: + with write_txn(connection): + reconcile_send_consent(connection, True, now=later) + reconcile_send_consent( + connection, True, now=later + timedelta(days=10) + ) + _add_package(store, "after-re-optin", (later + timedelta(days=1)).date().isoformat()) transport = FakeTransport(FakeResponse(202)) SharedMetricsSender( - store, ENDPOINT, post=transport, sleep=lambda _s: None, now=lambda: later + store, ENDPOINT, post=transport, sleep=lambda _s: None, + now=lambda: later + timedelta(days=2), ).send_pending() assert len(transport.calls) == 1 diff --git a/tests/hermes_cli/test_shared_metrics_sender_e2e.py b/tests/hermes_cli/test_shared_metrics_sender_e2e.py index 552ae93552..9ff8bf9caf 100644 --- a/tests/hermes_cli/test_shared_metrics_sender_e2e.py +++ b/tests/hermes_cli/test_shared_metrics_sender_e2e.py @@ -77,10 +77,28 @@ def server(): @pytest.fixture def store(tmp_path): - return SharedMetricsStore( + built = SharedMetricsStore( database_path=tmp_path / "metrics.sqlite3", outbox_directory=tmp_path / "outbox", ) + # Open a consent window covering the fixture packages; the interval gate + # fails closed without one, and this file tests transport, not consent. + from datetime import datetime, timezone + + from hermes_cli.observability.shared_metrics_sender import ( + reconcile_send_consent, + ) + from hermes_cli.sqlite_util import write_txn + + with built._connection() as connection: + with write_txn(connection): + reconcile_send_consent( + connection, True, now=datetime(2026, 8, 20, tzinfo=timezone.utc) + ) + reconcile_send_consent( + connection, True, now=datetime(2026, 10, 1, tzinfo=timezone.utc) + ) + return built def _endpoint(server): From 67d152bc7e63048f6e92b82c84cdc21a84a1492d Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 13:30:25 +1000 Subject: [PATCH 14/20] fix(telemetry): bound forward-clock damage to the consent horizon Sixth review - the first against the interval architecture - verdict: the architecture holds (idempotence, order-independence, 4-process concurrent-writer safety, rollback immunity, format consistency, and a 120-permutation order sweep all verified), with ONE high finding, which I had independently reproduced while the review ran: the FORWARD clock adversary was unhandled, and unlike every other failure mode in this subsystem it failed OPEN. The 'obs' mark is a MAX-upsert - monotonic in the leak direction. One glitched-forward sample (NTP flap reading 2099) while consented dragged last_confirmed_at to 2099; a later revoke stamped closed_at = 2099; the closed window then CONTAINED every refused period that followed. Both the reviewer and I reproduced refused packages becoming gate-eligible. The rollback twin was mutation-tested since round 5; nobody had asked whether the mirror image existed. Two clamps, each covering what the other cannot: - The obs mark advances at most MAX_OBS_ADVANCE_SECONDS (30 days) per call. Honest heartbeats never bind it; a machine off for months catches up in a few hook fires (fail-closed latency only); one insane sample moves the horizon by a bounded step that real time overtakes. - A close is MIN(last_confirmed_at, closing observation's raw stamp). Confirmed-time keeps unobserved gaps out of windows (v1's leak); the raw stamp lets an honest clock at revoke time pull a poisoned horizon back to the true revoke moment. A rolled-back clock at close time only closes earlier - fail-closed. Also from the review: - D2: the data-mark advance in the REAL package writer had no coverage (the harness re-implemented the insert; deleting the production line survived 314 tests). Now driven through create_and_export_package_if_due. - D3: the "don't create ~/.hermes/telemetry for fully-disabled users" skip was dead code - the store constructor creates the directory before the exists() check ran. The probe now checks the default path without constructing; verified empirically on a fresh HERMES_HOME. - Upgrade note in A.4: pre-interval backlog is never transmitted after upgrade (fail-closed; deliberate). New harness scenarios: forward-poison-then-revoke (the leak), and forward-poison-cannot-wedge (the cap). Mutation check: unclamping the close, removing the cap, and removing the real writer's data-mark advance each fail the suite. 273 tests pass; ruff and windows-footguns clean; staging E2E 202. --- docs/observability/relay-shared-metrics.md | 15 +++- .../observability/relay_shared_metrics.py | 12 ++- .../observability/shared_metrics_sender.py | 57 ++++++++++-- .../test_shared_metrics_consent_windows.py | 89 +++++++++++++++++++ .../test_shared_metrics_send_wiring.py | 12 ++- 5 files changed, 175 insertions(+), 10 deletions(-) diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index b538011675..cb3ea2197a 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -403,12 +403,23 @@ Turning sending off also **closes the consent window** — at the last moment consent was actually observed, not at the wall clock. Packages whose periods fall between one window and the next are never transmitted, even if sending is later re-enabled, and this holds for any number of on/off cycles, across -hand-edits with no process running, and under a clock that jumps backwards -(window opens are clamped above every timestamp already in the store). +hand-edits with no process running, and under a clock that jumps in either +direction (window opens are clamped above every timestamp already in the +store; observation marks advance by a bounded step per call, so one glitched +forward sample cannot drag the confirmation horizon years ahead; a close +never lands after the closing observation's own clock). Unlike the earlier single moving opt-in date, closing and reopening does NOT discard the still-undelivered backlog from a previous consented window — those packages stay inside their own interval and remain eligible. +One deliberate upgrade-path consequence: packages exported under the +pre-interval consent model (before `send_consent_windows` existed) predate +the first recorded window and are therefore never transmitted after an +upgrade. This is the fail-closed direction — re-importing the old moving +day-stamp to release them would re-import the semantics five review rounds +showed to be unsound — and it costs at most the undelivered backlog, never +collected data. + ### A.5 Retention - **Local:** unchanged — 30 days for successfully exported history, and pending diff --git a/hermes_cli/observability/relay_shared_metrics.py b/hermes_cli/observability/relay_shared_metrics.py index 978497945d..5a97c8a18d 100644 --- a/hermes_cli/observability/relay_shared_metrics.py +++ b/hermes_cli/observability/relay_shared_metrics.py @@ -1256,11 +1256,19 @@ def _reconcile_send_consent_once() -> None: reconcile_send_consent, ) from hermes_cli.sqlite_util import write_txn + from hermes_constants import get_hermes_home resolved = resolve_send_config(read_raw_config_readonly() or {}) - store = SharedMetricsStore() - if not resolved.send and not store.database_path.exists(): + # Probe for an existing store WITHOUT constructing one: the + # constructor creates the directory and schema as a side effect, + # which round 6 caught making this skip dead code — every + # fully-disabled user was getting a ~/.hermes/telemetry directory. + default_path = ( + get_hermes_home() / "telemetry" / "shared_metrics" / "metrics.sqlite3" + ) + if not resolved.send and not default_path.exists(): return + store = SharedMetricsStore() with store._connection() as connection: with write_txn(connection): reconcile_send_consent(connection, resolved.send) diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 389611ed37..8a01a44790 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -100,6 +100,13 @@ def _isoformat(value: datetime) -> str: return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") +def _parse_stamp(value: str) -> datetime: + """Parse a stamp this module itself wrote (Z-suffixed ISO-8601, UTC).""" + return datetime.fromisoformat(value.replace("Z", "+00:00")).astimezone( + timezone.utc + ) + + @dataclass class SendOutcome: """What one pass did. Returned for tests and diagnostics.""" @@ -164,6 +171,18 @@ def _retry_after_seconds(value: str | None, default: int) -> int: return default +#: Maximum distance one reconcile call can advance the 'obs' mark. Honest +#: heartbeats arrive hours apart at most, so the cap never binds in normal +#: operation; a machine legitimately off for months catches up in a few +#: hook fires (fail-closed latency only). What it bounds is FORWARD clock +#: poison: without it, a single glitched sample (NTP flap reading 2099) +#: permanently drags the mark — and with it every window open and every +#: confirmation horizon — decades ahead, which round 6 reproduced as a +#: refused-data leak. Capped, one insane sample moves the mark at most +#: this far, and real time overtakes it again. +MAX_OBS_ADVANCE_SECONDS = 30 * 24 * 3600 + + def reconcile_send_consent( connection: sqlite3.Connection, send_enabled: bool, @@ -182,9 +201,15 @@ def reconcile_send_consent( Timestamp discipline (each rule is load-bearing; see the validation harness in tests/hermes_cli/test_shared_metrics_consent_windows.py): - - The 'obs' mark advances to every observation stamp, monotonically. - An open window's ``last_confirmed_at`` follows it: consent is asserted - only for time that was actually observed. + - The 'obs' mark advances to every observation stamp, monotonically — + but by at most ``MAX_OBS_ADVANCE_SECONDS`` per call. Unbounded, the + mark is monotonic in the LEAK direction: one glitched-forward sample + would drag ``last_confirmed_at`` decades ahead, a later close would + stamp that horizon, and the closed window would contain every future + refused period (reproduced in round 6). Bounded, a poisoned sample + costs at most one cap's width, and real time overtakes it. + An open window's ``last_confirmed_at`` follows the mark: consent is + asserted only for time that was actually observed. - A close is stamped at ``last_confirmed_at`` — never "now" — so an unobserved gap (hand-edited config, machine off for 90 days) is never inside a window and fails closed. @@ -193,6 +218,16 @@ def reconcile_send_consent( make the new window adjacent to the previous close. """ stamp = _isoformat(now or _utc_now()) + raw_stamp = stamp # pre-cap observation time, used to clamp closes + previous_obs = connection.execute( + "SELECT stamp FROM consent_marks WHERE name = 'obs'" + ).fetchone() + if previous_obs is not None: + ceiling = _isoformat( + _parse_stamp(str(previous_obs[0])) + + timedelta(seconds=MAX_OBS_ADVANCE_SECONDS) + ) + stamp = min(stamp, ceiling) connection.execute( """ INSERT INTO consent_marks(name, stamp) VALUES ('obs', ?) @@ -226,10 +261,22 @@ def reconcile_send_consent( (obs, open_row[0]), ) elif open_row is not None: + # Close at the last CONFIRMED moment, but never after the closing + # observation's own raw stamp. The two clamps serve different + # adversaries and both are load-bearing: + # - min with last_confirmed_at: an unobserved gap (machine off, + # hand-edited config) is never asserted as consented (v1's leak). + # - min with the RAW stamp (pre-cap, pre-MAX): if last_confirmed_at + # was poisoned by a glitched-forward sample, an honest clock at + # revoke time pulls the close back to the true revoke moment, so + # the refused era that follows falls OUTSIDE the closed window + # (round 6's D1 leak). A rolled-back clock at close time only + # closes EARLIER — fail-closed. connection.execute( - "UPDATE send_consent_windows SET closed_at = last_confirmed_at" + "UPDATE send_consent_windows" + " SET closed_at = MIN(last_confirmed_at, ?)" " WHERE rowid = ?", - (open_row[0],), + (raw_stamp, open_row[0]), ) diff --git a/tests/hermes_cli/test_shared_metrics_consent_windows.py b/tests/hermes_cli/test_shared_metrics_consent_windows.py index 83da1a4749..66b58d5dd3 100644 --- a/tests/hermes_cli/test_shared_metrics_consent_windows.py +++ b/tests/hermes_cli/test_shared_metrics_consent_windows.py @@ -131,6 +131,62 @@ class TestRefusedWindowIsNeverReleased: class TestClockAdversaries: + def test_forward_poison_then_revoke_releases_nothing(self, store): + """Round 6 D1: one glitched-forward sample must not defeat a close. + + Unfixed, the poisoned obs mark dragged last_confirmed_at to 2099, a + later revoke stamped closed_at = 2099, and the closed window then + CONTAINED every refused period that followed — all 8 refused + packages became eligible. The close now clamps to the closing + observation's own raw stamp, so an honest clock at revoke time pulls + the window back to the true revoke moment. + """ + _observe(store, True, dt(0)) + _observe(store, True, datetime(2099, 1, 1, tzinfo=timezone.utc)) + _observe(store, False, dt(1)) # honest clock at revoke + for n in range(2, 10): + _add(store, f"REFUSED-{n}", ts(days=n), ts(days=n, hours=2)) + + leaked = [p for p in _eligible(store) if p.startswith("REFUSED")] + assert not leaked, f"poisoned horizon released refused data: {leaked}" + + def test_forward_poison_cannot_wedge_consent_forever(self, store): + """The obs-advance cap bounds the damage of one insane sample. + + Uncapped, a 2099 sample would clamp every future window open at + 2099, suppressing consented data for decades (fail-closed but + permanent). Capped, the mark moves at most MAX_OBS_ADVANCE_SECONDS + past its previous value, so honest time overtakes it. + """ + from hermes_cli.observability.shared_metrics_sender import ( + MAX_OBS_ADVANCE_SECONDS, + ) + + _observe(store, True, dt(0)) + _observe(store, True, datetime(2099, 1, 1, tzinfo=timezone.utc)) + with store._connection() as connection: + stamp = connection.execute( + "SELECT stamp FROM consent_marks WHERE name = 'obs'" + ).fetchone()[0] + ceiling = ts(days=MAX_OBS_ADVANCE_SECONDS // 86_400) + assert stamp <= ceiling, ( + f"one glitched sample advanced the mark unboundedly: {stamp}" + ) + + # Consented data from shortly after the cap horizon still flows once + # honest observations catch the marks up. + horizon_days = MAX_OBS_ADVANCE_SECONDS // 86_400 + _add( + store, + "post-glitch", + ts(days=horizon_days + 1), + ts(days=horizon_days + 1, hours=4), + ) + _observe(store, True, dt(days=horizon_days + 2)) + assert "post-glitch" in _eligible(store), ( + "consent wedged after a forward glitch" + ) + def test_rollback_at_re_enable_releases_nothing(self, store): """Round 5 D2: the data mark clamps opens above existing packages.""" _observe(store, True, dt(0)) @@ -201,6 +257,39 @@ class TestReconcilerProperties: ).fetchone()[0] assert stamp == ts(5), f"obs mark moved backwards: {stamp}" + def test_the_real_package_writer_advances_the_data_mark(self, store): + """Round 6 D2: the harness's _add re-implements the data-mark insert, + so deleting the advance from the REAL writer survived 314 tests. + This drives the production exporter instead. + """ + from datetime import date, timedelta as _td + + yesterday = (date.today() - _td(days=1)).isoformat() + with store._connection() as connection: + with write_txn(connection): + connection.execute( + "INSERT INTO counter_aggregates(" + " period_start, metric_name, hermes_version, os_family," + " architecture, install_method, dimensions_json, value," + " packaged_value" + ") VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)", + ( + yesterday, "hermes.client.active", "0.0.0-test", + "macos", "arm64", "git", "{}", 1, 0, + ), + ) + + exported = store.create_and_export_package_if_due() + assert exported, "the generator was expected to export yesterday's period" + + with store._connection() as connection: + row = connection.execute( + "SELECT stamp FROM consent_marks WHERE name = 'data'" + ).fetchone() + assert row is not None and row[0] >= yesterday, ( + "the production package writer did not advance the data mark" + ) + def test_the_gate_is_read_only(self, store): _observe(store, True, dt(0)) before = _windows(store) diff --git a/tests/hermes_cli/test_shared_metrics_send_wiring.py b/tests/hermes_cli/test_shared_metrics_send_wiring.py index 8ed7724b1e..29f572c697 100644 --- a/tests/hermes_cli/test_shared_metrics_send_wiring.py +++ b/tests/hermes_cli/test_shared_metrics_send_wiring.py @@ -315,8 +315,18 @@ class TestConsentWindows: ) from hermes_cli.sqlite_util import write_txn + # Lay the store out exactly as production does, under a redirected + # HERMES_HOME: the boot reconciler probes the default path (without + # constructing the store — the constructor creates directories), so + # the probe and the store must agree the way they do in production. + home = tmp_path / "home" + monkeypatch.setattr( + "hermes_constants.get_hermes_home", lambda: home + ) + root = home / "telemetry" / "shared_metrics" store = SharedMetricsStore( - database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o" + database_path=root / "metrics.sqlite3", + outbox_directory=root / "outbox", ) # A consent window is open from an earlier consented era. with store._connection() as connection: From 60addb16e28eec4923c1e891bfbeaf3d2f0d7c8d Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 15:04:02 +1000 Subject: [PATCH 15/20] fix(telemetry): fence send authority on a per-claim token Responds to the independent PR review (andrexibiza). Both P1s were checked against current HEAD rather than taken on authority - the review was written against 613849c190, before the interval-model consent replacement landed. P1-1 (same-UTC-day revoke/re-enable releases refused data): already fixed by the interval model. The reviewer's exact reproduction - opt in 06:00, revoke 12:00, package collected 18:00, re-enable 20:00 same day - was re-run at HEAD: the off-window package stays local, and a full-day aggregate straddling the revocation boundary also stays local (period containment, timestamp precision). The consent-windows harness already pins both. The reviewer's related ask that consent-ledger persistence failures fail closed also holds structurally now: reconciliation derives state rather than recording transitions, so a lost write means a shorter confirmed horizon - less is released, never more. P1-2 (lease has no owner) was VALID at head. Reproduced exactly as described: A claims, is suspended past the 300s lease, B reclaims and POSTs, A resumes and POSTs again - and the ingest key is minute- prefixed, so the duplicate lands as a DISTINCT stored object, making this worse than a benign idempotent overwrite. Fix: every claim now mints a claim_token (additive nullable column, schema version unchanged). Ownership is revalidated immediately before every external POST, and every settlement, rejection, and backoff write is compare-and-set on (package_id, claim_token, pending). A lapsed claimant that resumes yields without transmitting, and its stale backoff cannot move next_attempt_at under the live claim's lease. Two deterministic regressions ship with it: expiry -> reclaim -> resume (the reviewer's schedule), and the subtler stale-backoff-clobber case. Honest scope, documented on _send_one: delivery remains at-least-once. The token closes the claim->POST gap; a suspension landing mid-POST (bytes already on the wire) is not client-revocable. The residual duplicate is byte-identical content; collapsing it fully needs package_id-keyed dedupe at the ingest service. 275 tests pass; ruff + footguns clean; staging E2E 202. --- hermes_cli/observability/shared_metrics.py | 6 + .../observability/shared_metrics_sender.py | 104 ++++++++++++++++-- .../hermes_cli/test_shared_metrics_sender.py | 81 ++++++++++++++ 3 files changed, 182 insertions(+), 9 deletions(-) diff --git a/hermes_cli/observability/shared_metrics.py b/hermes_cli/observability/shared_metrics.py index 32dfc4fff6..ddf570b6f9 100644 --- a/hermes_cli/observability/shared_metrics.py +++ b/hermes_cli/observability/shared_metrics.py @@ -378,6 +378,12 @@ class SharedMetricsStore: # Only the ~36-byte id is stored: the body is recomputed from # payload_json, whose serialisation is deterministic. ("sent_install_id", "TEXT"), + # NULL until first claimed; rewritten on every claim. Settlement + # and the pre-POST revalidation are compare-and-set on this, so a + # claimant whose lease lapsed loses authority the moment another + # process reclaims (PR-review finding: without it, a suspended + # sender resuming after a reclaim double-POSTs the package). + ("claim_token", "TEXT"), ): if column not in existing: connection.execute( diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 8a01a44790..1c06acc686 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -33,6 +33,7 @@ import sqlite3 import time import urllib.error import urllib.request +import uuid from dataclasses import dataclass from datetime import datetime, timedelta, timezone @@ -396,24 +397,31 @@ class SharedMetricsSender: # caller to continue rather than stop. return {"package_id": package_id, "skip": True} + token = str(uuid.uuid4()) connection.execute( """ UPDATE package_outbox SET send_state = 'pending', send_attempts = send_attempts + 1, - next_attempt_at = ? + next_attempt_at = ?, + claim_token = ? WHERE package_id = ? """, # Lease INTO THE FUTURE: selection requires # next_attempt_at <= now, so no other process can take # this row while it is in flight. Success or a real # backoff overwrites it; if this process dies, it expires. - (_isoformat(lease_until), package_id), + # The token is this claim's identity: a reclaim after + # expiry mints a new one, and every later write by THIS + # claimant is compare-and-set against it, so a lapsed + # claimant that resumes cannot settle or transmit. + (_isoformat(lease_until), token, package_id), ) return { "package_id": package_id, "payload_json": str(row[1]), "derived": str(derived), + "claim_token": token, "skip": False, } @@ -479,12 +487,25 @@ class SharedMetricsSender: payload = substitute_install_id(json.loads(payload_json), derived) return json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") - def _mark(self, package_id: str, *, only_if_pending: bool = True, **columns) -> None: + def _mark( + self, + package_id: str, + *, + only_if_pending: bool = True, + token: str | None = None, + **columns, + ) -> None: """Write send state for one package. Guarded on send_state so a pass whose lease lapsed cannot resurrect a row another process has already finished: without this, a slow sender could overwrite 'sent' back to 'pending' and cause a re-send. + + When ``token`` is given, the write is additionally compare-and-set on + claim_token: it lands only if THIS claim is still the current one. A + claimant that lapsed and was superseded writes zero rows — its + settlement, backoff, and error strings all silently lose to the + newer claim's, which is the correct outcome. """ assignments = ", ".join(f"{name} = ?" for name in columns) predicate = ( @@ -492,15 +513,48 @@ class SharedMetricsSender: if only_if_pending else "" ) + params: list = [*columns.values(), package_id] + if token is not None: + predicate += " AND claim_token = ?" + params.append(token) with self._store._connection() as connection: with write_txn(connection): connection.execute( f"UPDATE package_outbox SET {assignments} " f"WHERE package_id = ?{predicate}", - (*columns.values(), package_id), + params, ) - def _defer(self, package_id: str, delay_seconds: int, reason: str) -> None: + def _still_owns(self, package_id: str, token: str | None) -> bool: + """Return whether this pass's claim on the row is still current.""" + if token is None: + # Defensive: a package dict without a token (not produced by + # _claim_next today) gets no authority rather than unlimited. + return False + try: + with self._store._connection() as connection: + row = connection.execute( + "SELECT 1 FROM package_outbox" + " WHERE package_id = ? AND claim_token = ?" + " AND (send_state IS NULL OR send_state = 'pending')", + (package_id, token), + ).fetchone() + return row is not None + except Exception: + # If the check itself fails, do not transmit on stale authority. + logger.warning( + "Unable to verify shared-metrics claim ownership", exc_info=True + ) + return False + + def _defer( + self, + package_id: str, + delay_seconds: int, + reason: str, + *, + token: str | None = None, + ) -> None: # Defence in depth: no current caller can pass a non-positive delay # (Retry-After is already clamped to [1, 86400] when parsed, and every # other call site passes a positive constant), so this clamp is @@ -512,6 +566,7 @@ class SharedMetricsSender: retry_at = self._now().timestamp() + delay self._mark( package_id, + token=token, send_state="pending", next_attempt_at=_isoformat( datetime.fromtimestamp(retry_at, tz=timezone.utc) @@ -520,11 +575,33 @@ class SharedMetricsSender: ) def _send_one(self, package: dict) -> str: - """Try one package. Returns 'sent', 'rejected', or 'deferred'.""" + """Try one package. Returns 'sent', 'rejected', or 'deferred'. + + Delivery is at-least-once. The pre-POST ownership check plus the + token-fenced writes close the claim->POST and settle-after-reclaim + gaps, but a suspension landing MID-POST (bytes already on the wire + when the machine sleeps) can still duplicate: no client-side check + can revoke a request in flight. The body is byte-identical across + retries by construction, so the residual duplicate is exactly one + redundant copy of identical content; collapsing it fully would need + package_id-keyed dedupe at the ingest service. + """ package_id = package["package_id"] + token = package.get("claim_token") body = self._body(package["payload_json"], package["derived"]) for attempt in range(1, self._max_attempts + 1): + # Revalidate ownership immediately before the external POST. The + # claim can lapse between claiming and here — a suspended laptop, + # a GC pause, a long gzip — and another process may have + # reclaimed and transmitted. Without this check the resumed + # claimant POSTs a duplicate; the ingest key is minute-prefixed, + # so duplicates become distinct stored objects, not overwrites. + if not self._still_owns(package_id, token): + logger.info( + "Shared-metrics claim on %s superseded; yielding", package_id + ) + return "deferred" try: response = self._post( self._endpoint, body, timeout=REQUEST_TIMEOUT_SECONDS @@ -532,7 +609,9 @@ class SharedMetricsSender: except Exception as exc: # transport failure: offline, DNS, TLS reason = f"{type(exc).__name__}: {exc}" if attempt >= self._max_attempts: - self._defer(package_id, _FAILURE_BACKOFF_SECONDS, reason) + self._defer( + package_id, _FAILURE_BACKOFF_SECONDS, reason, token=token + ) return "deferred" self._sleep(self._backoff(attempt)) continue @@ -540,6 +619,7 @@ class SharedMetricsSender: if response.status == 202: self._mark( package_id, + token=token, send_state="sent", sent_at=_isoformat(self._now()), last_error=None, @@ -560,6 +640,7 @@ class SharedMetricsSender: ) self._mark( package_id, + token=token, send_state="rejected", last_error=f"HTTP {response.status}: {response.body[:400]}", ) @@ -570,17 +651,22 @@ class SharedMetricsSender: package_id, _retry_after_seconds(response.retry_after, _FAILURE_BACKOFF_SECONDS), "rate limited", + token=token, ) return "deferred" # 5xx and anything unexpected: retryable. reason = f"HTTP {response.status}" if attempt >= self._max_attempts: - self._defer(package_id, _FAILURE_BACKOFF_SECONDS, reason) + self._defer( + package_id, _FAILURE_BACKOFF_SECONDS, reason, token=token + ) return "deferred" self._sleep(self._backoff(attempt)) - self._defer(package_id, _FAILURE_BACKOFF_SECONDS, "attempts exhausted") + self._defer( + package_id, _FAILURE_BACKOFF_SECONDS, "attempts exhausted", token=token + ) return "deferred" @staticmethod diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index 6b59d5d69d..0080224ade 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -649,6 +649,87 @@ class TestClaimingAndBounds: f"{len(attempts)} requests burned on one doomed package" ) + def test_a_lapsed_claimant_resuming_after_reclaim_cannot_double_post( + self, store + ): + """PR-review P1: expiry -> reclaim -> old claimant resumes. + + A claims, then is suspended (laptop lid) BEFORE its POST. The lease + expires; B reclaims and POSTs; A wakes and proceeds. The pre-POST + ownership check must make A yield without transmitting. + + Scope note: the check closes the claim->POST gap. A suspension that + lands mid-POST (bytes already leaving) is not client-fixable — that + residual needs server-side dedupe and is documented on _send_one. + """ + _add_package(store, "pkg-1", "2026-08-26") + + posts = [] + + def post_a(endpoint, payload, *, timeout): + posts.append("A") + return FakeResponse(202) + + def post_b(endpoint, payload, *, timeout): + posts.append("B") + return FakeResponse(202) + + sender_a = SharedMetricsSender( + store, ENDPOINT, post=post_a, sleep=lambda _s: None, now=lambda: NOW + ) + # A claims, then the process is suspended before _send_one runs. + claimed_a = sender_a._claim_next(NOW, set()) + assert claimed_a is not None and not claimed_a["skip"] + + # 400s later (past the 300s lease) B claims and completes the send. + later = NOW + timedelta(seconds=400) + sender_b = SharedMetricsSender( + store, ENDPOINT, post=post_b, sleep=lambda _s: None, now=lambda: later + ) + outcome_b = sender_b.send_pending() + assert outcome_b.sent == 1 + + # A resumes exactly where it left off. + result_a = sender_a._send_one(claimed_a) + + row = _row(store, "pkg-1") + assert posts == ["B"], ( + f"a lapsed claimant transmitted after reclaim: {posts}" + ) + assert result_a == "deferred" + assert row["send_state"] == "sent", "B's settlement must stand" + + def test_a_lapsed_claimants_backoff_cannot_clobber_the_new_claim(self, store): + """The token must fence DEFERS too, not just the 202 settlement. + + A's transport fails after B has reclaimed; A's backoff write must + not move next_attempt_at under B's live lease. + """ + _add_package(store, "pkg-1", "2026-08-26") + sender_a = SharedMetricsSender( + store, ENDPOINT, + post=FakeTransport(OSError("net"), OSError("net"), OSError("net")), + sleep=lambda _s: None, now=lambda: NOW, + ) + claimed_a = sender_a._claim_next(NOW, set()) + assert claimed_a is not None and not claimed_a["skip"] + + later = NOW + timedelta(seconds=400) + sender_b = SharedMetricsSender( + store, ENDPOINT, post=FakeTransport(), + sleep=lambda _s: None, now=lambda: later, + ) + claimed_b = sender_b._claim_next(later, set()) + assert claimed_b is not None and not claimed_b["skip"] + lease_b = _row(store, "pkg-1")["next_attempt_at"] + + # A's exhausted retries try to write a 15-minute backoff. + result = sender_a._send_one(claimed_a) + assert result == "deferred" + assert _row(store, "pkg-1")["next_attempt_at"] == lease_b, ( + "a lapsed claimant's backoff overwrote the live claim's lease" + ) + def test_an_expired_lease_is_reclaimed(self, store): """A process killed mid-pass must not strand its packages.""" _add_package(store, "pkg-1", "2026-08-26") From 4bdabb21ed68da01012a4491f621ba02b45e3ba4 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 15:21:30 +1000 Subject: [PATCH 16/20] fix(telemetry): renew the claim atomically before every POST Seventh review found the claim-token fix incomplete, and its reproduction is exact: the pre-POST check was READ-ONLY. A claimant whose lease expired while suspended still passes it when it wakes BEFORE anyone reclaims - its token is still in the row - and then a second process legitimately reclaims while the first one's POST is in flight. Both send. Reproduced at 60addb16e2: posts ['B', 'A'], both reporting 'sent'. This is the check-to-POST expiry race, not the documented mid-POST residual: A's lease was already dead before its authority check passed. The check is now an atomic RENEWAL (single CAS UPDATE): it requires the token to match, the row to be pending, AND the current lease to be unexpired, and only then extends next_attempt_at a fresh lease into the future. rowcount == 1 is the only grant. A claimant that wakes past its own lease fails the unexpired condition and yields even though its token was never replaced - expiry alone means another process may claim at any moment, so waking stale is disqualifying regardless of whether anyone has taken the row yet. The renewed lease (300s) covers the POST (30s timeout) with margin, and renewal runs before every retry, not just the first attempt. Regressions: the reviewer's exact ordering (expired wake before any reclaim -> zero POSTs, row stays claimable), plus a healthy-claimant renewal test. Mutation-checked: dropping the lease-unexpired condition or the token condition each fails the suite. The at-least-once scope note on _send_one stands: a suspension landing mid-POST remains client-unfixable; the fixable window is now closed on both sides (before the check, and between check and POST). 277 tests pass; ruff + footguns clean; staging E2E 202. --- .../observability/shared_metrics_sender.py | 73 +++++++++++++------ .../hermes_cli/test_shared_metrics_sender.py | 47 ++++++++++++ 2 files changed, 99 insertions(+), 21 deletions(-) diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 1c06acc686..8e6f81c4b8 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -525,25 +525,53 @@ class SharedMetricsSender: params, ) - def _still_owns(self, package_id: str, token: str | None) -> bool: - """Return whether this pass's claim on the row is still current.""" + def _renew_claim(self, package_id: str, token: str | None) -> bool: + """Atomically re-assert ownership and extend the lease. CAS, one row. + + A read-only ownership check is not enough: a claimant whose lease + expired while suspended can pass the check (its token is still in + the row if no one reclaimed yet) and then POST while another process + legitimately reclaims — the check-to-POST expiry race a seventh + review reproduced. Renewal closes it by requiring, in ONE statement: + + - the token still matches (nobody reclaimed), AND + - the current lease is UNEXPIRED (this claimant is not stale), AND + - the row is still pending, + + and only then pushing next_attempt_at a fresh lease into the future, + so the upcoming POST (30s timeout, well under the 300s lease) runs + entirely inside renewed authority. rowcount == 1 is the only grant. + A claimant that wakes past its own lease fails the unexpired + condition and yields even though its token was never replaced. + """ if token is None: - # Defensive: a package dict without a token (not produced by - # _claim_next today) gets no authority rather than unlimited. return False try: + now = self._now() + lease_until = now + timedelta(seconds=_CLAIM_LEASE_SECONDS) with self._store._connection() as connection: - row = connection.execute( - "SELECT 1 FROM package_outbox" - " WHERE package_id = ? AND claim_token = ?" - " AND (send_state IS NULL OR send_state = 'pending')", - (package_id, token), - ).fetchone() - return row is not None + with write_txn(connection): + cursor = connection.execute( + """ + UPDATE package_outbox + SET next_attempt_at = ? + WHERE package_id = ? + AND claim_token = ? + AND (send_state IS NULL OR send_state = 'pending') + AND next_attempt_at > ? + """, + ( + _isoformat(lease_until), + package_id, + token, + _isoformat(now), + ), + ) + return cursor.rowcount == 1 except Exception: - # If the check itself fails, do not transmit on stale authority. + # If renewal itself fails, do not transmit on unproven authority. logger.warning( - "Unable to verify shared-metrics claim ownership", exc_info=True + "Unable to renew shared-metrics claim", exc_info=True ) return False @@ -591,15 +619,18 @@ class SharedMetricsSender: body = self._body(package["payload_json"], package["derived"]) for attempt in range(1, self._max_attempts + 1): - # Revalidate ownership immediately before the external POST. The - # claim can lapse between claiming and here — a suspended laptop, - # a GC pause, a long gzip — and another process may have - # reclaimed and transmitted. Without this check the resumed - # claimant POSTs a duplicate; the ingest key is minute-prefixed, - # so duplicates become distinct stored objects, not overwrites. - if not self._still_owns(package_id, token): + # Atomically renew the claim before EVERY external POST. The + # renewal is compare-and-set on (token, pending, lease unexpired) + # and extends the lease past the request, so a suspended-then- + # resumed claimant whose lease lapsed yields here even if nobody + # has reclaimed yet — a read-only ownership check passed in that + # state and still double-sent (check-to-POST expiry race). The + # ingest key is minute-prefixed, so duplicates become distinct + # stored objects, not overwrites. + if not self._renew_claim(package_id, token): logger.info( - "Shared-metrics claim on %s superseded; yielding", package_id + "Shared-metrics claim on %s superseded or expired; yielding", + package_id, ) return "deferred" try: diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index 0080224ade..63fe022475 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -649,6 +649,53 @@ class TestClaimingAndBounds: f"{len(attempts)} requests burned on one doomed package" ) + def test_a_lapsed_claimant_yields_even_before_anyone_reclaims(self, store): + """Seventh review: the check-to-POST expiry race. + + A claims, sleeps past its own lease, and wakes BEFORE any other + process reclaims. Its token is still in the row, so a read-only + ownership check passes — and then B reclaims while A's POST is in + flight: both send. The pre-POST renewal must instead REJECT a + claimant whose lease already expired, whether or not anyone has + reclaimed yet, because expiry alone means another process may claim + at any moment. + """ + _add_package(store, "pkg-1", "2026-08-26") + + posts = [] + sender_a = SharedMetricsSender( + store, ENDPOINT, + post=lambda e, p, *, timeout: (posts.append("A"), FakeResponse(202))[1], + sleep=lambda _s: None, + now=lambda: clock["t"], + ) + clock = {"t": NOW} + claimed = sender_a._claim_next(NOW, set()) + assert claimed is not None and not claimed["skip"] + + # Suspended past the 300s lease; wakes with the row NOT yet reclaimed. + clock["t"] = NOW + timedelta(seconds=400) + result = sender_a._send_one(claimed) + + assert posts == [], ( + "a claimant with an expired lease transmitted before renewal" + ) + assert result == "deferred" + # The row must remain claimable by the next process. + row = _row(store, "pkg-1") + assert row["send_state"] == "pending" + + def test_renewal_extends_the_lease_across_the_post(self, store): + """A healthy in-lease claimant renews and its POST is covered.""" + _add_package(store, "pkg-1", "2026-08-26") + sender = _sender(store, FakeTransport(FakeResponse(202))) + claimed = sender._claim_next(NOW, set()) + assert claimed is not None + lease_before = _row(store, "pkg-1")["next_attempt_at"] + + assert sender._renew_claim("pkg-1", claimed["claim_token"]) is True + assert _row(store, "pkg-1")["next_attempt_at"] >= lease_before + def test_a_lapsed_claimant_resuming_after_reclaim_cannot_double_post( self, store ): From ecf327c87277aa3d71addaa7f0191a943c8b35c7 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Thu, 27 Aug 2026 16:36:11 +1000 Subject: [PATCH 17/20] test(telemetry): make the renewal-extension regression falsifiable Eighth review round (the first against the atomic-renewal fix) verdict: the production code holds - CAS exclusivity across real processes, lease-extension schedules, clock skew both directions, renew-per-attempt under 5xx backoff, defer accounting, and the author's mutants all verified - but one shipped regression test could not fail against the property it is named for. test_renewal_extends_the_lease_across_the_post asserted next_attempt_at >= lease_before under a frozen clock. A renewal that matches the row but never extends the lease (M4: SET next_attempt_at = next_attempt_at) satisfies >= trivially, and that mutant double-POSTs: the un-extended lease expires mid-POST and a second process reclaims. The reviewer demonstrated M4 surviving the whole suite while producing a real duplicate send in a two-process schedule. The test now renews 100s into the lease from an advanced clock and requires the deadline to move strictly forward to exactly renewal-clock + 300s. Verified: M4 now fails this test (61 others unaffected); clean HEAD passes all 62. No production code change. 277 tests; ruff + footguns clean. --- .../hermes_cli/test_shared_metrics_sender.py | 31 +++++++++++++++++-- 1 file changed, 28 insertions(+), 3 deletions(-) diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index 63fe022475..c6a7455fe2 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -686,15 +686,40 @@ class TestClaimingAndBounds: assert row["send_state"] == "pending" def test_renewal_extends_the_lease_across_the_post(self, store): - """A healthy in-lease claimant renews and its POST is covered.""" + """A healthy in-lease claimant renews and its POST is covered. + + Round-8 review: the original assertion was `>=` under a frozen + clock, which a renewal that matches the row but never extends the + lease also satisfies — the exact mutant that double-POSTs (the + un-extended lease expires mid-POST and a second process reclaims). + The renewal must move the deadline STRICTLY forward to now + lease, + so renew from a later clock and require the exact new deadline. + """ _add_package(store, "pkg-1", "2026-08-26") - sender = _sender(store, FakeTransport(FakeResponse(202))) + clock = {"t": NOW} + sender = SharedMetricsSender( + store, + ENDPOINT, + post=lambda e, p, *, timeout: FakeResponse(202), + sleep=lambda _s: None, + now=lambda: clock["t"], + ) claimed = sender._claim_next(NOW, set()) assert claimed is not None lease_before = _row(store, "pkg-1")["next_attempt_at"] + # 100s into the (300s) lease: still healthy, renews mid-flight. + clock["t"] = NOW + timedelta(seconds=100) assert sender._renew_claim("pkg-1", claimed["claim_token"]) is True - assert _row(store, "pkg-1")["next_attempt_at"] >= lease_before + lease_after = _row(store, "pkg-1")["next_attempt_at"] + assert lease_after > lease_before, ( + "renewal granted authority without extending the lease" + ) + # And not just 'later': the full fresh lease from the renewal clock. + expected = (NOW + timedelta(seconds=100 + 300)).strftime( + "%Y-%m-%dT%H:%M:%SZ" + ) + assert lease_after == expected def test_a_lapsed_claimant_resuming_after_reclaim_cannot_double_post( self, store From a69a9c351dfa7cc37b804259b19c4f4a6a3a7e44 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Fri, 28 Aug 2026 15:22:16 +1000 Subject: [PATCH 18/20] feat(telemetry): transmit the stable install_id as-is Product-owner decision, 2026-08-27: the analytical need is stable cross-window identity (retention curves, longitudinal install behaviour), which the rotating pseudonym destroyed by design. The feature has not shipped - zero consented users, zero production transmissions - so identity semantics can change without breaking any promise made to a user; existing (dev-only) consent windows carry forward unchanged. Removed in full rather than weakened in place: - shared_metrics_identity.py (salt generation/rotation, HMAC-SHA256 derivation, payload substitution) and its 19-test file. - The sender's derivation step. _freeze_identity keeps its validation role (unreadable/non-object/id-less payloads still reject rather than block the queue) and now records the raw install_id in sent_install_id; _body rewrites the payload's install_id from that frozen column, keeping byte-identical resends anchored to one recorded value. Consent surface updated in the same change: the setup wizard now states plainly that packages carry the stable profile-scoped install ID (a random UUID, no personal information, reset by deleting the shared-metrics directory). No consent was ever collected under the old wording in any shipped build. Docs A.2/A.3 rewritten as decision records rather than silently edited: A.2 records what is transmitted now and states the consequences plainly (indefinite cross-package correlation is the designed behaviour); A.3 records why rotation existed and why its removal was accepted. The main-body "must not reuse the persistent local identifier by default" escape hatch is exercised, not deleted: that paragraph required exactly this product decision, which has now been made. A.6's deletion note updated: install_id is now itself the lookup key, so a future delete-on-request needs only a service-side API, not a mapping. Tests: the two privacy assertions invert deliberately (test_the_stable_install_id_is_transmitted_as_is and the e2e wire variant); freezing/byte-identical-retry coverage unchanged. Staging E2E script now asserts transmitted == install_id. 258 targeted tests pass; ruff + footguns clean; both staging E2E harnesses green with the raw id observed on the wire (202s). --- docs/observability/relay-shared-metrics.md | 138 +++++++------- hermes_cli/observability/shared_metrics.py | 5 +- .../observability/shared_metrics_identity.py | 131 ------------- .../observability/shared_metrics_sender.py | 36 ++-- hermes_cli/setup.py | 10 +- scripts/e2e_shared_metrics_staging.py | 12 +- .../test_shared_metrics_identity.py | 179 ------------------ .../hermes_cli/test_shared_metrics_sender.py | 14 +- .../test_shared_metrics_sender_e2e.py | 7 +- 9 files changed, 119 insertions(+), 413 deletions(-) delete mode 100644 hermes_cli/observability/shared_metrics_identity.py delete mode 100644 tests/hermes_cli/test_shared_metrics_identity.py diff --git a/docs/observability/relay-shared-metrics.md b/docs/observability/relay-shared-metrics.md index cb3ea2197a..a736bf6edd 100644 --- a/docs/observability/relay-shared-metrics.md +++ b/docs/observability/relay-shared-metrics.md @@ -230,17 +230,18 @@ packages from that profile and can therefore link those local packages. Deleting `$HERMES_HOME/telemetry/shared_metrics` resets the identifier together with all aggregates and package files. -Remote delivery is opt-in and off by default. A remote exporter must not reuse -the persistent local identifier by default. It requires a separate product and -privacy decision covering consent, identity scope, rotation or keyed -pseudonymization, reset behavior, retention, and deletion. +Remote delivery is opt-in and off by default. Reusing the persistent local +identifier remotely required a separate product and privacy decision covering +consent, identity scope, reset behavior, retention, and deletion — that +decision has been made. > Those decisions are recorded in > [Appendix A](#appendix-a-remote-exporter-decisions-phase-2), and the exporter > implementing them has shipped. Collection alone still transmits nothing: the -> sender runs only when `telemetry.shared_metrics.send` is also true, and it -> transmits a rotating HMAC of the install identity rather than the identifier -> itself. +> sender runs only when `telemetry.shared_metrics.send` is also true. Each +> transmitted package carries the stable `install_id` as-is (product decision, +> 2026-08-27 — see A.2 for the record, including the superseded +> HMAC-pseudonym design). The install identity is scoped to one `HERMES_HOME`. To reset it, stop Hermes processes and remove `$HERMES_HOME/telemetry/shared_metrics`. This deliberately @@ -327,60 +328,66 @@ Local history can be up to 30 days old, and that data was collected under a promise that nothing is uploaded. Honouring consent forward-only costs at most 30 days of backlog we never had permission to send. -### A.2 Identity scope — the transmitted identifier is derived, not the local one +### A.2 Identity scope — the stable install_id is transmitted as-is -`install_id` is the persistent profile-scoped identifier described above. It is -**not transmitted**. Each package sent carries a derived value instead: +**Decision record.** The original design of this exporter (and revisions 1–8 +of this appendix) transmitted a keyed pseudonym instead of the identifier: +`HMAC-SHA256(key = locally-held rotating salt, message = install_id)`, with +the salt rotating every 30 days. On **2026-08-27**, before the feature +shipped (zero consented users, zero production transmissions), the product +owner decided the analytical need is a **stable cross-window identity** — +retention curves, longitudinal install behaviour — which rotation by design +destroys. The pseudonymization layer was removed in full rather than +weakened in place. -```text -transmitted_id = HMAC-SHA256(key = rotation_salt, message = install_id) -``` +What is transmitted now: -- `rotation_salt` is random, generated locally, and never leaves the machine. -- The derivation is one-way: the service cannot recover `install_id`. -- Within a rotation window, packages from one profile correlate — so distinct - installs remain countable, which is the primary analytical question. -- Across windows, they do not. +- Each package carries `install_id` verbatim: the persistent, profile-scoped + random UUID described above. +- It is generated locally (`uuid4`), contains no hardware, account, user, or + machine-derived information, and identifies a *profile*, not a person. +- It is stable until the user deletes the shared-metrics directory, which + regenerates it (see A.4). -This satisfies "must not reuse the persistent local identifier by default" -while keeping the data useful. Stripping the identifier entirely was rejected -because "how many installs are reporting" is the first question the data must -answer; sending `install_id` unchanged was rejected because it contradicts the -commitment made above. +Consequences stated plainly rather than papered over: -**Byte-identical resends still hold.** The derived value is computed **once**, -when the package is first prepared for sending, and stored alongside the -package (the derived id only — not a second copy of the payload, which is -recomputed deterministically from the stored package). A retry therefore -rebuilds identical bytes even if the salt rotated in between. The contract -requires this: resending a `package_id` with different content is undefined -behaviour. +- Packages from one profile correlate **indefinitely**, not per-window. + Long-term linkability of one install's daily envelope sequence is now the + designed behaviour, not a residue. +- The A.3 residue analysis of the old design (stable `resource` tuple + + contiguous periods bridging rotation windows) is moot — there is no window + boundary left to bridge. +- The setup wizard's consent language states this identity model explicitly; + it was updated in the same change that removed the derivation, so no + consent was ever collected under the old wording in any shipped build. -### A.3 Rotation +**Byte-identical resends still hold.** The transmitted id is recorded on the +row (`sent_install_id`) when the package is first prepared, and the wire body +is always rebuilt from that recorded value, so a retry rebuilds identical +bytes. The contract requires this: resending a `package_id` with different +content is undefined behaviour. (With a stable id the recorded copy is no +longer load-bearing against rotation — it remains as the audit column and as +cheap insurance against any future change to identity semantics.) -`rotation_salt` rotates on a fixed schedule (default: every 30 days, aligned to -local history retention). Rotation only affects packages prepared after it; -already-prepared packages keep their derived value so retries stay -byte-identical. +### A.3 Rotation — removed (decision record) -Rotation bounds long-term linkability without destroying short-term cohort -analysis. A profile is one identity for the length of a window, and an -unrelated identity after it. +Salt rotation was deleted together with the derivation (product decision, +2026-08-27). This section is retained as a record of what the earlier design +did and why the removal was accepted: -**What rotation does not bound.** The identifier changes; the rest of the -envelope does not. `resource` (`os_family`, `architecture`, `install_method`, -`hermes_version`) is stable and low-entropy, and `period_start` / -`period_end` are contiguous across a rotation boundary. For a common -configuration this is no help to an observer — measured against the 11 real -packages in a development outbox, every one shares the same -`arm64 / macos / git` tuple. For a **rare** configuration it is a plausible -re-identification aid: an unusual architecture or install method, combined -with an uninterrupted daily period sequence, can bridge two windows. The -claim this design makes is therefore "rotation raises the cost of long-term -correlation", not "rotation makes it impossible". Narrowing that residue -would mean coarsening `resource` or jittering period boundaries, and neither -is worth the analytical loss today — but it should be a conscious decision, -not an unexamined one. +- Rotation existed to bound long-term linkability: one identity per 30-day + window, unrelated identities across windows. +- The documented residue (see git history for the full analysis): the + envelope's stable, low-entropy `resource` tuple plus contiguous daily + periods could plausibly bridge windows for rare configurations anyway, so + the boundary was a cost-raiser, not a wall. +- The product need that killed it: cross-window continuity is precisely what + retention analysis requires. A boundary that mostly inconveniences honest + analysis while only raising costs for a determined correlator was judged + the wrong trade once stable identity became a requirement. + +There is no salt in the store, no rotation schedule, and no derived +identifier anywhere in the pipeline. ### A.4 Reset behavior @@ -388,11 +395,11 @@ Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity, aggregates, and package files, exactly as documented above. Two honest qualifications now apply: -- Reset also discards `rotation_salt`, so subsequent packages derive a **new** - transmitted identity. Local reset does give a new remote identity. +- Reset regenerates `install_id`, so subsequent packages transmit a **new** + identity. Local reset does give a new remote identity. - Reset **cannot unsend**. Packages already transmitted remain in the ingest - service's storage under their derived identifier. There is no read-back or - delete API in the v1 contract. + service's storage under the identifier they were sent with. There is no + read-back or delete API in the v1 contract. Setting `send: false` stops transmission immediately: consent is re-read before every package, so a pass already in flight stops after the package it @@ -439,13 +446,13 @@ invent one. What a user can do: |---|---| | `send: false` | No further packages leave the machine | | `enabled: false` | Collection stops; existing local state remains | -| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new derived identity | +| Remove `.../shared_metrics` | Local identity, aggregates, and files reset; future sends use a new install_id | | Delete already-sent data | Not self-service — requires an operator acting on the S3 bucket | -If a deletion-on-request obligation is ever taken on, it needs a lookup path -from a user to their derived identifiers. That is deliberately **not** built: -it would require retaining the mapping this design exists to avoid. Any such -change is a new product decision, not an implementation detail. +If a deletion-on-request obligation is ever taken on, the lookup path is now +direct: the user's `install_id` (readable from their local store) is the key +their data is stored under. Building the service-side delete API remains a +new product decision, not an implementation detail. ### A.7 What the outbox directory is @@ -464,7 +471,8 @@ state they were promised. Send state lives in new columns on the ### A.8 Scope note -The `install_id` field inside the package body is what gets replaced by the -derived value. No other payload field changes, nothing is added, and the -service treats the whole body as opaque. Payload schema evolution therefore -stays a sender-side concern, as before. +The `install_id` field inside the package body is transmitted as the +generator wrote it (rewritten from the row's frozen `sent_install_id`, which +records the same value). No other payload field changes, nothing is added, +and the service treats the whole body as opaque. Payload schema evolution +therefore stays a sender-side concern, as before. diff --git a/hermes_cli/observability/shared_metrics.py b/hermes_cli/observability/shared_metrics.py index ddf570b6f9..87094922d9 100644 --- a/hermes_cli/observability/shared_metrics.py +++ b/hermes_cli/observability/shared_metrics.py @@ -373,8 +373,9 @@ class SharedMetricsStore: # Earliest next attempt; enforces backoff across process restarts. ("next_attempt_at", "TEXT"), ("last_error", "TEXT"), - # The derived identifier actually transmitted, frozen on the first - # attempt so retries stay byte-identical across a salt rotation. + # The identifier actually transmitted, frozen on the first + # attempt so retries stay byte-identical. Since the 2026-08-27 + # product decision this is the stable install_id itself. # Only the ~36-byte id is stored: the body is recomputed from # payload_json, whose serialisation is deterministic. ("sent_install_id", "TEXT"), diff --git a/hermes_cli/observability/shared_metrics_identity.py b/hermes_cli/observability/shared_metrics_identity.py deleted file mode 100644 index b4f8cda8e3..0000000000 --- a/hermes_cli/observability/shared_metrics_identity.py +++ /dev/null @@ -1,131 +0,0 @@ -"""Keyed pseudonymization of the shared-metrics install identity. - -``install_id`` is a persistent, profile-scoped identifier. It is deliberately -NOT transmitted: ``docs/observability/relay-shared-metrics.md`` commits that a -remote exporter "must not reuse the persistent local identifier by default". - -Each transmitted package instead carries:: - - HMAC-SHA256(key=rotation_salt, message=install_id) - -where ``rotation_salt`` is generated locally, never leaves the machine, and -rotates on a fixed schedule. Within a rotation window the value is stable, so -distinct installs stay countable — the primary analytical question. Across -windows it changes, bounding long-term linkability. - -The derivation is one-way: the service cannot recover ``install_id`` from what -it receives. - -See Appendix A.2 and A.3 of the doc above for the decision record. -""" - -from __future__ import annotations - -import hashlib -import hmac -import secrets -import sqlite3 -from datetime import datetime, timedelta, timezone - -#: Salt lifetime. Matches local history retention so the two ages line up. -ROTATION_INTERVAL = timedelta(days=30) - -#: ``telemetry_state`` keys. The salt lives in the same store as install_id, so -#: deleting the shared-metrics directory resets both together — the documented -#: reset behaviour keeps working without a second cleanup path. -SALT_KEY = "send_rotation_salt" -SALT_ISSUED_AT_KEY = "send_rotation_salt_issued_at" - -_SALT_BYTES = 32 - - -def _isoformat(value: datetime) -> str: - return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z") - - -def _parse(value: str | None) -> datetime | None: - if not value: - return None - try: - parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) - except ValueError: - return None - if parsed.tzinfo is None: - parsed = parsed.replace(tzinfo=timezone.utc) - return parsed.astimezone(timezone.utc) - - -def _read(connection: sqlite3.Connection, key: str) -> str | None: - row = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (key,) - ).fetchone() - if row is None: - return None - # sqlite3.Row and plain tuples both index by position. - return str(row[0]) - - -def _write(connection: sqlite3.Connection, key: str, value: str) -> None: - connection.execute( - """ - INSERT INTO telemetry_state(key, value) VALUES (?, ?) - ON CONFLICT(key) DO UPDATE SET value = excluded.value - """, - (key, value), - ) - - -def current_salt( - connection: sqlite3.Connection, - *, - now: datetime | None = None, -) -> str: - """Return the active salt, generating or rotating it when due. - - Must be called inside a write transaction: it can write to - ``telemetry_state``. - """ - moment = now or datetime.now(timezone.utc) - salt = _read(connection, SALT_KEY) - issued_at = _parse(_read(connection, SALT_ISSUED_AT_KEY)) - - fresh = ( - salt is not None - and issued_at is not None - # Strictly within the window. A future issued_at means the clock moved - # backwards (or the value was tampered with), so the recorded age - # cannot be trusted and we reissue rather than keep using a salt of - # unknown vintage. Reissuing is the safe direction: it shortens - # linkability, and already-prepared packages keep their frozen - # identifier so retries stay byte-identical. - and issued_at <= moment < issued_at + ROTATION_INTERVAL - ) - if fresh: - return str(salt) - - salt = secrets.token_hex(_SALT_BYTES) - _write(connection, SALT_KEY, salt) - _write(connection, SALT_ISSUED_AT_KEY, _isoformat(moment)) - return salt - - -def derive_install_id(install_id: str, salt: str) -> str: - """Return the transmitted identifier for ``install_id`` under ``salt``.""" - return hmac.new( - salt.encode("utf-8"), - install_id.encode("utf-8"), - hashlib.sha256, - ).hexdigest() - - -def substitute_install_id(payload: dict, derived: str) -> dict: - """Return ``payload`` with its ``install_id`` replaced by ``derived``. - - This is the ONLY field the exporter changes. Everything else is - transmitted exactly as the generator wrote it, so payload schema evolution - stays a sender-side concern. A shallow copy is enough — only a top-level - key is replaced — and the caller's dict is left untouched. - """ - updated = dict(payload) - updated["install_id"] = derived - return updated diff --git a/hermes_cli/observability/shared_metrics_sender.py b/hermes_cli/observability/shared_metrics_sender.py index 8e6f81c4b8..9418353c9b 100644 --- a/hermes_cli/observability/shared_metrics_sender.py +++ b/hermes_cli/observability/shared_metrics_sender.py @@ -39,12 +39,6 @@ from datetime import datetime, timedelta, timezone from hermes_cli.sqlite_util import write_txn -from .shared_metrics_identity import ( - current_salt, - derive_install_id, - substitute_install_id, -) - logger = logging.getLogger(__name__) #: Contract recommends timing out at 30s and treating a timeout as retryable. @@ -432,13 +426,17 @@ class SharedMetricsSender: payload_json, now: datetime, ) -> str | None: - """Derive and persist the transmitted id, or reject an unusable row. + """Record the transmitted id on the row, or reject an unusable one. - Returns None when the package can never be sent. Rejecting rather than - raising matters: an exception here rolls back the claim transaction - and blocks every healthy package behind this one. + The stable install_id is transmitted as-is (product decision, + 2026-08-27 — see the doc's A.2). What remains of "freezing" is the + validation and the audit column: ``sent_install_id`` records exactly + what the wire will carry, and rejecting unusable rows here rather + than raising matters because an exception rolls back the claim + transaction and blocks every healthy package behind this one. """ reason = None + install_id = None try: payload = json.loads(payload_json) except (TypeError, ValueError): @@ -467,24 +465,26 @@ class SharedMetricsSender: ) return None - salt = current_salt(connection, now=now) - derived = derive_install_id(payload["install_id"], salt) connection.execute( "UPDATE package_outbox SET sent_install_id = ? WHERE package_id = ?", - (derived, package_id), + (install_id, package_id), ) - return derived + return str(install_id) # -- transmission ------------------------------------------------------ - def _body(self, payload_json: str, derived: str) -> bytes: + def _body(self, payload_json: str, transmitted_id: str) -> bytes: """Rebuild the exact bytes to send. The payload is recomputed from the stored package rather than kept as - a second copy: json.dumps with these options is deterministic, and the - only mutable input (the derived id) is frozen in the row. + a second copy: json.dumps with these options is deterministic. The + install_id is written from the frozen ``sent_install_id`` column + rather than trusted implicitly, keeping "a resend is byte-identical" + anchored to one recorded value. """ - payload = substitute_install_id(json.loads(payload_json), derived) + payload = json.loads(payload_json) + payload = dict(payload) + payload["install_id"] = transmitted_id return json.dumps(payload, indent=2, sort_keys=True).encode("utf-8") def _mark( diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 4772929b88..2ec08da8af 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2464,10 +2464,12 @@ def setup_telemetry(config: dict): print_success("Local shared metrics enabled.") print_info("") print_info("Sending uploads each daily package to the Nous telemetry") - print_info("service. Your profile-scoped install ID is NOT sent: packages") - print_info("carry a rotating HMAC of it instead. Only packages from the") - print_info("day you opt in onwards are ever sent, and sending can be") - print_info("turned off again at any time.") + print_info("service. Packages carry your profile-scoped install ID, a") + print_info("stable random UUID that identifies this profile across days") + print_info("(it contains no personal information and is reset by deleting") + print_info("the shared-metrics directory). Only packages from the day you") + print_info("opt in onwards are ever sent, and sending can be turned off") + print_info("again at any time.") shared_metrics["send"] = prompt_yes_no( "Send shared metrics to Nous?", default=shared_metrics.get("send") is True, diff --git a/scripts/e2e_shared_metrics_staging.py b/scripts/e2e_shared_metrics_staging.py index e0c497a342..666c9e51e9 100644 --- a/scripts/e2e_shared_metrics_staging.py +++ b/scripts/e2e_shared_metrics_staging.py @@ -171,10 +171,12 @@ def main() -> int: print(f" last_error : {row[5]}") if row[1] != "sent": failures.append(f"{row[0]} is {row[1]}: {row[5]}") - if row[4] == real_install_id: - failures.append(f"{row[0]} LEAKED the real install_id") - if not row[4] or len(str(row[4])) != 64: - failures.append(f"{row[0]} has a malformed derived id") + # Product decision 2026-08-27: the stable install_id is transmitted + # as-is; the transmitted value must be exactly the local id. + if row[4] != real_install_id: + failures.append( + f"{row[0]} transmitted {row[4]!r}, expected the install_id" + ) print() if failures: @@ -183,7 +185,7 @@ def main() -> int: print(f" ✗ {failure}") return 1 - print("PASS: every package acknowledged 202 with a derived identifier.") + print("PASS: every package acknowledged 202 with the stable install_id.") print() print("Verify the objects in S3 with the package ids above:") print(" aws s3 ls --recursive " diff --git a/tests/hermes_cli/test_shared_metrics_identity.py b/tests/hermes_cli/test_shared_metrics_identity.py deleted file mode 100644 index 1ea1d95961..0000000000 --- a/tests/hermes_cli/test_shared_metrics_identity.py +++ /dev/null @@ -1,179 +0,0 @@ -"""Tests for keyed pseudonymization of the shared-metrics install identity. - -The load-bearing property: install_id must never be transmitted, and the -value that IS transmitted must stay stable for a package even across a salt -rotation, or a retry would change the body under an already-used package_id. -""" - -from __future__ import annotations - -import sqlite3 -from datetime import datetime, timedelta, timezone - -import pytest - -from hermes_cli.observability.shared_metrics_identity import ( - ROTATION_INTERVAL, - SALT_ISSUED_AT_KEY, - SALT_KEY, - current_salt, - derive_install_id, - substitute_install_id, -) - -INSTALL_ID = "12a73e97-4de9-4766-830d-9ca1192c0420" -T0 = datetime(2026, 8, 26, 12, 0, tzinfo=timezone.utc) - - -@pytest.fixture -def connection(): - conn = sqlite3.connect(":memory:") - conn.execute( - "CREATE TABLE telemetry_state (key TEXT PRIMARY KEY, value TEXT NOT NULL)" - ) - yield conn - conn.close() - - -class TestSaltLifecycle: - def test_first_call_generates_a_salt(self, connection): - salt = current_salt(connection, now=T0) - assert len(salt) == 64 # 32 bytes hex - assert int(salt, 16) >= 0 # valid hex - - def test_salt_is_stable_within_the_window(self, connection): - first = current_salt(connection, now=T0) - later = current_salt(connection, now=T0 + timedelta(days=29, hours=23)) - assert first == later - - def test_salt_rotates_after_the_interval(self, connection): - first = current_salt(connection, now=T0) - after = current_salt(connection, now=T0 + ROTATION_INTERVAL + timedelta(seconds=1)) - assert first != after - - def test_salt_is_persisted(self, connection): - salt = current_salt(connection, now=T0) - stored = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (SALT_KEY,) - ).fetchone()[0] - assert stored == salt - - def test_issued_at_is_recorded(self, connection): - current_salt(connection, now=T0) - stored = connection.execute( - "SELECT value FROM telemetry_state WHERE key = ?", (SALT_ISSUED_AT_KEY,) - ).fetchone()[0] - assert stored.startswith("2026-08-26T12:00") - - def test_two_installs_get_different_salts(self): - salts = set() - for _ in range(5): - conn = sqlite3.connect(":memory:") - conn.execute( - "CREATE TABLE telemetry_state (key TEXT PRIMARY KEY, value TEXT NOT NULL)" - ) - salts.add(current_salt(conn, now=T0)) - conn.close() - assert len(salts) == 5, "salts must be random per install, not derived" - - def test_clock_rollback_reissues_rather_than_trusting_the_stamp(self, connection): - """A future issued_at means the clock moved; the age is unknowable. - - Reissuing is the safe direction — it shortens linkability rather than - extending it, and packages already prepared keep their frozen id. - """ - first = current_salt(connection, now=T0) - rolled_back = current_salt(connection, now=T0 - timedelta(days=5)) - assert rolled_back != first - - def test_corrupt_issued_at_reissues_rather_than_crashing(self, connection): - current_salt(connection, now=T0) - connection.execute( - "UPDATE telemetry_state SET value = 'not-a-date' WHERE key = ?", - (SALT_ISSUED_AT_KEY,), - ) - assert current_salt(connection, now=T0) is not None - - -class TestDerivation: - def test_derivation_is_deterministic(self): - salt = "a" * 64 - assert derive_install_id(INSTALL_ID, salt) == derive_install_id(INSTALL_ID, salt) - - def test_derivation_hides_the_install_id(self): - derived = derive_install_id(INSTALL_ID, "a" * 64) - assert INSTALL_ID not in derived - assert derived != INSTALL_ID - - def test_different_salts_give_different_values(self): - assert derive_install_id(INSTALL_ID, "a" * 64) != derive_install_id( - INSTALL_ID, "b" * 64 - ) - - def test_different_installs_give_different_values(self): - salt = "a" * 64 - assert derive_install_id(INSTALL_ID, salt) != derive_install_id("other", salt) - - def test_output_shape_is_sha256_hex(self): - derived = derive_install_id(INSTALL_ID, "a" * 64) - assert len(derived) == 64 - int(derived, 16) - - -class TestSubstitution: - def _package(self): - return { - "schema_version": "hermes.shared_metrics.v2", - "package_id": "3a63d27e-f170-4d4c-8c4d-ebd80feac592", - "install_id": INSTALL_ID, - "generated_at": "2026-08-26T01:01:25.311956Z", - "period_start": "2026-08-26T00:00:00Z", - "period_end": "2026-08-27T00:00:00Z", - "resource": {"hermes_version": "0.20.5", "os_family": "macos"}, - "metrics": [{"name": "hermes.client.active", "type": "counter", "value": 1}], - } - - def test_install_id_is_replaced(self): - result = substitute_install_id(self._package(), "derived-value") - assert result["install_id"] == "derived-value" - - def test_no_other_field_changes(self): - original = self._package() - result = substitute_install_id(original, "derived-value") - for key in original: - if key != "install_id": - assert result[key] == original[key] - - def test_the_caller_dict_is_not_mutated(self): - original = self._package() - substitute_install_id(original, "derived-value") - assert original["install_id"] == INSTALL_ID - - def test_no_fields_are_added_or_removed(self): - original = self._package() - assert set(substitute_install_id(original, "x")) == set(original) - - def test_the_raw_install_id_never_survives_substitution(self): - import json - - body = json.dumps(substitute_install_id(self._package(), "derived-value")) - assert INSTALL_ID not in body - - -class TestRetryStability: - """The property that keeps retries contract-compliant.""" - - def test_a_frozen_derived_id_survives_a_rotation(self, connection): - salt_before = current_salt(connection, now=T0) - frozen = derive_install_id(INSTALL_ID, salt_before) - - # Time passes, the salt rotates, and the package is retried. - salt_after = current_salt(connection, now=T0 + ROTATION_INTERVAL + timedelta(days=1)) - assert salt_after != salt_before - - # Rebuilding from the FROZEN value reproduces identical bytes; deriving - # afresh would not. - assert substitute_install_id({"install_id": INSTALL_ID}, frozen) == { - "install_id": frozen - } - assert derive_install_id(INSTALL_ID, salt_after) != frozen diff --git a/tests/hermes_cli/test_shared_metrics_sender.py b/tests/hermes_cli/test_shared_metrics_sender.py index c6a7455fe2..cbaff4c9ea 100644 --- a/tests/hermes_cli/test_shared_metrics_sender.py +++ b/tests/hermes_cli/test_shared_metrics_sender.py @@ -374,15 +374,19 @@ class TestConsentGate: class TestIdentity: - def test_install_id_is_never_transmitted(self, store): + def test_the_stable_install_id_is_transmitted_as_is(self, store): + """Product decision 2026-08-27: no pseudonymization. + + The wire body carries the profile-scoped install_id verbatim. This + test is the deliberate inversion of the pre-decision assertion that + the raw id never crossed the wire. + """ _add_package(store, "pkg-1", "2026-08-26") transport = FakeTransport(FakeResponse(202)) _sender(store, transport).send_pending() - raw = transport.calls[0]["payload"].decode("utf-8") - assert INSTALL_ID not in raw - assert transport.bodies[0]["install_id"] != INSTALL_ID + assert transport.bodies[0]["install_id"] == INSTALL_ID - def test_derived_id_is_frozen_on_the_row(self, store): + def test_transmitted_id_is_frozen_on_the_row(self, store): _add_package(store, "pkg-1", "2026-08-26") transport = FakeTransport(FakeResponse(503), FakeResponse(202)) _sender(store, transport).send_pending() diff --git a/tests/hermes_cli/test_shared_metrics_sender_e2e.py b/tests/hermes_cli/test_shared_metrics_sender_e2e.py index 9ff8bf9caf..85be9b2388 100644 --- a/tests/hermes_cli/test_shared_metrics_sender_e2e.py +++ b/tests/hermes_cli/test_shared_metrics_sender_e2e.py @@ -171,12 +171,11 @@ class TestRealTransport: ).fetchone()[0] assert state == "sent" - def test_the_install_id_never_crosses_the_wire(self, store, server): + def test_the_stable_install_id_crosses_the_wire_as_is(self, store, server): + """Product decision 2026-08-27: the raw install_id is transmitted.""" _add(store, "pkg-1", metrics=40) _sender(store, server).send_pending() - body = json.dumps(Ingest.received[0]["body"]) - assert INSTALL_ID not in body - assert len(Ingest.received[0]["body"]["install_id"]) == 64 + assert Ingest.received[0]["body"]["install_id"] == INSTALL_ID def test_content_type_is_json(self, store, server): _add(store, "pkg-1") From 24ecc2a7693d71bf18f71a3bf13cde6de8e2768f Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Fri, 28 Aug 2026 17:03:22 +1000 Subject: [PATCH 19/20] docs(telemetry): update cli-config.yaml.example for the stable-id decision PR review (andrexibiza, post-a69a9c351d) caught the one operator-facing surface the identity change missed: the example config still promised the profile-scoped ID is NOT sent and described the 30-day rotating HMAC. Rewritten to state the stable install_id is transmitted as-is, matching the sender, wizard, and docs A.2. Swept the repo for further stale references: none remain (the HMAC text in relay-shared-metrics.md A.2/A.3 is the intentional decision record). --- cli-config.yaml.example | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/cli-config.yaml.example b/cli-config.yaml.example index 2fe2c5e610..0675156986 100644 --- a/cli-config.yaml.example +++ b/cli-config.yaml.example @@ -1794,11 +1794,11 @@ display: # When sending is on: # * only packages whose period starts on or after the day you opted in are # ever transmitted, so data collected beforehand stays on this machine; -# * the profile-scoped ID is NOT sent. Each package carries an HMAC of it, -# keyed by a local-only salt that rotates every 30 days, so installs stay -# countable without shipping a durable identifier. +# * each package carries the profile-scoped ID as-is. It is a random UUID +# with no hardware, account, or host-derived content, and deleting the +# shared-metrics directory resets it. # See docs/observability/relay-shared-metrics.md (Appendix A) for the full -# consent, identity, rotation, retention, and deletion decisions. +# consent, identity, retention, and deletion decisions. telemetry: shared_metrics: enabled: false From 12bba22dc95b5fbbdc0a26461d8aed5a65ea9ce0 Mon Sep 17 00:00:00 2001 From: Ben Barclay Date: Wed, 2 Sep 2026 07:16:57 +1000 Subject: [PATCH 20/20] docs(telemetry): state the consent containment rule on operator surfaces PR re-review caught that cli-config.yaml.example and the setup wizard still described the OLD day-stamp gate ("period starts on or after the day you opted in"). The actual gate (CONSENT_GATE_SQL) requires the entire package period to be contained in one recorded consent window - stricter, and privacy-significant at opt-in/revocation boundaries: a package straddling a revocation starts after opt-in yet is correctly held back. Both surfaces now state the containment rule literally; docs A.1 already did. --- cli-config.yaml.example | 7 +++++-- hermes_cli/setup.py | 8 +++++--- 2 files changed, 10 insertions(+), 5 deletions(-) diff --git a/cli-config.yaml.example b/cli-config.yaml.example index 0675156986..6cb99466dc 100644 --- a/cli-config.yaml.example +++ b/cli-config.yaml.example @@ -1792,8 +1792,11 @@ display: # Nothing is uploaded unless you also set `send: true`. That is a separate # opt-in and requires `enabled`; it never turns collection on by itself. # When sending is on: -# * only packages whose period starts on or after the day you opted in are -# ever transmitted, so data collected beforehand stays on this machine; +# * only packages whose entire collection period falls within one +# continuous recorded consent window are ever sent. Consent windows open +# when you enable sending and close when you disable it, so data +# collected before you opted in — or during any gap between opt-ins — +# stays on this machine; # * each package carries the profile-scoped ID as-is. It is a random UUID # with no hardware, account, or host-derived content, and deleting the # shared-metrics directory resets it. diff --git a/hermes_cli/setup.py b/hermes_cli/setup.py index 2ec08da8af..e1baacd0ca 100644 --- a/hermes_cli/setup.py +++ b/hermes_cli/setup.py @@ -2467,9 +2467,11 @@ def setup_telemetry(config: dict): print_info("service. Packages carry your profile-scoped install ID, a") print_info("stable random UUID that identifies this profile across days") print_info("(it contains no personal information and is reset by deleting") - print_info("the shared-metrics directory). Only packages from the day you") - print_info("opt in onwards are ever sent, and sending can be turned off") - print_info("again at any time.") + print_info("the shared-metrics directory). Only packages whose entire") + print_info("collection period falls inside a recorded consent window are") + print_info("ever sent — data from before you opt in, or from any gap") + print_info("while sending was off, stays on this machine. Sending can be") + print_info("turned off again at any time.") shared_metrics["send"] = prompt_yes_no( "Send shared metrics to Nous?", default=shared_metrics.get("send") is True,