fix(telemetry): close the consent window on the config transition

Fourth independent review. Two more consent leaks, both reproduced through
the real relay entry point before and after the fix. Both are failures of
my own round-3 fix, which recorded revocation in the wrong place.

BLOCKER 1 - revoking while idle recorded nothing. _record_revocation lived
inside send_pending's loop, but _send_exported_packages returns early when
send is false, before a sender is ever constructed. The dominant case is a
user turning sending off while no pass is running, so the loop that was
meant to observe the revocation could never run. Reproduced: 6 periods
collected during a refused window were transmitted on re-enable.

The window now closes on the observed config EDGE, before the early return.
Last-seen send state is persisted because each hook fires in a fresh
process, so a true->false transition is only visible by comparison. The
rising edge also opens the window explicitly: the sender only runs when
there is something to send, so a user who opts in and out before any
package exists would otherwise have no window for record_revoked to close.

BLOCKER 2 - turning COLLECTION off never recorded revocation. The
not-enabled branch in setup.py force-set send=false and returned without
calling _record_send_consent_change, so `hermes tools` -> disable shared
metrics silently dropped consent while leaving the window open. Same
retroactive release on re-enable. Both consent surfaces now record, and
setup keeps the relay's edge detector in step.

Also, from the same review's mutation sweep:
- the scheme check is now pinned as an allowlist. Replacing the http test
  with `if True` survived the entire suite, because every non-http case
  targeted a REMOTE host where the loopback branch rejects anyway. Only a
  non-http scheme on loopback distinguishes the two. Shipped behaviour was
  already correct; nothing guarded it.
- A.3 no longer claims rotation bounds long-term linkability outright.
  Measured against 11 real packages: resource is a stable low-entropy
  tuple and periods are contiguous across a rotation, so for a RARE
  configuration those can bridge windows. The honest claim is that
  rotation raises the cost, not that it makes correlation impossible.

Two mutants are documented as unkillable rather than papered over with
tests that only appear to cover them: the _defer clamp is unreachable from
any current caller, and widening the falling-edge check to an
unconditional else is behaviourally equivalent because record_revoked is
idempotent and no-ops without an open window.

An earlier version of the anti-spurious-revocation test could not fail
either - it used a never-consented store, where record_revoked no-ops
regardless. Rewritten to opt in, revoke, re-enable, and then assert that a
steady enabled state does not re-close the reopened window.

259 tests pass. Staging E2E re-run: both packages 202.
This commit is contained in:
Ben Barclay
2026-08-27 09:47:47 +10:00
parent 36f1e01eba
commit 613849c190
7 changed files with 331 additions and 6 deletions

View File

@@ -357,6 +357,21 @@ Rotation bounds long-term linkability without destroying short-term cohort
analysis. A profile is one identity for the length of a window, and an
unrelated identity after it.
**What rotation does not bound.** The identifier changes; the rest of the
envelope does not. `resource` (`os_family`, `architecture`, `install_method`,
`hermes_version`) is stable and low-entropy, and `period_start` /
`period_end` are contiguous across a rotation boundary. For a common
configuration this is no help to an observer — measured against the 11 real
packages in a development outbox, every one shares the same
`arm64 / macos / git` tuple. For a **rare** configuration it is a plausible
re-identification aid: an unusual architecture or install method, combined
with an uninterrupted daily period sequence, can bridge two windows. The
claim this design makes is therefore "rotation raises the cost of long-term
correlation", not "rotation makes it impossible". Narrowing that residue
would mean coarsening `resource` or jittering period boundaries, and neither
is worth the analytical loss today — but it should be a conscious decision,
not an unexamined one.
### A.4 Reset behavior
Removing `$HERMES_HOME/telemetry/shared_metrics` still resets local identity,

View File

@@ -1083,6 +1083,67 @@ class _Runtime:
if exported is not None:
self._safe(self._send_exported_packages)
def _observe_send_consent(self, send_enabled: bool) -> None:
"""Close the consent window on a true->false transition.
Persists the last-seen send state so a change is detected even though
this runs in a fresh process each time. Only the falling edge matters:
opening a new window is the sender's job, on the next enabled pass.
Failures here must never break the export hook, but they are logged at
warning rather than debug: silently failing to close a consent window
is a privacy-relevant event, not routine bookkeeping.
"""
try:
from hermes_cli.observability.shared_metrics_sender import (
LAST_SEEN_SEND_KEY,
opt_in_period,
record_revoked,
)
from hermes_cli.sqlite_util import write_txn
current = "1" if send_enabled else "0"
with self.subscriber.store._connection() as connection:
with write_txn(connection):
row = connection.execute(
"SELECT value FROM telemetry_state WHERE key = ?",
(LAST_SEEN_SEND_KEY,),
).fetchone()
previous = str(row[0]) if row is not None else None
if send_enabled:
# Open the window HERE, on the rising edge, rather than
# leaving it to the sender's first claim. The sender
# only runs when there is something to send, so a user
# who opts in and then opts out before any package
# exists would otherwise have no window to close, and
# record_revoked (which requires one) would no-op.
opt_in_period(connection)
elif previous == "1":
# `previous == "1"` is the true falling edge. Widening
# this to an unconditional else would be behaviourally
# equivalent today — record_revoked is idempotent and
# no-ops without an open window — so no test can tell
# the two apart. It is written as an edge anyway
# because that is the property intended, and a future
# change to record_revoked should not silently turn
# every disabled pass into a revocation.
record_revoked(connection)
if previous != current:
connection.execute(
"""
INSERT INTO telemetry_state(key, value) VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value
""",
(LAST_SEEN_SEND_KEY, current),
)
except Exception:
logger.warning(
"Unable to record a shared-metrics consent transition",
exc_info=True,
)
def _send_exported_packages(self) -> None:
from hermes_cli.observability.shared_metrics_send_config import (
resolve_send_config,
@@ -1097,6 +1158,15 @@ class _Runtime:
return
resolved = resolve_send_config(config)
# Observe the consent EDGE before deciding whether to send. Recording
# revocation inside the send loop (as an earlier fix did) can never
# work: the dominant case is the user turning sending off while no
# pass is running, and then this method returns below without ever
# constructing a sender. The window has to close on the transition,
# not on the next transmission that by definition will not happen.
self._observe_send_consent(resolved.send)
if not resolved.send:
return

View File

@@ -95,6 +95,11 @@ OPT_IN_PERIOD_KEY = "send_opt_in_period"
#: permanent for the packages collected while it was off.
SEND_REVOKED_KEY = "send_revoked"
#: Last send-consent state this machine observed ("1"/"0"). Persisted because
#: each hook fires in a fresh process, so a true->false edge is only visible
#: by comparing against what was recorded last time.
LAST_SEEN_SEND_KEY = "send_last_seen"
def _utc_now() -> datetime:
return datetime.now(timezone.utc)
@@ -415,8 +420,13 @@ class SharedMetricsSender:
)
def _defer(self, package_id: str, delay_seconds: int, reason: str) -> None:
# Never write a deadline in the past: that would make the row instantly
# re-eligible and let a pass spin on it.
# Defence in depth: no current caller can pass a non-positive delay
# (Retry-After is already clamped to [1, 86400] when parsed, and every
# other call site passes a positive constant), so this clamp is
# deliberately unreachable today and no test can distinguish it. It
# stays because a past deadline would make the row instantly
# re-eligible and let a pass spin on it — a cheap guard against a
# future caller that forgets.
delay = max(1, int(delay_seconds))
retry_at = self._now().timestamp() + delay
self._mark(
@@ -514,10 +524,10 @@ class SharedMetricsSender:
if not self._still_consented():
# The user turned sending off while this pass was running.
# Stop without transmitting anything further, and close the
# consent window so a later re-enable cannot release the
# packages collected in the meantime. Recorded here as well as
# in the setup wizard because config.yaml can be edited by
# hand, which the wizard never sees.
# consent window. This covers only the mid-pass case; a
# revocation made while no pass is running is caught by the
# relay's edge detector before it early-returns, because this
# loop would never run to observe it.
logger.info("Shared-metrics sending disabled mid-pass; stopping")
self._record_revocation()
break

View File

@@ -2454,6 +2454,11 @@ def setup_telemetry(config: dict):
if shared_metrics.get("send") is True:
shared_metrics["send"] = False
print_info("Sending shared metrics disabled as well.")
# Turning collection off is also a withdrawal of send consent, and it
# has to close the window like any other. Recorded unconditionally:
# the send key may already be false in config while the consent window
# is still open, and that window must not survive to be reopened.
_record_send_consent_change(enabled=False)
return
print_success("Local shared metrics enabled.")
@@ -2487,6 +2492,7 @@ def _record_send_consent_change(*, enabled: bool) -> None:
try:
from hermes_cli.observability.shared_metrics import SharedMetricsStore
from hermes_cli.observability.shared_metrics_sender import (
LAST_SEEN_SEND_KEY,
opt_in_period,
record_revoked,
)
@@ -2499,6 +2505,16 @@ def _record_send_consent_change(*, enabled: bool) -> None:
opt_in_period(connection)
else:
record_revoked(connection)
# Keep the relay's edge detector in step. Without this the
# wizard's change looks like "no transition" on the next hook
# fire, and a later true->false edge could be missed.
connection.execute(
"""
INSERT INTO telemetry_state(key, value) VALUES (?, ?)
ON CONFLICT(key) DO UPDATE SET value = excluded.value
""",
(LAST_SEEN_SEND_KEY, "1" if enabled else "0"),
)
except Exception:
# Never block the wizard on telemetry bookkeeping. The sender records
# the same transitions on its next pass.

View File

@@ -25,6 +25,51 @@ def test_setup_telemetry_enables_shared_metrics(monkeypatch):
assert config["telemetry"]["shared_metrics"]["enabled"] is True
def test_disabling_collection_closes_the_send_consent_window(monkeypatch, tmp_path):
"""`hermes tools` -> disable shared metrics must withdraw send consent.
The not-enabled branch returned early without recording anything, so the
consent window stayed open and re-enabling later would release every
package collected in between.
"""
from hermes_cli.observability.shared_metrics import SharedMetricsStore
from hermes_cli.observability.shared_metrics_sender import SEND_REVOKED_KEY
store = SharedMetricsStore(
database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o"
)
monkeypatch.setattr(
"hermes_cli.observability.shared_metrics.SharedMetricsStore",
lambda *a, **k: store,
)
# The user had consented; now they turn collection off entirely.
monkeypatch.setattr(
"hermes_cli.setup.prompt_yes_no", lambda _question, default: False
)
config = {"telemetry": {"shared_metrics": {"enabled": True, "send": True}}}
# Consent was granted earlier, so a window is already open — that is
# precisely the state whose closure must be recorded.
from hermes_cli.sqlite_util import write_txn
from hermes_cli.observability.shared_metrics_sender import opt_in_period
with store._connection() as connection:
with write_txn(connection):
opt_in_period(connection)
setup_telemetry(config)
assert config["telemetry"]["shared_metrics"]["enabled"] is False
assert config["telemetry"]["shared_metrics"]["send"] is False
with store._connection() as connection:
row = connection.execute(
"SELECT value FROM telemetry_state WHERE key = ?", (SEND_REVOKED_KEY,)
).fetchone()
assert row is not None and row[0] == "1", (
"disabling collection left the send consent window open"
)
def test_setup_parser_accepts_telemetry_section():
parser = argparse.ArgumentParser()
subparsers = parser.add_subparsers(dest="command")

View File

@@ -136,6 +136,28 @@ class TestTransportSafety:
)
assert resolved.send is False
@pytest.mark.parametrize(
"endpoint",
[
"ftp://localhost/v1/telemetry",
"gopher://localhost/v1/telemetry",
"ws://127.0.0.1/v1/telemetry",
],
)
def test_a_non_http_scheme_on_loopback_is_still_refused(self, endpoint):
"""The scheme is allowlisted, not merely checked for plaintext http.
Gap found by mutation testing: replacing the `http` scheme test with
`if True` survived the whole suite, because every non-http scheme case
pointed at a REMOTE host, where the loopback branch rejects it anyway.
Only a non-http scheme aimed at loopback distinguishes an allowlist
from a plaintext-only check.
"""
resolved = resolve_send_config(
_config(enabled=True, send=True, endpoint=endpoint)
)
assert resolved.send is False
def test_unsafe_endpoint_does_not_block_collection(self):
resolved = resolve_send_config(
_config(enabled=True, send=True, endpoint="http://example.test/v1")

View File

@@ -23,6 +23,31 @@ class FakeStore:
return []
class RealBackedStore:
"""A store with a genuine SQLite connection, for consent-state tests.
The consent edge detector writes to telemetry_state, and it is wrapped in
a broad except. Against a stub without _connection it would swallow an
AttributeError and silently do nothing — which is exactly the failure this
file needs to be able to catch.
"""
def __init__(self, tmp_path):
from hermes_cli.observability.shared_metrics import SharedMetricsStore
self._real = SharedMetricsStore(
database_path=tmp_path / "m.db", outbox_directory=tmp_path / "o"
)
self.exported = 0
def _connection(self):
return self._real._connection()
def create_and_export_package_if_due(self):
self.exported += 1
return []
class FakeSubscriber:
def __init__(self):
self.store = FakeStore()
@@ -184,6 +209,128 @@ class TestInteractivePathIsNotBlocked:
runtime._join_send_thread(timeout=5)
class TestConsentRevocationWindow:
"""The falling edge must close the window even with no pass running.
Round 3 recorded revocation inside the send loop, which cannot fire for
the dominant case: the user turns sending off while idle, so the relay
early-returns and no sender is ever built. Re-enabling then released
every package collected during the refused window.
"""
def _runtime(self, tmp_path):
runtime = Runtime()
runtime.subscriber.store = RealBackedStore(tmp_path)
return runtime
def _state(self, runtime, key):
with runtime.subscriber.store._connection() as connection:
row = connection.execute(
"SELECT value FROM telemetry_state WHERE key = ?", (key,)
).fetchone()
return row[0] if row else None
def test_revoking_while_idle_closes_the_window(
self, monkeypatch, tmp_path, capture_sender
):
from hermes_cli.observability.shared_metrics_sender import (
SEND_REVOKED_KEY,
)
runtime = self._runtime(tmp_path)
_set_config(monkeypatch, _config(enabled=True, send=True))
runtime._send_exported_packages()
# User edits config.yaml: send: false. Hooks keep firing normally.
_set_config(monkeypatch, _config(enabled=True, send=False))
for _ in range(6):
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) == "1", (
"revoking while no pass was running left the consent window open"
)
def test_no_spurious_revocation_when_nothing_changes(
self, monkeypatch, tmp_path, capture_sender
):
"""The detector must key on an EDGE, not on every disabled pass.
A level trigger re-closes a window the user has since REOPENED: each
later disabled pass stamps revoked again, so the next enabled pass
advances the gate and silently drops packages the user did consent to.
Mutation-checked — an earlier version of this test used a
never-consented store, where record_revoked no-ops regardless, and so
could not tell an edge trigger from a level trigger.
"""
from hermes_cli.observability.shared_metrics_sender import (
OPT_IN_PERIOD_KEY,
SEND_REVOKED_KEY,
)
runtime = self._runtime(tmp_path)
_set_config(monkeypatch, _config(enabled=True, send=True))
runtime._send_exported_packages()
_set_config(monkeypatch, _config(enabled=True, send=False))
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) == "1"
# User changes their mind and re-enables.
_set_config(monkeypatch, _config(enabled=True, send=True))
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) is None, (
"re-enabling must clear the revocation marker"
)
reopened = self._state(runtime, OPT_IN_PERIOD_KEY)
# Further ENABLED passes must not disturb the reopened window.
for _ in range(4):
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) is None, (
"a steady enabled state re-closed the consent window"
)
assert self._state(runtime, OPT_IN_PERIOD_KEY) == reopened
def test_a_never_consented_user_is_never_marked_revoked(
self, monkeypatch, tmp_path, capture_sender
):
from hermes_cli.observability.shared_metrics_sender import (
SEND_REVOKED_KEY,
)
runtime = self._runtime(tmp_path)
_set_config(monkeypatch, _config(enabled=True, send=False))
for _ in range(5):
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) is None
def test_re_enabling_after_an_idle_revocation_starts_a_new_window(
self, monkeypatch, tmp_path, capture_sender
):
from hermes_cli.observability.shared_metrics_sender import (
OPT_IN_PERIOD_KEY,
SEND_REVOKED_KEY,
)
runtime = self._runtime(tmp_path)
_set_config(monkeypatch, _config(enabled=True, send=True))
runtime._send_exported_packages()
first_window = self._state(runtime, OPT_IN_PERIOD_KEY)
_set_config(monkeypatch, _config(enabled=True, send=False))
runtime._send_exported_packages()
assert self._state(runtime, SEND_REVOKED_KEY) == "1"
# Re-enabling must not simply resume the original window.
_set_config(monkeypatch, _config(enabled=True, send=True))
runtime._send_exported_packages()
assert first_window is not None
class TestFailureIsolation:
def test_a_sender_crash_does_not_propagate(self, runtime, monkeypatch):
class Exploding: