Rebased onto main (~9700 commits): the branch predates the god-file
decomposition and the tests/state -> tests/hermes_state move.
Fixes for seams that moved:
- cron memory contract patches cron.scheduler_delivery._resolve_origin and
hermes_state_registry.acquire (where run_job now reads them).
- state.db conformance imports _live_writer_holds_db from
hermes_state_repair and drives the public create_quick_snapshot.
- update receipt: serve runtimes are reconciled in their own unit
vocabulary since #100479, so the "full accounting" row names the serve
unit instead of borrowing a gateway relaunch.
Deleted (change-detectors / source-greps / duplicates / dead code):
- source-text scan of hermes_state*.py for "maintenance-shaped" defs and
the symbol REGISTRY it fed (renames are not regressions).
- per-op exact-outcome table under a live writer -> one invariant: refuse
with a lock error or report zero work, DB stays intact.
- copy_db_and_verify pins: the symbol is a revert-scheduled plugin-compat
pointer, not production code (check_compat_pointers.py).
- cron ON-direction tests already pinned by tests/cron/test_scheduler.py,
plus a tautology that never called production code.
- exact warning-wording asserts in the env deprecation truth table.
- same-file duplicates: fixed-point (implied by idempotence), tripwire
round-trips, absent-registry, sequential "race" re-enactment.
Two weeks of closed issues/merged PRs show the same areas regenerating:
each salvage pinned its instance while the class invariant had no test.
These suites pin the invariants themselves:
- tests/conformance/test_profile_write_tripwire.py — no writes to the
default profile tree while a profile is active (#88532#92662#89190#89625#92156); reusable tripwire fixture, 4 surfaces
- tests/hermes_cli/test_env_deprecation_truthtable.py — 18-row truth
table for the Deprecated-.env warning (#88829#89016#89389#90299)
- tests/cron/test_cron_memory_contract.py — cron<->memory contract that
flipped twice in Aug (#91269 -> #91384 -> #91447)
- tests/agent/test_injected_param_strip_retry_registry.py — every
strippable injected param x real 400 shapes must strip-and-retry;
unknown params must still fail (#90257#89897#91164#89503)
- tests/agent/test_transcript_decoration_idempotence.py — f(f(x))==f(x)
law + 4-breakpoint budget for apply_anthropic_cache_control (#90971)
- tests/state/test_state_db_maintenance_conformance.py — registry-
enumerated maintenance ops refuse/degrade under a live writer; copies
of corrupt DBs are refused or flagged (#91839#90806#90613#88235)
- tests/tools/test_bot_mode_canonical_chat_resolution.py — canonical
Bot Chat resolution is idempotent, never mints, unique per profile,
race-safe (#92040#90705#92692#90005#90732, PR #92129)
- tests/hermes_cli/test_update_receipt_truthfulness.py — receipts:
crash never claims success; success requires full fleet accounting;
refusal != failure (#91283#91439#92902#92780)
117 tests, all sabotage-verified (each suite proven to FAIL when its
bug class is reintroduced).
Salvage round on the Phase 1 skeleton (three findings from review):
1. The DELETE leg was silently upgraded to WAL on healthy SQLite:
pre-seeding the file via PRAGMA was undone by SessionDB.__init__'s
apply_wal_with_fallback(), which upgrades any non-WAL file whenever
the configured mode (default wal) says so — only WAL-reset-vulnerable
interpreters preserved DELETE, i.e. the leg tested the advertised
mode only where CI wasn't running. Each matrix leg now pins
database.journal_mode in an isolated HERMES_HOME for the child and
audits the ON-DISK mode after the run (effective_mode_or_skip):
a leg that ran in a different mode skips instead of double-counting.
2. Cell 2 exit-code conflation: a claimant crashing with an unhandled
exception exits 1 — indistinguishable from the clean "lost the
claim" exit(1), so one winner + seven crashes passed as consume-once
proof. Codes are now disjoint (0=won, 10=lost, anything else=crash).
3. Crashed-writer diagnostics: wait_for now fails immediately with the
child's stderr when the writer dies before reaching the kill window
(was: 60s opaque deadline, stderr discarded). spawn_child prepends
to an inherited PYTHONPATH instead of clobbering it.
Plus: dead `if False` scaffolding removed from cell 1's writer; README
matrix section corrected (cell 2 is default-mode-only by design — the
consume-once property rests on a single predicated UPDATE).
Deterministic, LLM-free conformance cells against the real SessionDB with
real SIGKILL mid-write, per the tracking issue's spot-probe method:
- cell 1: acknowledged-append durability + recovery determinism (adapted
from the issue's 29.5K probe, scaled kill window, identical assertions)
- cell 2: consume-once under 8-process concurrent claim_handoff
- cell 3 (new): compression-rotation atomicity — never a compression-ended
parent without a continuation (#80337 contract; #80487 recovery context)
- cells 4-5: documented stubs interlocked with #82956-#82959 and
#83197/#83557
Journal-mode matrix (resolver default / DELETE / WAL-with-skip-gate) per
cell; every wait deadline-bounded; writers asserted alive at kill time.
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
