Two related races in credential-pool cooldown state:
1. Lost update across processes: write_credential_pool merged only
entries missing from the caller's snapshot; for entries present on
both sides the caller's in-memory copy won wholesale. A process
holding a snapshot taken before another process marked a key
exhausted would, on its next persist (e.g. a round-robin rotation),
write the key back as healthy — erasing the cooldown so every
process resumes hammering a rate-limited key. Merge status fields by
last_status_at recency: adopt the on-disk status only when it is
strictly newer AND still binding (DEAD, or EXHAUSTED with an
unexpired cooldown), and never onto re-authed (token-changed)
entries, so legitimate expiry-clears and fresh logins are preserved.
2. Wrong-key quarantine: when mark_exhausted_and_rotate received an
api_key_hint that matched no entry, it fell through to
current()/_select_unlocked() — on a freshly loaded pool that selects
the NEXT healthy key and benches it for the full cooldown TTL,
punishing an innocent credential. When a hint is provided but
unmatched, rotate without marking anything instead of guessing.
Includes regression tests.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
3d67f00fe1