Two review findings on the salvage stack:
- _write_turns' _quietly consolidation keyed failure on a None result,
implicitly assuming add_memory never legitimately returns None. A None-
returning stub (the most common mock idiom) would mark every successful
write failed and re-append the batch forever. Module-level _FAILED
sentinel: only a raised exception re-queues.
- _pending_turns had no bound: a persistently failing service accumulated
one entry per turn for the process lifetime (gateway runs never
re-initialize), and every retry re-sent the whole accumulated payload —
O(n^2) upload bytes, and a size-rejected batch could never shrink. Cap
the buffer (50 turns / 256 KiB, drop oldest, warn once per trim).
The per-turn capture rewrite removed the raw urllib /v4/conversations
ingest, but stale references survived outside the diff hunks:
- README "Behavior" still carried the "written once via the conversations
endpoint" paragraph contradicted by the new bullets right above it.
- website memory-providers (EN + zh-Hans) still listed full-session ingest,
session-end /v4/conversations ingest, and ingest in the base-url and
api_timeout rows — the PR had updated one line per file but missed the
rest of the section.
Now every surface describes per-turn documents.add capture with retry.
Two cleanups on the salvaged per-turn capture path (review of #109359):
- _write_turns hand-rolled the try/except/log that the file's own _quietly
helper already provides (same exception class, level, exc_info shape);
the deleted _ingest path used _quietly for the identical call. Consolidate:
add_memory always returns a dict, so a None result is the failure sentinel.
- _capture_custom_id's `or 'hermes'` never fires: _sanitize_tag already
returns _DEFAULT_CONTAINER_TAG on empty input, and the literal duplicated
the constant the helper owns. Verified behavior-preserving (tests green
with the fallback artificially restored).
sync_turn runs on the MemoryManager worker, but on_session_switch and
shutdown run on the caller thread, so two _write_turns() calls could
snapshot the same pending batch and each replace the whole list (duplicate
append or lost pending turn). A capture lock now covers the
snapshot/write/replace sequence; _write_turns() reads pending turns inside
the lock instead of taking a caller-built list.
Retries are at-least-once: the documents API appends on a shared custom_id
and does not dedupe by content, so a write it accepted but whose response
was lost is appended again. Stated in the docstring and README instead of
implied.
Addresses review on #109359.
A failed flush at on_session_switch() restored the pending buffer and
then cleared it on the next line, so an unavailable service at the switch
boundary still lost every pending turn. Pending turns now carry their own
session_id; _write_turns() batches per session, so a later retry (next
turn, session end, shutdown) writes old-session turns under the old
session's custom_id even after the switch.
Addresses review on #109359.
sync_turn now writes each completed turn through the SDK's documents.add,
keyed by custom_id "<session>_<date>_b<0-5>" so all turns of a session in
one 4-hour window append to a single document. This matches the capture
shape of the other Supermemory agent integrations and removes the raw
urllib POST to /v4/conversations, which the self-hosted server does not
implement (#101270).
Failed turn writes stay pending and are retried with the next turn, at
session end, on session switch, and at shutdown. Previously a failed
session-end ingest was logged once and the whole session was lost.
Inline base64 data URIs in captured text are replaced with "[image]" so
pasted screenshots no longer land in the document as megabytes of text.
Metadata stays type/session_id/timestamp plus the existing sm_source.
Under gateway.multiplex_profiles os.environ holds the default profile's .env, so `_run_brv`
building the child env from raw os.environ curated a secondary profile's turns into the DEFAULT
profile's ByteRover cloud account (and prefetched the default's memories into the secondary's
context). The local half was already profile-scoped (`_get_brv_cwd`).
The child env now comes from `build_subprocess_env` and, under multiplex, strips the launch
profile's residue and sets BRV_API_KEY only from the served profile's secret scope — a miss means
no cloud key. Single-profile installs pass the process env through unchanged.
Closes#108993 (report and fix direction by @jonpol01).
`langfuse._secret` and `azure_identity_adapter._scoped_env` were changed to
raise rather than fall back, because swallowing `UnscopedSecretError` hides the
spawn-site bug the exception exists to surface. `_scoped_setting` looked like it
contradicted that, so make the split explicit and pin it.
Hindsight already follows the contract for everything that decides WHERE data
goes: `mode`, `apiKey` and the `bankId` partition read through bare
`get_secret`, so a scopeless multiplexed read raises. In `_load_config` that
raise happens on `HINDSIGHT_MODE` before any shaping value is reached, so the
swallow below cannot mask an isolation failure.
Presentation shaping is deliberately not in that class. `MemoryManager._each_provider`
logs an `initialize` failure at WARNING and drops the provider for the session,
so raising there would cost the whole memory provider because a speaker prefix
could not be resolved. It degrades to the provider's own default instead —
never to `os.environ`, which under multiplex is the default profile's.
The test names the offending key rather than asserting that something raised:
routing `mode` through the shaping helper shifts the failure to
`HINDSIGHT_API_KEY`, which a bare `pytest.raises` would still accept.
_load_config() reads the Hindsight bank, mode and retain tags through the profile
secret scope, but _apply_retain_settings() then discarded that answer and re-read
os.environ whenever the config value was falsy:
return cfg.get(key) or os.environ.get(env_var, default)
Under gateway.multiplex_profiles os.environ holds the DEFAULT profile's .env, so a
secondary profile's scoped miss came back as the default profile's retain tags,
observation scopes, source and speaker prefixes — the fallback-after-miss shape
gateway/AGENTS.md forbids. Tags are Hindsight's retrieval partition and
metadata.source is opt-in by design, so the secondary's memories were both
mislabelled and selectable by the default profile's tag filters.
Both halves now go through _scoped_setting(), which resolves the value with
get_secret() and falls back to the provider's OWN default — a miss is a miss, the
same rule embedded.py already applies to the daemon's key and base URL. The three
raw reads left inside _load_config() (retain_source, retain_user_prefix,
retain_assistant_prefix), directly under the comment declaring them per-profile,
go through it too.
Single-profile deployments are unchanged: with no scope installed get_secret()
still reads the process env, where the value IS this profile's own.
Fixes#108865
_render_peers_map_view called _sibling_resolutions per account, and each call
ran _all_profile_host_configs(), which parses every profile's config.yaml via
list_profiles() and re-reads its honcho.json; show() re-runs after every
workspace switch. cmd_peers_map now scans once and threads the rows through.
_seen_gateway_accounts turned a locked or corrupt state.db into the same [] that
means "no gateway traffic yet"; it now notes the error on stderr before returning
the empty list. The missing-table fallback for old schemas is unchanged.
Iterating a honcho SyncPage walks every following page, so _api_workspace_peers
pulled the whole workspace on its first call, never hit the `< 50` break, re-walked
pages 2..N and returned duplicates; the 200-peer cap that exists so a public bot's
workspace cannot stall the CLI was defeated and the `p<N>` picks pointed at the
wrong peers. It now reads `.items` per page and stops at the cap; `_api_workspaces`
reads `.items` too.
The wizard's `new_host` probe looked only at the host block, so an install that
keeps peerName/enabled/workspace at the root with no hosts.hermes block read as
fresh and Enter defaulted to pinning every account onto one peer. The probe
includes the root.
`_seen_gateway_accounts` dropped rows whose origin said `is_bot`, but
SessionSource.to_dict never serializes that field, so the filter was dead; removed
with its test row. `_sanitize_peer_id` was a copy of session_peers.sanitize_peer_id.
A confirmed workspace repoint is persisted, so the exit line no longer says
"Nothing changed" after one.
The identity step treated any host block without a mapping key as a new install and defaulted the choice to single peer. An install with enabled, workspace and peerName then had Enter write pinUserPeer: true and merge every gateway account onto the operator. cmd_setup now decides new-ness before its prompts populate the block, and only a block with none of the mapping, peerName, workspace or enabled keys defaults to single.
The docstring said grouping session rows by (source, user_id) enumerates every account the gateway handled. record_gateway_session_peer overwrites a row's user_id, so a shared thread keeps only its last author. The docstring now says that, and a test pins it.
clone_honcho_for_profile copied sessionPeerPrefix but not sessionAiPeerPrefix. A new profile cloned from a default block with the AI prefix on fell back to unprefixed session names and collided with the default profile's gateway sessions.
peers map showed sanitize(prefix + user_id) for prefixed accounts. The runtime appends a sha256 suffix when sanitizing changed the id or it collides with an explicit peer, so the preview named a peer the gateway never writes to. The preview now builds a HonchoSessionManager with no client and asks it.
Clearing the last alias in peers map popped userPeerAliases from the host block, and the host then inherited the root aliases again. The host block now keeps an empty map, which both config readers treat as an explicit override. The summary says when that empty map hides root aliases.
_api_workspace_peers returns peer IDs instead of dicts. The created date
it carried was never printed. Callers index the list directly.
_preview_peer_resolution uses _sanitize_peer_id instead of a local copy.
_sibling_resolutions strips the display suffix itself; its one caller did.
_seen_gateway_accounts opens the database under contextlib.closing and
parses origin_json once into a dict. _save_alias_map picks the target
block first and writes it once. cmd_peers_map renders through one local
show() and computes the previous resolution on one path for both account
numbers and typed runtime IDs.
peers map read seen accounts from state.db but required a non-empty
session_key. every telegram row on a long-lived install written before the
gateway stamped that column was dropped, so the picker showed no accounts
while the same rows grouped fine by (source, user_id). the key was never
used by the grouping.
The picker-cap comment repeated the constant's name; the
sessionAiPeerPrefix field comment restated the field name and its
symmetry with sessionPeerPrefix before reaching the collision it
prevents. Both now state only the why.
Extends the read-only 'hermes honcho peers' view with a 'map' action
that joins two sources: workspace peers fetched from the Honcho API,
labeled from local config (your peer, each profile's AI peer, alias
targets, runtime peers of seen accounts, user-* fallback peers,
'unrecognized' otherwise), and the gateway accounts recorded in
state.db with what each currently resolves to. Targets are picked
from the workspace list so a typo cannot silently create a peer;
every assignment states its consequence (aliases move future
messages only; a runtime peer left behind keeps its history).
'w' lists every workspace the key can reach — the wrong-workspace
fallback — and can repoint the profile's workspace on explicit
confirmation. With multiple profiles, saving a root-cascading map
asks whether to write root or fork the host block, root writes warn
when sibling profiles sit on other workspaces, and the accounts
table marks siblings that resolve an account differently. Offline
the command degrades to typed targets over the local account list.
The setup wizard's gateway step closes by pointing at the command.
Address review feedback on #39130: the new sessionAiPeerPrefix setting
affects the resolved Honcho session key, which HonchoMemoryProvider freezes
at construction (self._session_key). Because it wasn't part of the gateway's
cached-agent signature, a live config flip left an existing gateway session
bound to its old, AI-peer-agnostic Honcho session until an unrelated eviction
or restart.
Add honcho.session_ai_peer_prefix to _HONCHO_CACHE_BUSTING_KEYS and the
_extract_honcho_cache_busting_config values so a flip rebuilds the cached
agent on the next turn, mirroring the existing aiPeer / pin_peer_name /
runtime_peer_prefix contracts.
Also close the symmetric gap for the pre-existing user-side sessionPeerPrefix:
it feeds the same resolve_session_name output (per-session/title/per-repo/
per-directory strategies) into the same frozen _session_key, so it had the
identical live-flip staleness bug and was likewise absent from the cache
signature. Fixing both keeps the two prefixes consistent.
Add one config-flip regression test covering both keys, alongside the
existing Honcho cache-signature test.
The gateway_session_key branch of resolve_session_name() returns an
AI-peer-agnostic name, so multiple AI peers sharing one workspace +
peerName + gateway chat key collide on a single Honcho session.
Add sessionAiPeerPrefix (symmetric counterpart to sessionPeerPrefix):
when set, the resolved session name is prefixed with {ai_peer}- on every
resolution path. The prefixed name is re-run through the session-id
length cap so the prefix can never exceed Honcho's limit.
- config field + host/root parsing in client.py
- public resolve_session_name() wraps a new _resolve_session_name_base()
- tests covering parsing, the gateway-key case, cross-peer disjointness,
the length cap, and a disabled-by-default regression guard
- README: config table + resolution notes
The step declares its scope up front: human mapping only, with each
Hermes profile bringing its own AI peer. A note explains aliases as
the join between platform accounts and named peers. Each shape now
says when it fits. Fresh configs default the choice to [1] single
peer — the common personal setup — instead of [3], which silently
fragmented a solo operator's gateway account away from their
peerName history. Configured setups keep their detected shape as
the default.
main folded the plugin's contextvars-inheriting thread wrapper into
agent.memory_provider.spawn_context_thread. The registry this stack adds for
join_plugin_threads keeps a thin client.spawn_context_thread that calls the core
spawner and records the thread under its owner; the four spawn sites import it
from client. main's inherit-profile test builds a real provider now that
_spawn_write is an instance method (the owner is what gets joined).
_SESSION_CACHE_MAX_SIZE was assigned twice with two comments describing one
constant. _retain_for_retry and _keep_until_flushed shared the has-unsynced /
current-owner / reclaim body and differed only in what to do when a newer object
owns the key; _reclaim_key_locked returns that owner and each caller keeps its
tail. _ensure_async_writer had no production caller (save() uses the _locked
form under _async_thread_lock); removed, tests retargeted, constructor comment
fixed. shutdown() sets _shutting_down under _async_thread_lock like the other
site so save()'s flag check and the enqueue cannot interleave with it.
tests/test_honcho_session_cache_bounds.py hoists its mid-file imports.
A flush that rebuilds an evicted session's SDK session stores its observation
flags again, and the cap pass pruned every per-session dict but that one, so the
dict grew one entry per evicted-then-flushed session. The cap pass now prunes it
with the rest.
The peer-failure notice classified platforms with its own {"cli","tui","desktop"}
set, so an ACP session read the gateway wording ("do not suggest peerName"); it
now uses agent.coding_context.INTERACTIVE_CODING_PLATFORMS, which includes acp.
Three getattr/try-except guards existed only for test doubles (a bare __new__
provider, a SimpleNamespace config); the real types always carry the attribute.
Removed, and the two tests build real objects. An unset timeout resolves to the
client's 30s default, so the join-budget test expects 30, not the 5s floor.
Timing tests keep the >= 2s wall-clock bound the testing rules ask for.
main's _join_observation_flags returned the manager-wide booleans with a comment
saying #103889 would plug the per-session flags in here; this is that plug. The
two main-side tests that stubbed the old two-tuple _get_or_create_honcho_session
return move to the three-tuple.
The generic panel writes every field flat, and the plugin reads `injection` as one object, so a dotted `injection.sessionStart` field would never be read. Declare `injection` as a JSON field instead. Blank clears the pin.
stop_async_writer bounded only the join and then drained the queue with no deadline, so a shutdown whose budget was already spent could still start uploads, and honcho-ai's add_messages has no per-call timeout. The join and the drain now share the shutdown deadline, no upload starts once it has passed, the writer skips its 2s retry once shutdown began, and shutdown logs one warning with the count left unsynced. The docstrings now say what the budget can do: stop new uploads and bound lock waits, while an upload already in flight runs to the client's HTTP timeout.
A clean session can be evicted while its caller still holds it, and the caller's next save in turn mode flushed inline and never put the object back, so a failed upload left the batch nowhere flush_all() looks. Main reinserted after every flush. A failed save-time flush now reinserts the session when its key is free, or holds it in a retry list when a newer object owns the key, and flush_all() covers both; the collision path in _keep_until_flushed honors the flush result the same way.
Provider shutdown handed the manager a remaining budget, but flush_all ran first with no deadline and blocked on each session's flush lock. An async upload still in flight held that lock, so shutdown waited the full HTTP timeout past its declared budget. flush_all and the queue drain now take the deadline, skip a session whose lock or budget is gone, and log one warning with the count of messages that stayed unsynced.
With recallSync on, prefetch popped only the auth notice and returned without writing the injection log. A session whose init failed for a missing user peer never told the model that memory was off, and the audit file stayed empty for every turn. The recall sync branch now pops the peer notice the way the async branch does and records each turn as injected or recall-sync-empty.
Parametrize the sessionStart, injection-log, dashboard user_id, unresolved-peer
and deferred-save tests that differed only in their inputs. Share the blocking
remote in the concurrent flush tests. Fold _as_flag onto a word table. Cut the
added docstrings to the what and the one non-obvious why.
recall_sync.py spawned honcho-recall-sync without owner=, so shutdown's
join_plugin_threads((self, manager), ...) never saw it. The worker now
registers under the provider like the other provider threads.
`_flush_session` discarded the observation flags when it rebuilt an evicted SDK session, and the
cached path returned none, so recall fell back to the config snapshot. Both paths now return and
store the flags. A deferred `save()` on a session the cap evicted puts it back in the cache, or
flushes it inline when a newer object owns the key. `save()` and `stop_async_writer()` share the
writer lock, and the writer drains its queue after the join, so a put that raced shutdown is
written. The trim after a flush runs under the cache lock. The shutdown join takes the remaining
budget instead of a fixed ten seconds.
The injection audit file is created owner-only, and `logging: "false"` reads as off. The desktop
passes `<provider>:<user id>` so a basic-auth alice and an OIDC alice are two peers. When a
gateway platform supplies no user id, the peer notice and tool error no longer recommend
peerName, which would merge every user of that gateway onto one peer. README documents
`injection.sessionStart`, `logging`, and what a dashboard login does to peer resolution.
a desktop or cli session with no peerName in honcho.json and no gateway
user id landed on a peer derived from the session key: user-default-<dir>
for per-directory sessions, user-<channel>-<chat> for keyed ones. every
directory got its own phantom peer, so the operator's turns and memory
never reached their real peer and the injected representation went stale
(#93326).
_resolve_user_peer_id now raises HonchoPeerUnresolvedError instead of
deriving a name. a peer is either the declared peerName or an identity
the transport supplied. the provider records the failure, tells the model
once that memory is off and which key to set, returns the same detail
from tool calls, and stops retrying init because a missing config key
does not heal mid-session. the memory-file migration gate loses its
"no owner and no runtime identity" branch: that cohort no longer has a
session to migrate into. whitespace-only peerName is treated as unset
rather than sanitized to "--".
provider shutdown joined the dialectic, sync and memwrite threads for 5s
each and then the async writer, but the session-init thread,
honcho-base-first and honcho-context-prefetch were never joined. any of
them still blocked in httpx when the interpreter finalized aborted the
process with SIGABRT 134 (#37632, #60616, #33485). the 5s join was also
shorter than the 30s http timeout a blocked call can hold (#33485).
spawn_context_thread now registers each thread under its owner (the
provider or the manager) in a weak registry, and shutdown joins every live
thread of both owners inside one deadline: at least 5s, or the configured
http timeout when longer. the manager refuses new prefetch threads once
shutdown began and flushes a late save() inline instead of respawning the
writer. threads that outlive the budget are named in a warning.
the sdk client exposes close() on its http pool. one client is shared by
every manager with the same identity in a gateway process, so a per-agent
shutdown cannot close it; close_honcho_clients() closes all pools and is
registered with atexit when the first client is built, the pattern the
hindsight and mem0 plugins use. follows #69070, #33701, #7627.
the idle sweep from #71463 left three growth paths open. _peers_cache had no
bound at all, _session_observation (from #98941) grew one entry per session
id and kept orphans when an init failed after add_peers, and a burst of
distinct sessions inside one ttl window was not bounded. the sweep also
evicted sessions whose messages had not reached honcho yet, which in
"session" write mode drops the only copy, and _flush_session re-inserted an
evicted session into the cache with no observation flags, so recall for it
routed from the config snapshot instead of its server config.
_cache and _sessions_cache now cap at 128 entries and _peers_cache at 512,
evicting least recently used first (dict order, refreshed on every hit).
a session with unsynced messages is never evicted by the sweep or the cap.
_configure_session_peers returns the synced flags and get_or_create stores
them under _cache_lock next to the cache entry, so the observation dict can
hold no id the cache does not; eviction drops both. recall reads go through
_cached_session, which stamps updated_at, so a read-only session survives
the idle sweep. follows #71461, #71463, #98936.
_flush_session read the unsynced messages, posted them, and only then set
_synced. in a short-lived run the async writer draining save()'s queue and
the exit-time flush_all() both saw the same batch unsynced, so every turn
of a one-shot run landed in honcho twice (#92458).
each HonchoSession now carries an RLock that _flush_session holds around
the whole select, send, mark sequence. the second flusher enters after the
first marked the batch and sends nothing; different sessions still flush
in parallel. the lock lives on the session object instead of a manager
dict keyed by session id (#86094, #92787): it needs no eviction, and two
flushers of one message list can never hold different locks. tests
adapted from #86094 and #92787.
HonchoSession.messages grew forever -- add_message() appended every
turn and _flush_session() only marked entries synced, never trimmed
them. HonchoSessionManager's four caches (_cache, _peers_cache,
_sessions_cache, _context_cache) had no eviction path besides an
explicit /new reset. A long-lived channel that is never manually
reset accumulates both for the gateway's entire uptime.
Honcho is the durable source of truth (get_or_create() already
re-fetches history from Honcho on a cache miss), so both bounds are
safe: evicting an idle local entry only costs one extra Honcho
round-trip next time that key is used.
- Trim already-synced messages beyond a retention window right after
a successful flush; unsynced messages are never touched.
- Add a rate-limited idle-TTL sweep triggered opportunistically from
get_or_create(), so no new background task/watcher wiring is
needed.
Fixes#71461
The four observation booleans on HonchoSessionManager were manager-wide
mutable state initialized from per-session server configs: every session
setup overwrote them, so the last session to initialize retuned recall
routing for all other sessions the manager serves. Store the server-synced
flags under each session's own id instead; the manager-level fields stay
as the config snapshot and sessions that never synced fall back to them.
injection.sessionStart: unset renders every component in table order, an
empty list renders nothing, a pinned list renders only those names in table
order (not config order), a host block pin beats root, a non-list value is
treated as unset, and initialize() reads the pin.
logging: off by default, the logging key or HONCHO_LOGGING turns it on, a
host block can turn it back off, HONCHO_INJECTION_LOG overrides the path,
each record carries reason/turn/session_key/bytes/payload, an unwritable
path never raises, and a tools-mode prefetch logs its reason. the record
holds the user's representation verbatim, which is why the default stays off.
the two session-context tests that asserted exact context() kwargs now expect
tokens= as well (adopted from #92964). config_schema declares the logging
switch so hermes memory setup shows it.
Salvage of #70951 (Willkons / Alice-Willk-bot). Two of three
session.context() sites never forwarded the configured cap, so Honcho
always returned honcho_chat_summary_long.
Adds the missing get_session_context tokens= regression the sweeper
asked for on that PR.
force_refresh_token gated the adopt-from-disk paths behind the failure cooldown.
After one of our exchanges failed transiently, a 401 within the next 30s returned
None even when a sibling process had already rotated and written a valid grant,
so the operation raised HonchoAuthError with a good token sitting on disk.
Adopting is a disk read, not an exchange; the cooldown exists to stop replaying a
single-use refresh token, so the two adopt checks now run before the gates.
_write_config parsed honcho.json twice under the lock and only wrapped the first
read into ConfigWriteRefused; _refuse_unparseable now returns the parsed dict and
the branches use it. The getattr/isinstance duck-typing collapses to one guard.
cli._read_config reuses oauth's tolerant reader (BOM-tolerant, like the strict
reader the write side uses) instead of its own utf-8 copy.
The CLI's _write_config took only the best-effort file lock, so an in-process
refresh thread could still interleave with a command's read-modify-write. The
dashboard's Honcho save (_write_provider_honcho) still seeded its whole-file
rewrite from a tolerant reader, so a honcho.json that exists but does not parse
was replaced by the active host's block alone from the UI - the same bug class
this PR closes on the CLI and refresh paths. `hermes profile create --clone`
swallowed the new ConfigWriteRefused as "plugin not installed".
One parametrized test covers the web writer for the corrupt and parseable cases.
_write_config applied a command's edits relative to the snapshot the read took, but never moved that snapshot after a successful write. A second write on the same object therefore compared A -> B -> A against A, saw no change, and left disk at B. After a write the snapshot and path now follow the caller's dict, so the next write applies only the edits made since.
