Found by the pre-merge correctness and hermes-specific review of this stack:
- A fold after a row whose one response repeated a provider id (one shared uid) appended the
absorbed turn's uid in the wrong slot. The shared uid now fills each of the row's own
occurrences before the fold appends (per_occurrence_tool_call_uids). A single response keeps one
shared uid for a repeated id: every result carries it, so no call looks unanswered.
- A digest-less re-flush of a restored fold (replay heal, adopt after a digest mismatch) let the
stored pre-fold map overwrite the longer live list, so the later occurrence's results paired
with nothing. A live list that starts with every stored occurrence is kept (stored uid or list).
- A dict that lost its uid map (a clone) rewrote tool_call_uids to NULL. The row-addressed
rewrite now fills stored uids for the calls the dict still names; a call it dropped keeps none.
- append_messages_batch(chunk_rows=...) reset the pairing index per chunk, so a result in the chunk
after its call was stored without tool_call_uid (branch/seed copies). The whole batch is paired
before it is chunked.
- _merge_assistant_into keeps its dict guard on the absorbed turn's map (a plugin-built dict with a
non-dict map no longer raises).
_merge_assistant_into never joins multimodal (list) content, so a non-empty later turn
beside a list (or a list beside non-empty text) is dropped, yet the fold still recorded its
uid in _absorbed_message_uids, a false claim that its text lives on. The merge now reports
whether the later text was kept and the witness follows it; the retired row id is still
recorded.
Reported by @andrexibiza on #126307.
Folding two assistant turns that reuse a provider id (llama.cpp-style constant ids) leaves
both calls in tool_calls, but the {id: uid} union kept only the later uid, so the first
occurrence lost its identity. The id now maps to a list of uids, one per occurrence in
tool_calls order (merge_tool_call_uids); index_tool_call_uids pairs each call with its own
occurrence, so a following result still pairs with the nearest call. Provider ids are never
rewritten. Persisted and restored as-is (the JSON map accepts list values).
Reported by @andrexibiza on #126307.
Alternation repair replaces a provisional verification candidate with the final answer instead of
folding it. Its row is still retired through _absorbed_row_ids, but its uid must not appear in the
replacement's _absorbed_message_uids: the witness names text that lives on in the composite.
(cherry picked from commit fd284d1fd35371215a616efab965ef117348f3c0)
- Strip persistence-only fields from a select_context() selection: the request copy was stripped before
the hook, but an engine handing back the conversation_messages clones put timestamp, _row_id and the
new ids on the wire (pre-existing for the old fields).
- Every row carries a message_uid whoever wrote it: an AFTER INSERT trigger mints one for a row inserted
without it (an older build writing into a v31 store); the one-time backfill is gated by its own
state_meta marker so a store whose schema version cannot advance (no FTS5) never rescans.
- One merge-witness helper (record_absorbed_message) for every fold of a durable message: alternation
repair's user and assistant merges, the compressor's in-flight restatement onto the carrier, the real
user anchor folded into a scaffolding turn (the anchor's uid leads), micro-compaction's adjacent-user
merge.
- tool_call_uids follows tool_calls: it is a payload column, so an assistant merge's unioned map is
rewritten with the row instead of being overwritten by the stale stored map on adoption.
- A result pairs with the nearest call naming its provider id: each assistant row's ids shadow earlier
occurrences in the batch, restore and live-list indexes, a call row without a map leaves its result
unpaired, and nothing pairs across a user turn.
- A composite rewind returns and installs the replacement row's uid on the live scaffold.
- Tests: import_foreign_history, hermes sessions recover and backup import carry the ids; the ACP
restore test accounts for the uid on restored rows.
(cherry picked from commit b30636a49cb29fdd21ba4100eef9d810897338be)
Provider tool-call ids repeat: Hermes mints deterministic call_<12hex> ids for identical calls and models
reuse ids, so tool_call_id cannot identify an occurrence. An assistant row now carries
messages.tool_call_uids, a {tool_call_id: uuid4} map for its tool_calls, and each tool-result row carries
the matching messages.tool_call_uid. The provider-facing tool_calls JSON is untouched, so row identity,
display identity and the CAS digest are unchanged and nothing nested reaches the wire.
Minted at the assistant row's first insert; paired onto the result in the same batch, from the live list
when the result is flushed later, or on restore from the preceding assistant row; copied by every clone
and re-insert; restored as the live _tool_call_uids / _tool_call_uid; stripped from provider requests.
(cherry picked from commit aa49c0f59788676d4beca288181711c558fca165)
A merge survivor's _absorbed_message_uids lived only in memory, so an engine that restarted after a
merge was back to parsing "\n\n" to learn which stored messages a composite user row contains. Store
the list in messages.absorbed_message_uids (an owned column: the survivor's row-addressed rewrite and
every re-insert carry it), restore it as the live key, and round-trip it through export/import.
(cherry picked from commit 0bc05f2656229b1010f330e1998684b04d5f963f)
The physical messages.id is re-issued by every copy (in-place compaction generations and their
concurrent-tail clones, rotation children, replace_messages), and _row_id is opt-in on restore, so a
context engine that keeps its own per-message state had nothing stable to key on across a restart or a
compaction boundary and re-identified rows by content and timestamp.
Add messages.message_uid (uuid4 hex), minted once at a row's first insert and stamped on the caller's
dict, copied by every SQL clone, kept by every re-insert of the same dict, left alone by row-addressed
rewrites, restored on every projection, and stripped from provider requests. A consecutive-user merge
survivor keeps its own uid and records the absorbed uids in _absorbed_message_uids. Schema v31 backfills
a uid onto rows that predate the column.
(cherry picked from commit 20d67ffb5e50e0ed956e56ddc7f8e28d36059e2e)
plugins/context_engine.load_context_engine scanned only the bundled directory. An engine
dropped into $HERMES_HOME/plugins/<name> with `context.engine: <name>` was reachable only
through the general plugin system, which skips any user plugin not listed in
plugins.enabled — so every agent init logged "Context engine '<name>' not found — falling
back to built-in compressor" although the engine was installed and named in config.
Live probe on base (fake HOME, plugins/ctx_demo with register(ctx), context.engine:
ctx_demo): the warning fired on EVERY init, not only the first; adding the plugin to
plugins.enabled made it load through the general fallback. `context.engine` is the
activation signal (as memory.provider / cron.provider are for their kinds), so the engine
loader now resolves bundled then user dirs the way plugins/cron_providers does: same
`user_plugins_dir()` seam, cheap source heuristic (register_context_engine / ContextEngine),
user engines imported under a synthetic namespace, bundled wins on collision, and
discover_context_engines() lists them for `hermes plugins` / the dashboard.
Fixes#61839
credit: @giggling-ginger #61995
A general-plugin context engine is one shared instance; agent init copied it per agent
with copy.deepcopy() only. Engines that hold a SQLite connection or lock (hermes-lcm)
already expose clone_for_agent() for exactly this, but it was never called, so every
init logged "could not be safely copied … falling back to built-in compressor" and the
engine was unusable through the plugin system.
ContextEngine grows clone_for_agent() (default: deepcopy, the previous behaviour) and
_select_context_engine calls it; the failure message now names the hook to override.
Docs: context-engine-plugin.md documents the per-agent clone contract.
Test change (existing on main): tests/agent/test_context_engine.py::
test_agent_init_source_deepcopies_singleton_not_aliases was a source-reading pin on the
literal `copy.deepcopy(_candidate)` line, which this fix intentionally replaces. It is
superseded by tests/agent/test_plugin_context_engine_clone.py, which drives the real
_select_context_engine seam and asserts the invariant it guarded (child update_model()
never mutates the shared singleton) plus the new clone_for_agent() path.
Fixes#99640
credit: @stephenschoettler #62374
credit: @686f6c61 #99677
Mechanical `check_doc_links.py --fix` pass over website/docs (hand-authored
and generated pages) and the zh-Hans mirror: 1,868 route-style links
(`](/section/page#anchor)`, `](/docs/...)`) become `](../section/page.md#anchor)`.
Every target was asserted to exist on disk; anchors and query strings are
preserved; fenced code blocks and inline-code examples are untouched.
Two dead targets found by the converter were fixed by hand first:
memory-providers.md linked `/user-guide/plugins` (page is
`user-guide/features/plugins`), and the zh-Hans learning-path still linked the
removed `rl-training` page — ported the EN treatment (external Atropos link).
Docusaurus build after: EN locale 0 unresolved Markdown links, 0 broken links,
0 broken anchors.
The pooled worker closure captured the caller's live `messages` list and
compress_context explicitly supports plugin/legacy context engines that
mutate that list in place — so after a host timeout, a late engine could
rewrite the live conversation (roles, ordering, persisted content)
concurrently with the resumed turn.
The worker now deep-snapshots the transcript on the worker thread before
any engine code runs; the caller's list object is never handed to pooled
code. Results reach caller-visible state only through the returned value
of an ADMITTED commit (the host discards results on timeout/cancel), and
durable SessionDB mutation was already gated behind the commit fence.
No-op passes map the unchanged snapshot back to the caller's original
list so identity-based no-op detection and flush dedup keep working.
Document the thread-safety contract for context-engine and
memory-provider extension points (they now run on pooled threads) in the
module docstring and the context-engine plugin guide.
Regression: an in-place-mutating engine plus host timeout proves the
caller's live transcript is byte-identical WHILE the worker is still
blocked inside the engine (released only after the assertions).
PR #76354 review, blocking finding 3 / merge gate 3.
Maintainer scoping decision for the #51226 salvage: document that
select_context() is for engines that must REPLACE per-request context
(retrieval/routing) — pre_llm_call is inject-only by documented cache
design; that observation-only plugins should implement a MemoryProvider
(sync_turn) rather than a context engine, with on_turn_complete scoped
as the observation mirror for engines that already select; and that a
non-no-op select_context naturally changes the prompt-cache prefix on
turns where the selection changes — engines should return stable
selections when nothing changed.
- website guide: on_turn_complete() now carries the same best-effort coverage
caveat as the ABC docstring (fires from the finalization seam; abnormal
early-return paths bypass it) — removes the doc/code inconsistency.
- test: finalization seam emits on_turn_complete with usage=None + the
interrupted flag for an interrupted finalized turn. Docstring records that
the negative early-return-bypass half is best-effort and deferred to a
shared-seam follow-up rather than pinned via a full run_conversation harness.
- _apply_context_engine_selection: reject an empty list. all([]) is True, so
a [] returned by a failing/buggy engine previously replaced a valid request
with an empty message list the downstream sanitizers can't restore; now it
falls open to the unmodified request (honors the fail-open contract).
Thanks @johnnykor82 for catching this on #41918's review.
- test: empty list keeps the original request (fail-open regression).
- docs: document select_context()/on_turn_complete() in the public
context-engine plugin guide (were still describing only the old contract).
Follow-up to the salvaged contributor commit, closing the three gaps
flagged in the sweeper review:
1. Init ordering: assign compression.model_thresholds to a selected
plugin context engine BEFORE the initial update_model() call in
agent_init.py, so the initial model's override applies from init
(previously it only took effect after the first /model switch).
Base-class ContextEngine.update_model() now snapshots the
pre-override percent once so repeated switches fall back to the
engine's configured threshold, not a previous model's override.
2. DEFAULT_CONFIG: add compression.model_thresholds (empty map) to
hermes_cli/config.py — additive key, no _config_version bump.
3. Docs: document the key in
website/docs/developer-guide/context-compression-and-caching.md
(yaml example, parameter table, dedicated section) and update the
plugin-boundary note in context-engine-plugin.md to state the
explicit context-engine contract for model_thresholds.
Adds tests/run_agent/test_per_model_threshold_init_ordering.py:
plugin-engine AIAgent init regression (override applies at init,
empty map unchanged), DEFAULT_CONFIG key presence, floor interaction
on the model-switch path (override below the small-context floor is
raised to the floor; above the floor wins), and base-class config
snapshot across repeated switches. Also maps @bennybuoy in
contributors/emails/.
The locale switcher appeared broken because hardcoded markdown links
(`](/docs/X)`) got double-prefixed by Docusaurus to `/docs/<locale>/docs/X`
(404) in non-English locales, and the MDX hero `<a href>` on the index
page escaped locale routing entirely.
Changes:
- Rewrite 922 `](/docs/X)` -> `](/X)` across 166 docs files (strip trailing
.md too). Docusaurus prepends locale + baseUrl itself.
- docs/index.md -> index.mdx; hero "Get Started" anchor -> Docusaurus
<Link> so it stays inside the active locale.
- Drop `ko` locale entirely from docusaurus.config.ts + delete i18n/ko/
(4 stale auto-translated kanban pages, <2% coverage, misleading).
Verified `npm run build` succeeds for both en and zh-Hans; `build/zh-Hans/
index.html` has no /docs/zh-Hans/docs/... double-prefixed paths.
PR2 will translate the 335 English docs into i18n/zh-Hans/.
Broad drift audit against origin/main (b52b63396).
Reference pages (most user-visible drift):
- slash-commands: add /busy, /curator, /footer, /indicator, /redraw, /steer
that were missing; drop non-existent /terminal-setup; fix /q footnote
(resolves to /queue, not /quit); extend CLI-only list with all 24
CLI-only commands in the registry
- cli-commands: add dedicated sections for hermes curator / fallback /
hooks (new subcommands not previously documented); remove stale
hermes honcho standalone section (the plugin registers dynamically
via hermes memory); list curator/fallback/hooks in top-level table;
fix completion to include fish
- toolsets-reference: document the real 52-toolset count; split browser
vs browser-cdp; add discord / discord_admin / spotify / yuanbao;
correct hermes-cli tool count from 36 to 38; fix misleading claim
that hermes-homeassistant adds tools (it's identical to hermes-cli)
- tools-reference: bump tool count 55 -> 68; add 7 Spotify, 5 Yuanbao,
2 Discord toolsets; move browser_cdp/browser_dialog to their own
browser-cdp toolset section
- environment-variables: add 40+ user-facing HERMES_* vars that were
undocumented (--yolo, --accept-hooks, --ignore-*, inference model
override, agent/stream/checkpoint timeouts, OAuth trace, per-platform
batch tuning for Telegram/Discord/Matrix/Feishu/WeCom, cron knobs,
gateway restart/connect timeouts); dedupe the Cron Scheduler section;
replace stale QQ_SANDBOX with QQ_PORTAL_HOST
User-guide (top level):
- cli.md: compression preserves last 20 turns, not 4 (protect_last_n: 20)
- configuration.md: display.platforms is the canonical per-platform
override key; tool_progress_overrides is deprecated and auto-migrated
- profiles.md: model.default is the config key, not model.model
- sessions.md: CLI/TUI session IDs use 6-char hex, gateway uses 8
- checkpoints-and-rollback.md: destructive-command list now matches
_DESTRUCTIVE_PATTERNS (adds rmdir, cp, install, dd)
- docker.md: the container runs as non-root hermes (UID 10000) via
gosu; fix install command (uv pip); add missing --insecure on the
dashboard compose example (required for non-loopback bind)
- security.md: systemctl danger pattern also matches 'restart'
- index.md: built-in tool count 47 -> 68
- integrations/index.md: 6 STT providers, 8 memory providers
- integrations/providers.md: drop fictional dashscope/qwen aliases
Features:
- overview.md: 9 image models (not 8), 9 TTS providers (not 5),
8 memory providers (Supermemory was missing)
- tool-gateway.md: 9 image models
- tools.md: extend common-toolsets list with search / messaging /
spotify / discord / debugging / safe
- fallback-providers.md: add 6 real providers from PROVIDER_REGISTRY
(lmstudio, kimi-coding-cn, stepfun, alibaba-coding-plan,
tencent-tokenhub, azure-foundry)
- plugins.md: Available Hooks table now includes on_session_finalize,
on_session_reset, subagent_stop
- built-in-plugins.md: add the 7 bundled plugins the page didn't
mention (spotify, google_meet, three image_gen providers, two
dashboard examples)
- web-dashboard.md: add --insecure and --tui flags
- cron.md: hermes cron create takes positional schedule/prompt, not
flags
Messaging:
- telegram.md: TELEGRAM_WEBHOOK_SECRET is now REQUIRED when
TELEGRAM_WEBHOOK_URL is set (gateway refuses to start without it
per GHSA-3vpc-7q5r-276h). Biggest user-visible drift in the batch.
- discord.md: HERMES_DISCORD_TEXT_BATCH_SPLIT_DELAY_SECONDS default
is 2.0, not 0.1
- dingtalk.md: document DINGTALK_REQUIRE_MENTION /
FREE_RESPONSE_CHATS / MENTION_PATTERNS / HOME_CHANNEL /
ALLOW_ALL_USERS that the adapter supports
- bluebubbles.md: drop fictional BLUEBUBBLES_SEND_READ_RECEIPTS env
var; the setting lives in platforms.bluebubbles.extra only
- qqbot.md: drop dead QQ_SANDBOX; add real QQ_PORTAL_HOST and
QQ_GROUP_ALLOWED_USERS
- wecom-callback.md: replace 'hermes gateway start' (service-only)
with 'hermes gateway' for first-time setup
Developer-guide:
- architecture.md: refresh tool/toolset counts (61/52), terminal
backend count (7), line counts for run_agent.py (~13.7k), cli.py
(~11.5k), main.py (~10.4k), setup.py (~3.5k), gateway/run.py
(~12.2k), mcp_tool.py (~3.1k); add yuanbao adapter, bump platform
adapter count 18 -> 20
- agent-loop.md: run_agent.py line count 10.7k -> 13.7k
- tools-runtime.md: add vercel_sandbox backend
- adding-tools.md: remove stale 'Discovery import added to
model_tools.py' checklist item (registry auto-discovery)
- adding-platform-adapters.md: mark send_typing / get_chat_info as
concrete base methods; only connect/disconnect/send are abstract
- acp-internals.md: ACP sessions now persist to SessionDB
(~/.hermes/state.db); acp.run_agent call uses
use_unstable_protocol=True
- cron-internals.md: gateway runs scheduler in a dedicated background
thread via _start_cron_ticker, not on a maintenance cycle; locking
is cross-process via fcntl.flock (Unix) / msvcrt.locking (Windows)
- gateway-internals.md: gateway/run.py ~12k lines
- provider-runtime.md: cron DOES support fallback (run_job reads
fallback_providers from config)
- session-storage.md: SCHEMA_VERSION = 11 (not 9); add migrations
10 and 11 (trigram FTS, inline-mode FTS5 re-index); add
api_call_count column to Sessions DDL; document messages_fts_trigram
and state_meta in the architecture tree
- context-compression-and-caching.md: remove the obsolete 'context
pressure warnings' section (warnings were removed for causing
models to give up early)
- context-engine-plugin.md: compress() signature now includes
focus_topic param
- extending-the-cli.md: _build_tui_layout_children signature now
includes model_picker_widget; add to default layout
Also fixed three pre-existing broken links/anchors the build warned
about (docker.md -> api-server.md, yuanbao.md -> cron-jobs.md and
tips#background-tasks, nix-setup.md -> #container-aware-cli).
Regenerated per-skill pages via website/scripts/generate-skill-docs.py
so catalog tables and sidebar are consistent with current SKILL.md
frontmatter.
docusaurus build: clean, no broken links or anchors.
