The quiet sibling of `tour`, in the same `desktop_ui` toolset and reading the
same `tour(action='targets')` discovery call: one bubble with an arrow, for a
sentence that would be clearer with a finger on the thing it's about. Dimming
the whole app to say "the model name is a button" is the wrong weight.
Fire-and-forget rather than a round-trip, because a tip is not a question and
blocking the turn on one would stall the reply it belongs to. The renderer
enforces the user's opt-out itself, so a stale config read can never put a
bubble on a screen that asked for none.
The in-app browser was a one-way mirror. open_preview put a page in the pane
and read_preview read its text back, but nothing could touch it. A click meant
falling back to the browser_* tools, which drive a separate Chromium the user
cannot see — so "log into this and pull my invoices" happened in a different
browser from the one on screen, with none of the sessions the user is already
signed into.
Four pieces, and they only make sense together:
· an in-page engine that inventories what is interactable and performs the
verb, injected as source because it has to run inside the guest page;
· the preview.act.request bridge from the gateway into the pane;
· drive_preview, for acting: elements, click, type, scroll, press, and the
pane's own back/forward/reload;
· annotate_preview, for marking without acting.
Those last two started as one tool doing two unrelated jobs. Leaving a mark is
not an action — it outlives the turn that drew it — so it gets its own verb,
and the interaction verb gets a name that says what it does.
Gating is the existing surface rule: desktop_ui folds in on session
source: 'desktop', and the bridge refuses to act for a background session, so a
turn running behind the user's back cannot reach into the page they are working
in.
Two details worth a reviewer's attention. Typing assigns through the
prototype's value setter, because React shadows value with its own accessor and
ignores an input event whose value it believes it already wrote — a plain
el.value = … types into a field that snaps back on the next render. And
clicking replays the pointer/mouse pair before activation, because frameworks
bind to mousedown as often as to click.
open_preview and read_preview could drive the page, but nothing could close
the pane. Same desktop_ui / session-source gate as the rest of the GUI tools.
One generic tool in the desktop_ui toolset: discover what is on screen,
highlight an element with narration, or hand the user a paged tour. No tour
content lives in the code — the agent authors each one live, which is what
makes 'how does this work?' answerable as a walkthrough instead of a wall
of text.
Rides the existing blocking-prompt bridge (tour.request/.respond) like
read_preview, so it works on every connection topology.
- The readonly-loader completer test now stubs
get_portable_mcp_server_names_nowait — real plugin discovery runs
load_config() during one-time process init, which is not the
per-keystroke read the test guards against.
- Cap _resolve_toolset_memo at 256 entries: generation-keyed entries
from stale generations are never hit again, so clear on overflow to
keep long sessions bounded.
resolve_toolset() recursively walks toolset includes and, with
include_registry=True, merges registry-registered tools on every call —
each external call re-runs the includes walk and takes a fresh registry
snapshot under the registry lock. It is called dozens of times per
_get_platform_tools() (every /tools completion keystroke, per picker
render) and at 19 call sites across the CLI/gateway.
Memoise the external-entry result keyed on (name, include_registry,
registry id, registry generation). tools.registry exposes a monotonic
_generation counter bumped on every register/deregister/alias/MCP
refresh (its docstring explicitly invites generation-keyed memoisation),
so a cache entry is valid until the registry changes. External callers
never pass visited, so the memo engages exactly at the public entry
and the internal cycle-detection recursion is untouched.
Measured: _get_platform_tools drops 165us -> 59us per call (3x) with the
xAI credential fix simulated; /tools completion ~2ms -> ~77us/keystroke
combined. Regression tests: repeat resolution is a memo hit (get_toolset
called once), a generation bump forces a fresh resolve, and the memoised
result is identical to a fresh resolution.
(cherry picked from commit 3d36ecb273f6652d00556a7b9d846c08047801aa)
New desktop_ui tool: the agent proposes an MCP server (install/enable/
authorize + a one-line reason) and blocks on mcp.setup.request until the
renderer's consent card answers mcp.setup.respond with the outcome
(installed/enabled/authorized/declined/unanswered/error). Same lifecycle
as clarify: 10-min timeout, allow_expired late answers, tool lifecycle
events forced on so the card mounts even with tool progress off. Desktop
prompt hint steers the model to the tool instead of hand-editing config;
every other surface keeps the schema out and is pointed at hermes mcp
install.
Add a non-terminal "review" status so a worker that finished implementation
can hand off for human review without abusing kanban_block. The old
kanban_block(reason="review-required: ...") convention routed the handoff
through the unblock-loop breaker, so a normal review -> changes -> review
cycle was falsely escalated to triage.
- kanban_db: request_review (running/ready -> review, non-block, emits
review_requested), reopen_review_task (review -> ready/todo, review_reopened),
complete_task accepts review -> done, and a review_dispatch gate (default off,
shared by the dispatcher loop and the gateway health probe).
- kanban_request_review worker tool + `request-review` / `reopen-review` CLI
verbs; tool wired through toolsets, EXPOSED_TOOLS, _POLISHED_TOOLS.
- Gateway notifier wakes the origin subscriber on review_requested and
block_loop_detected; the subscription survives until done/archived, so every
review cycle re-notifies.
- Dashboard PATCH + bulk route the review transitions (request_review /
reopen_review_task) and render the review column.
- goals.py goal-loop and KANBAN_GUIDANCE recognize review as a terminator.
- Docs (reference tables, user guide, AGENTS.md, zh-Hans mirrors) + tests.
needs_input / failed are unchanged: they still route through kanban_block,
still count toward block_recurrences, and still escalate to triage.
Desktop-gated (desktop_ui toolset) metadata-only window awareness: the agent
can ask which application window sits directly behind the Hermes window
(app, title, bounds — never pixels). Rides the same blocking bridge as
read_terminal: the gateway emits window.read.request and the renderer
answers window.read.respond.
The pane, in-app browser, and reaction tools were gated on HERMES_DESKTOP=1 —
an env var set only on backends Electron spawns itself (local and SSH). A
desktop client connected to a plain URL gateway or Hermes Cloud lost all six:
they were stripped from the schema before the model saw them, on the same
backend whose platform hint was telling it "you are chatting inside the Hermes
desktop app". open_preview, read_preview, read_terminal, close_terminal,
focus_pane, and react_to_message were all silently absent.
The client is not the host. Capability now resolves from the session's own
source, which session.create already carries:
- The six tools move into a `desktop_ui` toolset, off _HERMES_CORE_TOOLS so no
other platform pays their schema.
- _gui_surface_toolsets(platform) folds `desktop_ui` (and the existing
`project` tools) into the GUI gateway's resolution when the session's
platform is the desktop app — the same answer on every topology.
- check_fn drops the env probe. It kept the one thing that is genuinely a
per-process/user fact: react_to_message's display.message_reactions opt-in,
which the desktop mirrors onto whichever gateway it is connected to.
react_to_message was doubly broken: it read that toggle behind the env gate, so
even a local-backend user's Settings toggle could not reach a remote session.
The embedded terminal pane keeps working correctly the other way round: it runs
`hermes --tui` against a desktop-spawned backend, and a tui-sourced session
gets no GUI tools even though HERMES_DESKTOP=1 is set on that process.
* feat(agent): read_preview — the desktop-gated tool that reads the in-app browser
The agent could open the preview pane (open_preview) and read the embedded
terminal (read_terminal), but the browser it had just opened was a black box —
'what does this page say?' had no answer. read_preview mirrors read_terminal
end to end: HERMES_DESKTOP-gated via check_fn (zero schema footprint outside
the GUI), dispatched through the same agent callback pattern, windowed with
start/count so a long page pages instead of flooding context.
* feat(gateway): preview.read blocking bridge
Same lifecycle as terminal.read: the tool blocks on preview.read.request, the
renderer answers preview.read.respond (allow_expired — a slow page extraction
losing the 45s race must not surface a raw 4009), and a timeout emits
preview.read.expire so late answers resolve quietly.
* feat(desktop): the renderer serializes the active preview tab for the agent
preview-reader.ts is the preview analog of the terminal's buffer registry: the
URL pane registers a page reader (webview executeJavaScript → title + visible
innerText) keyed by tab id; readActivePreview resolves the ACTIVE tab, windows
the text (24k cap per read), and answers file/artifact tabs with identity plus
a note pointing at the tool that reads that content directly. The gateway
event handler answers preview.read.request beside terminal.read.request.
Reactions live in the existing messages.display_metadata JSON column (no new
table), with iOS Tapback semantics enforced DB-side: one reaction per author
per message, re-tap retracts, different emoji replaces. The desktop catches up
to the reaction contract five platform adapters already ship.
- SessionDB: set/get_message_reaction, latest_message_row_id (role + offset +
require_text so invisible tool-call-only rows are never targeted),
take_unseen_reactions (announce-exactly-once), get_message_role
- message.react RPC: accepts row_id or newest_role for live messages that
haven't learned their durable id yet
- react_to_message tool: desktop-gated (check_fn), defaults to the user's
latest visible message, messages_back for retroactive reactions
- Model context rides run_message only (beside the speech-interrupted note):
the persisted prompt stays clean, so no [The user reacted …] scaffolding in
transcripts, and no cached prefix ever changes
- Resume projection forwards row_id + reactions; _row_id is stripped from
outgoing API copies next to display_metadata
Make the x_search / xurl boundary explicit in the skill, feature docs,
toolset metadata, setup note, and reference pages, while keeping the
model-facing x_search schema generic (no static xurl name).
Regression tests assert behavioral routing invariants rather than frozen
prose snapshots. Drop the stale CI-only plugin/hangup hunks already on
main so this rebases cleanly.
Extract the open_preview emitter into a shared tools/desktop_ui bridge
(one gateway-injected sink, routed by HERMES_UI_SESSION_ID) and add a
second desktop-gated tool on top of it:
- focus_pane(chat|files|terminal|review|sessions) -> pane.reveal event.
The desktop runs each pane's own reveal path (revealDesktopPane table)
and only acts on the active window -- a background turn never moves the
user's focus (desktop AGENTS.md: offer, don't hijack).
open_preview now emits through the same bridge. Both tools are check_fn
on HERMES_DESKTOP (zero footprint elsewhere), sitting beside
read_terminal/close_terminal in _HERMES_CORE_TOOLS.
Deliberately not adding run_slash: letting the agent fire slash commands
mid-turn (/model, /new, /clear) fights prompt-cache + conversation
invariants.
Add a desktop-gated open_preview tool so 'open cnn.com in the preview
pane' works. The tool (check_fn on HERMES_DESKTOP, zero footprint
elsewhere) emits a preview.open event through a gateway-injected emitter,
mirroring the close_terminal -> terminal.close bridge. The desktop
handles it in usePreviewRouting, normalizing the target and opening the
pane for the active session only -- a background turn never hijacks it.
Bare domains and localhost are coaxed into fetchable URLs (www.cnn.com ->
https://, localhost:3000 -> http://); file paths and schemes pass through
to the renderer's normalizer.
The kanban board has had full attachment storage and a dashboard HTTP
API (upload/list/download/delete) since #35338, but there was no agent
toolset tool and no `hermes kanban` CLI verb for attachments. Agents and
scripts that don't go through the dashboard server (or can't touch the DB
directly) had no way to create or read real attachments — only links in
comments.
Close that gap by mirroring the existing comment surface:
- `kanban_db.store_attachment_bytes()` — one shared write path (validate
name, enforce the 25 MB cap, write the blob under the per-task dir with
collision-free naming, insert the metadata row, clean up an orphan blob
if the insert fails). `_MAX_ATTACHMENT_BYTES`, `_safe_attachment_name`,
and a new `_collision_free_path` move here so the dashboard, the tool,
and the CLI all share one implementation and can't drift.
- Tools (`tools/kanban_tools.py`): `kanban_attach` (inline base64),
`kanban_attach_url` (server-side http/https fetch with the same cap),
`kanban_attachments` (list). Write tools respect worker task-ownership;
list is read-only. Registered in the `kanban` toolset.
- CLI (`hermes_cli/kanban.py`): `attach <id> <path>`, `attachments <id>`,
`attach-rm <attachment_id>`.
- Dashboard `upload_task_attachment` now imports the shared helpers and
uses `_collision_free_path` — behavior identical (still streams to disk
with the cap, still 413 on overflow).
- Docs (AGENTS.md, kanban-worker skill) and toolset membership updated.
Tests: tool round-trip + oversize + bad base64 + ownership; attach_url
against a local HTTP fixture incl. oversize-mid-stream and non-http
scheme rejection; CLI attach/attachments/attach-rm; shared-helper unit
tests; dashboard parity preserved.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Follow-up on the #56480 salvage: the include_registry=False docstring said
None is returned only for registry/MCP-only toolsets; it also applies to
registry-derived aliases, which have no static TOOLSETS counterpart.
Adds include_registry=True kwarg to resolve_toolset/get_toolset. When False,
returns only the static TOOLSETS view with no registry-merged tools — the
composite-authored membership platform reverse-mapping must compare against.
Default True preserves all existing behavior; this is the enabling half of
the api_server toolset-drop fix (#49622).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add durable public-URL output and URL-based chaining to xAI Grok Imagine:
- Store generated media on files-cdn with permanent public HTTPS URLs
(public_url: true, no expiry by default).
- Chain by URL: generate -> edit -> extend each take a prior result's
public HTTPS URL (or a data URI / local file for inputs).
- Add provider-specific xai_video_edit and xai_video_extend tools.
- Image generation: public-URL/storage output, multi-reference edits,
and ~/ local-path support for image edits.
Credentials use xAI Grok device-code OAuth (separate PR).
Make the read-only agent terminal mirrors stream in real time and give
the agent a desktop-only way to dismiss its own tabs.
- Stream background output live: the local reader used a blocking
read(4096) that buffered small periodic output until EOF, so agent
tabs only "filled in" at process exit. Switch to buffer.read1(4096)
(decoded) for incremental chunks.
- Route agent.terminal.output / terminal.close to the window that owns
the process (its gateway session) instead of an empty session id, so
events actually reach the desktop renderer.
- Add close_terminal: a HERMES_DESKTOP-gated tool (sibling of
read_terminal) that drops a process's read-only tab WITHOUT killing it
via process_registry.on_close; output keeps buffering and the user can
reopen from the status stack.
- ⌘W now closes a focused agent tab: mark the agent instance
data-terminal and focus it on activation so isFocusWithin routes there.
- ensureTerminal() no longer spawns an extra user shell when a tab
already exists (e.g. opening a background task from the status stack).
* feat(moa): expose MoA presets as selectable virtual models
Reconstructed onto current main (PR #46081's base had diverged with no common
ancestor, marking the PR dirty so CI never dispatched). MoA is now a virtual
provider: each named preset is a selectable model under provider 'moa', and the
preset's aggregator is the acting model that answers and calls tools.
Reference models fan out in parallel via a bounded ThreadPoolExecutor (the same
batch pattern delegate_task uses) — all references dispatched at once, collected
when every one finishes, then handed to the aggregator. Output order is
preserved, failures and the MoA-recursion guard stay isolated per reference.
- Removed the old mixture_of_agents model tool and moa toolset.
- Added moa as a virtual provider in the provider/model inventory.
- /moa is shortcut behavior over model selection (default preset / named preset
/ one-shot prompt).
- Dashboard + Desktop manage named presets; presets appear in model pickers.
- Parallel reference fan-out in agent/moa_loop.py with regression test.
* fix(moa): thread moa_config through _run_agent to _run_agent_inner
The reconstructed gateway MoA wiring declared moa_config on _run_agent (the
profile-scoping wrapper) and used it inside _run_agent_inner, but the wrapper
never forwarded it — _run_agent_inner had no such parameter, so the runtime hit
NameError: name 'moa_config' is not defined on the compression-failure session
sync path. Add moa_config to _run_agent_inner's signature and forward it from
both wrapper call sites (multiplex and non-multiplex). Caught by
tests/gateway/test_compression_failure_session_sync.py on CI shard test(4).
* fix(moa): classify moa as a virtual provider in the catalog
The moa virtual provider has no PROVIDER_REGISTRY/ProviderProfile entry, so
provider_catalog() fell through to the default auth_type="api_key" with no
env vars — tripping two catalog invariants:
- test_provider_catalog: api_key providers must expose a credential env var
- test_provider_parity: every hermes-model provider must be desktop-configurable
moa already declares auth_type="virtual" in HERMES_OVERLAYS; consult that
overlay as an auth_type fallback so the catalog reports moa as virtual (no real
credential, no network endpoint). Exempt virtual providers from the desktop
parity union check the same way 'custom' is exempt — derived from the catalog,
not a hardcoded slug, so future virtual providers are covered too.
The runtime gate (check_computer_use_requirements) and the hermes tools
platform_gate both enable linux alongside darwin/win32, but several
docstrings/comments still described Linux as "alpha, gated off until it
flips upstream" — contradicting the code that ships it. Bring the prose in
line with the gate that's actually live:
- tool.py / cua_backend.py module docstrings: Linux is enabled (X11 today,
Wayland via XWayland), not gated off.
- toolsets.py description and hermes tools display name: (macOS/Windows) ->
(macOS/Windows/Linux).
No behavior change — the gate already allowed all three platforms.
Make the computer_use toolset platform-agnostic by driving cua-driver on
macOS, Windows, and Linux. Consumes the 8 cua-driver decoupling surfaces
(capability discovery, structuredContent AX tree, opaque element_token,
click button enum, explicit mimeType, machine-readable manifest,
structured list_windows, structured health_report), each degrading
gracefully on older drivers.
Adds `hermes computer-use doctor` (drives cua-driver health_report with a
per-OS check matrix and an exit 0/1/2 ok/degraded/blocked contract), full
typed wrappers for the previously-uncovered cua-driver tools plus a generic
call_tool escape hatch, per-session agent-cursor lifecycle, platform-aware
system-prompt guidance (host-deterministic, cache-safe), and honors
HERMES_CUA_DRIVER_CMD end-to-end.
Replaces the macOS-only skills/apple/macos-computer-use skill with a
cross-platform skills/computer-use skill, and refreshes the EN + zh-Hans
docs.
Supersedes #44221 (Windows-enablement salvage of #30660).
Co-authored-by: Teknium <127238744+teknium1@users.noreply.github.com>
When a platform-bundle name (e.g. `hermes-yuanbao`, or any `hermes-*`) lands
in `agent.disabled_toolsets`, the shared tool-assembly path
(`model_tools._compute_tool_definitions`, used by the gateway, cron, AND the
CLI) subtracted the WHOLE bundle from the enabled set. Because every platform
bundle is defined as `_HERMES_CORE_TOOLS + [platform extras]`, and core tools
are shared by every other enabled toolset, the subtraction emptied the tool
list entirely — the model received `tools: []` / `tool_choice: null` and
started replying "I cannot execute shell commands" with no error, no warning,
and `hermes tools list` / `hermes doctor` still green. For unattended cron
jobs this fails silently for days. (#33924)
(The original report framed this as gateway-only; it actually affects every
caller of `_compute_tool_definitions`, including the CLI — the reporter's
follow-up confirms this. Fixing the shared chokepoint covers all paths.)
Fix: for a `hermes-*` bundle in `disabled_toolsets`, subtract only its
*non-core delta* (its platform-specific tools plus those of any `includes`),
leaving `_HERMES_CORE_TOOLS` intact. Disabling a bundle now removes its
platform tools (e.g. the `yb_*` tools for `hermes-yuanbao`) while terminal,
read_file, web, etc. survive. A `logger.warning` notes that core tools are
preserved and that bundle names usually belong in `toolsets:`, not
`disabled_toolsets` — informative, not destructive (the subtraction still
behaves sensibly).
Salvaged from #33941 by @liuhao1024 (authorship preserved). Extracted the
inline bundle-resolution into a module-level `_bundle_non_core_tools` helper
(was re-importing `toolsets` inside the disable loop), and added the
informative warning folding in the UX intent of #34073 (@ousiaresearch)
without its hard "ignore the bundle name" behavior — which would have undone
this fix's sensible-subtraction.
Verified empirically: disabling `hermes-yuanbao` from a gateway-style enabled
set keeps all core tools (18→18) and would remove only the 5 `yb_*` tools;
disabling `hermes-discord` removes only `discord`/`discord_admin`.
Fixes#33924
Co-authored-by: liuhao1024 <sunsky.lau@gmail.com>
* feat(mcp): raise default tool-call timeout 120s -> 300s
Port from openai/codex#28234. Long-running MCP tools (web fetches,
sandboxed builds, deep-research servers) routinely exceed 120s, causing
spurious timeout failures. Codex bumped its default MCP tool timeout from
120 to 300 for the same reason.
- _DEFAULT_TOOL_TIMEOUT 120 -> 300 in tools/mcp_tool.py (per-server
'timeout' config override unchanged)
- update test_default_timeout assertion
- document the default in mcp-config-reference.md
* refactor: remove agent-callable send_message tool
The agent should not decide on its own to fire off cross-platform
messages or reactions. Outbound platform messaging is handled outside
the agent loop — cron delivery, the gateway kanban notifier
(dashboard-toggled), and the `hermes send` CLI.
Removes the model-tool registration only; the send engine in
send_message_tool.py (_send_to_platform, _send_via_adapter,
_parse_target_ref, per-platform _send_* helpers) is kept intact for
those non-agent callers. Drops the now-empty 'messaging' toolset and
its `hermes tools` toggle. Yuanbao DM guidance now points at the
native yb_send_dm tool.
* feat(agent): coding-context posture with per-model edit-format tuning
Hermes detects when it's running in a coding context — an interactive
surface (CLI, TUI, ACP, desktop) sitting in a code workspace (git repo or
recognised project root) — and shifts into a coding posture. Outside that
(chat platforms, non-workspaces) nothing changes.
The posture is modelled as a frozen RuntimeMode selected from a small
ContextProfile registry (coding/general). A profile is data: the toolset to
collapse to, the operating brief to inject, and seams for model routing and
memory. Every domain reads the same resolved object instead of re-probing
git/config on its own:
- System prompt — RuntimeMode.system_blocks(): an operating brief (gather
context before editing, edit through tools not chat, verify with terminal,
cap retry loops) plus a live git/workspace snapshot, built once and baked
into the stable prompt tier so per-conversation caching is preserved.
- Per-model edit-format tuning — the brief nudges each model family toward
the patch mode it handles best: OpenAI/Codex toward mode='patch' (V4A
multi-file diffs), Anthropic toward mode='replace' (string replacement).
The model id rides on RuntimeMode; unknown families keep neutral wording.
- Skill index — non-coding skill categories are pruned from the prompt's
skill index (discovery-only; skills_list/skill_view still reach the full
catalog, with a disclosure note).
- Toolset — only under the opt-in 'focus' mode does the posture collapse to
the coding toolset + enabled MCP servers; the default posture is
prompt-only and never overrides configured toolsets.
Activation via agent.coding_context: auto (default), focus, on, off.
Subagents inherit the posture for free via toolset inheritance + the shared
prompt builder. Detection is not memoized so a long-lived gateway/TUI
process can't pin a stale posture across working directories.
* feat(agent): cover new-file authoring in the coding edit-format nudge
The per-model edit-format guidance only addressed editing existing code
(patch mode='patch' vs 'replace'), but authoring a brand-new file —
write_file, not patch — is a large fraction of real coding work and the
nudge was silent on it. Surfaced when building a single-file artifact where
the dominant operation was write_file and the steering offered no guidance.
Both family lines now lead with "author new files with write_file; for
edits to existing code prefer ...". Tests assert write_file appears in each
family's brief; unknown families still get neutral wording.
* docs(agent): correct memoization docstring + clarify TUI config-load asymmetry
* feat(agent): sharpen the coding posture — verify-loop facts, wider edit steering, $HOME guard
Tuning pass on the coding posture from dogfooding it as a harness:
- Workspace snapshot now hands the model its verify loop up front:
detected manifests + package manager (lockfile sniff), the exact
verify commands (package.json scripts, Makefile targets,
scripts/run_tests.sh, pytest config), and which context files
(AGENTS.md / CLAUDE.md / .cursorrules) exist at the root. Marker-only
(non-git) projects get the snapshot too instead of nothing. The
"verify before claiming done" brief line was the highest-value piece
in evals — this turns it from advice into an executable loop instead
of making the model rediscover the test command every session. Still
stat-cheap, size-guarded reads, built once at prompt time.
- Edit-format steering covers the families Hermes actually serves:
Gemini and open-weight coding models (DeepSeek, Qwen, Kimi, GLM,
Grok, Hermes, Llama, Mistral, Devstral, MiniMax) steer to
mode='replace' — their RL scaffolds use str_replace-style editors.
Previously only GPT/Codex and Claude families got steering; the
models Hermes users disproportionately run all fell to neutral.
- Operating brief gains four behaviors elite harnesses encode: batch
independent reads/searches in one turn; fix root causes and the bug
class (sibling call paths), not the reported site; no drive-by
refactors/renames/reformatting; never read, print, or commit secrets.
Plus a patch-failure escalation ladder: after the same region fails
twice, rewrite the enclosing function/file with write_file instead of
a third patch attempt.
- $HOME dotfiles guard: a git repo rooted exactly at the home directory
(or a marker sitting in it, e.g. a global ~/AGENTS.md) is user config,
not a code workspace — without the guard, every session anywhere under
a dotfiles-managed home silently flipped to the coding posture. Real
projects under such a home still detect via their own markers/repos;
'on' mode bypasses the guard.
* refactor(desktop): dock terminal under chat and simplify file rail
Keep the right rail focused on file browsing while moving the persistent terminal into the chat column bottom slot, and make terminal colors follow the active light/dark mode instead of a fixed Solarized palette.
* fix(desktop): make the terminal a resizable, themed side pane
- Move the terminal into a resizable pane (viewport-% widths) that shares
<main>'s stacking context, so its drag handle no longer sits under the
fixed terminal overlay; works on either rail side.
- Restore +x on node-pty's spawn-helper before the first spawn to fix
"posix_spawnp failed" on macOS prebuilds (real cause; drop the redundant
shell-candidate retry loop).
- Gate terminal open/fit/start on document.fonts.ready and strip leading
blank rows (re-armed before the resize Ctrl-L redraw) so the prompt sits
flush at the top with no starship add_newline gap.
- Inherit the app editor-surface color as the terminal background.
- Bind Ctrl+` (⌃` on macOS) to toggle the terminal; add a palette entry.
* feat(desktop): show platform hotkey hints in the command palette
- Render each palette item's live binding as a <KbdGroup> hint via a new
comboTokens() helper (mac shows ⌘/⌃/⌥/⇧, every other platform shows
Ctrl/Alt/Shift — never a ⌘ on PC).
- Default the terminal toggle to ⌘` / Ctrl+` (the ~ key) on both platforms.
- Drop the hardcoded (⌘⏎) baked into the composer steer tooltip; render it
platform-aware with formatCombo instead.
* fix(desktop): drop the active check on the command-palette terminal item
* fix(desktop): remove active/check states from the command palette
* fix(desktop): allow ⌥/Shift-drag selection over mouse-mode TUIs
Full-screen apps (hermes --tui, vim) enable mouse reporting, so a plain
drag can't select text and ⌘/Ctrl+L (add-selection-to-chat) had nothing
to send. Enable macOptionClickForcesSelection so ⌥-drag on macOS (Shift
elsewhere) forces a native selection over mouse-mode apps.
* feat(desktop): tell the in-pane agent it's embedded in the GUI
Set HERMES_DESKTOP_TERMINAL=1 on the terminal pane's shell env and surface
it in build_environment_hints, so a hermes/--tui launched inside the pane
knows it's next to the GUI chat and that ⌥/Shift-drag + ⌘/Ctrl+L sends a
selection to the composer. Distinct from HERMES_DESKTOP (agent backend).
* refactor(desktop): drop the redundant Ctrl+` terminal-toggle fallback
The toggle now ships as mod+` on both platforms, so the standard combo
index handles it — the bespoke fallback (and its stale 'old default'
comment) is dead weight.
* fix(desktop): read live terminal selection for ⌘/Ctrl+L
A redraw-heavy TUI (spinners/clocks) outruns onSelectionChange, leaving the
React selection state empty so the state-gated shortcut listener never
attached and ⌘L no-op'd. Always listen and read xterm's live selection (with
a native fallback) at press time; only swallow the key when there's text to
send. Drops the now-redundant custom key handler.
* feat(desktop): make any agent aware it's in the Hermes desktop GUI
Generalize the runtime-surface hint: fire for HERMES_DESKTOP (the backend
powering the GUI chat) as well as HERMES_DESKTOP_TERMINAL (a hermes in the
embedded terminal pane), so it's about being inside the desktop GUI, not
about being a TUI. The terminal-pane selection note stays pane-specific.
* feat(desktop): give the GUI agent a read_terminal tool
The in-app terminal buffer lives in the renderer (xterm), so expose it to the
chat agent over the same blocking bridge clarify uses: read_terminal emits
terminal.read.request, the renderer serializes the buffer (visible screen by
default, or a start_line/count range against total_lines) and answers
terminal.read.respond. Gated to the GUI via HERMES_DESKTOP.
Also restores the flipped-layout titlebar inset fix (app-shell +
desktop-controller) for terminal/preview rails at the window's left edge.
* chore(desktop): trim read_terminal comments
* feat(desktop): add a terminal toggle to the statusbar
The file rail lost its terminal icon, leaving ⌘` and the command palette
as the only ways in. Add a one-click toggle to the statusbar's left
cluster, mirroring the command-center item: it reads $terminalTakeover so
it lights up while the pane is open and stays in sync with the hotkey, and
is gated to chat view (the only place the pane can show).
* fix(desktop): relabel the terminal header button to what it does
The in-pane button claimed a focus/split fullscreen toggle ("Focus
terminal view" / "Return to split view", screen-full/normal icons), but
the terminal is just a resizable side pane — there's no fullscreen. The
button only mounts while the pane is open, so the focus branch was dead
and clicking it merely closed the terminal. Relabel to "Hide terminal"
with a close icon, drop the dead conditional and the now-unused takeover
read.
* fix(desktop): move the terminal toggle next to the version item
Relocate it from the left cluster to the right of the statusbar, just
left of the client version item.
* feat(desktop): default the terminal to PowerShell on Windows
Prefer pwsh (7+) then Windows PowerShell 5.1 over cmd.exe, falling back to
comspec only when neither is present. -NoLogo drops the startup banner so
the prompt sits flush like the POSIX shells.
* feat(desktop): show a persistent divider on the terminal pane
The resize sash only painted on hover, so the terminal/chat boundary was
invisible at rest. Add an opt-in `divider` prop to Pane that paints a thin
resting hairline on the resize edge (side-aware, so it tracks the rail when
the layout flips) and enable it on the terminal pane.
* refactor(desktop): resolve the terminal shell instead of hardcoding it
Make shell selection a real resolver: an explicit override wins
(HERMES_DESKTOP_SHELL on both platforms, $SHELL on POSIX), otherwise
auto-detect the best installed shell — pwsh > Windows PowerShell 5.1 > cmd
on Windows, zsh > bash > sh on POSIX. A shared shellSpecFor() picks the
interactive flags by family, so an overridden bash/pwsh/cmd all launch
correctly.
* fix(desktop): repaint the terminal on light/dark switch
Setting term.options.theme updated colors for the DOM renderer but not the
WebGL one, which caches glyph colors in a texture atlas — so already-drawn
cells kept their old palette after a mode switch. Hold the WebglAddon in a
ref and clear its atlas when the theme changes.
* fix(desktop): match the terminal palette to VS Code Light+/Dark+
Adopt VS Code's exact default ANSI palette (the terminalColorRegistry
defaults), enable minimumContrastRatio: 4.5 so foregrounds are clamped
against the background the way the integrated terminal does, and key the
light/dark choice off renderedMode (the painted surface) instead of
resolvedMode so it can't invert. The canvas + inset paint the live skin
surface (--ui-editor-surface-background) so the terminal blends with the
app and follows light/dark, while the contrast clamp keeps colors crisp.
* fix(desktop): tighten command palette search to substring matching
cmdk's default fuzzy scorer matched anything with the query letters
scattered across an item, so e.g. "color" never narrowed to color
entries. Add a substring filter: every typed word must literally appear
in an item's value/keywords, keeping results tight and predictable.
* fix(desktop): blend the terminal header into the skin surface
The persistent-terminal overlay painted the static palette background
(#1e1e1e/#ffffff), so the transparent header strip revealed a near-black
slab above the surface-colored body. Paint the overlay with the live
--ui-editor-surface-background so header and body read as one pane.
* fix(desktop): re-resolve the terminal surface on skin switch
The canvas surface only re-resolved on light/dark change, so switching
skins at the same mode left the WebGL canvas painted with the old tint
until reload. Key the resolve off themeName too. Also trim the palette
comments.
* chore(desktop): drop redundant terminal theming header comment
ntfy now ships as a self-contained plugin under plugins/platforms/ntfy/
instead of editing 8 core files (gateway/config.py Platform enum,
gateway/run.py factory + auth maps, cron/scheduler.py, toolsets.py,
hermes_cli/status.py, agent/prompt_builder.py, gateway/channel_directory.py,
tools/send_message_tool.py).
All routing goes through gateway/platform_registry via register_platform():
- adapter_factory, check_fn, validate_config, is_connected
- env_enablement_fn seeds PlatformConfig.extra from NTFY_* env vars so
gateway status reflects env-only setups without instantiating httpx
- standalone_sender_fn handles deliver=ntfy cron jobs when cron runs
out-of-process from the gateway
- allowed_users_env / allow_all_env hook into _is_user_authorized
- cron_deliver_env_var=NTFY_HOME_CHANNEL for cron home routing
- platform_hint surfaces in the system prompt
- pii_safe=True (topic names are the only identifier; no PII to redact)
Tests moved to tests/gateway/test_ntfy_plugin.py using _plugin_adapter_loader
so the module lives under plugin_adapter_ntfy in sys.modules and cannot
collide with sibling plugin-adapter tests on the same xdist worker. The
core-file grep tests (Platform.NTFY in source, hermes-ntfy in toolsets,
etc.) are replaced with plugin-shape tests covering register() metadata,
env_enablement_fn output, and standalone_sender_fn behavior.
68 tests pass under scripts/run_tests.sh.
* feat(x_search): gated X (Twitter) search tool with OAuth-or-API-key auth
Salvages tools/x_search_tool.py from the closed PR #10786 (originally by
@Jaaneek) and reworks its credential resolution so the tool registers
when EITHER xAI credential path is available:
* XAI_API_KEY (paid xAI API key) is set in ~/.hermes/.env or the env, OR
* The user is signed in via xAI Grok OAuth — SuperGrok subscription —
i.e. hermes auth add xai-oauth has been run
Both paths route through xAI's built-in x_search Responses tool at
https://api.x.ai/v1/responses. When both credentials exist OAuth wins,
matching tools/xai_http.py's existing preference order (uses SuperGrok
quota instead of paid API spend).
The check_fn calls resolve_xai_http_credentials() which auto-refreshes
the OAuth access token if it's within the refresh skew window, so a
True return means the bearer is fetchable AND non-empty.
Wiring
- tools/x_search_tool.py — new tool, ~370 LOC. Schema gated by check_fn,
bearer resolved per-call so revoked OAuth surfaces a clean tool_error
rather than an HTTP 401.
- toolsets.py — "x_search" toolset def. NOT added to _HERMES_CORE_TOOLS;
users opt in via hermes tools.
- hermes_cli/tools_config.py — CONFIGURABLE_TOOLSETS entry + TOOL_CATEGORIES
block with two provider options (OAuth + API key) sharing the existing
xai_grok post_setup hook for credential bootstrap.
- hermes_cli/config.py — DEFAULT_CONFIG["x_search"] with model /
timeout_seconds / retries. Additive nested key; no version bump.
- tests/tools/test_x_search_tool.py — 13 tests covering HTTP shape,
handle validation, citation extraction, 4xx/5xx/timeout handling,
and the full credential-resolution matrix (OAuth-only, API-key-only,
both-set, neither-set, resolver-raises, config overrides, registry
registration).
- website/docs/guides/xai-grok-oauth.md — adds X Search to the
direct-to-xAI tools section with off-by-default note.
- website/docs/user-guide/features/tools.md — new row in the tools table.
Off by default — users enable via `hermes tools` → 🐦 X (Twitter) Search.
Schema only appears to the model when xAI credentials are configured.
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
* docs(x_search): add dedicated feature page + reference entries
- website/docs/user-guide/features/x-search.md (new) — full feature
walkthrough: authentication, enablement, configuration, parameters,
returned fields, example, troubleshooting, see-also links.
- website/docs/reference/tools-reference.md — new "x_search" toolset
section with parameter docs and credential gating note.
- website/docs/reference/toolsets-reference.md — new row in the
toolset catalog table.
- website/sidebars.ts — wires the new feature page under
Media & Web, after web-search.
---------
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
* feat(video_gen): unified video_generate tool with pluggable provider backends
One core video_generate tool, every backend a plugin. Mirrors the
image_gen + memory_provider + context_engine architecture: ABC, registry,
plugin-context registration hook, and per-plugin model catalogs surfaced
through hermes tools.
Surface (one schema, every backend):
- operation: generate / edit / extend
- modalities: text-to-video (prompt only), image-to-video (prompt +
image_url), video edit (prompt + video_url), video extend (video_url)
- reference_image_urls, duration, aspect_ratio, resolution,
negative_prompt, audio, seed, model override
- Providers ignore unknown kwargs and declare what they support via
VideoGenProvider.capabilities() — backend-specific quirks stay in the
backend, the agent learns one tool
Backends shipped:
- plugins/video_gen/xai/ — Grok-Imagine, full generate/edit/extend +
image-to-video + reference images (salvaged from PR #10600 by
@Jaaneek, reshaped into the plugin interface)
- plugins/video_gen/fal/ — Veo 3.1 (t2v + i2v), Kling O3 i2v,
Pixverse v6 i2v with model-aware payload building that drops keys a
model doesn't declare
Wiring:
- agent/video_gen_provider.py — VideoGenProvider ABC, normalize_operation,
success_response / error_response, save_b64_video / save_bytes_video,
$HERMES_HOME/cache/videos/
- agent/video_gen_registry.py — thread-safe register/get/list +
get_active_provider() reading video_gen.provider from config.yaml
- hermes_cli/plugins.py — PluginContext.register_video_gen_provider()
- hermes_cli/tools_config.py — Video Generation category in
hermes tools, plugin-only providers list, model picker per plugin,
config write to video_gen.{provider,model}
- toolsets.py — new video_gen toolset
- tests: 31 new tests covering ABC, registry, tool dispatch, both plugins
- docs: developer-guide/video-gen-provider-plugin.md (parallel to the
image-gen guide), sidebar + toolsets-reference + plugin guides updated
Supersedes: #25035 (FAL), #17972 (FAL), #14543 (xAI), #13847 (HappyHorse),
#10458 (provider categories), #10786 (xAI media+search bundle), #2984
(FAL duplicate), #19086 (Google Veo standalone — easy port to plugin
interface).
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
* feat(video_gen): dynamic schema reflects active backend's capabilities
Address the 'capability variance' question — instead of one tool with a
static schema that lies about what every backend supports, the
video_generate tool now rebuilds its description at get_definitions()
time based on the configured video_gen.provider and video_gen.model.
The agent sees backend-specific guidance up-front:
- 'fal-ai/veo3.1/image-to-video': 'image-to-video only — image_url is
REQUIRED; text-only prompts will be rejected'
- 'fal-ai/veo3.1' (t2v): no image_url restriction shown
- xAI grok-imagine-video: 'operations: generate, edit, extend; up to 7
reference_image_urls'
- Backends without edit/extend: 'not supported on this backend — surface
that they need to switch backends via hermes tools'
This is the same pattern PR #22694 used for delegate_task self-capping —
documented in the dynamic-tool-schemas skill. Cache invalidation is
free: get_tool_definitions() already memoizes on config.yaml mtime, so a
mid-session backend swap rebuilds the schema automatically.
Tested:
- Empirical FAL OpenAPI schema check confirms image-to-video models
require image_url (FAL returns HTTP 422 otherwise) — client-side
rejection in FALVideoGenProvider.generate() now prevents the wasted
round-trip
- Live E2E: fal-ai/veo3.1/image-to-video + prompt-only → clean
missing_image_url error; fal-ai/veo3.1 + prompt-only → dispatches
- 6 new tests cover the builder (no config / image-only / full-surface /
text-only / unknown provider / registry wiring), all passing
- 37/37 in the slice, 134/134 in the broader regression set
* test(video_gen/xai): full surface integration tests + cleaner schema
Verified end-to-end that the xAI plugin handles every documented mode
from PR #10600's surface: text-to-video, image-to-video,
reference-images-to-video, video edit, video extend (with and without
prompt). All five modes route to the correct xAI endpoint
(/videos/generations, /videos/edits, /videos/extensions) with the right
payload shape (image / reference_images / video keys), and all five
client-side rejections fire before the network: edit-without-prompt,
extend-without-video_url, image+refs conflict, >7 references, and
duration/aspect_ratio clamping.
15 new integration tests grouped into four classes (endpoint routing,
modalities, validation, clamping). httpx is stubbed via a small fake
AsyncClient that records POSTs so the tests assert the actual payload
the plugin would send to xAI — not just the success/error envelope.
Also cleaned up a description redundancy: when a model's operations
match the backend's overall set, we no longer print the duplicate
'operations supported by this model' line. xAI's description now reads:
Active backend: xAI . model: grok-imagine-video
- operations supported by this backend: edit, extend, generate
- modalities supported by this backend: image, reference_images, text
- aspect_ratio choices: 16:9, 1:1, 2:3, 3:2, 3:4, 4:3, 9:16
- resolution choices: 480p, 720p
- duration range: 1-15s
- reference_image_urls: up to 7 images
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
* feat(video_gen): collapse surface to t2v + i2v, family-based auto-routing
Two design changes per Teknium:
1) Drop edit/extend from the tool surface entirely. Only text-to-video
and image-to-video remain. The agent sees a clean tool with two
modalities; backend-specific quirks like xAI's edit/extend endpoints
stay out of the unified schema.
2) FAL: pick a model FAMILY once, the plugin routes between the
family's text-to-video and image-to-video endpoints based on whether
image_url was passed. Users no longer pick 'fal-ai/veo3.1' AND
'fal-ai/veo3.1/image-to-video' as separate options — they pick
'veo3.1', and the plugin handles the rest.
Catalog rewritten as families:
veo3.1 fal-ai/veo3.1 / fal-ai/veo3.1/image-to-video
pixverse-v6 fal-ai/pixverse/v6/text-to-video / fal-ai/pixverse/v6/image-to-video
kling-o3-standard fal-ai/kling-video/o3/standard/text-to-video / fal-ai/kling-video/o3/standard/image-to-video
xAI uses a single endpoint (/videos/generations) for both modes,
routed by the presence of the 'image' field in the payload — no
edit/extend exposure.
Schema changes:
- VIDEO_GENERATE_SCHEMA: drop operation, drop video_url. Final params:
prompt (required), image_url, reference_image_urls, duration,
aspect_ratio, resolution, negative_prompt, audio, seed, model.
- VideoGenProvider ABC: drop normalize_operation, VALID_OPERATIONS,
DEFAULT_OPERATION. capabilities() drops 'operations' key.
- success_response: add 'modality' field ('text' | 'image') so the
agent and logs can see which endpoint was actually hit.
Dynamic schema builder simplified — no operations bullet, no
'switch backends if you need edit/extend' guidance. When the active
backend supports both modalities (the common case), description reads:
Active backend: FAL . model: pixverse-v6
- supports both text-to-video (omit image_url) and image-to-video
(pass image_url) - routes automatically
- aspect_ratio choices: 16:9, 9:16, 1:1
- resolution choices: 360p, 540p, 720p, 1080p
- duration range: 1-15s
- audio: pass audio=true to enable native audio (pricing tier)
- negative_prompt: supported
Tests: 51 in the video_gen slice, 216 across the broader image+video
sweep, all passing. New FAL routing tests prove pixverse-v6 + no image
hits text-to-video endpoint, pixverse-v6 + image_url hits
image-to-video endpoint, same for veo3.1 and kling-o3-standard.
Docs updated: developer-guide page rewrites the 'model families' pattern
as a first-class section so external plugin authors know the convention.
toolsets-reference and toolsets.py descriptions match the new surface.
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
* feat(video_gen/fal): expand catalog to 6 families, cheap + premium tiers
Catalog now covers everything Teknium specced from FAL:
Cheap tier:
ltx-2.3 fal-ai/ltx-2.3-22b/text-to-video / image-to-video
pixverse-v6 fal-ai/pixverse/v6/text-to-video / image-to-video
Premium tier:
veo3.1 fal-ai/veo3.1 / fal-ai/veo3.1/image-to-video
seedance-2.0 bytedance/seedance-2.0/text-to-video / image-to-video
kling-v3-4k fal-ai/kling-video/v3/4k/text-to-video / image-to-video
happy-horse fal-ai/happy-horse/text-to-video / image-to-video
DEFAULT_MODEL moved from veo3.1 (premium) to pixverse-v6 (cheap, sane
defaults, both modalities) — better first-run UX for users who haven't
explicitly picked a model.
New family-entry knob: image_param_key. Kling v3 4K's image-to-video
endpoint expects start_image_url instead of image_url; declaring
image_param_key='start_image_url' on the family lets _build_payload
remap correctly. Other families default to plain image_url.
Per-family capability flags reflect each model's docs:
- LTX 2.3 + Happy Horse: minimal payloads (no duration/aspect/resolution
enum exposed by FAL — let endpoint apply defaults)
- Seedance: 6 aspect ratios incl 21:9, durations 4-15, audio supported,
negative prompts NOT supported per docs
- Kling v3 4K: 16:9/9:16/1:1, 3-15s, audio + negative
- Veo 3.1: unchanged, 16:9/9:16, 4/6/8s
Tests: +5 covering the new families (full catalog, Kling 4K
start_image_url remap, Seedance routing, LTX payload minimality, Happy
Horse minimality). 56/56 in the slice green.
Note: I did NOT add the FAL-hosted xAI Grok-Imagine variant. Hermes
already has a direct xAI plugin that talks to xAI's own API; routing
the same model through FAL's wrapper would duplicate the surface
without adding capabilities. Users on FAL who want Grok-Imagine should
use the xAI plugin directly; flag if you want both routes available.
* test(video_gen): tool-surface routing matrix — every model x modality
End-to-end matrix test driven through _handle_video_generate() — the
actual function the agent's video_generate tool call lands in. Writes
config.yaml, invokes the registered handler with a raw args dict, then
asserts the outbound HTTP/SDK call hit the right endpoint with the right
payload shape.
Parametrized over FAL_FAMILIES.keys() so the matrix auto-discovers new
families as they're added (add a family to FAL_FAMILIES and you get
both modalities tested for free).
Coverage:
- All 6 FAL families x {text-only, text+image} = 12 cases
- xAI x {text-only, text+image} = 2 cases
- tool-level model= arg overrides config = 2 cases
For each case, verifies:
- result['success'] is True
- result['modality'] matches input shape ('text' if no image_url, 'image' otherwise)
- outbound endpoint URL matches the family's text_endpoint or image_endpoint
- text-only payloads carry no image-shaped keys
- text+image payloads carry the family's image key (image_url for most,
start_image_url for kling-v3-4k, wrapped 'image' object for xAI)
All 16 cases passing. Confirms the tool surface routes every
(provider, model, modality) combination correctly with zero leakage.
* feat(video_gen): keep video_gen out of first-run setup, surface in status
Two changes:
1. video_gen joins _DEFAULT_OFF_TOOLSETS, so it is NOT pre-selected in
the first-run toolset checklist. Video gen is niche, paid, and slow —
most users don't want it nagging them during initial setup. Anyone
who wants it opts in via 'hermes tools' -> Video Generation, which
already routes to the provider+model picker.
2. The 'hermes setup' status panel learns about video_gen — but only
shows the row when a plugin reports available. Users without
FAL_KEY/XAI_API_KEY see nothing about video gen; users with one of
those keys see 'Video Generation (FAL) ✓' as confirmation it's wired.
Verified live:
- Fresh install (no creds): zero video_gen mentions in wizard.
- With FAL_KEY: status row appears with active backend name.
- 160/160 in the setup + tools_config + video_gen test slice.
Rationale: image_gen is on by default because it's a featured creative
tool used in casual chat (telegrams, etc). Video gen is heavier — long
wait, paid per-second pricing. Default-off matches user intent better.
---------
Co-authored-by: Jaaneek <Jaaneek@users.noreply.github.com>
Background macOS desktop control via cua-driver MCP — does NOT steal the
user's cursor or keyboard focus, works with any tool-capable model.
Replaces the Anthropic-native `computer_20251124` approach from the
abandoned #4562 with a generic OpenAI function-calling schema plus SOM
(set-of-mark) captures so Claude, GPT, Gemini, and open models can all
drive the desktop via numbered element indices.
- `tools/computer_use/` package — swappable ComputerUseBackend ABC +
CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary).
- Universal `computer_use` tool with one schema for all providers.
Actions: capture (som/vision/ax), click, double_click, right_click,
middle_click, drag, scroll, type, key, wait, list_apps, focus_app.
- Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style
`content: [text, image_url]` parts) that flows through
handle_function_call into the tool message. Anthropic adapter converts
into native `tool_result` image blocks; OpenAI-compatible providers
get the parts list directly.
- Image eviction in convert_messages_to_anthropic: only the 3 most
recent screenshots carry real image data; older ones become text
placeholders to cap per-turn token cost.
- Context compressor image pruning: old multimodal tool results have
their image parts stripped instead of being skipped.
- Image-aware token estimation: each image counts as a flat 1500 tokens
instead of its base64 char length (~1MB would have registered as
~250K tokens before).
- COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset
is active.
- Session DB persistence strips base64 from multimodal tool messages.
- Trajectory saver normalises multimodal messages to text-only.
- `hermes tools` post-setup installs cua-driver via the upstream script
and prints permission-grant instructions.
- CLI approval callback wired so destructive computer_use actions go
through the same prompt_toolkit approval dialog as terminal commands.
- Hard safety guards at the tool level: blocked type patterns
(curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash,
force delete, lock screen, log out).
- Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic)
workflow guide.
- Docs: `user-guide/features/computer-use.md` plus reference catalog
entries.
44 new tests in tests/tools/test_computer_use.py covering schema
shape (universal, not Anthropic-native), dispatch routing, safety
guards, multimodal envelope, Anthropic adapter conversion, screenshot
eviction, context compressor pruning, image-aware token estimation,
run_agent helpers, and universality guarantees.
469/469 pass across tests/tools/test_computer_use.py + the affected
agent/ test suites.
- `model_tools.py` provider-gating: the tool is available to every
provider. Providers without multi-part tool message support will see
text-only tool results (graceful degradation via `text_summary`).
- Anthropic server-side `clear_tool_uses_20250919` — deferred;
client-side eviction + compressor pruning cover the same cost ceiling
without a beta header.
- macOS only. cua-driver uses private SkyLight SPIs
(SLEventPostToPid, SLPSPostEventRecordTo,
_AXObserverAddNotificationAndCheckRemote) that can break on any macOS
update. Pin with HERMES_CUA_DRIVER_VERSION.
- Requires Accessibility + Screen Recording permissions — the post-setup
prints the Settings path.
Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic-
native schema). Credit @0xbyt4 for the original #3816 groundwork whose
context/eviction/token design is preserved here in generic form.
* feat: add video_analyze tool for native video understanding
Adds a video_analyze tool that sends video files to multimodal LLMs
(e.g. Gemini) for analysis via the OpenRouter-compatible video_url
content type. Mirrors vision_analyze in structure, error handling,
and registration pattern.
Key design:
- Base64 encodes entire video (no frame extraction, no ffmpeg dep)
- Uses 'video_url' content block type (OpenRouter standard)
- Supports mp4, webm, mov, avi, mkv, mpeg formats
- 50 MB hard cap, 20 MB warning threshold
- 180s minimum timeout (videos take longer than images)
- AUXILIARY_VIDEO_MODEL env override, falls back to AUXILIARY_VISION_MODEL
- Same SSRF protection, retry logic, and cleanup as vision_analyze
Default disabled: registered in 'video' toolset (not in _HERMES_CORE_TOOLS).
Users opt in via: hermes tools enable video, or enabled_toolsets=['video'].
* feat(video): add models.dev capability pre-check + CONFIGURABLE_TOOLSETS entry
- Pre-checks model video capability via models.dev modalities.input
before expensive base64 encoding. Fails early with helpful message
suggesting video-capable alternatives (gemini, mimo-v2.5-pro).
- Passes optimistically if model unknown or lookup fails.
- Adds ModelInfo.supports_video_input() helper.
- Adds 'video' to CONFIGURABLE_TOOLSETS and _DEFAULT_OFF_TOOLSETS
so 'hermes tools enable video' works from CLI.
- 8 new tests for the capability check (37 total).
* refactor(video): remove models.dev capability pre-check
Removes _check_video_model_capability and ModelInfo.supports_video_input.
The vision_analyze tool doesn't pre-check image capability either — both
tools rely on the same pattern: send request, handle API errors gracefully
with categorized user-facing messages. The pre-check was inconsistent
(only worked for some providers/models) so drop it for parity.
* cleanup: compress comments, fix fragile timeout coupling
- Replace _VISION_DOWNLOAD_TIMEOUT * 2 with hardcoded 60s (no silent
breakage if vision timeout changes independently)
- Strip verbose comments and redundant log lines throughout
- No behavioral changes
Plugin platforms now get full toolset support without any entries in
toolsets.py.
tools_config._get_platform_tools(): Falls back to 'hermes-<name>'
when the platform isn't in the static PLATFORMS dict. No more
KeyError for plugin platforms.
toolsets.resolve_toolset(): Auto-generates a toolset for plugin
platforms (hermes-<name>) containing _HERMES_CORE_TOOLS plus any
tools the plugin registered into a matching toolset name. This means
a plugin can call ctx.register_tool(toolset='irc', ...) and those
tools will be included in the hermes-irc toolset automatically.
webhook.py: Registry-aware cross-platform delivery.
run_agent.py: Platform hints from plugin registry.
IRC adapter: Token lock + platform hint.
Removed dead token-empty-warning extension.
Updated docs.
Salvage-follow-up to @shannonsands's /reload-skills PR. Trims the feature to
match the design: user-initiated rescan, no prompt-cache reset, no new
schema surface, no phantom user turn, and the next-turn note carries each
added/removed skill's 60-char description (not just its name).
Changes vs the original PR:
* Drop the in-process skills prompt-cache clear in reload_skills(). Skills
are invoked at runtime via /skill-name, skills_list, or skill_view —
they don't need to live in the system prompt for the model to use them.
Keeping the cache intact preserves prefix caching across the reload so
/reload-skills pays no cache-reset cost. (MCP has to break the cache
because tool schemas must be known at conversation start; skills do not.)
* Drop the skills_reload agent tool and SKILLS_RELOAD_SCHEMA from
tools/skills_tool.py, plus the four skills_reload enumerations in
toolsets.py. No new schema surface — agents can already see a freshly-
installed skill via skill_view / skills_list the moment it's on disk.
* Replace the phantom 'role: user' turn injection with a one-shot queued
note. CLI uses self._pending_skills_reload_note (same pattern as
_pending_model_switch_note, prepended to the next API call and cleared).
Gateway uses self._pending_skills_reload_notes[session_key]. The note
is prepended to the NEXT real user message in this session, so message
alternation stays intact and nothing out-of-band is persisted to the
transcript.
* reload_skills() now returns added/removed as
[{'name': str, 'description': str}, ...] (description truncated to 60
chars — matches the curator / gateway adapter budget). The injected
next-turn note formats each entry as 'name — description' so the model
can actually reason about which new skills to call without running
skills_list first.
* Only emit the note when the diff is non-empty. On empty diff, print
'No new skills detected' and do nothing else.
* Tests rewritten to cover the queue semantics, the description payload,
and a regression guard that the prompt-cache snapshot is preserved.
Adds a public reload path for the in-process skill caches so newly
installed (or removed) skills become visible mid-session without a
gateway restart. Mirrors the shape of /reload-mcp.
Three surfaces:
* /reload-skills slash command — CLI (cli.py) and gateway (gateway/run.py),
with /reload_skills alias for Telegram autocomplete and an explicit
Discord registration.
* skills_reload agent tool (tools/skills_tool.py) — lets agents/subagents
pick up freshly-installed skills via tool call.
* agent.skill_commands.reload_skills() — shared helper that clears
_skill_commands, _SKILLS_PROMPT_CACHE (in-process LRU), and the
on-disk .skills_prompt_snapshot.json, then returns an added/removed
diff plus the new total count.
Tested:
* tests/agent/test_skill_commands_reload.py (9 cases)
* tests/cli/test_cli_reload_skills.py (3 cases)
* tests/gateway/test_reload_skills_command.py (4 cases)
Use case: NemoClaw / OpenShell-style sandboxed orchestrators that drop
skills into ~/.hermes/skills mid-session, plus agentic flows where the
agent itself installs a skill via the shell tool and needs it bound
without a gateway restart. The Python helper
clear_skills_system_prompt_cache(clear_snapshot=True) already exists
internally — this PR just exposes it via slash command and tool.
Background macOS desktop control via cua-driver MCP — does NOT steal the
user's cursor or keyboard focus, works with any tool-capable model.
Replaces the Anthropic-native `computer_20251124` approach from the
abandoned #4562 with a generic OpenAI function-calling schema plus SOM
(set-of-mark) captures so Claude, GPT, Gemini, and open models can all
drive the desktop via numbered element indices.
- `tools/computer_use/` package — swappable ComputerUseBackend ABC +
CuaDriverBackend (stdio MCP client to trycua/cua's cua-driver binary).
- Universal `computer_use` tool with one schema for all providers.
Actions: capture (som/vision/ax), click, double_click, right_click,
middle_click, drag, scroll, type, key, wait, list_apps, focus_app.
- Multimodal tool-result envelope (`_multimodal=True`, OpenAI-style
`content: [text, image_url]` parts) that flows through
handle_function_call into the tool message. Anthropic adapter converts
into native `tool_result` image blocks; OpenAI-compatible providers
get the parts list directly.
- Image eviction in convert_messages_to_anthropic: only the 3 most
recent screenshots carry real image data; older ones become text
placeholders to cap per-turn token cost.
- Context compressor image pruning: old multimodal tool results have
their image parts stripped instead of being skipped.
- Image-aware token estimation: each image counts as a flat 1500 tokens
instead of its base64 char length (~1MB would have registered as
~250K tokens before).
- COMPUTER_USE_GUIDANCE system-prompt block — injected when the toolset
is active.
- Session DB persistence strips base64 from multimodal tool messages.
- Trajectory saver normalises multimodal messages to text-only.
- `hermes tools` post-setup installs cua-driver via the upstream script
and prints permission-grant instructions.
- CLI approval callback wired so destructive computer_use actions go
through the same prompt_toolkit approval dialog as terminal commands.
- Hard safety guards at the tool level: blocked type patterns
(curl|bash, sudo rm -rf, fork bomb), blocked key combos (empty trash,
force delete, lock screen, log out).
- Skill `apple/macos-computer-use/SKILL.md` — universal (model-agnostic)
workflow guide.
- Docs: `user-guide/features/computer-use.md` plus reference catalog
entries.
44 new tests in tests/tools/test_computer_use.py covering schema
shape (universal, not Anthropic-native), dispatch routing, safety
guards, multimodal envelope, Anthropic adapter conversion, screenshot
eviction, context compressor pruning, image-aware token estimation,
run_agent helpers, and universality guarantees.
469/469 pass across tests/tools/test_computer_use.py + the affected
agent/ test suites.
- `model_tools.py` provider-gating: the tool is available to every
provider. Providers without multi-part tool message support will see
text-only tool results (graceful degradation via `text_summary`).
- Anthropic server-side `clear_tool_uses_20250919` — deferred;
client-side eviction + compressor pruning cover the same cost ceiling
without a beta header.
- macOS only. cua-driver uses private SkyLight SPIs
(SLEventPostToPid, SLPSPostEventRecordTo,
_AXObserverAddNotificationAndCheckRemote) that can break on any macOS
update. Pin with HERMES_CUA_DRIVER_VERSION.
- Requires Accessibility + Screen Recording permissions — the post-setup
prints the Settings path.
Supersedes PR #4562 (pyautogui/Quartz foreground backend, Anthropic-
native schema). Credit @0xbyt4 for the original #3816 groundwork whose
context/eviction/token design is preserved here in generic form.
