Files

Brooklyn Nicholson c57581cd0d feat(tools): drive_preview and annotate_preview — the agent can use the page it opened

The in-app browser was a one-way mirror. open_preview put a page in the pane
and read_preview read its text back, but nothing could touch it. A click meant
falling back to the browser_* tools, which drive a separate Chromium the user
cannot see — so "log into this and pull my invoices" happened in a different
browser from the one on screen, with none of the sessions the user is already
signed into.

Four pieces, and they only make sense together:

  · an in-page engine that inventories what is interactable and performs the
    verb, injected as source because it has to run inside the guest page;
  · the preview.act.request bridge from the gateway into the pane;
  · drive_preview, for acting: elements, click, type, scroll, press, and the
    pane's own back/forward/reload;
  · annotate_preview, for marking without acting.

Those last two started as one tool doing two unrelated jobs. Leaving a mark is
not an action — it outlives the turn that drew it — so it gets its own verb,
and the interaction verb gets a name that says what it does.

Gating is the existing surface rule: desktop_ui folds in on session
source: 'desktop', and the bridge refuses to act for a background session, so a
turn running behind the user's back cannot reach into the page they are working
in.

Two details worth a reviewer's attention. Typing assigns through the
prototype's value setter, because React shadows value with its own accessor and
ignores an input event whose value it believes it already wrote — a plain
el.value = … types into a field that snaps back on the next render. And
clicking replays the pointer/mouse pair before activation, because frameworks
bind to mousedown as often as to click.

2026-08-20 05:26:37 -05:00

13 KiB

Raw Blame History

sidebar_position, title, description

sidebar_position	title	description
4	Toolsets Reference	Reference for Hermes core, composite, platform, and dynamic toolsets

Toolsets Reference

Toolsets are named bundles of tools that control what the agent can do. They're the primary mechanism for configuring tool availability per platform, per session, or per task.

How Toolsets Work

Every tool belongs to exactly one toolset. When you enable a toolset, all tools in that bundle become available to the agent. Toolsets come in three kinds:

Core — A single logical group of related tools (e.g., file bundles read_file, write_file, patch, search_files)
Composite — Combines multiple core toolsets for a common scenario (e.g., debugging bundles file, terminal, and web tools)
Platform — A complete tool configuration for a specific deployment context (e.g., hermes-cli is the default for interactive CLI sessions)

Configuring Toolsets

Per-session (CLI)

hermes chat --toolsets web,file,terminal
hermes chat --toolsets debugging        # composite — expands to file + terminal + web
hermes chat --toolsets all              # everything

Per-platform (config.yaml)

toolsets:
  - hermes-cli          # default for CLI
  # - hermes-telegram   # override for Telegram gateway

Interactive management

hermes tools                            # curses UI to enable/disable per platform

Or in-session:

/tools list
/tools disable browser
/tools enable homeassistant

Core Toolsets

Toolset	Tools	Purpose
`browser`	`browser_back`, `browser_cdp`, `browser_click`, `browser_console`, `browser_dialog`, `browser_get_images`, `browser_navigate`, `browser_press`, `browser_scroll`, `browser_snapshot`, `browser_type`, `browser_vision`, `web_search`	Core browser automation. Includes `web_search` as a fallback for quick lookups. `browser_cdp` and `browser_dialog` are gated at runtime — registered only when a CDP endpoint is reachable at session start (via `/browser connect`, `browser.cdp_url` config, Browserbase, or Camofox). `browser_dialog` works together with the `pending_dialogs` and `frame_tree` fields that `browser_snapshot` adds when a CDP supervisor is attached.
`clarify`	`clarify`	Ask the user a question when the agent needs clarification.
`code_execution`	`execute_code`	Run Python scripts that call Hermes tools programmatically.
`coding`	composite (`file` + `terminal` + `search` + `web` + `skills` + `browser` + `todo` + `memory` + `session_search` + `clarify` + `code_execution` + `delegation` + `vision`)	Coding-focused bundle for software work: file editing, terminal, search, web docs, skills, browser, delegate, and code execution.
`cronjob`	`cronjob`	Schedule and manage recurring tasks.
`debugging`	composite (`file` + `terminal` + `web`)	Debug bundle — file, process/terminal, web extract/search.
`delegation`	`delegate_task`	Spawn isolated subagent instances for parallel work.
`discord`	`discord`	Core Discord text/embed/DM actions (gateway-only). Active on the `hermes-discord` toolset.
`discord_admin`	`discord_admin`	Discord moderation (bans, role changes, channel management). Active on the `hermes-discord` toolset; requires the bot to hold the relevant Discord permissions.
`feishu_doc`	`feishu_doc_read`	Read Feishu/Lark document content. Used by the Feishu document-comment intelligent-reply handler.
`feishu_drive`	`feishu_drive_add_comment`, `feishu_drive_list_comments`, `feishu_drive_list_comment_replies`, `feishu_drive_reply_comment`	Feishu/Lark drive comment operations. Scoped to the comment agent; not exposed on `hermes-cli` or other messaging toolsets.
`file`	`patch`, `read_file`, `search_files`, `write_file`	File reading, writing, searching, and editing.
`homeassistant`	`ha_call_service`, `ha_get_state`, `ha_list_entities`, `ha_list_services`	Smart home control via Home Assistant. Only available when `HASS_TOKEN` is set.
`computer_use`	`computer_use`	Background desktop control via cua-driver — does not steal cursor/focus. Works with any tool-capable model. macOS, Windows, and Linux; requires `cua-driver` on `$PATH`.
`context_engine`	(varies)	Runtime tools exposed by the active context-engine plugin (empty until a plugin populates it).
`image_gen`	`image_generate`	Text-to-image generation via FAL.ai (with opt-in OpenAI / xAI backends).
`video_gen`	`video_generate`, `xai_video_edit`, `xai_video_extend`	Text-to-video and image-to-video via plugin-registered backends (xAI Grok-Imagine, FAL.ai Veo 3.1 / Pixverse v6 / Kling O3). Pass `image_url` to animate an image; omit it for text-to-video. `xai_video_edit` / `xai_video_extend` are provider-specific edit/extend tools, gated on xAI Imagine credentials.
`kanban`	`kanban_attach`, `kanban_attach_url`, `kanban_attachments`, `kanban_block`, `kanban_comment`, `kanban_complete`, `kanban_create`, `kanban_heartbeat`, `kanban_link`, `kanban_list`, `kanban_request_changes`, `kanban_request_review`, `kanban_show`, `kanban_unblock`	Multi-agent coordination tools. Registered for dispatcher-spawned task workers (`HERMES_KANBAN_TASK`) and for profiles that explicitly list the `kanban` toolset by name (the `all`/`` wildcard does not* enable it). Workers mark tasks done, request first-class review, block, heartbeat, comment, and create/link follow-up tasks; orchestrator profiles additionally get board-routing tools like list/unblock. `delegate_task` children are not Kanban run owners: their schema strips/disables this toolset and runtime guards reject direct board mutations, even if parent `HERMES_KANBAN_*` env vars are present.
`memory`	`memory`	Persistent cross-session memory management.
`desktop_ui`	`annotate_preview`, `close_preview`, `close_terminal`, `drive_preview`, `focus_pane`, `open_preview`, `react_to_message`, `read_preview`, `read_terminal`, `read_window_below`, `tour`	Affordances that act on the Hermes desktop app itself — read/close the embedded terminal pane, open, read, close, interact with, and annotate the in-app browser, identify the OS window behind the app, reveal a pane, react to a message, run a guided tour (highlight + narrate UI elements in the app or the preview pane). Enabled for sessions whose source is the desktop app, whichever backend it's connected to (local, SSH, URL, or Hermes Cloud). Never present on CLI, TUI, messaging, or cron sessions.
`project`	`project_create`, `project_list`, `project_switch`	Create and switch desktop Projects (named, multi-folder workspaces). GUI / desktop sessions only.
`safe`	`image_generate`, `vision_analyze`, `web_extract`, `web_search` (via `includes`)	Read-only research + media generation. No file writes, no terminal, no code execution.
`search`	`web_search`	Web search only (without extract).
`session_search`	`session_search`	Search past conversation sessions.
`skills`	`skill_manage`, `skill_view`, `skills_list`	Skill CRUD and browsing.
`spotify`	`spotify_albums`, `spotify_devices`, `spotify_library`, `spotify_playback`, `spotify_playlists`, `spotify_queue`, `spotify_search`	Native Spotify control (playback, queue, search, playlists, albums, library). Registered by the bundled `spotify` plugin.
`terminal`	`process`, `terminal`	Shell command execution and background process management.
`todo`	`todo`	Task list management within a session.
`tts`	`text_to_speech`	Text-to-speech audio generation.
`vision`	`vision_analyze`	Image analysis via vision-capable models.
`video`	`video_analyze`	Video analysis and understanding tools (opt-in, not in the default toolset — add explicitly via `--toolsets`).
`web`	`web_extract`, `web_search`	Web search and page content extraction.
`x_search`	`x_search`	Read-only public X discovery via xAI's built-in `x_search` Responses tool. Use the `xurl` skill for authenticated X API reads and account actions. Off by default; opt in via `hermes tools`. Schema only registered when xAI credentials (SuperGrok OAuth or `XAI_API_KEY`) are configured.
`yuanbao`	`yb_query_group_info`, `yb_query_group_members`, `yb_search_sticker`, `yb_send_dm`, `yb_send_sticker`	Yuanbao DM/group actions and sticker search. Registered only on `hermes-yuanbao`.

Platform Toolsets

Platform toolsets define the complete tool configuration for a deployment target. Most messaging platforms use the same set as hermes-cli:

Toolset	Differences from `hermes-cli`
`hermes-cli`	Full toolset — the default for interactive CLI sessions. Includes file, terminal, web, browser, memory, skills, vision, image_gen, todo, tts, delegation, code_execution, cronjob, session_search, clarify, computer_use, Home Assistant, and the kanban tools (all check_fn-gated at runtime).
`hermes-acp`	Drops `clarify`, `cronjob`, `image_generate`, `text_to_speech`, `computer_use`, all four Home Assistant tools, and the kanban tools. Focused on coding tasks in IDE context.
`hermes-api-server`	Drops `clarify`, `text_to_speech`, `computer_use`, and the kanban tools. Keeps everything else — suitable for programmatic access where user interaction isn't possible.
`hermes-cron`	Same as `hermes-cli`.
`hermes-telegram`	Same as `hermes-cli`.
`hermes-discord`	Adds `discord` and `discord_admin` on top of `hermes-cli`.
`hermes-slack`	Same as `hermes-cli`.
`hermes-whatsapp`	Same as `hermes-cli`.
`hermes-signal`	Same as `hermes-cli`.
`hermes-matrix`	Same as `hermes-cli`.
`hermes-mattermost`	Same as `hermes-cli`.
`hermes-email`	Same as `hermes-cli`.
`hermes-sms`	Same as `hermes-cli`.
`hermes-bluebubbles`	Same as `hermes-cli`.
`hermes-dingtalk`	Same as `hermes-cli`.
`hermes-feishu`	Adds the five `feishu_doc_` / `feishu_drive_` tools (only used by the document-comment handler, not the regular chat adapter).
`hermes-qqbot`	Same as `hermes-cli`.
`hermes-wecom`	Same as `hermes-cli`.
`hermes-wecom-callback`	Same as `hermes-cli`.
`hermes-weixin`	Same as `hermes-cli`.
`hermes-yuanbao`	Adds the five `yb_*` tools (DM/group/sticker) on top of `hermes-cli`.
`hermes-homeassistant`	Same as `hermes-cli` (the Home Assistant tools are already present by default and activate when `HASS_TOKEN` is set).
`hermes-webhook`	Restricted safe subset — only `web_search`, `web_extract`, `vision_analyze`, and `clarify`. Webhook-triggered runs get no terminal, file, or browser access.
`hermes-gateway`	Internal gateway orchestrator toolset — union of every `hermes-<platform>` toolset; used when the gateway needs to accept any message source.

Dynamic Toolsets

MCP server toolsets

Each configured MCP server generates a mcp-<server> toolset at runtime. For example, if you configure a github MCP server, a mcp-github toolset is created containing all tools that server exposes.

# config.yaml
mcp_servers:
  github:
    command: npx
    args: ["-y", "@modelcontextprotocol/server-github"]

This creates a mcp-github toolset you can reference in --toolsets or platform configs.

Plugin toolsets

Plugins can register their own toolsets via ctx.register_tool() during plugin initialization. These appear alongside built-in toolsets and can be enabled/disabled the same way.

Custom toolsets

Define custom toolsets in config.yaml to create project-specific bundles:

toolsets:
  - hermes-cli
custom_toolsets:
  data-science:
    - file
    - terminal
    - code_execution
    - web
    - vision

Wildcards

all or * — expands to every registered toolset (built-in + dynamic + plugin)

A handful of tools have an additional availability check on top of toolset membership and are not turned on by all/* alone:

Capability-gated tools (browser, computer_use, code_execution, Feishu, Home Assistant, cronjob) appear only when their backend/credential prerequisite is configured.
Workflow-gated tools — the kanban toolset — are deliberately opt-in. all/* does not enable kanban; you must list kanban explicitly (or be a dispatcher-spawned worker with HERMES_KANBAN_TASK set). Kanban tools mutate shared board state, so they stay off by default even under all.

Relationship to `hermes tools`

The hermes tools command provides a curses-based UI for toggling individual tools on or off per platform. This operates at the tool level (finer than toolsets) and persists to config.yaml. Disabled tools are filtered out even if their toolset is enabled.

See also: Tools Reference for the complete list of individual tools and their parameters.

13 KiB Raw Blame History