afc3b7c6f3
* feat(connectors): the desktop connects apps through one backend-owned operation Re-based onto main after #109517, #110368, #110574 and #110843 landed as squash merges (e0ef0eb9c3,d105376b21,ee2f5629b8,1ab32b212b): the branch's history no longer shared a base with main, so this is the PR's exact delta against +3762/-1470, identical to the branch tip 05d7f2d3d5. The fifteen commits it carried, in order: --- feat(connectors): the wire follows the connector contract (six-state status, required toolkit metadata, connectionId on mint) Hermes types exactly what the contract page writes: /tmp/magic/CONTRACT-TOOLKIT-METADATA.resolved.md (carriers: portal PR2 `sid/connection-api` for the enum and account rows, the portal contract branch on top of it for the list metadata). No optional-for-compat fields, no fallback branch, no seven-state word left in the tree. A gateway that does not speak this contract fails validation loudly. Wire (tools/connectors/gateway/wire.py): `ConnectionStatus` is the six values, `pending` covering the vendor's INITIALIZING and INITIATED; `ConnectorListItem` requires title, description, https iconUrl and authKind, and carries activeConnectionId when the session binds an account; `ConnectorListResponse` types a page whole with its total; `ConnectorAccount` and `ConnectorAccountsResponse` type the account routes; a mint result names its connectionId on `initiated` and never on `failed`; CONNECTION_REQUIRED carries connectUrl and connectionId together or not at all; `account` exists on the execute call and is never sent while multi-account is off. Client: `list_connectors(search, connected)` refuses a search under three characters before any request and parses every page whole; `list_accounts` and `account_status` read the account routes, 404 `connection_not_found` is None and 429 raises `RateLimited(retry_after)`. Watcher: the target keeps the account id the mint named and the toolkit's title and icon from one list read before the card is emitted; `_status_for` is the one seam between the watcher and the gateway's status source (the list walk today; the per-account route replaces that body only). `pending` and a missing status move nothing; revoked and inactive are failed. Desktop: `ConnectorRow` is the strict list item; `ConnectorRowSeed` is what a tool call's args can say; `connection.request`/`update` targets carry `connection_id`, `title`, `icon_url`; the mark ladder is glyph, vendor icon as a plain image, favicon, monogram. Tests run red first: wire rejects the retired words and the missing metadata; client search minimum, execute omits account, account_status 404/429; managed targets carry the metadata and the account id, a pending row moves nothing; the logo ladder order; the store passes the fields through. --- feat(connectors): onboarding connect-first rides the connection operation The guided first build had its own connector surface: a 2 s connectors.list poll loop, an auto-open of every minted link, a hidden "[setup] links opened" note that painted as a user bubble, CONNECT FIRST rows in the transcript, and a "Start with N apps connected" composer pill that sent the go signal early. Delete all of it. The build session's first manage_connections connect now shows the same card as any chat, and the settled tool result is the go signal. Deleted: store/first-build-connectors.ts, assistant-ui/first-build-connectors.tsx (nothing rendered it any more), lib/first-build-start.ts, onboarding-chat/start.tsx, the first-build session markers in handoff-receipt.ts, the connectionRows / latestConnectorPart helpers they were the last readers of, and the five strings only they used. The runbook (setup-profile.ts::connectFirstRunbook) now describes the operation's real contract: one connect call with every picked slug, one card with a row per app, blocks until settled, a per-app result of connected / skipped / not_connected. No wait, no "Start with", no "already active" branch (the result never carried it), and no offer to re-mint: Try again and Continue live on the card, and a Continue means the user moved on. Tests (red first): the runbook names one connect call and its per-app result and never the retired model actions; the no-account rule survives an empty pick. --- feat(connectors): the registry is keyed by profile, every watch read is bounded by the deadline, connectionId is optional, connect and execute carry returnTo and op Registry (P1-14): tools/connectors/live.py keys an operation by (profile home, session key). Two multiplexed profiles can carry the same timestamp-based session key; the tool thread opens under its turn's profile override and the RPC side names the session record's profile home, the default profile resolving to the process home on both sides. Watch reads (P1-8 residual): list_connectors takes a per-page timeout and the watcher bounds each read by the operation's remaining deadline (floor one second), so a stalled gateway page cannot hold the operation past its deadline. Contract follow-up from the portal handoff (/tmp/magic/HANDOFF-HERMES-PR3-PORTAL-CONTRACT.md): connectionId on a mint result and on CONNECTION_REQUIRED is plain Optional (a no-auth toolkit answers active with none; a failed mint answers with neither link nor id); a target without one has nothing to watch. Connect and execute requests carry returnTo (hermes-desktop | hermes-desktop-dev | portal) and op, so the vendor's done page can send the browser back to the app; the call sites and the deep-link handler follow in the next commit. Tests, red first: two profiles share a session key without seeing each other; each watch read shrinks with the deadline; the client honours a per-page timeout; the optional id; the two request fields. The fakes' list_connectors accept the timeout keyword; two wrapper spies that did not were the cause of a 300 s hang under the per-file harness. --- feat(connectors): an MCP target runs on the shared connection operation manage_connections ran MCP targets as a renderer errand: the desktop card ran the catalog lookup, the install and the OAuth flow, then told the backend what had happened, and every surface without a card got `unavailable` plus two terminal commands. That left the outcome in the renderer's word, and left the model unable to connect an MCP server anywhere but the desktop. The backend now owns the work, the way it already owns a managed connector. `prepare` starts the OAuth flow in process and points the browser at the backend's own callback route; an install that still needs credentials waits pending and publishes their names as `required_env`; `observe` reads the flow or the install worker on every tick. The worker itself moved out of tui_gateway/mcp_oauth_sessions.py into tools/connectors/mcp_oauth.py, and that module now calls it, so the Capabilities tab keeps its RPC session table. The card may only say approved, skipped or continue: a claim of any other state moves nothing. With that, `renderer_flow` and the `unavailable` state have no producer left and are gone from the contract. Off the desktop there is no card, so the action runs at once and the result carries the authorization URL for the user to open. Try again reaches the same work: `_reissue` dispatches on `target.kind`, so an MCP row re-runs its own install, enable or OAuth instead of minting a managed gateway link. (cherry picked from commit fa0438c4121807604e7c983ba42b901314a1305b) --- feat(connectors): the MCP card projects the operation instead of running the flow The MCP setup card used to own the work: it called the catalog, ran installMcpCatalogEntry, polled the action, drove the OAuth window, and then told the backend what had happened through connection.respond. That made the renderer a second authority on target state, so an install that finished after the window closed, or a card that never mounted, left the operation with a state nobody could correct. PR3 moves that work to the backend, so the card has one job left: show the operation and send the user's consent. McpSetupPending now renders request.targets the way ConnectorOffer renders them: one row per target, one verb by (action, state), Continue below the shell. Install and Enable send {status: 'approved'} and nothing else; an authorize row opens the link the backend minted, with no OAuth RPC of its own; a running install holds its verb with the busy mark; a failed row sends connectors.connect {reconnect: true} on the open operation, the same call the managed card makes. The owner lookup and that call are now shared helpers on connector-tool.tsx rather than a second copy here. ConnectionTargetOutcome loses connected / initiated / failed: those were the renderer reporting state, which it no longer may do. ConnectionActor loses renderer_flow for the same reason. ConnectionOperationTarget gains required_env, the credentials an MCP install is still waiting for; the card renders a field per entry under the pending row and holds Install until every required one has text, so the values travel with the approval instead of through a separate catalog call. (cherry picked from commit 2e42ad19f63e41f274a87199d07eb99a7c995cbe) --- fix(connectors): the toolkit metadata leaves the wire; the vendor logo is derived from the slug Sid cancelled portal PR3 (the toolkit metadata on the list item) on 2026-09-14, so the fields the contract commit typed as required come off the wire: no title, description, iconUrl, authKind or activeConnectionId on the list item, no total on the page, no search or connected query, no title or icon_url on a target snapshot. The list item is exactly what portal #1220 emits. The desktop derives the vendor logo from the toolkit slug instead (`connectorIconUrl`, https://logos.composio.dev/api/{slug}; the gateway slug is the vendor slug, checked for every lead-order pick), rendered as a plain image because the host sends no CORS header. The title stays `connectorTitle(slug)`. The mark ladder is unchanged: glyph, derived vendor icon, favicon, monogram. Everything else in the contract stands: six-state status, optional connectionId on mint results and on CONNECTION_REQUIRED, returnTo and op, the account routes. The contract commit's message still names the metadata; this commit is the correction. --- fix(connectors): an MCP target survives the card's Continue, and the model is not sent to an action that refuses it The review of the MCP backend found eight defects; each one is a test first. - The off-desktop note told the model to confirm with action 'status', which refuses MCP names outright. It now says the authorization finishes in the background and the tools arrive on the next turn. - The card's existence is a property of the session. A missing callback no longer routes a desktop session down the off-desktop path, where the model would be handed a live link. - A second failure of a backend attempt was published with actor 'user'. ``refresh`` takes the actor, so the frame says who produced the text. - Continue on the RPC thread can settle the operation between any read of a target's state and the transition that follows it. A settled operation has a frozen result, so the lost move is dropped; an IllegalTransition no longer escapes into the tool result. The same rule covers a worker whose outcome arrives late and a Try again that arrives after Continue. - Try again on an install carried no credentials, so the second install ran with an empty env. The approved map is kept on the runner (never on the target: target fields are serialised to the model) and reused. - An approval that does not cover a required credential no longer reaches the worker, where ``install_entry``'s prompt would block on stdin forever; the row waits for the card's fields instead. - ``enable`` writes ``mcp_servers`` under the scope and lock the dashboard's toggle route uses, so the two read-modify-write paths in one process cannot drop each other's write. - Several authorize targets start their flows together and share one wait; one wait per target kept the card empty for minutes. - The off-desktop operation is in no session's registry, so it now emits no connection.update. Try again is also refused for a target the transition table cannot move back to 'initiated' (an MCP target has no move out of 'expired') and for an operation that settled during the call. The contract test that froze the Actor enum is replaced by two behaviour tests: no actor but the backend watcher can connect an MCP target, and the card's word never moves one. (cherry picked from commit 081c16412b48667758a77a7e7c29eaf2be6a93dd) --- fix(connectors): the watcher reads one account route per target, and every frame carries its seq The watch loop walked the whole toolkit list once per tick to learn whether one target had connected. That read costs a vendor call per page, cannot tell one account from another, and forced `awaiting_new_attempt`: after a forced reconnect the list still reported the OLD account `active`, so the row had to be disbelieved until it read as something else once. The gateway now serves `GET /v1/connectors/accounts/{connectionId}`, so each pending target reads its own account: the id the mint named, one read per target per tick at 1 Hz (the route's bucket is 180/min per principal), each bounded by the operation's remaining deadline. A 429 parks that one target until its Retry-After passes and leaves the others reading. A 404 is "not yet" until the deadline. A target the mint gave no account for has nothing to read, so it is not read. A forced reconnect watches the new account, which is why `awaiting_new_attempt` and its two tests are gone. `mint` and `run_remote` now name where the browser should come back to (`returnTo`, plus the operation id on a mint), so the vendor's done page can hand the user back to the desktop app that asked instead of stranding them on a web page. Only the desktop registers that URL scheme, so no other surface sends either field. `connection.update` frames were built by re-reading the operation after the lock was released, so a second writer could give an older frame a newer state and the renderer could not tell which frame was last. Every write now advances a monotonic `seq` and takes its snapshot under the same lock, and the emitter sends that snapshot; a renderer that keeps the highest seq per operation can drop a frame that arrives out of order. `connectors.operation.wake` lets the desktop's deep-link return ask for a read now instead of at the next tick; it only shortens the wait and trusts nothing else in the link. (cherry picked from commit f5423cf72302b92b26c043495184ad6426fa2b36) --- fix(connectors): the desktop card follows the operation, and says so out loud The MCP card was a second implementation of the connector card with the review's defects: it painted for any request on the session, kept its controls after the operation settled, offered an approve verb while the backend was still minting an authorize link, and re-enabled Install when the RPC returned rather than when the state frame moved the row. A second click in that window sent the consent twice. The renderer now reads the operation's `seq`: an update or a status frame whose sequence is not greater than the one already applied changes nothing, and a resume snapshot neither revives a settled card nor puts back a row a newer frame has moved. Without it the transport's ordering decided what the user saw. `hermes://connections/done?op=…` brings the user back from the browser to the session that opened the operation and wakes its watcher, so the row moves at once instead of at the watcher's next tick. Only the op id is used; the link's status moves no row. Each row's mark and cue are one polite live region, so a row that flips is heard and not only seen, focus follows the row the backend moved while the card holds it, and the waiting mark stops spinning under reduced motion. (cherry picked from commit 7c30d252556d7d496ddb354b50bcecc41bceef27) --- fix(connectors): the renderer types seq as the wire carries it The watcher commit made `seq` a required field on every operation frame. The desktop store still declared its own optional `seq` so it would compile against a backend that predates the field; that backend no longer exists on this branch, so the hedge is dead code and the fixtures were short one field. The store now reads `seq` from the shared types, and the fixtures count the way the backend does. The repeat-frame test asserted the whole request keeps its reference. With a real rising `seq` the request must change; the invariant the test guards is that the target row keeps its identity so open credential inputs do not remount. --- fix(connectors): a URL that arrives after the shared wait still lands on its row The shared prepare wait failed every row still pending when it ran out, while that row's own thread was still waiting on the provider. When the URL arrived a moment later the thread's move raised inside the daemon thread, the link was lost, and Try again started a second flow. The wait now bounds only how long prepare blocks; a row still pending afterwards is left to its own thread, which is the only writer of that row and ends with the URL or the flow's own failure. The comment on the approved credentials said every target field is serialised to the model; it is not (the snapshot names its keys). The reason they live on the runner is that they are secrets and the runner's life is exactly the operation's. --- fix(connectors): the review findings the operation must survive before the fold The MCP prepare threads and the install worker started with an empty context, so a named-profile turn's home override never reached them: the flow resolved the process home's `mcp_servers` and stored the token there. Each thread now runs in a copy of the calling thread's context. `connection.respond` had the same gap on the RPC thread: an approval ran the enable, which writes config.yaml, with no profile bound, so the flag landed in the launch home. The handler binds the session record's profile the way `_connector_rpc` does; `config_write_scope(None)` keeps that override, so the enable needs no change. The watcher raised out of the tool when a per-target Skip landed while that target's read was in flight: the operation stayed open with `live` closed, and every later answer got 4004. A read for a row that is no longer live is dropped at debug; only a refusal on a live row is still a contract violation. The same skip from the card raced a row the backend had just connected and aborted the rest of the answer; a skip for a resolved row is ignored and every entry, then the settle check, still runs. A read was bounded by the whole remaining deadline, so a hung gateway held the first read for 300 s and Continue could not return the tool; a read now waits ten seconds at most. A 429 parked only the target that read, but the budget is the principal's, so the next target's read in the same tick spent it again: every live target waits out the one Retry-After. The MCP surface rule read the platform alone, so a desktop call without the callback (registry dispatch from execute_code) opened an operation nobody rendered and blocked for the deadline; it now uses the managed rule, surface and callback. A write after settlement advanced `seq` while emitting the frozen frame, so the resume snapshot named a seq no frame carried; the counter stops at the settle frame. A mint that reports `initiated` with no account id logs that the watcher cannot read the row. (cherry picked from commit 587b228016bf8ee2b022e7fa58c2d9f6057809e9) --- fix(connectors): the desktop card holds a verb until the backend answers, and never takes the keyboard from a credential field The review of the desktop card found five defects; each one is a test first. - A resume snapshot was refused whenever the cache held a settled operation, whichever operation it was, so a session that opened a second operation after settling the first never got its card back from a resume. And the refused snapshot handed the caller the settled cache as "the request", so the session was flagged as needing input behind a summary with no controls. Only the same operation can refuse the snapshot now, and a refused one is no pending card. - The focus handoff picked the row's first button, which after pending -> initiated is the disabled working verb; the focus call was a no-op and the keyboard landed on the document body. It picks the first control that can take focus, else the row. It also moved focus out of a credential field the user was typing in whenever another row moved; it leaves an editable alone. The "focus Continue once every row resolved" branch was dead (Continue unmounts the moment nothing is unresolved), so it and its ref plumbing are gone. - The done link navigated to a settled operation's session and rejected when the wake RPC did (4004 once the operation left the live registry). A settled request is ignored, and a refused wake is nothing: the wake only shortens the wait, the watcher still ticks. - Install spun forever when the store refused to send the consent (the operation gone or settled under the card): `respondToConnectionRequest` resolves false in that case and the verb was only released in `catch`. - After a partial approval (a required credential missing) the backend answers with a same-state frame whose detail names what is missing; nothing released the verb because it was held until the row's state moved. The row now remembers the seq the click saw and holds the verb only until a frame past it arrives, which is the backend's word on the click whether or not the row moved. (cherry picked from commit 4c534d6e2af979778d9a2423cf97d717edeaa1a9) --- fix(connectors): a skip that loses the race to the watcher is ignored, not raised The skip guard read the row's state and then moved it; the watcher can connect the row between the two, and the refused move aborted the rest of the card's answer. The refusal itself is now the witness: a move refused for a row that is resolved, or on a settled operation, is the same nothing-to-do as a row resolved earlier. Two recording fakes in the managed tests kept their lists on the class; they now start per instance so a lifted fake cannot share reads between tests. --- fix(connectors): a resume that lands behind a newer live frame still reports the pending card The refused-snapshot branch answered "no pending card" for both reasons it can refuse: the operation settled, or a newer frame already moved a row. Only the first is no card. For the second the live card is still open and blocking the turn, so the caller must keep the session flagged as waiting on it. * test(connectors): defer new connection coverage until implementation settles Remove PR-added test cases and their unused helpers while retaining existing tests adapted to the changed connection contract. The three PR-only renderer test files are removed for now. Focused behavioral coverage will be added as the final implementation step before verification. Existing main coverage is not being removed wholesale, and this does not declare the feature merge-ready. * fix(connectors): commit MCP authorization at initialize, save setup values after success The OAuth probe treated one exception as one outcome: any failure after the browser step restored the token snapshot and manager entry, so a server that accepted the token but failed tools/list discarded a completed consent. Now the probe reports whether initialize succeeded (details["initialized"], read from the claimed MCPServerTask). Failure before that point rolls back as before. Failure after it saves the server config, keeps the tokens, and reports tools unavailable through flow.discovery_error; the card can retry discovery without repeating consent. Catalog install wrote the submitted values to .env before install_entry and the probe ran. The values now live in the secret scope for the duration of the install (get_env_value reads through get_secret, so install_entry finds them without a prompt), and .env is written only after the probe returns tools. A failed probe removes the server block and writes nothing. _probe_tool_names returns None on a failed probe instead of an empty list, so failure and a valid empty listing are distinct. Failure text is redacted before it reaches target detail: every value the card submitted for that target is replaced by exact match, then the pattern redactor runs. required_env now carries the manifest's secret and default flags; Target carries the manifest's post_install text as instructions. The wire contract gains secret, default, instructions and discovery_error; generated TS and OpenRPC regenerated. * fix(connectors): one OAuth callback receiver picker for the connection card The card's authorize target built its redirect from the dashboard web server and raised when none was bound in the process, so a standalone hermes --tui session could never authorize an MCP server. The receiver is now chosen in one place (choose_callback_receiver): a pinned pre-registered client keeps the SDK's own listener on the registered port; a client-advertised loopback URI is used as-is and its callback arrives through the mcp.servers.oauth.callback relay; otherwise the backend binds a one-shot loopback listener and feeds it into the flow. The dashboard route stays with the dashboard web page, which cannot bind a port. tui_gateway/mcp_oauth_sessions.py had a second copy of the loopback listener and a _worker that referenced _probe_with_rollback, set_hermes_home_override, reset_hermes_home_override and Path without importing them, so every RPC-started flow raised NameError. Both are deleted; start_flow spawns run_worker directly and uses the same receiver picker. Flow registration is shared (register_flow / finish_flow) so a card-started flow with a client URI is reachable by the relay. Under an SSH session with no client listener the attempt's detail carries the existing paste-the-redirect instructions. Nothing on the card path opens a browser. * feat(desktop): setup-form modal for MCP connection cards The connection card rendered an MCP server's setup fields inline: every field as a password input, no default value, no instructions. A URL such as the n8n MCP server URL was typed blind, and the manifest's setup text never reached the user. Two components carry the form now. SetupFieldList renders the ordered fields the backend declares (a plain field as text prefilled with the manifest default, a secret field masked and empty). SetupFormDialog composes it with a one-line title, the manifest instructions, an inline error for a failed attempt, and Cancel / Connect. The row's Install action opens the dialog when the target has fields; Connect sends {status: approved, env}; Cancel sends {status: skipped}. A failed attempt keeps the dialog open with the draft intact. The draft lives in the dialog component only; nothing reaches the store or the resume snapshot. Once the backend publishes the authorization URL the dialog shows it as text with an Open in browser button. Nothing opens a browser on a state change: the Try again path on both cards used to open the re-minted link at once; it now waits for the row's update frame and the user's click. The store types gain the wire's secret, default, instructions and discoveryError fields and normalise them; a connected target with discoveryError renders as authorized with tools unavailable. * feat(tui): connection card in the Ink TUI The Ink TUI had no client for the connection operation: connection.request, connection.update, connection.respond and the pending_connection resume snapshot were unhandled, so a manage_connections call in a TUI session could only print a link through the model. connectionOperationStore.ts holds the backend snapshot: a request opens only for a new operation id, an update applies only to the live operation with a higher seq, a settled update freezes the id so a late request frame cannot reopen the card. The gateway event handler feeds it; session resume hydrates it from pending_connection. connectionSetupOverlay.tsx renders one callout in the prompt zone: a one-line title, the manifest instructions, every field (plain rows prefilled with the default, secret rows masked), then a Connect / Cancel selector. Connect sends the draft through connection.respond; Cancel skips the active target. When the backend publishes the authorization URL the callout shows it as text under "Press Enter to open in browser"; Enter is the only thing that opens it. A failed attempt unlocks the fields with the draft kept. An accepted secret renders as "Set" and is never echoed. A target authorized without tools shows that state and Continue. The overlay joins the existing input-owner set in overlayStore so typing and other prompts are blocked while it is up. * feat(cli): connection panel in the classic CLI The classic hermes CLI passed no connection_callback, so a manage_connections call could only print an authorization link through the model and could not take a setup value at all. The CLI now renders the connection operation as a prompt_toolkit panel, on the same queue mechanism as the clarify panel: the agent thread's callback opens the panel, blocks until the first decision, then returns so the operation's watch loop runs; every later state reaches the panel through ConnectionOperation.on_change (installed only when no gateway hook is set, restored on close). Panel: one-line title, the manifest instructions, one row per field (plain rows prefilled with the default, secret rows masked in the input buffer and rendered as "Set" once accepted), then Connect / Cancel. Connect sends the draft through apply_answer; Esc skips the active target; Ctrl-C sets the tool-thread interrupt so the operation settles as interrupted. Once the backend publishes the authorization URL the panel shows it with the target detail and "Press Enter to open in browser"; Enter is the only thing that opens it. A failed attempt unlocks the fields with the draft kept; an authorized target without tools offers Retry discovery / Continue. Up/Down move between rows, Left/Right toggle the action, and the panel joins the blocking-overlay guards so chat input, history and voice stay out while it is up. The single-query (headless) mode passes no callback, as it does for clarify. * feat(connectors): register a connected MCP server and report its tools in the result After a successful install or authorization the target carried the probe's tool names and the model was told the tools "become available on your next turn". MCP tool schemas are deferrable by construction, so nothing about them lives in the sent tool array; a server registered in the scoped registry is callable through tool_describe/tool_call in the same turn. The operation now registers the server (register_mcp_servers under the owner's home scope) once authorization is committed, records the registered names on the target, and the settled result carries a tools_listing block in the deferred-catalog format plus a note that the tools are callable now. A registration failure keeps the target connected with tools: [] and a sanitized discovery_error. agent.tools and the system prompt are untouched; the between-turns refresh updates the catalog block as before. The card gate no longer asks for the desktop platform. Every surface that renders the card attaches a connection callback (Desktop, the Ink TUI, the classic CLI); registry dispatch and messaging sessions attach none and keep the link result. Pre-commit rollback in probe_with_rollback used restore(only_if_absent=True), which skips the rollback when a token file exists. On a first-ever authorization the only file is the one this attempt wrote, so a token the resource rejected was kept. Live E2E (controlled provider answering 403 to the issued token) showed the row fail and the token survive; the rollback now restores the snapshot outright, and the same run shows the token file removed. * fix(connectors): install an OAuth catalog entry through the card's own flow The card's install ran install_entry and then a plain probe. For an OAuth entry that probe has no card flow around it: in the desktop backend it failed at once ("non-interactive environment and no cached tokens"), and in the classic CLI it saw a TTY, opened a browser by itself and drew install_entry's curses tool checklist over the panel. Since a probe failure is now an error, the failed install was rolled back with _remove_mcp_server, and authorize refuses a server that is not configured. A clean home had no path to a connected OAuth entry, which is 55 of the 65 catalog entries. Found by the live three-surface run. An install now builds the entry's configuration in memory (card_install_config: no prompts, no probe, no checklist; a prior tool selection or the manifest's curated filter applies). An OAuth entry starts the same flow authorize uses with that configuration and the setup values in the attempt's secret scope, so the row reaches the URL step, and the configuration and the setup values are saved together when initialize accepts the token. Every other entry is probed in memory and saved after the server answers. A failure writes nothing, so a failed reinstall keeps the previous configuration, and the failed row asks for its fields again so the card can reopen the form over the draft it kept. * fix(agent): make a server connected in this turn callable in this turn tool_describe and tool_call resolve names inside the agent's toolset selection, which is fixed when the agent is built. A server that manage_connections had just registered was therefore "not found" for the rest of the turn whose result calls its tools available, and stayed out of the next turn's catalog too. The live Desktop run showed it: the result listed mcp__fx_oauth_fields__echo and the tool_describe that followed answered not_found. The executor now adds the MCP servers the call connected to the selection. Only the selection changes; agent.tools does not, so the sent tool schema bytes stay the same. A selection of None (every toolset) and the no_mcp sentinel are left alone. * fix(cli): reopen the connection form when a required field is still empty When the backend refuses an answer because a required field is missing it keeps the row pending and names the fields. The panel mapped that frame to its waiting phase, which draws the detail line and nothing else, so the user saw "waiting for FX_API_KEY" with no fields and no buttons until the 300 s deadline. The panel now returns to the form on the first missing field, over the draft it kept. * test(connectors): the no-card path is the one with no callback attached The card gate is "a connection callback is attached" since the classic CLI got its own panel. This test still attached one under platform "cli" and expected no card, so it opened a real operation, waited out the 300 s deadline and failed. It now drives the path that has no card: no callback. * fix(connectors): order a cancel against the commit and stop deleting a working grant Found by the live runs on Desktop, the Ink TUI and the classic CLI. A user's skip did not stop the OAuth attempt. With the worker parked in the token request, the row settled "skipped" and the token was written 33 s later. A skip now cancels the attempt, and one lock orders that cancel against the commit: the attempt is either canceled with the earlier tokens restored, or committed and kept. When the commit won, the skipped row says the authorization was kept. An interrupted turn cancels its attempts the same way. Every attempt began by deleting the saved tokens, so retrying discovery for an authorized server demanded consent again, and a cancel in between left no grant at all. The card's flow now connects with the saved tokens first, with no browser step; only when they do not work does it replace them. The RPC session surface keeps the old behavior because its caller waits for an authorization URL. A second Connect from the form a failed row reopened was dropped without a frame, which left the Desktop dialog with Connect loading and Cancel disabled until the deadline. An approval on a failed or expired row is now a retry with the new values. The reported tool names came from the registration call, which returns nothing for a server the process already holds. A retry after a failed listing therefore said "no tools" while the server had them. The names are read from the registry, and a parked server is woken first. An authorization that commits after its card closed by deadline was never registered, so the next turn still could not reach it. The runner keeps such attempts, and the between-turns refresh adopts the ones that were approved. An error with no message reached the user as a class name ("CancelledError"). The tool description still said a server's tools arrive on the next turn. * fix(desktop): let an OAuth install open its link, and keep the form's draft An install of an OAuth catalog entry with no setup fields reached the URL step and gave the user nothing to click: an initiated install was always drawn as a disabled spinner, and the only other place the link is shown is the setup dialog, which opens for entries with fields. That is most of the catalog. An initiated row that carries a link now offers Open, whatever the action. The setup dialog reset its draft whenever the field list changed identity. The backend sends a fresh list with every frame and an empty one while an attempt runs, so a failed Connect erased what the user had typed. Fields now only fill in what the draft lacks, and closing the dialog drops the draft. The settled summary dropped the "tools unavailable" fact, and the live row offered a Try again for that state which the gateway refuses for a connected target. The summary keeps the fact and the row offers no dead control. * fix(tui): keep the typed draft when a Connect fails The overlay reset its draft whenever required_env changed identity, and every backend snapshot delivers a freshly parsed array. A failed Connect therefore came back as a form with the default region and an empty secret. The draft now resets per target only, and a snapshot's fields fill in what the draft lacks. * fix(cli): show the URL step and start an install that has no fields The panel applied the user's answer and then set its phase to "waiting". The backend applies the answer on the same thread and its change hook had already set the next phase, so the URL step of an OAuth install and the reopened form for a missing field were both overwritten, and the card sat on "Waiting…" until the deadline. The waiting phase is now set before the answer is applied. A pending install or enable with no fields opened in the waiting phase, which sends no approval, so the flow never started. It now opens on Connect/Cancel. A target that already carries its link opens on the URL step. Connect on a failed row re-ran the attempt without the values now in the draft; it sends them. The authorized-without-tools phase offered a "Retry discovery" that a connected target cannot run inside the same operation; it offers Continue. * fix(connectors): a newer OAuth attempt replaces the older one for the same server A card that closed by deadline leaves its worker waiting on the browser for up to 300 s, and a tampered callback leaves one waiting too. A retry or a new operation for the same server then ran beside it: both wrote the same token files, and the older one's rollback could write over the newer attempt's grant. The newest attempt per home and server is recorded. Starting one cancels the older attempt, takes over its pre-attempt snapshot so a later failure still restores the state from before either, and the older attempt's rollback leaves the files alone. * fix(desktop): label an install's link control "Open in browser" The control that hands an install's authorization link to the browser reused the action's verb, so the row showed "Install" before the click and "Install" again at the link step, told apart only by the cue. It now reads "Open in browser", the label the setup dialog already uses for the same act. Authorize keeps its verb. * docs(mcp): describe the setup card on the desktop, the terminal UI and the CLI The MCP guide said the chat install exists only in the desktop app and that the CLI relays commands. All three surfaces now show the same card: fields, Connect or Cancel, an authorization link the user opens, one save when the server has accepted the token, and tools the agent can call in the same turn. The tools reference gains the result fields (tools, tools_listing, discovery_error) and the no-card behavior of an OAuth install. * fix(cli): keep the layout hook callable without a connection widget The CLI panel commit added connection_widget to _build_tui_layout_children as a required keyword. That method is the documented override point for wrappers, and five existing tests (extension hooks, prompt stash, subagent dock) call it without the new argument, so CI failed with a TypeError. The argument is now optional, like the other widgets added after the hook was published; a missing widget is left out of the layout. The settled tool result also carried the target's setup instructions. Those are the card's text for the user, and a catalog entry's notes can predate this flow ("restart your session so the tools are loaded"), which contradicts a result that says the tools are callable now. The model-facing result drops them; the card payloads keep them. This restores test_connector_local_batches. * fix(connectors): wait for an in-progress registration before reporting a server's tools Found with the real Vercel MCP server. Saving the configuration wakes the config watcher, which starts its own connect for the new server. The operation's registration call then skips the server as "already connecting" and returns at once, so the card settled "connected" with no tools, and the 214 tools were registered three seconds later. With no names in the result the model searched, found the hosted connector of the same vendor and asked the user to connect that instead. The registered names are now read once the registration has finished: while another task is connecting the server the read waits (30 s at most), a parked server is woken once, and a server that finished registering with no tools is still a valid empty list.
38 KiB
38 KiB
sidebar_position, title, description
| sidebar_position | title | description |
|---|---|---|
| 3 | Built-in Tools Reference | Authoritative reference for Hermes built-in tools, grouped by toolset |
Built-in Tools Reference
This page documents Hermes' built-in tools, grouped by toolset. Availability varies by platform, credentials, and enabled toolsets.
Quick counts (current registry): ~100 tools — 10 browser tools (core) + 2 CDP-gated browser tools + 5 browser-vault tools + browser_exec, 4 file tools, 4 Home Assistant tools, 2 terminal tools (terminal, process_manage), 11 desktop-GUI tools (read_terminal, close_terminal, desktop_preview, drive_preview, annotate_preview, read_window_below, focus_pane, react_to_message, gui_tour, show_tip, apply_layout — desktop-app sessions only), 2 web tools, 5 Feishu tools, 7 Spotify tools (registered by the bundled spotify plugin), 5 Yuanbao tools, 14 kanban tools (registered when the kanban dispatcher spawns the agent), 1 project tool (desktop_project; desktop/GUI sessions), 2 Discord tools, 3 video tools (video_generate, xai_video_edit, xai_video_extend), and a handful of standalone tools (memory, clarify, delegate_task, execute_code, cronjob_manage, session_search, skill_view/skill_manage/skills_list, text_to_speech, image_generate, vision_analyze, video_analyze, todo_list, computer_use, x_search).
:::tip MCP Tools
In addition to built-in tools, Hermes can load tools dynamically from MCP servers. MCP tools appear with the prefix mcp__<server>__ (e.g., mcp__github__create_issue for the github MCP server). See MCP Integration for configuration.
:::
browser toolset
| Tool | Description | Requires environment |
|---|---|---|
browser_back |
Navigate back to the previous page in browser history. Requires browser_navigate to be called first. | — |
browser_click |
Click on an element identified by its ref ID from the snapshot (e.g., '@e5'). The ref IDs are shown in square brackets in the snapshot output. Requires browser_navigate and browser_snapshot to be called first. | — |
browser_console |
Get browser console output and JavaScript errors from the current page. Returns console.log/warn/error/info messages and uncaught JS exceptions. Use this to detect silent JavaScript errors, failed API calls, and application warnings. Requi… | — |
browser_get_images |
Get a list of all images on the current page with their URLs and alt text. Useful for finding images to analyze with the vision tool. Requires browser_navigate to be called first. | — |
browser_navigate |
Navigate to a URL in the browser. Initializes the session and loads the page. Must be called before other browser tools. For simple information retrieval, prefer a lightweight retrieval tool when one is available (faster, cheaper). Use browser tools when you need… | — |
browser_press |
Press a keyboard key. Useful for submitting forms (Enter), navigating (Tab), or keyboard shortcuts. Requires browser_navigate to be called first. | — |
browser_scroll |
Scroll the page in a direction. Use this to reveal more content that may be below or above the current viewport. Requires browser_navigate to be called first. | — |
browser_snapshot |
Get a text-based snapshot of the current page's accessibility tree. Returns interactive elements with ref IDs (like @e1, @e2) for browser_click and browser_type. full=false (default): compact view with interactive elements. full=true: comp… | — |
browser_type |
Type text into an input field identified by its ref ID. Clears the field first, then types the new text. Requires browser_navigate and browser_snapshot to be called first. | — |
browser_vision |
Take a screenshot of the current page so you can inspect it visually. Use this when you need to understand what the page looks like — especially for CAPTCHAs, visual verification challenges, complex layouts, or cases where the text snapshot misses important visual information. On native-vision models the screenshot is attached directly; otherwise falls back to an auxiliary vision mo… | — |
browser toolset (CDP-gated tools)
These two tools live in the browser toolset but only register when a Chrome DevTools Protocol endpoint is reachable at session start — via /browser connect, browser.cdp_url config, a Browserbase session, or Camofox.
| Tool | Description | Requires environment |
|---|---|---|
browser_cdp |
Send a raw Chrome DevTools Protocol command. Escape hatch for browser operations not covered by the higher-level browser_* tools. See https://chromedevtools.github.io/devtools-protocol/ |
CDP endpoint |
browser_dialog |
Respond to a native JavaScript dialog (alert / confirm / prompt / beforeunload). Call browser_snapshot first — pending dialogs appear in its pending_dialogs field. Then call browser_dialog(action='accept'|'dismiss'). |
CDP endpoint |
clarify toolset
| Tool | Description | Requires environment |
|---|---|---|
clarify |
Ask the user a question when you need clarification, feedback, or a decision before proceeding. Supports three modes: 1. Single-select multiple choice — up to 4 choices; the user picks one or types their own answer via a 5th 'Other' option. 2. Multi-select multiple choice — multi_select=true renders checkboxes and returns a list of selected choices. 3. Open-ended — no choices; the user types a free-form response. Choices are ordered best-first, so the first one is labelled (Recommended) on every surface and is the default highlight; the label is presentation only and is stripped from the answer the agent reads. On the classic CLI multi-select uses Space-to-toggle checkboxes; on messaging platforms without native checkbox UIs the user replies with comma/space-separated numbers (e.g. "1, 3") or the option text. |
— |
Asking multiple questions at once
The clarify tool also accepts a questions array (2–5 independent questions, each with its own choices and multi_select) so the agent can batch several clarification needs into a single prompt instead of asking sequentially. The result is a responses array in the same order, with each question's id (when supplied) echoed back.
Per-surface behavior:
- Desktop shows every question on one card. Picks and typed answers stage locally, and one Confirm and continue button (enabled once every question has an answer) submits the whole batch. Staged answers stay editable until that confirm. Skip cancels the whole batch.
- TUI and CLI show a compact status list (
✓answered /▸active /·pending) with only the active question's choices expanded. Enter locks the active answer and jumps to the next unanswered question; Tab moves between questions to answer in any order; Esc cancels the batch. - Messaging platforms (Telegram, Discord, …) fall back to asking the questions one at a time through the existing single-question prompt. If the user stops responding, the remaining questions are not sent.
If the prompt times out part-way, answers the user already locked are kept: the tool result carries them plus "timed_out": true, with the unanswered entries left blank, so the agent can distinguish a deliberate skip from an absent user. On messaging platforms the result also carries a "notice" saying why the wait ended ([user did not respond within Nm], or [clarify prompt could not be delivered] when the platform rejected the card — Hermes first retries the question as a plain numbered-list message, and only reports this when that fails too; [clarify prompt could not be delivered: no chat surface] when the run has no chat to prompt in), so an undelivered prompt is never reported as user inactivity.
connections toolset
One tool for both kinds of external app. A target is a managed connector ("gmail" or
{"name": "gmail"}, authorized through the Nous gateway) or a local MCP server
({"name": "linear", "mcp": true}, an entry in mcp_servers).
| Tool | Description | Requires environment |
|---|---|---|
manage_connections |
Managed actions: status, connect, reconnect (repairs only what is not connected; force: true restarts a working one). MCP actions, for mcp: true targets only: install a catalog entry, enable a disabled configured server, authorize (OAuth). In the desktop app, the terminal UI and the classic CLI every action shows a card and blocks until each target is connected, skipped, or the 300-second deadline passes; the result lists targets as connected, skipped or not_connected and carries no link. An MCP install collects the entry's setup values in the card, runs the entry's OAuth from the card when it has one (the user opens the link; nothing opens by itself), and saves configuration, tokens and values together when the server accepts the token. A connected MCP target carries tools (the registered names) and tools_listing, and those tools are callable through tool_describe/tool_call in the same turn. A target with discovery_error is authorized but its tools could not be listed; call install or authorize for it again to retry discovery without new consent. On surfaces with no card (messaging, scripted dispatch) managed targets return a connect_url per app for the user to open, and an MCP target runs at once: authorize and an OAuth install return the authorization URL, other installs and enable report the outcome, and a missing credential comes back as failed naming the variable to set. Cannot disconnect or revoke an account. |
— |
The deadline for one call is five minutes, fixed by the backend when the call starts;
reopening the chat or restarting the desktop never extends it. The tool is present only when the
Nous Portal has enabled connectors for the signed-in account (the managed_tools claim on its
token). Other sessions do not see it.
code_execution toolset
| Tool | Description | Requires environment |
|---|---|---|
execute_code |
Run a Python script that can call Hermes tools programmatically. Use this when you need 3+ tool calls with processing logic between them, need to filter/reduce large tool outputs before they enter your context, need conditional branching (… | — |
cronjob toolset
| Tool | Description | Requires environment |
|---|---|---|
cronjob_manage |
Unified scheduled-task manager. Use action="create", "list", "update", "pause", "resume", "run", or "remove" to manage jobs. Supports skill-backed jobs with one or more attached skills, and skills=[] on update clears attached skills. Cron runs happen in fresh sessions with no current-chat context. |
— |
delegation toolset
| Tool | Description | Requires environment |
|---|---|---|
delegate_task |
Spawn subagents in isolated contexts; each gets its own conversation, terminal session, and toolset, and only its final summary returns to you. Provide 'goal' for a single task or 'tasks' for a parallel batch (limits and nesting rules… | — |
feishu_doc toolset
Scoped to the Feishu document-comment intelligent-reply handler (gateway/platforms/feishu_comment.py). Not exposed on hermes-cli or the regular Feishu chat adapter.
| Tool | Description | Requires environment |
|---|---|---|
feishu_doc_read |
Read the full text content of a Feishu/Lark document (Docx, Doc, or Sheet) given its file_type and token. | Feishu app credentials |
feishu_drive toolset
Scoped to the Feishu document-comment handler. Drives comment read/write operations on drive files.
| Tool | Description | Requires environment |
|---|---|---|
feishu_drive_add_comment |
Add a top-level comment on a Feishu/Lark document or file. | Feishu app credentials |
feishu_drive_list_comments |
List whole-document comments on a Feishu/Lark file, most recent first. | Feishu app credentials |
feishu_drive_list_comment_replies |
List replies on a specific Feishu comment thread (whole-doc or local-selection). | Feishu app credentials |
feishu_drive_reply_comment |
Post a reply on a Feishu comment thread, with optional @-mention. |
Feishu app credentials |
file toolset
| Tool | Description | Requires environment |
|---|---|---|
patch |
Targeted find-and-replace edits in files. Use this instead of sed/awk in terminal. Uses fuzzy matching (9 strategies) so minor whitespace/indentation differences won't break it. Returns a unified diff. Auto-runs syntax checks after editing… | — |
read_file |
Read a text file with line numbers and pagination. Use this instead of cat/head/tail in terminal. Output format: 'LINE_NUM|CONTENT'. Suggests similar filenames if not found. Use offset and limit for large files. Reads exceeding ~100K characters are truncated on a line boundary and return a next_offset. Jupyter notebooks (.ipynb), Word documents (.docx), and Excel workbooks (.xlsx) a… | — |
search_files |
Search file contents or find files by name. Use this instead of grep/rg/find/ls in terminal. Ripgrep-backed, faster than shell equivalents. Content search (target='content'): Regex search inside files. Output modes: full matches with line… | — |
write_file |
Write content to a file, completely replacing existing content. Use this instead of echo/cat heredoc in terminal. Creates parent directories automatically. OVERWRITES the entire file — use 'patch' for targeted edits. For an existing file, call read_file first: write_file refuses (file untouched) when the task has no current full read/write of the file or the file changed on disk since — on refusal, read_file, merge, retry. Auto-runs syntax checks on .py/.json/.yaml/.toml and other linted languages; only NEW errors introduced by the write are surfaced. | — |
homeassistant toolset
| Tool | Description | Requires environment |
|---|---|---|
ha_call_service |
Call a Home Assistant service to control a device. Use ha_list_services to discover available services and their parameters for each domain. | — |
ha_get_state |
Get the detailed state of a single Home Assistant entity, including all attributes (brightness, color, temperature setpoint, sensor readings, etc.). | — |
ha_list_entities |
List Home Assistant entities. Optionally filter by domain (light, switch, climate, sensor, binary_sensor, cover, fan, etc.) or by area name (living room, kitchen, bedroom, etc.). | — |
ha_list_services |
List available Home Assistant services (actions) for device control. Shows what actions can be performed on each device type and what parameters they accept. Use this to discover how to control devices found via ha_list_entities. | — |
computer_use toolset
| Tool | Description | Requires environment |
|---|---|---|
computer_use |
Background desktop control via cua-driver — screenshots (SOM / vision / AX), click / drag / scroll / type / key / wait, list_apps, focus_app. Does NOT steal the user's cursor or keyboard focus. Works with any tool-capable model. macOS, Windows, and Linux. | cua-driver on $PATH (install via hermes tools). |
:::note
Honcho tools (honcho_profile, honcho_search, honcho_context, honcho_reasoning, honcho_conclude) are no longer built-in. They are available via the Honcho memory provider plugin at plugins/memory/honcho/. See Memory Providers for installation and usage.
:::
image_gen toolset
| Tool | Description | Requires environment |
|---|---|---|
image_generate |
Generate images from text prompts (text-to-image) or edit/transform an existing image (image-to-image) via the user-configured backend (FAL.ai, OpenAI, OpenAI Codex auth, xAI, Krea). Pass image_url to edit an image and reference_image_urls for style references; omit both for text-to-image. The model is user-configured and not selectable by the agent. Returns a single image URL or local path. |
FAL_KEY / OPENAI_API_KEY / Codex OAuth / xAI OAuth / KREA_API_KEY |
kanban toolset
Registered when the agent is either (a) spawned by the kanban dispatcher (HERMES_KANBAN_TASK env set) or (b) running in a profile that explicitly enables the kanban toolset. Task-scoped workers use lifecycle tools for their assigned task; orchestrator profiles additionally get board-routing tools like kanban_list and kanban_unblock. See Kanban Multi-Agent for the full workflow.
| Tool | Description | Requires environment |
|---|---|---|
kanban_show |
Show the active kanban task assigned to this worker (title, description, comments, dependencies). | HERMES_KANBAN_TASK or kanban toolset |
kanban_list |
List board tasks with filters. Orchestrator-only; hidden from dispatcher-spawned task workers. | profile with kanban toolset |
kanban_complete |
Mark the current task done with a structured handoff payload (results, artifacts, follow-ups). | HERMES_KANBAN_TASK or kanban toolset |
kanban_block |
Block the current task on a question for the user — the dispatcher pauses, surfaces the question, and resumes once a human replies. | HERMES_KANBAN_TASK or kanban toolset |
kanban_request_review |
Hand the implementation to a reviewer with summary, optional structured metadata, and an optional reviewer profile. Moves the same task to review; it is not a block and does not affect block-loop accounting. |
HERMES_KANBAN_TASK or kanban toolset |
kanban_request_changes |
Reviewer verdict for an actively claimed review run. Closes the review run, reapplies parent gating, and routes the task back to the original implementer without using a block. | HERMES_KANBAN_TASK or kanban toolset |
kanban_heartbeat |
Send a progress heartbeat during a long-running operation so the dispatcher knows the worker is still alive. | HERMES_KANBAN_TASK or kanban toolset |
kanban_comment |
Add a comment to the task thread without changing its state — useful for surfacing intermediate findings. | HERMES_KANBAN_TASK or kanban toolset |
kanban_create |
Fan out child tasks from the current task. Used by orchestrators and follow-up-spawning workers. | HERMES_KANBAN_TASK or kanban toolset |
kanban_link |
Link tasks with a parent → child dependency edge. | HERMES_KANBAN_TASK or kanban toolset |
kanban_unblock |
Move a blocked task to ready when all parents are done, or todo while any parent remains open. Orchestrator-only; hidden from dispatcher-spawned task workers. |
profile with kanban toolset |
kanban_attach |
Attach a file to a task by passing its bytes inline (base64). Stored as a real attachment under the task's attachments dir, capped at 25 MB. | HERMES_KANBAN_TASK or kanban toolset |
kanban_attach_url |
Attach a file to a task by URL — Hermes downloads it server-side and stores it as a real attachment (capped at 25 MB). Only http/https URLs. | HERMES_KANBAN_TASK or kanban toolset |
kanban_attachments |
List the files attached to a task: id, filename, content_type, size, uploader, and the absolute on-disk path. | HERMES_KANBAN_TASK or kanban toolset |
project toolset
Tools for driving desktop Projects — named, multi-folder workspaces. Registered when the project toolset is enabled (primarily the desktop app / dashboard surfaces).
| Tool | Description | Requires environment |
|---|---|---|
desktop_project |
One action enum for the three Project verbs: create makes a desktop Project (a named workspace) and switches this chat into it — pass path to anchor it to a repo/folder; list shows the desktop Projects and which one is active; switch moves this chat into an existing Project (by name, slug, or id), moving the session workspace to the project's primary folder. |
— |
memory toolset
| Tool | Description | Requires environment |
|---|---|---|
memory |
Save important information to persistent memory that survives across sessions. Your memory appears in your system prompt at session start -- it's how you remember things about the user and your environment between conversations. WHEN TO SA… | — |
session_search toolset
| Tool | Description | Requires environment |
|---|---|---|
session_search |
Search past sessions stored in the local session DB, or scroll inside one. FTS5-backed retrieval; returns actual messages from the DB (no LLM calls). Four shapes: discovery (pass query), scroll (pass session_id + around_message_id), read (pass session_id only), browse (no args). Discovery supports time bounds (after/before — ISO dates or relative durations like 7d, 24h, 2w) and exclude_session_ids for iterative re-finding. |
— |
skills toolset
| Tool | Description | Requires environment |
|---|---|---|
skill_manage |
Manage skills (create, update, delete). Skills are your procedural memory — reusable approaches for recurring task types. New skills go to ~/.hermes/skills/; existing skills can be modified wherever they live. Actions: create (full SKILL.m… | — |
skill_view |
Skills allow for loading information about specific tasks and workflows, as well as scripts and templates. Load a skill's full content or access its linked files (references, templates, scripts). First call returns SKILL.md content plus a… | — |
skills_list |
List available skills (name + description). Use skill_view(name) to load full content. | — |
terminal toolset
| Tool | Description | Requires environment |
|---|---|---|
process_manage |
Manage background processes started with terminal(background=true). Actions: 'list' (show all), 'poll' (check status + new output), 'log' (full output with pagination), 'wait' (block until done or timeout), 'kill' (terminate), 'write' (sen… | — |
terminal |
Execute shell commands on a Linux environment. Filesystem persists between calls. Set background=true for long-running servers. Set notify_on_complete=true (with background=true) to get an automatic notification when the process finishes — no polling needed. Do NOT use cat/head/tail — use read_file. Do NOT use grep/rg/find — use search_files. |
— |
desktop_ui toolset
Enabled for sessions whose source is the Hermes desktop app, on any backend it is connected to (local, SSH, URL, or Hermes Cloud). Absent from CLI, TUI, messaging, and cron sessions.
| Tool | Description | Requires environment |
|---|---|---|
read_terminal |
Read what's currently shown in the in-app terminal pane of the Hermes desktop GUI (the embedded shell beside this chat). | — |
close_terminal |
Close the read-only terminal tab for a background process in the Hermes desktop GUI. Does NOT kill the process — only drops the tab/view; use process_manage(action='kill') to stop it. | — |
desktop_preview |
Drive the preview pane beside the chat in the Hermes desktop app: open a web URL, localhost dev-server URL, or file path (HTML renders live); close the whole pane (omit url) or one tab inside it (pass the URL or file path); read what the pane currently shows — the in-app Browser's page text (URL + title + rendered text, pageable with start/count) or a file/artifact tab's identity. |
— |
drive_preview |
Interact with the page open in the in-app browser: elements inventories what's clickable and typable (each with a ref that names it, like btn-sign-in or inp-email, plus role, label, and value), then click, hover, type, scroll, and press act on a ref, and back/forward/reload drive the pane's history. The pointer and keyboard are real input, so hover menus open. A ref lasts until the page navigates, including across a re-render that rebuilds the element, so after the first inventory every action answers with just a delta — what was added, removed, changed, or rebound — instead of the whole page again. |
— |
annotate_preview |
Outline an element in the in-app browser and leave the mark up until it's removed — the deliberate counterpart to the transient cues drive_preview draws as it works. add marks a ref with an optional short label, remove takes one down, clear takes them all. Marks follow their element and vanish with it, so a navigation clears them. |
— |
read_window_below |
Identify the OS window directly underneath the Hermes desktop window — app name, title, bounds (metadata only, never pixels). On macOS, other apps' titles appear only when Screen Recording is already granted; the tool never prompts for it. | — |
focus_pane |
Reveal and focus a pane in the Hermes desktop app (chat, files, terminal, review, sessions). | — |
react_to_message |
React to a message with a single emoji, iMessage-tapback style. Opt-in via Settings → Appearance (display.message_reactions). |
— |
gui_tour |
Give a live guided tour: dim the screen, highlight an element, and attach a narrated popover (driver.js). Works on the Hermes app's own UI and on any page open in the preview pane; targets discovers what's on screen, show narrates step-by-step, start hands the user Next/Prev controls. |
— |
show_tip |
Point at one element with a small accent bubble and an arrow — the quiet sibling of gui_tour, with no dimming, no spotlight, and no Next/Prev. Same data-tour handles and the same tour(action='targets') discovery call. |
— |
apply_layout |
Apply a saved layout preset to the Hermes desktop app when the user asks to rearrange the workspace. Built-ins: default (chat + sidebars), focus (chat only), terminal-deck, quad; plugin/user presets by id. To reveal ONE pane, use focus_pane instead. |
— |
Tours
The gui_tour tool discovers its own targets — call action='targets' and it returns every addressable element on screen with a selector, a label, and a stable flag. Stable selectors key off identity (data-tour, id, data-testid, aria-label) and survive a re-render; positional nth-child paths don't, so stable ones sort first and should be preferred.
To give an element a durable handle of your own, mark it up:
<div data-tour="composer">…</div>
Handles are applied at the primitive, not the call site, so one edit names every instance. The ones that already exist:
| Handle | What it names |
|---|---|
overlay-nav |
the left nav of any route overlay (settings, cron, profiles, agents) |
nav-<id> |
one row in that nav — nav-models, nav-appearance, … |
field-<schemaKey> |
one settings row, by its config key — field-model, field-provider, … |
page-tabs |
the filter tabs on any PageSearchShell page (artifacts, skills, …) |
artifact-card |
an artifact card in the grid |
When adding a surface, tag its shared primitive the same way rather than tagging screens one by one — that keeps the tour vocabulary small and stops selectors from rotting.
The same engine backs curated (non-agent) tours in the desktop app, so a feature can ship its own walkthrough:
import { startTour, showTourStep, stopTour } from '@/lib/tour'
startTour([
{ selector: '[data-tour="composer"]', title: 'Composer', text: 'Type here.' },
{ selector: '[data-tour="files"]', title: 'Files', text: 'Browse your project.' }
])
A step can also move the app to where its target lives, and the tour puts things back when it ends:
startTour([
{ navigate: '/artifacts', selector: '[data-tour="page-tabs"]', title: 'Filters', text: '…' },
{ pane: 'sessions', selector: '[data-slot="sidebar"]', title: 'Sessions', text: '…' }
])
navigate takes a route path and pane a desktop pane name. Both run as the step is entered, targets that mount late are waited for, and closing the tour — by any route, including Esc — returns to wherever it started.
Pass 'preview' as the second argument to run against the page in the preview pane instead of the app.
Tips
A tip is a tour step without the production: one bubble, one arrow, no scrim and nothing to page through. It's the right weight for a sentence that would be clearer with a finger on the thing it's about — "the model name is a button" — where dimming the whole app would not be.
The show_tip tool takes the same selectors gui_tour(action='targets') reports, so
discovery is one call for both, and the durable data-tour handles above name
targets for either. One tip is on screen at a time; a new one replaces the last.
The app can also show its own, walking a built-in catalog of app features in order, paced like a game's loading-screen tips rather than a notification: a few minutes into a launch at the earliest, then at most one every six hours, and only at a genuinely idle moment. A tip from Hermes shares that cooldown, so it also buys the user six hours of quiet from the rotation. The rotation is a single lap: each catalog tip shows once, whether it timed out or was closed with the ✕, and once every tip has had its turn the app goes quiet. The settings row starts the lap over.
Both tips and tours are on by default and switched off in Settings → Appearance
(display.in_app_tips, display.in_app_tours). Off covers Hermes as well as
the app: the switch reaches the connected gateway's config and the tool leaves
the model's schema, so the agent is never told about a surface it isn't allowed
to use. Like every schema change, that lands on the next session — a running
conversation keeps the toolset it started with, and the app declines the call in
the meantime.
todo toolset
| Tool | Description | Requires environment |
|---|---|---|
todo_list |
Manage your task list for the current session. Use for complex tasks with 3+ steps or when the user provides multiple tasks. Call with no parameters to read the current list. Items may nest: an item's optional parent field points at another item's id, making it a subtask — surfaces render the tree indented. |
— |
vision toolset
| Tool | Description | Requires environment |
|---|---|---|
vision_analyze |
Analyze images using AI vision. On vision-capable main models, returns the raw image pixels as a multimodal tool result so the model sees them natively on its next turn. On text-only main models, falls back to an auxiliary vision model that describes the image and returns the description as text. Tool signature is identical either way. | — |
video toolset
Opt-in toolset (not loaded in the default hermes-cli set). Add via --toolsets video or include video in your toolsets: config.
| Tool | Description | Requires environment |
|---|---|---|
video_analyze |
Analyze video content from a URL or file path — captions, scene breakdowns, key timestamps, and visual descriptions. | — |
video_gen toolset
Opt-in toolset (not loaded in the default hermes-cli set). Add via --toolsets video_gen or enable it in hermes tools → Video Generation, which also walks you through picking a backend.
Backends ship as plugins under plugins/video_gen/<name>/:
- xAI Grok-Imagine — text-to-video and image-to-video (SuperGrok OAuth or
XAI_API_KEY). - FAL.ai — Veo 3.1, Pixverse v6, Kling 3.0 / O3 (requires
FAL_KEY). - OpenRouter — every generative model on OpenRouter's video API (Veo 3.1, Sora 2 Pro, Kling 3, Seedance 2, Wan 3, Hailuo 3, Grok Imagine, FLUX 3 Video, …); text-to-video, image-to-video and reference-to-video; catalog and per-model limits fetched live (requires
OPENROUTER_API_KEY, billed to your OpenRouter credit). - DeepInfra — live
video-gencatalog over the OpenAI-compatible videos endpoint (requiresDEEPINFRA_API_KEY).
The single video_generate tool covers both modalities — pass image_url to animate a still, omit it to generate from text alone. The active backend auto-routes to the right endpoint. As with image_generate, the model is user-configured (video_gen.model) and not selectable by the agent — none of the video tools take a model argument. The tool's description is rebuilt at session start to reflect the active backend's actual capabilities (modalities, aspect ratios, resolutions, duration range, max reference images, audio support). See Video Generation Provider Plugins for backend authoring.
| Tool | Description | Requires environment |
|---|---|---|
video_generate |
Generate a video from a text prompt (text-to-video) or animate a still image (image-to-video) using the user's configured video generation backend. Pass image_url to animate that image; omit it to generate from text alone. The backend auto-routes to the right endpoint. Returns either an HTTP URL or an absolute file path in the video field. |
Active video_gen plugin + its credential (e.g. XAI_API_KEY, FAL_KEY) |
xai_video_edit |
Edit an existing video with xAI Imagine. Provider-specific (separate from video_generate). video_url must be the public HTTPS MP4 URL from a prior Imagine result. |
xAI Imagine credentials (SuperGrok OAuth or XAI_API_KEY) |
xai_video_extend |
Extend an existing video with xAI Imagine. Provider-specific (separate from video_generate). video_url must be the public HTTPS MP4 URL from a prior Imagine result. |
xAI Imagine credentials (SuperGrok OAuth or XAI_API_KEY) |
web toolset
| Tool | Description | Requires environment |
|---|---|---|
web_search |
Search the web for information. Returns up to 5 results by default with titles, URLs, and descriptions. Accepts an optional limit (1-100, default 5). The query is passed through to the configured backend, so operators such as site:domain, filetype:pdf, intitle:word, -term, and "exact phrase" may work when the backend supports them. |
EXA_API_KEY or PARALLEL_API_KEY or FIRECRAWL_API_KEY or TAVILY_API_KEY or PERPLEXITY_API_KEY or KEENABLE_API_KEY |
web_extract |
Extract content from web page URLs. Returns clean page content in markdown/text (no LLM summarization — fast). Also works with PDF URLs (arxiv papers, documents) — pass the PDF link directly. Pages within the char budget (default 15000) return whole; larger pages return a head+tail window with a footer pointing at the full text saved on disk. Max 5 URLs per call. | EXA_API_KEY or PARALLEL_API_KEY or FIRECRAWL_API_KEY or TAVILY_API_KEY or PERPLEXITY_API_KEY or KEENABLE_API_KEY |
x_search toolset
| Tool | Description | Requires environment |
|---|---|---|
x_search |
Search X (Twitter) posts, profiles, and threads using xAI's built-in x_search Responses tool. Read-only public X discovery for current discussion, reactions, or claims on public X (not general web pages). Does not post, reply, like, DM, upload media, delete, or inspect the authenticated X account — those need a separate authenticated X API surface (e.g. the xurl skill). Off by default — opt in via hermes tools → 🐦 X (Twitter) Search. Schema is only registered when xAI credentials are configured (check_fn-gated). |
XAI_API_KEY or xAI Grok OAuth (SuperGrok / Premium+) login |
tts toolset
| Tool | Description | Requires environment |
|---|---|---|
text_to_speech |
Convert text to speech audio. Returns a MEDIA: path that the platform delivers as a voice message. On Telegram it plays as a voice bubble, on Discord/WhatsApp as an audio attachment. In CLI mode, saves to ~/voice-memos/. Voice and provider… | — |
discord toolset
Registered on the hermes-discord platform toolset (gateway only). Uses the same bot token as the messaging adapter.
| Tool | Description | Requires environment |
|---|---|---|
discord |
Read and participate in a Discord server. Actions include search_members, fetch_messages, send_message, react, fetch_channel, list_channels, and more. |
DISCORD_BOT_TOKEN |
discord_admin toolset
Registered on the hermes-discord platform toolset. Moderation actions require the bot to hold the matching Discord permissions.
| Tool | Description | Requires environment |
|---|---|---|
discord_admin |
Manage a Discord server via the REST API: list guilds/channels/roles, create/edit/delete channels, manage role grants, timeouts, kicks, and bans. | DISCORD_BOT_TOKEN + bot permissions |
spotify toolset
Registered by the bundled spotify plugin. Requires an OAuth token — run hermes auth spotify once to authorize.
| Tool | Description | Requires environment |
|---|---|---|
spotify_playback |
Control Spotify playback, inspect the active playback state, or fetch recently played tracks. | Spotify OAuth |
spotify_devices |
List Spotify Connect devices or transfer playback to a different device. | Spotify OAuth |
spotify_queue |
Inspect the user's Spotify queue or add an item to it. | Spotify OAuth |
spotify_search |
Search the Spotify catalog for tracks, albums, artists, playlists, shows, or episodes. | Spotify OAuth |
spotify_playlists |
List, inspect, create, update, and modify Spotify playlists. | Spotify OAuth |
spotify_albums |
Fetch Spotify album metadata or album tracks. | Spotify OAuth |
spotify_library |
List, save, or remove the user's saved Spotify tracks or albums. | Spotify OAuth |
hermes-yuanbao toolset
Registered only on the hermes-yuanbao platform toolset. Yuanbao is Tencent's chat app; these tools drive its DM/group/sticker APIs.
| Tool | Description | Requires environment |
|---|---|---|
yb_query_group_info |
Query basic info about a group (called "派/Pai" in the app): name, owner, member count. | Yuanbao credentials |
yb_query_group_members |
Query members of a group (for @-mentions, finding a user by name, listing bots). |
Yuanbao credentials |
yb_send_dm |
Send a private/direct message to a user in a group, with optional media files. | Yuanbao credentials |
yb_search_sticker |
Search the built-in Yuanbao sticker (TIM face) catalogue by keyword. | Yuanbao credentials |
yb_send_sticker |
Send a built-in sticker to the current Yuanbao chat. | Yuanbao credentials |