Files
hermes-agent/tools/AGENTS.md
unsupportedpastels e36a818033 fix(process): a completion is published once, by its owner, with its final output
Child exit, result finalization, notice publication and output consumption are
separate steps, but several callers treated an earlier one as the last:

- A duplicate _move_to_finished set _completion_event while the owning call was
  still before its notice, so a one-shot parent could exit without the follow-up.
  Only the call that moved the session out of _running releases waiters now, and
  it always does, even when its publish raises.
- _finish_exited ran mark_exited without the session lock, so a kill committed
  between its `killed` check and its writes was overwritten with exited/0.
- After a reconcile flips `exited` ahead of a reader still on its final drain,
  wait/read_log/kill no longer consume the partial output as the completion,
  and poll no longer marks it observed. poll/read_log/kill decide that under
  the same lock hold as their snapshot, so a reader finishing in between cannot
  make an earlier partial snapshot count as the whole output.
- kill_process ignored a failed systemd scope stop. It now returns an error
  with scope_stop_failed whenever the stop fails, including when the root
  exited meanwhile (that proves nothing about detached descendants); the
  session keeps its unit so a later kill retries the stop. An already-exited
  session reports scope_stop_failed too.

Process-tree termination and post-kill survivor checks move into
tools/process_registry_termination.py (ProcessTerminationMixin) to keep
process_registry.py under its size cap. The only change in the move:
_live_descendants logs the traceback of the failure it swallows.

Refs #132358. The races were found by @andrexibiza and @jonpol01 in that thread.
2026-10-05 14:35:39 -04:00

14 KiB

tools/ + toolsets.py + model_tools.py — model tools

Applies on top of the root AGENTS.md: settle the Footprint Ladder before adding anything here. Most capabilities should NOT be core tools. Long-form: website/docs/developer-guide/adding-tools.md, tools-runtime.md.

Registry and discovery

tools/registry.py has no deps and is imported by every tool file; each tools/*.py calls registry.register() at import time; model_tools.py imports the registry and triggers discovery (discover_builtin_tools()), then run_agent.py, cli.py, batch_runner.py, environments/ consume it. Any tools/*.py with a top-level registry.register() is imported automatically — no manual import list. A tool that is a whole package (tools/connectors/) registers from tools/<pkg>/tool.py, the only file discovery scans inside a package; every sibling in the package is a library by construction, and the package needs an __init__.py or discovery skips it with a warning (setuptools would drop it from the wheel). The registry handles schema collection, dispatch (handle_function_call()), availability (check_fn, TTL-cached process-wide), and error wrapping. All handlers return a JSON string.

Adding a core tool (2 files) — only when the user is explicitly contributing a core tool

For custom/local-only tools do NOT edit core: create ~/.hermes/plugins/<name>/plugin.yaml + __init__.py and call ctx.register_tool(...); plugin toolsets are discovered automatically and toggled without touching tools/ or toolsets.py (plugins/AGENTS.md).

  1. tools/your_tool.py:
    from agent.secret_scope import get_secret
    from tools.registry import registry
    def check_requirements() -> bool: return bool(get_secret("EXAMPLE_API_KEY"))
    def example_tool(param: str, task_id: str = None) -> str: return json.dumps({"success": True, ...})
    registry.register(name="example_tool", toolset="example",
        schema={"name": "example_tool", "description": "...", "parameters": {...}},
        handler=lambda args, **kw: example_tool(param=args.get("param", ""), task_id=kw.get("task_id")),
        check_fn=check_requirements, requires_env=["EXAMPLE_API_KEY"])
    
  2. toolsets.py: add the name to _HERMES_CORE_TOOLS (all platforms) or a new toolset. Required — discovery registers the schema, but a tool is only exposed if a toolset names it. _HERMES_CORE_TOOLS is the default bundle every platform's base toolset inherits, not dead code.

Rules for tool code:

  • Schema descriptions must not name tools from other toolsets (browser_navigate saying "prefer web_search"). Those tools may be unavailable (missing key, disabled toolset) and the model hallucinates calls to them. Cross-references are added dynamically in get_tool_definitions() in model_tools.py — see the browser_navigate / execute_code post-processing blocks.
  • Paths in schema descriptions use display_hermes_home() (schema is built at import; under multiplex it shows the launch home, which is display-only). State files use get_hermes_home() at call time, never Path.home()/.hermes and never a module constant, so each served profile gets its own state.
  • No offset/limit on instructional tools (skills, prompts, playbooks) — models read page 1 and skip the rest (root rubric).
  • check_fn answers reachability/opt-in for the profile it runs under, never surface. Results are TTL-cached in registry.py::_check_fn_cache keyed by hermes_home_key(), and one process serves many sessions AND many profiles: a probe reads credentials through agent.secret_scope.get_secret, never bare os.getenv (that answers with the launch profile's .env for everyone). The registry classifies an UnscopedSecretError from current_secret_scope() at the catch site — a boot-time probe with no scope is DEBUG, not a traceback. GUI-only tools go in a named toolset (desktop_ui, project) folded in by _load_enabled_toolsets(platform) (root: capability is a property of the SESSION).
  • Agent-level tools (todo, memory) are intercepted before handle_function_call() via the INLINE_TOOL_EXECUTORS table (agent/inline_tool_executors.py; agent/AGENTS.md).
  • _last_resolved_tool_names is a process-global in model_tools.py; _run_single_child() in delegate_tool.py saves/restores it around child runs — readers may see it stale mid-delegation.
  • New tools integrate with existing setup UX (hermes tools, hermes setup, auto-install) rather than a raw env var; secrets go in OPTIONAL_ENV_VARS (hermes_cli/AGENTS.md).

Toolsets (toolsets.py)

Single TOOLSETS dict. Keys today: browser, clarify, code_execution, cronjob, debugging, delegation, discord, discord_admin, feishu_doc, feishu_drive, file, image_gen, kanban, memory, messaging, moa, rl, safe, search, session_search, skills, spotify, terminal, todo, tts, video, vision, web, yuanbao (don't assert the list in tests). Per-platform enable/disable via hermes tools (curses) or tools.<platform>.enabled/disabled in config.yaml. browser_exec replaces the other browser tools when browser.backend is browser-use.

Backends and providers inside tools/

Several tools front pluggable backends: terminal environments in tools/environments/ (local, docker, ssh, modal, daytona, singularity; terminal_tool_backends.py, tool_backend_helpers.py), browser (browser_tool_*.py: cdp, cloud, install, lifecycle, session, real_profile, vision), MCP client (mcp_tool_*.py: config, discovery, transport, registration, content, errors), TTS (tts_tool_providers.py, tts_command_provider.py), skills hub sources (skills_hub_official.py OptionalSkillSource). Adding a backend = a new sibling or provider entry in the existing table, never an elif on a backend name (root shape rules). Remote-backend file visibility problems are fixed at the mount, not by adding a tool.

Native vision embeds are history, not one-shot payloads. vision_tools.py::_vision_analyze_native (and the browser screenshot twins in browser_tool_vision.py / browser_use_cli.py) bake the image into a tool result that is re-sent on every later API call. Size and repeat policy live in vision_tools_history_budget.py (config section vision: embed_target_bytes, max_calls_per_image); the repeat counter is keyed on (session id, resolved source) so region crops share their file's count, and its default cap applies only inside agent.delegation_context.is_delegated_child_process_context(). Put new embed-cost rules there, never a second counter in a tool.

Every spawn goes through one env builder. environments/local.py::build_subprocess_env (+ hermes_constants.apply_subprocess_home_env, env_passthrough.py::resolve_passthrough_value) is how a terminal, execute_code, background process, delegation child, ACP or MCP stdio child gets its environment; a child that acts FOR the served profile (hermes -p X workers, key_cmd helpers, browser drivers, Bot Chat relay turns) uses environments/local.py:: served_profile_child_env(target_home=, inherit_credentials=): launch-profile .env / TERMINAL_* residue dropped (strip_launch_profile_env), the target home pinned, only the target's own secrets overlaid. os.environ.copy() / dict(os.environ) pins the launch profile; contextvars do not cross process boundaries, so resolve before Popen. A child's UnscopedSecretError is a spawn-site bug, never a reason to add environ fallthrough. New threads from scoped code use agent.memory_provider.spawn_context_thread (a bare threading.Thread drops the scope). MCP trust is a per-profile record: mcp_tool_registration.py:: _record_scope_trust keys trust on the home; a secondary never adopts the launch profile's trust for a same-named server, and mcp_tool_handlers.py::_trust_gate_check consults the calling session's profile.

Background-process teardown signals the parent first. process_registry_termination.py:: ProcessTerminationMixin._terminate_host_pid snapshots the descendants, SIGTERMs only the recorded parent, waits terminal.daemon_term_grace_seconds for it to exit and reap its own children, then SIGTERMs the snapshot survivors and SIGKILLs whatever ignored both (so a supervisor that reaps its tree — a Chromium/Electron browser reaping its zygotes, a shell trap — exits cleanly, while a shell whose children ignore SIGHUP still leaves no orphan). Never SIGTERM descendants before the parent: killing a browser's zygote mid-shutdown turns exit 0 into a SIGTRAP core dump. _stop_systemd_unit (scope teardown, kills the worker cgroup) runs only after that PID kill in kill(), or on a parent already proven dead/recycled (session.exited, _signal_kill recycled-PID path, checkpoint recovery) — it is never the first signal a live parent receives.

Delegation (tools/delegate_tool.py)

Spawns a subagent with isolated context + terminal session; the parent waits for the summary unless background=true, which returns a delegation id and re-enters the result via the async-delegation completion queue. Shapes: single (goal + optional context, toolsets) or batch (tasks: [...], concurrency capped by delegation.max_concurrent_children, default 3). A background batch returns as ONE completion by default; with delegation.independent_completions it is split into completion units (delegate_tool_dispatch._units_of): tasks sharing a group join and report together; each ungrouped task reports alone as it finishes. Units of one call share ONE pool slot (slot_key in async_delegation._dispatch) — never count units against capacity; the executor is sized by live UNITS and the stall clock arms when the runner starts, so a queued unit is never judged stalled. Roles: leaf (default; no delegate_task, clarify, memory, send_message, cronjob; keeps execute_code) and orchestrator (keeps delegate_task; gated by delegation.orchestrator_enabled, bounded by delegation.max_spawn_depth, default 2). Config knobs under delegation:: max_concurrent_children, independent_completions, max_spawn_depth, child_timeout_seconds, orchestrator_enabled, subagent_auto_approve, inherit_mcp_toolsets, max_iterations. Child processes: a child's background processes are killed at its teardown and their notices are suppressed in the parent; process_manage(action="handoff") (children only) flips ProcessSession.owner_task_id to the parent under the registry lock (process_registry.transfer_ownership) so the completion routes and reaps by the new owner; un-handed leftovers land on the result as orphaned_processes, exited-but-never-read notify processes as unread_completions (_ChildRun.account_background_processes, before cleanup kills them). Child kernels: a child's execute_code kernels are keyed <parent-owner>::child::<child-session-id>, pinned against the max_session_kernels LRU cap while the child runs and disposed by cleanup (code_kernel.shutdown_kernels_for_delegated_child) — never let a finished child's kernel squat the cap. output_schema: a miss after the one retry keeps status: completed with the raw text in summary plus schema_valid: false / schema_errors / schema_note — never discard a child's result. Durability: background delegation is process-local; work that must survive restart uses cronjob or terminal(background=True, notify_on_complete=True). heartbeat=N on the same spawn emits a heartbeat event (type on the completion queue; delta output only, HEARTBEAT_MIN_SECONDS floor, one daemon timer thread for all sessions in process_registry.py::ProcessRegistry._heartbeat_loop) — every surface that renders watch_match must render heartbeat (process_registry_notifications.py, gateway/run.py::_drain_gateway_watch_events, tui_gateway/session_notifications.py::_DEDUP_EXTRA_FIELDS). Stop fan-out: a turn's hard interrupt reaches only _active_children; background units are detached at dispatch, so every stop surface (gateway /stop busy AND idle paths, TUI session.interrupt, ACP cancel, CLI /stop via interrupt_all) calls async_delegation.interrupt_for_session too. Depth>0 delegations are always synchronous (_model_background_value), so the stop recurses through the child's own _active_children fan-out; an interrupted entry's summary is the child's last real assistant text (_build_result_entry). API: website/docs/developer-guide/subagent-lifecycle-api.md.

Tests

tests/tools/. Test the handler through the registry (real dispatch), not the bare function only; assert contracts ("every registered tool has a toolset", "no schema description names a tool from another toolset") rather than tool counts. Approval/security-boundary tools are E2E'd with real imports against a temp HERMES_HOME (see tests/tools/test_approval_config_readonly.py).

Surface capability is a property of the SESSION, never of the process env

A tool that works only because of who is on the other end (desktop panes, in-app browser, message reactions, Projects) must resolve availability from the session's own source, not from an env var on the backend. Client and backend are separate machines: the desktop app may drive a locally spawned backend, one over SSH, one behind URL + token, or Hermes Cloud, and only the first two carry HERMES_DESKTOP=1. An env-keyed gate is a silent no-op on the other topologies — the tool is stripped from the schema while the platform hint tells the model it is "inside the Hermes desktop app". The pattern:

  • The toolset is the surface gate. Keep such tools off _HERMES_CORE_TOOLS and in a named toolset (desktop_ui, project); the GUI gateway's _load_enabled_toolsets(platform) folds it in when the session's platform says GUI. One resolver, every topology.
  • check_fn answers reachability or opt-in, not surface. "Is the bridge wired?" — fine. "Was I spawned by Electron?" — not. check_fn results are TTL-cached process-wide (tools/registry.py); a per-session answer does not belong there.
  • Ask which identity you mean. HERMES_DESKTOP=1 legitimately means "this backend was spawned by the app" (cron ticker, web-dist handling). It does NOT mean "a GUI is watching"; the embedded terminal pane (hermes --tui against that backend) is the counterexample.

Test: if the capability still makes sense with the client on another machine, it is session-scoped. Assert the GUI session gets the tool with the env var absent.