The hook loop now treats SystemExit like any other callback failure; the
three sibling dispatch loops in this module (event subscribers,
invoke_middleware, system prompt section content) had the same
`except Exception` gap and let a plugin dependency's sys.exit() abandon
the loop and kill the turn. KeyboardInterrupt still propagates on all
of them.
The timeout branch of _run_hook_callback_bounded unconditionally added
gate_key to _hook_abandoned. A worker that finishes between done.wait()
returning False and the caller taking the lock has already popped its
token via _release_token, so nothing would ever clear that entry: the
callback stayed blocked for every later call id until reload with no
thread behind it. Guard the insert on the worker still being registered.
The new test makes the race deterministic by swapping the module's
threading.Event for one whose wait() lets the worker finish and then
reports a timeout, and asserts a fresh call id still runs.
Also pass tool_call_id inline from terminal_tool_result instead of the
conditional dict plumbing: an empty id is already treated as "no
identity" by _hook_call_identity and unknown fields are withheld from
narrow-signature callbacks (same shape as _fire_approval_hook). Update
the stale "(hook_name, id(cb))" comment above _hook_running_callbacks.
The same-call negative control fired two sequential calls with a 0.1 s
timeout, so the first call timed out and the second was dropped by the
60 s suppression window; the identity gate itself was never exercised.
Mirror the positive test instead: a 5 s timeout, two threads with the
same tool_call_id while the callback is held on an Event.
Add one test for the abandoned-worker gate: a hung callback followed by
a call with a fresh tool_call_id, with suppression zeroed, must start
exactly one worker. Reverting the gate change makes it fail.
Concurrent invocations of the same tool in one session collapsed into a single
busy key (hook_name, id(cb)): the second invocation was reported as 'still
running' and dropped. For pre_tool_call a drop is a fail-closed block, so the
gate silenced itself on an ordinary, healthy callback.
Measured on a busy profile: 3574 skip lines and 0 timeout lines in one hour —
every skip was the 'while still running' branch, i.e. pure key collision, not
slowness.
The gate now keys on the call identity that is already in the payload
(tool_call_id, else turn_id, else none — the last case behaves exactly as
before). Suppression stays keyed coarsely on (hook_name, id(cb)): a hung
callback is a fact about the callback, so its back-off must not be diluted
per call.
Refs #98382. Independent of #107894 (that one releases the slot on timeout;
this one stops healthy concurrency from colliding).
(cherry picked from commit 53b3dacd008418fcdf5fa6dfcadde575a35a776e)
Under a running loop `resolve_plugin_command_result` awaited the coroutine on
a raw thread, so an async hook saw the process-default HERMES_HOME and no
secret scope (get_secret -> UnscopedSecretError on a secondary profile).
Run the thread body through `contextvars.copy_context().run`, matching the
bounded hook worker. Also fixes async plugin slash commands the same way.
No runtime consumer read the proxy (terminal_tool/environments call is_interrupted()/set_interrupt()
directly); its only users were tests patching tools.interrupt._interrupt_event, which had no effect on
the code under test. tools/terminal_tool.py's own re-export of the name is owned by another worker.
tools/approval.py no longer re-exports sibling names (approval_context/prompt/floors/detection/
human_wait/smart/gateway_wait); it imports only what it uses. Siblings reference sibling-defined
names directly (module-attribute reads on tools.approval_context so patching the defining module
still works); only facade-owned state (_lock, _gateway_queues, _permanent_approved, _denied,
_denial_breaker_addendum, _gateway_notify_cb) is still read back through tools.approval.
approval_detection calls its own _command_detection_variants instead of late-binding through the facade.
re_register_config_hooks() cleared the entire process-global idempotence
set on every force-reload, so a profile-local plugin force-reload dropped
another live profile's ledger key without touching its still-registered
callback — the next registration call for that profile then appended a
duplicate. Scope the clear to the reloading profile's own home, and give
outbound webhooks the same force-reload restoration shell hooks already
had, since unload() wipes both from the shared _hooks dict.
Seven test files asserted sub-2s wall-clock bounds (elapsed < 0.5/1.0s,
stop(timeout=1.0), event waits of 0.5-2s). Under CI load these fired on
healthy code: main run 33455779041 alone flaked 6 of them in one pass
(observed 1.01s vs 0.5, 1.20s vs 1.0, 3.61s vs 3.0, 1.55s vs 1.0,
stop(1.0) returning False, lease TTL 0.1s expiring before the authority
change was observed).
Per the AGENTS.md flake policy (waits >= 2s), bounds are raised to 5s+
while keeping their teeth: every hang path they guard blocks for 10s+
(release.wait holds), so the loosened bounds still distinguish bounded
from unbounded behavior. The authority-loss test gets a 30s lease TTL so
lease expiry can no longer preempt the authority-change assertion.
New commands and plugins declare argument_mode and desktop availability
on CommandDef / register_command. commands.catalog ships that map so
desktop does not need a second command list.
Allowlist hot-path hooks for abandon-on-timeout, keep subagent_stop on the caller thread, suppress re-fires of hung callbacks, and block tools when pre_tool_call times out.
Adds a `modify` response type to pre_tool_call hooks so a hook can
transform tool arguments before the tool executes, instead of repairing
results afterwards via post_tool_call.
- hermes_cli/plugins.py: _dispatch_pre_tool_call_hooks() fires hooks once
and returns (block_message, modified_args); modify directives
shallow-merge into an accumulated dict built from the original args.
- agent/shell_hooks.py: _parse_response() accepts both the canonical
{"action": "modify", "args": {...}} and Claude Code-compatible
{"decision": "modify", "tool_input": {...}} wire formats.
- model_tools.py, agent/tool_executor.py, agent/agent_runtime_helpers.py:
dispatch sites migrated; modified args applied before execution.
- Docs + 10 new tests (merge semantics, precedence, block interplay).
Salvaged from PR #28953. Best fix for #18988.
Documents and tests the routing contract the sweeper review asked about:
classification records the manifest but does not activate anything.
- model-provider test now exercises providers.get_provider_profile() against
the pip-only name (None today — providers discovery is directory-based)
and asserts the module never leaks into sys.modules via that path.
- new test for the mnemosyne shape: a pip entry point duplicating a
same-name directory provider. The pip copy is classified exclusive and
never imported; the directory copy still activates through
plugins.memory discovery, exactly once.
- _classify_entrypoint_kind docstring now states the activation contract
explicitly: pip-only providers were equally unactivatable pre-change
(both destination systems are directory-only; the
hermes_agent.memory_providers entry-point group has no consumers), so
classification only removes the wasted import. Entry-point activation
is tracked upstream (#40644 for memory); this change is its
prerequisite, preventing double import once it lands.
find_spec() on a dotted module name imports the parent package first,
executing its __init__.py — which is exactly where a provider's heavy
imports typically live (fastembed -> onnxruntime and friends). The
previous classifier only preserved the no-import property for
top-level entry points.
_resolve_module_source() now resolves only the top-level name with
find_spec() (import-free for top-level names) and walks the remaining
dotted segments through submodule_search_locations by hand, mirroring
PathFinder's file conventions (part.py module / part/__init__.py
package). Namespace packages, zipped modules, extension modules, and
anything else unexpected fall back to standalone (the safe default).
.pyc origins map back to source via source_from_cache.
Regression: a dotted entry point whose parent __init__.py writes an
execution marker and imports the child — asserts the parent never
executed and neither module enters sys.modules during classification.
Fails against the previous implementation (marker written), passes now.
Entry-point (pip-installed) plugins exposing register_memory_provider()
or register_provider() + ProviderProfile were treated as plain
standalone plugins and eagerly imported by the general PluginManager,
even though memory and model providers have their own discovery
systems and the module has no register() for the general manager to
call. The import registered nothing and paid the module's full import
cost in every Hermes process (a pip memory provider pulls fastembed ->
onnxruntime, ~60 MB RSS).
Entry-point manifests now get the same source-scan classification as
directory plugins via a shared _detect_kind_from_source() helper: the
module is resolved with importlib.util.find_spec (no import) and its
first 8192 chars are scanned for provider markers. Memory providers ->
kind=exclusive, model providers -> kind=model-provider; both are
recorded for introspection and skipped by the general loader.
Unresolvable or non-Python modules stay standalone (default behavior
unchanged).
Tests: an enabled pip entry-point memory provider is never imported;
a pip entry-point model provider routes to providers/ discovery.
Salvaged from PR #64188 (@Bartok9), re-reviewed against the #64229
ownership ledger (landed in #84923).
Delivery parity (survived):
- Module-level invoke_hook/invoke_middleware/has_hook/has_middleware
lazily run plugin discovery via _delivery_manager(), so surfaces that
never import model_tools (dashboards, TUI slash workers, query mode,
cron, gateway platform events) deliver plugin callbacks instead of
silently dropping them (#50776, #67597, #67890, #50937).
- _delivery_manager() joins any in-flight background discovery first and
tolerates test doubles that monkeypatch get_plugin_manager().
Symmetric force-reload (survived):
- agent/shell_hooks.py gains re_register_config_hooks(); the force branch
of discover_and_load() calls it after a successful sweep, restoring
config.yaml shell hooks that the ledger-driven unload wiped but cannot
restore (they are config-owned, not plugin-owned) (#60036).
- unload(plugin=None) now sweeps pre-ledger _plugin_tool_names entries
out of the process-global tools.registry, mirroring the platform-name
sweep that already existed, so zombie tools cannot survive a force
reload in long-lived pre-ledger processes (#60050).
Superseded by the ownership ledger (dropped from #64188):
- _unload_global_plugin_registrations() bulk tool/platform teardown —
the ledger's reverse-order handle disposal with previous-entry
restoration covers it more precisely.
- tools.registry/platform_registry displaced-entry LIFO restore stacks —
the ledger's restore_registration() identity-checked previous-entry
restoration made them redundant.
- Discovery serialization lock + double-checked singleton — main already
has _discovery_lock on every discover/unload path and a keyed,
lock-guarded per-home manager cache (#24714 concern is covered).
Fixes tracked under #64178 (#50776, #60036, #60050, #24714, #67798,
#50937, #67597, #67890, #31480 — #31480 already handled on main by
_parse_hooks_block warn+suggest).
Second, deeper pass over tools/gateway/hermes_cli plus first pass over
the trees wave 1 missed (acp, acp_adapter, skills, computer_use, docker,
dashboard, conformance, monitoring, secret_sources, hermes_state,
providers). Same rubric as wave 1 (AGENTS.md test policy); security,
alternation/caching invariants, issue-number regressions, and E2E kept.
Real test-quality fixes found and rooted out along the way:
- tests/tools/test_command_guards.py made real auxiliary-LLM HTTPS calls
(DEFAULT_CONFIG smart-approval leaked in) — pinned approval
mode=manual via autouse fixture: 17.4s → 0.4s.
- test_model_switch_custom_providers.py / test_user_providers_model_switch.py
silently probed live provider catalogs (~2s/test) — stubbed
cached_provider_model_ids/provider_model_ids/fetch_api_models.
- test_telegram_noise_filter.py: 15-platform copy-paste matrix over
shared gateway.run logic → 3 representative platforms (55s → 3.9s).
- test_gateway_shutdown.py: stop()'s 5s interrupt-deadline loop spun on
MagicMock agents — interrupt.side_effect now clears _running_agents
(22s → 1.0s).
- test_gateway_inactivity_timeout.py poll-harness timings shrunk 3-5x
(24s → 1.1s); test_mcp_stability.py backoff/SIGTERM-grace sleeps
patched (15.4s → 2.5s); test_async_delegation.py negative-drain wait
5s → 0.5s.
- test_telegram_init_deadline.py: loop-block margin restored to 1.0s
with rationale comment — the watchdog-dump assertion needs the loop
blocked well past deadline+grace under parallel load (flaked once in
the 40-worker verification run at a 0.2s margin).
Verification: full hermetic suite via scripts/run_tests.sh —
2,438 files, 21,718 tests passed, 0 failed, 293.9s wall.
Suite totals vs original baseline: 46,820 → 19,757 test functions
(−57.8%), wall 583.5s → 293.9s (−50%), subprocess CPU 13,564s → 11,623s.
CI slices 2 and 7 caught three tests broken by always-defer:
- /tools (CLI show_tools + TUI gateway tools.show) now passes
skip_tool_search_assembly=True — it's a discovery/inspection surface,
so users verifying an MCP installed must see deferred tools, not a
collapsed bridge row. This also fixes
test_slash_worker_mcp_discovery (profile MCP tool visible in /tools).
- test_plugins.py::test_plugin_tools_in_definitions: 'visible' becomes
'reachable' — direct schema OR listed in the bridge description;
scope-exclusion assertions unchanged (not direct AND not listed).
- test_discord_tool.py dynamic-schema-rebuild test reads the
pre-assembly list (the rebuilt schema is what tool_describe serves).
Banner/status tool counts intentionally keep the post-assembly view —
they reflect what the model actually sees.
Extend the pre_tool_call plugin hook return contract with a new directive:
{"action": "approve", "message": "why this needs human confirmation"}
Previously a pre_tool_call hook could only veto a tool call (action: block)
or allow it silently. It could not escalate to the existing human-approval
flow. This unlocks user-defined runtime approval rules on ANY tool (HTTP
writes, file writes to sensitive paths, email sends), enforced at runtime —
resolving #51221 as a pure plugin, with no core approval.py rule schema.
Mechanism:
- get_pre_tool_call_directive() returns (action, message) for block|approve;
get_pre_tool_call_block_message() kept as a block-only back-compat shim.
- resolve_pre_tool_block() is the single dispatch-site chokepoint: fetches
the directive and, for approve, invokes the human gate; fail-closed to a
block on denial, timeout, or gate exception. ALL FOUR tool-dispatch sites
now call it: tool_executor (concurrent + sequential), agent_runtime_helpers,
and model_tools.handle_function_call.
- request_tool_approval() escalates via the SAME machinery as Tier-2
dangerous commands: session/permanent allowlist, prompt_dangerous_approval
(CLI) / submit_pending (gateway), [o]nce/[s]ession/[a]lways/[d]eny,
timeout fail-closed, approvals.cron_mode for cron contexts.
Architecture: extracted the shared decision core into _run_approval_gate(),
called by BOTH check_dangerous_command() and request_tool_approval() so the
fail-closed / cron / gateway / yolo / persist policy lives in ONE place and
cannot drift. Fixed a latent divergence — the plugin path now honors --yolo.
Approval grain: [a]lways is keyed on tool_name + a hash of the reason (an
explicit plugin rule_key overrides), so distinct reasons on the same tool
persist independently instead of one 'always' blanketing the whole tool.
Non-interactive: cron honors approvals.cron_mode (parity with commands); any
other non-interactive non-gateway context fails CLOSED for the plugin path
(the command path keeps its historical fail-open default, unchanged).
No new config schema, no new env vars, no new hook events.
egilewski found the prior sink gate was transient: it only applied while
PluginManager executed register(ctx). A plugin could defer a direct
registry.register(..., override=True) to a post-load callback/thread, after
the scope was cleared, and still replace a built-in.
Make authorization durable by binding it to where the handler is DEFINED
(handler.__globals__['__name__']) rather than to call timing. At load, each
plugin's module namespace is mapped to its allow_tool_override opt-in in a
table that is never cleared. The sink resolves the handler's owning plugin
module and rejects an override from any plugin namespace without opt-in,
regardless of when or on which thread the call happens. Plugin namespaces
with no recorded policy are treated as not-opted-in (fail-closed). Built-in
and MCP handlers live outside the plugin namespace and are unaffected.
Adds a regression test for the delayed/post-load direct-registry override.
The opt-in gate lived only in PluginContext.register_tool, so a plugin
could bypass it by importing tools.registry and calling
registry.register(..., override=True) directly. Enforce the same gate at
the sink: during plugin load, the registry rejects an override from a
plugin without operator opt-in regardless of the path taken. Built-in and
MCP registrations (no active plugin scope) are unaffected.
Adds a regression test covering the direct-registry bypass.
The tool_override flag landed in v0.14.0 (#26759) so plugins can replace
a built-in tool with their own implementation. It works as advertised
but there is no trust gate, so any enabled third-party plugin can
silently override any built-in like shell_exec, write_file, or web_fetch
and exfiltrate everything the agent invokes through it. The only trace
is a DEBUG-level log line.
Compare with ctx.llm (#23194) which does gate the equivalent privilege
escalation: overriding the provider requires
plugins.entries.<id>.llm.allow_provider_override: true in config.yaml.
The policy shape exists, it just was not extended to tool overrides.
Fix:
* Add PluginToolOverrideError(PermissionError) for the gate failure.
* register_tool() now checks _tool_override_allowed(name) when
override=True. Bundled plugins (manifest.source == 'bundled') are
trusted by default. Every other source requires
plugins.entries.<plugin_id>.allow_tool_override: true in config.yaml.
* fail-closed: if config.yaml cannot be loaded for any reason,
_tool_override_allowed returns False. Same posture as
MSGraphWebhookAdapter.connect() in #22353.
Backwards compatibility:
* Bundled plugins: no change (source == 'bundled' short-circuits the
gate).
* Third-party plugins not using override: no change (gate is only
consulted when override=True).
* Third-party plugins using override: registration fails until the
operator opts in. The error message includes the exact config path
to add, so the fix is one config edit away for legitimate use cases.
Same migration path users went through for allow_provider_override
after #23194 landed.
Regression tests:
* tests/hermes_cli/test_plugins.py::test_register_tool_override_replaces_existing
and ::test_register_tool_override_on_new_name_is_noop_path were
written before the gate existed. Updated their test configs to
include allow_tool_override: true under
plugins.entries.<plugin_id>, mirroring how a legitimate operator
would now grant the privilege.
* New regression test ::test_register_tool_override_blocked_without_operator_opt_in
exercises both the PluginManager-catches-error path (built-in tool is
preserved, attacker plugin is skipped) and the direct-call path
(PluginToolOverrideError is raised with a message that names the
config key to set). Verified the test fails without this fix and
passes with it.
* All 73 tests in test_plugins.py continue to pass.
Add a `pre_verify` user/plugin/shell hook fired once per turn when the agent
edited code and is about to finish, after the existing verify-on-stop guard. A
hook can keep the agent going one more turn (run a check, defer it, tidy the
diff) by returning {"action":"continue","message":...} (the Claude-Code Stop
shape {"decision":"block","reason":...} is accepted too). Hooks receive coding,
attempt, final_response, and sorted changed_paths so they can self-scope and
self-throttle; the path is bounded by agent.max_verify_nudges and preserves
message-role alternation.
Hermes still ships its default coding guidance (agent.verify_guidance, on by
default), but it now rides the evidence-based verify-on-stop missing-evidence
nudge instead of a separate default pre_verify continuation, so it costs no
extra model turn of its own. Guidance reuses the shared utils.is_truthy_value
parser rather than a local copy.
Answers a recurring plugin-author question: how to read the active
profile and drive Hermes from inside a hook callback when ctx._cli_ref
is None (gateway, hermes chat -q, and kanban-spawned worker sessions).
- Adds a 'Act from inside a hook' section to the plugin guide covering
ctx.profile_name and ctx.dispatch_tool as the session-agnostic APIs,
with a kanban_task_blocked example, and notes there is no in-process
slash-command bridge for headless workers (shell out via the terminal
tool instead).
- Adds the three kanban lifecycle hooks to the hook reference table with
their process semantics.
- Pins the contract with a regression test: ctx.dispatch_tool invokes a
tool handler with _cli_ref=None (worker/hook context).
Requested by @Smithangshu on Discord.
Plugins previously had no way to read the active profile name from the
PluginContext. The workaround in the wild — reaching into
ctx._manager._cli_ref — only works in an interactive CLI session;
_cli_ref is None in the gateway and in kanban-spawned worker sessions
(hermes -p <profile> chat -q ...), so the workaround breaks exactly
where multi-profile awareness matters most.
ctx.profile_name wraps hermes_cli.profiles.get_active_profile_name(),
which derives the name from HERMES_HOME and therefore works in every
execution context with zero dependency on _cli_ref.
list_plugins() attribution diffed registry names against all already-loaded
plugins, so when a plugin registered a hook / middleware / tool name an
earlier plugin had already used, the shared name was credited to the first
plugin only and later plugins under-reported (0 hooks) in hermes plugins
list. commands_registered right beside it already attributed correctly by
plugin ownership.
Snapshot per-registry counts before register() and attribute the entries
this plugin's register() actually added (per-registration delta). Add a
regression test: two plugins registering the same hook name are each
credited with 1 hook.
discover_and_load(force=True) cleared every per-plugin registry except
_plugin_platform_names, which register_platform() populates. A platform
plugin disabled between force-rediscovers left a stale name behind, so the
set diverged from the real platform_registry / _plugins state and never
shrank across repeated force passes.
Add the missing clear() and a regression test that seeds every per-plugin
registry, forces a rediscover, and asserts they all empty (so a future
registry addition can't silently leak across a force pass either).
Root-cause hardening for the stranded-empty-registry failure behind
'No web search/extract provider configured': discover_and_load() set
_discovered=True before scanning, so a sweep that raised partway was
swallowed by callers as a warning and every later call early-returned
against an empty registry for the process lifetime. The flag now acts
only as a re-entrancy guard and is reset when the sweep raises, so the
next call retries discovery.
Address the two non-blocking follow-ups from review:
- next_call is now single-use per middleware frame. A second invocation
raises instead of silently re-running the downstream provider/tool, so
the terminal call cannot execute twice via the chain. The error surfaces
through the existing handler, which preserves the first downstream result.
- Request-middleware payload copies go through _safe_copy(), which falls
back to a shallow dict copy when deepcopy() fails on a non-deepcopyable
member (clients, callbacks, file handles) instead of aborting the pass.
Adds regression coverage for both: double next_call() keeps the terminal
single-run, and a non-deepcopyable (threading.Lock) request payload still
runs middleware via the shallow fallback.
Track successful next_call completion separately from invocation so execution
middleware that catches and translates a downstream provider/tool failure does
not accidentally convert that failure into a successful None result.
Also avoid wrapping BaseException from downstream execution, and document the
execution middleware error semantics.
Tests cover:
- pre-next_call middleware failures fail open to the remaining chain
- post-next_call middleware failures preserve the downstream result
- translated downstream failures propagate instead of returning None
- downstream BaseException is not wrapped
Signed-off-by: Bryan Bednarski <bbednarski@nvidia.com>
