hermes-agent/website/docs/developer-guide/browser-supervisor.md

---
sidebar_position: 18
title: "Browser CDP Supervisor"
description: "How Hermes detects and responds to native JS dialogs and interacts with cross-origin iframes via a persistent CDP connection."
---

# Browser CDP Supervisor

The CDP supervisor closes two long-standing gaps in Hermes' browser tooling:

1. **Native JS dialogs** (`alert`/`confirm`/`prompt`/`beforeunload`) block the
   page's JS thread. Without supervision, the agent has no way to know a
   dialog is open — subsequent tool calls hang or throw opaque errors.
2. **Cross-origin iframes (OOPIFs)** are invisible to top-level
   `Runtime.evaluate`. The agent can see iframe nodes in the DOM snapshot but
   can't click, type, or eval inside them without a CDP session attached to
   the child target.

The supervisor solves both by holding a persistent WebSocket to the backend's
CDP endpoint per browser task, surfacing pending dialogs and frame structure
into `browser_snapshot`, and exposing a `browser_dialog` tool for explicit
responses.

## Backend support

| Backend | Dialog detect | Dialog respond | Frame tree | OOPIF `Runtime.evaluate` via `browser_cdp(frame_id=...)` |
|---|---|---|---|---|
| Local Chrome (`--remote-debugging-port`) / `/browser connect` | ✓ | ✓ full workflow | ✓ | ✓ |
| Browserbase | ✓ (via bridge) | ✓ full workflow (via bridge) | ✓ | ✓ |
| Camofox | ✗ no CDP (REST-only) | ✗ | partial via DOM snapshot | ✗ |

**Browserbase quirk.** Browserbase's CDP proxy uses Playwright internally and
auto-dismisses native dialogs within ~10ms, so `Page.handleJavaScriptDialog`
can't keep up. The supervisor injects a bridge script via
`Page.addScriptToEvaluateOnNewDocument` that overrides
`window.alert`/`confirm`/`prompt` with a synchronous XHR to a magic host
(`hermes-dialog-bridge.invalid`). `Fetch.enable` intercepts those XHRs before
they touch the network — the dialog becomes a `Fetch.requestPaused` event the
supervisor captures, and `respond_to_dialog` fulfills via
`Fetch.fulfillRequest` with a JSON body the injected script decodes.

From the page's perspective, `prompt()` still returns the agent-supplied
string. From the agent's perspective, it's the same `browser_dialog(action=...)`
API either way.

Camofox is unsupported — no CDP surface, REST-only.

## Architecture

### CDPSupervisor

One `asyncio.Task` running in a background daemon thread per Hermes `task_id`.
Holds a persistent WebSocket to the backend's CDP endpoint. Maintains:

- **Dialog queue** — `List[PendingDialog]` with `{id, type, message, default_prompt, session_id, opened_at}`
- **Frame tree** — `Dict[frame_id, FrameInfo]` with parent relationships, URL, origin, whether cross-origin child session
- **Session map** — `Dict[session_id, SessionInfo]` so interaction tools can route to the right attached session for OOPIF operations
- **Recent console errors** — ring buffer of the last 50 for diagnostics

Subscribes on attach:

- `Page.enable` — `javascriptDialogOpening`, `frameAttached`, `frameNavigated`, `frameDetached`
- `Runtime.enable` — `executionContextCreated`, `consoleAPICalled`, `exceptionThrown`
- `Target.setAutoAttach {autoAttach: true, flatten: true}` — surfaces child OOPIF targets; supervisor enables `Page`+`Runtime` on each

Thread-safe state access via a snapshot lock; tool handlers (sync) read the
frozen snapshot without awaiting.

### Lifecycle

- **Start:** `SupervisorRegistry.get_or_start(task_id, cdp_url)` — called by
  `browser_navigate`, Browserbase session create, `/browser connect`.
  Idempotent.
- **Stop:** session teardown or `/browser disconnect`. Cancels the asyncio
  task, closes the WebSocket, discards state.
- **Dropped endpoint:** after a successful attach the supervisor reconnects with
  backoff (≤10 s) but gives up after `MAX_POST_ATTACH_RECONNECT_FAILURES`
  consecutive failures — one final warning, the thread exits and the registry
  entry is dropped. A dead local Chrome (its task finished) therefore never leaves
  a retrying thread behind; the next browser call starts a fresh supervisor.
- **Rebind:** if the CDP URL changes (user reconnects to a new Chrome), the
  old supervisor is stopped and a fresh one started — state is never reused
  across endpoints.

### Dialog policy

Configurable via `config.yaml` under `browser.dialog_policy`:

- **`must_respond`** (default) — capture, surface in `browser_snapshot`, wait
  for explicit `browser_dialog(action=...)` call. After a 300s safety timeout
  with no response, auto-dismiss and log. Prevents a buggy agent from stalling
  forever.
- `auto_dismiss` — record and dismiss immediately; agent sees it after the
  fact via `browser_state` inside `browser_snapshot`.
- `auto_accept` — record and accept (useful for `beforeunload` where the
  workflow wants to navigate away cleanly).

Policy is per-task; no per-dialog overrides.

## Trusted plugin CDP access

In-process plugin code can send raw CDP commands over a task's supervisor
connection through one public call, instead of writing the supervisor's private
call table or socket:

```python
from tools.browser_supervisor import SUPERVISOR_REGISTRY
from tools.browser_supervisor_capture import CapturedCDPInvalid

cdp = SUPERVISOR_REGISTRY.capture(task_id)  # raises CapturedCDPInvalid if no attached supervisor
reply = cdp.call("Runtime.evaluate", {"expression": "document.title", "returnByValue": True},
                 session_id=cdp.page_session_id, timeout=5)
title = reply["result"]["result"]["value"]
```

- `SUPERVISOR_REGISTRY.capture(task_id, *, timeout=10.0) -> CapturedCDP` pins the
  supervisor's current WebSocket and the default page session attached on it. It
  never starts, reconnects or refocuses a supervisor.
- `CapturedCDP.call(method, params=None, *, session_id=None, timeout=10.0) -> dict`
  returns the raw CDP reply (`{"id", "result"}`). `session_id=None` addresses the
  browser endpoint (`Target.*`); pass `page_session_id` or a session you attached
  yourself for page domains. A CDP error reply raises `RuntimeError`; no reply
  within `timeout` raises `TimeoutError`. The call blocks its thread, so call it
  from a worker thread, never from the supervisor's own event loop.
- `CapturedCDP.is_valid()` stays true until the supervisor reconnects, stops, or
  is replaced in the registry. After that every `call` raises
  `CapturedCDPInvalid`, including a call already in flight when the socket
  drops. A handle never follows the supervisor to a new connection: capture
  again and re-check whatever state you depend on.
- `page_session_id` is the default page session at capture time. A later
  `focus_page` on the supervisor does not change it.
- Sessions you attach (`Target.attachToTarget`) are yours to detach. A reply
  that arrives after `timeout` is dropped, so a timed-out attach can leave a
  session attached until the connection closes; give attaches a generous timeout.

This is a trusted, in-process seam. It grants nothing in-process Python could
not already reach, and it enforces no origin, consent or target-ownership
policy: a plugin that acts on pages must enforce its own. It is not exposed as a
model tool and has no config switch.

## Agent surface

### `browser_dialog` tool

```
browser_dialog(action, prompt_text=None, dialog_id=None)
```

- `action="accept"` / `"dismiss"` → responds to the specified or sole pending dialog (required)
- `prompt_text=...` → text to supply to a `prompt()` dialog
- `dialog_id=...` → disambiguate when multiple dialogs are queued (rare)

Tool is response-only. The agent reads pending dialogs from `browser_snapshot`
output before calling.

### `browser_snapshot` extension

Adds three optional fields to the existing snapshot output when a supervisor
is attached:

```json
{
  "pending_dialogs": [
    {"id": "d-1", "type": "alert", "message": "Hello", "opened_at": 1650000000.0}
  ],
  "recent_dialogs": [
    {"id": "d-1", "type": "alert", "message": "...", "opened_at": 1650000000.0,
     "closed_at": 1650000000.1, "closed_by": "remote"}
  ],
  "frame_tree": {
    "top": {"frame_id": "FRAME_A", "url": "https://example.com/", "origin": "https://example.com"},
    "children": [
      {"frame_id": "FRAME_B", "url": "about:srcdoc", "is_oopif": false},
      {"frame_id": "FRAME_C", "url": "https://ads.example.net/", "is_oopif": true, "session_id": "SID_C"}
    ],
    "truncated": false
  }
}
```

- **`pending_dialogs`** — dialogs currently blocking the page's JS thread.
  The agent must call `browser_dialog(action=...)` to respond. Empty on
  Browserbase because their CDP proxy auto-dismisses within ~10ms.

- **`recent_dialogs`** — ring buffer of up to 20 recently-closed dialogs with
  a `closed_by` tag: `"agent"` (we responded), `"auto_policy"` (local
  auto_dismiss/auto_accept), `"watchdog"` (must_respond timeout hit), or
  `"remote"` (browser/backend closed it on us, e.g. Browserbase). This is
  how agents on Browserbase still get visibility into what happened.

- **`frame_tree`** — frame structure including cross-origin (OOPIF) children.
  Capped at 30 entries + OOPIF depth 2 to bound snapshot size on ad-heavy
  pages. `truncated: true` surfaces when limits were hit; agents needing
  the full tree can use `browser_cdp` with `Page.getFrameTree`.

No new tool schema surface for any of these — the agent reads the snapshot it
already requests.

### Availability gating

Both surfaces gate on `_browser_cdp_check` (supervisor can only run when a CDP
endpoint is reachable). On Camofox / no-backend sessions, the dialog tool is
hidden and the snapshot omits the new fields — no schema bloat.

## Cross-origin iframe interaction

`browser_cdp(frame_id=...)` routes CDP calls (notably `Runtime.evaluate`)
through the supervisor's already-connected WebSocket using the OOPIF's child
`sessionId`. Agents pick frame_ids out of
`browser_snapshot.frame_tree.children[]` where `is_oopif=true` and pass them
to `browser_cdp`. For same-origin iframes (no dedicated CDP session), the
agent uses `contentWindow`/`contentDocument` from a top-level
`Runtime.evaluate` instead — the supervisor surfaces an error pointing at that
fallback when `frame_id` belongs to a non-OOPIF.

On Browserbase, this is the only reliable path for iframe interaction —
stateless CDP connections (opened per `browser_cdp` call) hit signed-URL
expiry, while the supervisor's long-lived connection keeps a valid session.

## File layout

- `tools/browser_supervisor.py` — `CDPSupervisor`, `SupervisorRegistry`, `PendingDialog`, `FrameInfo`
- `tools/browser_supervisor_capture.py` — `CapturedCDP` / `CapturedCDPInvalid`, the trusted-plugin CDP seam behind `SUPERVISOR_REGISTRY.capture`
- `tools/browser_dialog_tool.py` — `browser_dialog` tool handler
- `tools/browser_tool.py` — `browser_navigate` start-hook, `browser_snapshot` merge, `/browser connect` reattach, `_cleanup_browser_session` teardown
- `toolsets.py` — registers `browser_dialog` in `browser`, `hermes-acp`, `hermes-api-server`, and core toolsets (gated on CDP reachability)
- `hermes_cli/config.py` — `browser.dialog_policy` and `browser.dialog_timeout_s` defaults

## Non-goals

- Detection/interaction for Camofox (upstream gap; tracked separately)
- Streaming dialog/frame events live to the user (would require gateway hooks)
- Persisting dialog history across sessions (in-memory only)
- Per-iframe dialog policies (agent can express this via `dialog_id`)
- Replacing `browser_cdp` — it stays as the escape hatch for the long tail (cookies, viewport, network throttling)

## Testing

Unit tests (`tests/tools/test_browser_supervisor.py`) use an asyncio mock CDP
server that speaks enough of the protocol to exercise all state transitions:
attach, enable, navigate, dialog fire, dialog dismiss, frame attach/detach,
child target attach, session teardown. Real-backend E2E (Browserbase + local
Chromium-family browser) is manual — exercise via `/browser connect` to a
live Chromium-family browser and run the dialog/frame test cases described
above.