fix(bot-relay): a relayed DM into a Bot Chat open in Desktop returns the target's answer, not a receipt

bot_relay.deliver has two live-owner branches: the in-process prompt.submit
handoff (#100523) and the sibling-process mailbox handoff (#113753). Both
answered the sender with a receipt sentence — "Delivered into / Queued for
@x's open Bot Chat; the reply will appear there" — so a bot on another machine
never received the target's answer whenever the target's Bot Chat happened to
be open. The owner's poller already settles a receipt carrying the reply, and
local DMs wait on it (bot_mode_dm._wait_live_dm); the relay just never read it.

Owner-first now: a live owner that advertises a mailbox — this process or a
sibling — gets the DM through it, and the handler waits on the receipt on the
local lane's budget: the reply comes back (a bare silence marker as ""), a
failed or cancelled turn as the typed 5092 refusal, and a DM still unanswered
at the budget is reported queued there with "do not resend". prompt.submit
stays as the fallback for a live session with no mailbox.

Fixes #115316

(cherry picked from commit 1f76e8f39be55ec63d7833da765ad5427a01e090)

Rebuilt onto main as one commit after #116903/#116968/#117055 landed: relay tests guard both child seams; the cross-connection docs bullets are deduplicated.

Co-authored-by: teknium1 <127238744+teknium1@users.noreply.github.com>

tests/tui_gateway/test_bot_relay_methods.py

@@ -17,6 +17,7 @@ import os
 import subprocess
 import sys
 import textwrap
+import threading
 import time
 from pathlib import Path
 from unittest import mock
@@ -250,42 +251,117 @@ def test_deliver_lands_in_live_bot_chat_instead_of_subprocess(home, monkeypatch)
     assert out["reply"] == "pong" and spawned and not submitted
 
 
-def test_deliver_hands_off_to_a_bot_chat_owned_by_another_process(home, monkeypatch):
-    """#113753: the relay RPC lands in whichever backend the Desktop routes the target CONNECTION
-    to, while the Desktop-opened Bot Chat can be live in a sibling process for that profile
-    (per-(connection, profile) pool, per-profile SSH dashboards). That owner's lease refuses the
-    subprocess transport with SESSION_NOT_OWNED, so the handler must hand the DM to the owner
-    through the durable mailbox local DMs use, and never spawn the CLI.
-    """
+def _lease_open_bot_chat(home, *, live_session_id="live-in-other-process"):
+    """A Bot Chat leased by a mailbox-capable live owner in the target's home (real state.db row,
+    real lease) — what a Desktop-opened Bot Chat looks like from the relay handler's side."""
     from hermes_cli.active_sessions import try_acquire_active_session
     from hermes_state import SessionDB
-    from tools import bot_live_delivery as mailbox
 
     ops_home = home / "profiles" / "ops"
     db = SessionDB(db_path=ops_home / "state.db")
     db.create_session(session_id="chat", source="desktop")
     db.set_session_title("chat", "Bot Chat")
     db.close()
-    # The sibling process's lease: a mailbox-capable live owner registered in the target's home.
     lease, refusal = try_acquire_active_session(
         session_id="chat", surface="desktop", config={}, registry_home=ops_home,
-        metadata={"live_session_id": "live-in-other-process", "bot_live_delivery_consumer": True})
+        metadata={"live_session_id": live_session_id, "bot_live_delivery_consumer": True})
     assert refusal is None
-    spawned = []
+    return ops_home, lease
 
+
+def _no_cli_transport(monkeypatch, spawned):
     def _fake_run(argv, *a, **k):
         if argv and argv[0] != "git":
             spawned.append(argv)
         raise AssertionError("the CLI transport collides with the live owner")
 
-    monkeypatch.setattr("hermes_cli.quiet_single_query.run_reported_turn", _fake_run)
+    # Guard the deliver child's runner, whichever this tree has: subprocess.run today, and
+    # quiet_single_query.run_reported_turn once the relay books turns from their report
+    # (#114980) — guarding only the first would let a real ``hermes chat -Q`` child spawn there.
+    monkeypatch.setattr("subprocess.run", _fake_run)
+    monkeypatch.setattr("hermes_cli.quiet_single_query.run_reported_turn", _fake_run, raising=False)
+
+
+def _owner_settles(ops_home, outcome: dict) -> threading.Thread:
+    """Stand in for the owner's poller (session_notifications._poll_bot_live_delivery_once): claim
+    the queued DM, run "the turn", write the terminal receipt."""
+    from tools import bot_live_delivery as mailbox
+
+    def run():
+        owner = mailbox.find_canonical_live_owner(ops_home)
+        deadline = time.monotonic() + 10
+        while time.monotonic() < deadline:
+            claimed = mailbox.claim_pending_delivery(ops_home, owner)
+            if claimed is not None:
+                mailbox.complete_delivery(ops_home, claimed["delivery_id"], **outcome)
+                return
+            time.sleep(0.05)
+
+    thread = threading.Thread(target=run, daemon=True)
+    thread.start()
+    return thread
+
+
+@pytest.mark.parametrize(
+    ("live_here", "outcome", "expect"),
+    [
+        (True, {"status": "settled", "reply": "pong from the open chat"}, ("reply", "pong from the open chat")),
+        (False, {"status": "failed", "error": "Error code: 429 - rate limit exceeded", "reason": "provider_rate_limit"},
+         ("reason", "provider_rate_limit")),
+    ],
+    ids=["answer-from-a-chat-live-here", "typed-failure-from-a-chat-live-elsewhere"],
+)
+def test_deliver_into_an_open_bot_chat_returns_the_owners_answer(home, monkeypatch, live_here, outcome, expect):
+    """#113753 put a relayed DM into the mailbox of a Bot Chat live in a sibling process, and the
+    owner settles a receipt carrying the reply — the receipt local DMs wait on (_wait_live_dm).
+    The relay answered with a receipt sentence instead, so the sending agent never got the target's
+    answer whenever its Bot Chat happened to be open. Now the handler waits on the receipt, on the
+    local lane's budget: the reply comes back, a failed turn as the typed 5092 refusal, and the
+    CLI is never spawned. A mailbox-capable chat live in THIS process takes the same door — one
+    door, one receipt — and prompt.submit (#100523) stays the fallback for a live session with no
+    mailbox (test_deliver_lands_in_live_bot_chat_instead_of_subprocess)."""
+    ops_home, lease = _lease_open_bot_chat(home)
+    spawned, submitted = [], []
+    _no_cli_transport(monkeypatch, spawned)
+    monkeypatch.setitem(
+        srv._methods, "prompt.submit", lambda rid, p: submitted.append(p) or srv._ok(rid, {"status": "streaming"}))
     monkeypatch.setattr(srv, "_profile_home", lambda name: ops_home)
-    monkeypatch.setattr(srv, "_sessions", {})  # THIS process hosts nothing for ops
+    monkeypatch.setattr(srv, "_sessions", (
+        {"live-ops": {"profile_home": str(ops_home), "pending_title": "Bot Chat", "history": []}} if live_here else {}))
+    monkeypatch.setattr("tools.bot_mode_dm._LIVE_WAIT_SECONDS", 10)
+    try:
+        owner = _owner_settles(ops_home, outcome)
+        out = srv._methods["bot_relay.deliver"](1, {
+            "profile": "ops", "message": "ping", "from_profile": "cody", "from_handle": "cody",
+            "from_connection": "conn-a"})
+        owner.join(timeout=10)
+        assert not spawned and submitted == []
+        key, value = expect
+        if key == "reply":
+            assert _result(out)["reply"] == value
+        else:
+            assert out["error"]["code"] == 5092 and out["error"]["data"]["reason"] == value
+    finally:
+        lease.release()
+
+
+def test_deliver_into_a_busy_open_bot_chat_reports_it_queued_and_keeps_the_receipt(home, monkeypatch):
+    """The owner admits at its next idle boundary; a DM not answered within the budget is reported
+    queued there — the record stays for the owner, carrying the message and the relayed sender as
+    the turn author, and the sender is told not to resend."""
+    from tools import bot_live_delivery as mailbox
+
+    ops_home, lease = _lease_open_bot_chat(home)
+    spawned = []
+    _no_cli_transport(monkeypatch, spawned)
+    monkeypatch.setattr(srv, "_profile_home", lambda name: ops_home)
+    monkeypatch.setattr(srv, "_sessions", {})
+    monkeypatch.setattr("tools.bot_mode_dm._LIVE_WAIT_SECONDS", 0.6)
     try:
         out = _result(srv._methods["bot_relay.deliver"](1, {
             "profile": "ops", "message": "ping", "from_profile": "cody", "from_handle": "cody",
             "from_connection": "conn-a"}))
-        assert not spawned and "open Bot Chat" in out["reply"]
+        assert not spawned and "open Bot Chat" in out["reply"] and "Do not resend" in out["reply"]
         (queued,) = [
             r for p in (ops_home / "runtime" / mailbox.DELIVERY_DIR_NAME).glob("*.json")
             if (r := json.loads(p.read_text(encoding="utf-8")))]

tui_gateway/methods_bot_relay.py

@@ -102,10 +102,7 @@ def _(rid, params: dict, _root=_relay_root, _run=_run_delivery,
                                        read_remote_roster(root), local_taken_forms(root))
 
         # When THIS gateway already hosts the target's Bot Chat live, the subprocess transport is
-        # fenced out by the single-owner lease and the payload dropped. Land the DM in the live
-        # session via prompt.submit — the composer's choke point, so role alternation, persistence
-        # and streaming behave as a typed message would.
-        # (Nested per method_ctx rebinding.) See #100523.
+        # fenced out by the single-owner lease and the payload dropped (#100523). See below.
         from tools.bot_mode_probe import BOT_CHAT_TITLE
         live_home = _profile_home(resolved)
         want_home = str(live_home) if live_home is not None else None
@@ -135,9 +132,45 @@ def _(rid, params: dict, _root=_relay_root, _run=_run_delivery,
             author = relaying_principal_author(_principal_digest(identity))
         else:
             author = delivery_turn_author(*(params.get(k) for k in sender_fields))
+
+        # This process's _sessions is not the ownership authority: the Desktop pools one backend per
+        # (connection, profile) and an SSH source runs one remote dashboard per profile, so the
+        # target's Bot Chat can be live in a sibling process on this host while the relay RPC lands
+        # here. The subprocess transport would then be refused SESSION_NOT_OWNED by that owner's
+        # lease (#113753). Hand the DM to the live owner — this process or a sibling — through the
+        # same mailbox local DMs use (tools/bot_mode_dm.py::_run_delivery); its poller admits it at
+        # the next idle boundary and settles a receipt carrying the reply. Local DMs wait on that
+        # receipt (_wait_live_dm); so does this relay, on the same budget, so the sender gets the
+        # target's answer rather than a receipt when its Bot Chat happens to be open.
+        import time
+        from tools.bot_live_delivery import deliver_to_live_owner, find_canonical_live_owner, read_delivery_result
+        from tools.bot_mode_dm import _LIVE_WAIT_SECONDS
+        owner_home = live_home if live_home is not None else Path(_hermes_home)
+        owner = find_canonical_live_owner(owner_home)
+        if owner is not None:
+            record = deliver_to_live_owner(owner_home, owner, message, author=author)
+            deadline = time.monotonic() + _LIVE_WAIT_SECONDS
+            while record["status"] in ("queued", "claimed") and time.monotonic() < deadline:
+                time.sleep(0.5)
+                record = read_delivery_result(owner_home, record["delivery_id"]) or record
+            if record["status"] == "settled":
+                from tui_gateway.prompt_turn import _bot_mode_delivery_text
+                return _ok(rid, {"reply": _bot_mode_delivery_text((record.get("reply") or "").strip(), successful=True)})
+            if record["status"] in ("queued", "claimed"):
+                # Admitted but not answered within the budget: the receipt stays, the turn still runs.
+                reply = (f"Queued for @{resolved}'s open Bot Chat; it runs as that chat's next turn and the reply "
+                         "will appear there. Do not resend.")
+                return _ok(rid, {"reply": reply})
+            from tools.bot_failure_reasons import CANCELLED, classify_agent_error
+            error = str(record.get("error") or f"Bot Chat delivery {record['status']}")
+            reason = record.get("reason") or (CANCELLED if record["status"] == "cancelled" else classify_agent_error(error))
+            return _err(rid, 5092, f"delivery turn failed: {error[-500:]}", data={"reason": reason})
+
         if live_sid:
-            # queued=True: a teammate's DM runs as the NEXT turn and never interrupts or steers a
-            # turn in flight (the default busy mode does); arrivals queue in order.
+            # A live Bot Chat here that advertises no mailbox: land the DM through prompt.submit, the
+            # composer's choke point, so role alternation, persistence and streaming behave as a
+            # typed message would (#100523). queued=True: a teammate's DM runs as the NEXT turn and
+            # never interrupts or steers a turn in flight (the default busy mode does).
             submit_params: dict = {"session_id": live_sid, "text": message, "queued": True}
             if author:
                 submit_params["_turn_author"] = DeliveryAuthor(author)
@@ -147,22 +180,6 @@ def _(rid, params: dict, _root=_relay_root, _run=_run_delivery,
             reply = f"Delivered into @{resolved}'s open Bot Chat; the reply will appear there."
             return _ok(rid, {"reply": reply})
 
-        # This process's _sessions is not the ownership authority: the Desktop pools one backend per
-        # (connection, profile) and an SSH source runs one remote dashboard per profile, so the
-        # target's Bot Chat can be live in a sibling process on this host while the relay RPC lands
-        # here. The subprocess transport would then be refused SESSION_NOT_OWNED by that owner's
-        # lease (#113753). Hand the DM to the live owner through the same mailbox local DMs use
-        # (tools/bot_mode_dm.py::_run_delivery); its poller admits it at the next idle boundary.
-        from tools.bot_live_delivery import deliver_to_live_owner, find_canonical_live_owner
-        owner_home = live_home if live_home is not None else Path(_hermes_home)
-        owner = find_canonical_live_owner(owner_home)
-        if owner is not None:
-            deliver_to_live_owner(owner_home, owner, message, author=author)
-            # The owner's poller admits the mailbox record at its next idle boundary; this
-            # process only queued it, so say so (the in-process branch above really submitted).
-            reply = f"Queued for @{resolved}'s open Bot Chat; it runs as that chat's next turn and the reply will appear there."
-            return _ok(rid, {"reply": reply})
-
         def _detail(p) -> str:
             from tools.bot_failure_reasons import turn_failure_text
             return turn_failure_text(p.stdout, p.stderr)

website/docs/user-guide/bot-mode.md

@@ -164,7 +164,7 @@ Bots message each other with attribution, and you can hand work off from any cha
 - **Remote Bots complete under their titles, from any chat.** The `@` autocomplete lists Bots on your other connected machines as soon as the Desktop is connected — you do not have to open the Bots pane first — and a remote `default` appears under its Bot Mode title (`@cos-bot` for a remote default titled *CoS Bot*, not a second `@hermes`). When two Bots would tag alike, the picker inserts the connection-qualified form (`@cos-bot@<connection>`), which resolves to exactly that machine's Bot. A relayed message signed `Message from 🤖 hermes (@hermes@<connection>)` still renders as an agent notice, not as your own text.
 - **Direct messages** — every Bot Chat carries the `message_agent` tool: a Bot messages a teammate by calling `message_agent(target="researcher", message="…")`. The target is the teammate's profile name, its friendly name (`hermes profile rename` or the Bot Mode title — `Scribe`, `Dr. Foo`) or the `@`-tag the Desktop inserts for it (`@scribe`, `@dr-foo`, `@drfoo`); `@hermes` always means the primary Bot. A profile name is matched first, so it can never be hijacked by another Bot's friendly name, and a friendly name shared by two Bots is refused with the roster instead of guessing. The tool validates the target against the live roster, prefixes the sender's `Message from 🤖 <friendly name> (@<handle>):` attribution automatically, and delivers into the teammate's canonical Bot Chat. Delivery is **fire-and-forget**: the sender gets a *dispatch* acknowledgement (`status: queued` plus a `delivery_id` — and a `process_id` for the background delivery process — means the message was handed to that process, not that it was delivered), finishes its turn, and that process's completion notification carries the outcome — the reply, or the delivery failure. On surfaces that cannot receive completion notifications (an `api_server` session, one-shot runners) the acknowledgement instead carries `reply_delivery: poll`: the sender retrieves the outcome with `process(action="wait", session_id=…)` before ending its turn, and the outcome is also saved into the sender's session transcript as a delivery row when the process exits, so the reply is never silently lost. The notification carries the reply whole up to the message size limit (16,000 characters); a longer one arrives as its tail and says how much was cut (`process(action="log", session_id=…)` has the rest). The message travels as a real parameter (nothing shell-interpreted — quotes, `$(...)`, and backticks arrive verbatim), and the Bot composes its own message rather than forwarding your words. The teammate roster — names **and roles** from each profile's title/description — is part of every Bot Chat's system prompt, so Bots know who does what before choosing a recipient. The tool exists **only** in canonical Bot Chat sessions on Bot-Mode-managed installs; regular chats, group-room member sessions, and CLI sessions never see it.
 
-Local messages also reach a Bot Chat that stays open in Desktop or the TUI. The receiving backend keeps ownership: it reads durable ingress on its existing notification poller, admits immediately when idle, or waits until the running turn and already queued human prompts finish. A `queued` acknowledgement confirms durable admission, **not** a completed reply. The target profile retains the delivery ID and receipt under `runtime/bot_live_delivery/`; `settled` confirms completion. A crashed or cancelled imported turn is not automatically replayed, and pending work pinned to a departed owner remains inspectable rather than being silently rerun. Do not resend a delivery whose outcome is unknown. Older backends without live-delivery capability retain the existing ownership refusal; restart that backend after upgrading.
+Local messages also reach a Bot Chat that stays open in Desktop or the TUI, and so do messages relayed from another machine. The receiving backend keeps ownership: it reads durable ingress on its existing notification poller, admits immediately when idle, or waits until the running turn and already queued human prompts finish. A `queued` acknowledgement confirms durable admission, **not** a completed reply. The target profile retains the delivery ID and receipt under `runtime/bot_live_delivery/`; `settled` confirms completion. A crashed or cancelled imported turn is not automatically replayed, and pending work pinned to a departed owner remains inspectable rather than being silently rerun. Do not resend a delivery whose outcome is unknown. Older backends without live-delivery capability retain the existing ownership refusal; restart that backend after upgrading.
 
 - **Staying silent** — a Bot that has nothing to add may end a turn with one of the [intentional silence tokens](./messaging/index.md#intentional-silence-tokens) (`[SILENT]`, `NO_REPLY`, …). The Bot Chat keeps that turn in its transcript but renders nothing, and a teammate that messaged it gets an empty reply instead of the token. Failed turns and prose that merely mentions a token are shown as-is.
 
@@ -226,10 +226,8 @@ A failed bot turn or relay delivery carries a machine-readable `reason` code alo
 Every gateway you register in **Settings → Connections** — local, remote URL, SSH, Hermes Cloud, docker — is a persistent line the Desktop holds open, and Bot Mode uses those lines for messaging automatically. No extra setup:
 
 - **Rosters propagate on their own.** While the Desktop runs, it periodically tells each connected gateway which agents live on the *other* connections. Every Bot Chat's teammate roster then lists them ("Teammates on OTHER connected machines"), with names, roles, and which machine they're on — and the roster refreshes when agents appear, disappear, or get renamed (capability epoch).
-- **`message_agent` reaches them directly.** A Bot on your laptop messages the cloud agent with `message_agent(target="moxie", …)` exactly like a local teammate. If the same handle exists on several machines, disambiguate with `target="moxie@<connection>"` (the tool's error tells the Bot the exact forms). Delivery rides the Desktop: the sending gateway queues the message, the Desktop relays it to the target connection's own gateway, the target Bot runs a turn in its canonical Bot Chat, and the reply comes back to the sender as the same background completion notification local DMs use (the process that waits for it is a Hermes entrypoint, so it starts under `approvals.single_query_mode: deny` — the default for a Bot's one-shot reply turn — too). Messages to different Bots are delivered side by side, so one Bot's long turn never delays another Bot's mail (or ages it past `bot_mode.envelope_ttl_seconds`); messages to the *same* Bot are delivered in order, one turn at a time.
+- **`message_agent` reaches them directly.** A Bot on your laptop messages the cloud agent with `message_agent(target="moxie", …)` exactly like a local teammate. If the same handle exists on several machines, disambiguate with `target="moxie@<connection>"` (the tool's error tells the Bot the exact forms). Each teammate is listed by the machine's name from **Settings → Connections**, so a Bot reads `@moxie on Homelab` rather than a bare connection id. Delivery rides the Desktop: the sending gateway queues the message, the Desktop relays it to the target connection's own gateway, the target Bot runs a turn in its canonical Bot Chat — as that chat's next turn when it is open in a Desktop — and the reply comes back to the sender as the same background completion notification local DMs use (the process that waits for it is a Hermes entrypoint, so it starts under `approvals.single_query_mode: deny` — the default for a Bot's one-shot reply turn — too; an open chat that has not answered within the live-delivery budget reports the message queued there instead; do not resend). Messages to different Bots are delivered side by side, so one Bot's long turn never delays another Bot's mail (or ages it past `bot_mode.envelope_ttl_seconds`); messages to the *same* Bot are delivered in order, one turn at a time.
 - **The Desktop is the courier.** Cross-connection delivery works while a Desktop that knows both connections is running (it holds the sockets and the credentials — gateways never see each other's auth). If the Desktop is closed mid-delivery, the sender's Bot is told the reply didn't arrive rather than left hanging; a message the Desktop picked up but never handed to the target (it disconnected in between) is offered to the next drain again — once, and only after the Desktop's own delivery deadline (~26 minutes) has passed with no reply, so a slow-but-live turn is never delivered twice. The sender keeps waiting long enough for that second delivery to finish (~52 minutes in all); if it still gets no reply, the sender is told so with reason `delivery_timeout`. While a re-offered message waits for a drain, `bot_mode.envelope_ttl_seconds` applies to it exactly as to a freshly queued one, and the first reply recorded for a message is the one that stands. For always-on machine-to-machine messaging with no Desktop in the loop, register a peer (`hermes peer`, below) — the two routes coexist.
-- **`message_agent` reaches them directly.** A Bot on your laptop messages the cloud agent with `message_agent(target="moxie", …)` exactly like a local teammate. If the same handle exists on several machines, disambiguate with `target="moxie@<connection>"` (the tool's error tells the Bot the exact forms). Each teammate is listed by the machine's name from **Settings → Connections**, so a Bot reads `@moxie on Homelab` rather than a bare connection id. Delivery rides the Desktop: the sending gateway queues the message, the Desktop relays it to the target connection's own gateway, the target Bot runs a turn in its canonical Bot Chat, and the reply comes back to the sender as the same background completion notification local DMs use (the process that waits for it is a Hermes entrypoint, so it starts under `approvals.single_query_mode: deny` — the default for a Bot's one-shot reply turn — too). Messages to different Bots are delivered side by side, so one Bot's long turn never delays another Bot's mail (or ages it past `bot_mode.envelope_ttl_seconds`); messages to the *same* Bot are delivered in order, one turn at a time.
-- **The Desktop is the courier.** Cross-connection delivery works while a Desktop that knows both connections is running (it holds the sockets and the credentials — gateways never see each other's auth). If the Desktop is closed mid-delivery, the sender's Bot is told the reply didn't arrive rather than left hanging. For always-on machine-to-machine messaging with no Desktop in the loop, register a peer (`hermes peer`, below) — the two routes coexist.
 - **A remote `default` goes by its title.** Every machine's `default` is `@hermes`, so a remote one is offered — and reachable — under its Bot Mode title instead (a remote `default` titled *CoS Bot* is `@cos-bot`; with no title, `hermes@<connection>`), and a DM it sends arrives signed with that same reply-safe form rather than a bare `@hermes` that would point back at your own default.
 - **The 10-minute cap bounds the target's turn, not its handoffs.** A relayed message gives the target Bot 10 minutes to finish its turn. When that turn messages a teammate itself, the delivery process stays alive afterwards — bounded by `terminal.oneshot_completion_wait_seconds` — so the teammate's reply can land in its Bot Chat (and, when it arrives in time, becomes the answer relayed back). That wait is never counted against the cap: a turn that finished is reported to the sender with its answer, not as `delivery_timeout`, and its handoff is left to complete.