c257e9196b
The sequential tool path only noticed a user interrupt after the running tool returned: with the deadline disabled it ran the tool inline (fully blocking), and with a deadline it waited in 5s slices without ever checking agent._interrupt_requested. Any tool without cooperative is_interrupted() polling (image_generate, tts, transcription, skills sync, ...) held the whole turn hostage — the reported symptom was a redirect queued ~40s behind a FAL image generation + upscale pass. Executor backstop (class fix, covers ALL tools): - _run_sequential_tool_execution_middleware always dispatches on the daemon worker (timeout None no longer means inline blocking) and polls the interrupt flag every 1s. - On interrupt: 3s cooperative grace (mirrors the concurrent path), then synthesize a cancelled tool result (_ToolCancelledResult), emit the terminal post_tool_call with status=cancelled, and abandon the worker. - _ToolCancelledResult suppresses downstream post-hook double emission exactly like _ToolTimeoutResult, so an abandoned worker finishing late cannot report success for a cancelled call. - clarify (interactive, _NEVER_PARALLEL_TOOLS) keeps the inline path — it owns its own human wait. Cooperative layer in the reported offender: - image_generation_tool: blind handler.get() (generation + Clarity upscale) replaced with _wait_fal_result(), which polls is_interrupted() in 0.5s slices and raises ImageGenerationInterrupted immediately. - _upscale_image propagates the interrupt instead of swallowing it into the "upscale failed, use original" fallback. Message alternation is preserved: the cancelled result is a normal tool result for the call_id. Sabotage-verified: with the old wait loop restored, the new tests fail (tool blocks full runtime); with the fix they pass in ~4s.
83 lines
2.2 KiB
Python
83 lines
2.2 KiB
Python
"""_wait_fal_result must notice a user interrupt while the FAL job runs."""
|
|
|
|
import threading
|
|
import time
|
|
|
|
import pytest
|
|
|
|
import tools.image_generation_tool as image_tool
|
|
from tools.interrupt import set_interrupt
|
|
|
|
|
|
class _SlowHandler:
|
|
"""Fake FAL handler whose get() blocks like the real SDK."""
|
|
|
|
def __init__(self, delay=30.0, result=None):
|
|
self.delay = delay
|
|
self._result = result if result is not None else {"images": []}
|
|
|
|
def get(self):
|
|
time.sleep(self.delay)
|
|
return self._result
|
|
|
|
|
|
class _FastHandler:
|
|
def __init__(self, result):
|
|
self._result = result
|
|
|
|
def get(self):
|
|
return self._result
|
|
|
|
|
|
@pytest.fixture(autouse=True)
|
|
def _clean_interrupt():
|
|
set_interrupt(False)
|
|
yield
|
|
set_interrupt(False)
|
|
|
|
|
|
def test_wait_fal_result_returns_result():
|
|
result = image_tool._wait_fal_result(_FastHandler({"images": [{"url": "u"}]}))
|
|
assert result == {"images": [{"url": "u"}]}
|
|
|
|
|
|
def test_wait_fal_result_raises_on_interrupt():
|
|
def _interrupt_soon(tid):
|
|
time.sleep(0.2)
|
|
set_interrupt(True, tid)
|
|
|
|
tid = threading.current_thread().ident
|
|
threading.Thread(target=_interrupt_soon, args=(tid,), daemon=True).start()
|
|
|
|
t0 = time.monotonic()
|
|
with pytest.raises(image_tool.ImageGenerationInterrupted):
|
|
image_tool._wait_fal_result(_SlowHandler(delay=30.0), poll_seconds=0.05)
|
|
assert time.monotonic() - t0 < 5.0
|
|
|
|
|
|
def test_wait_fal_result_propagates_handler_error():
|
|
class _ErrHandler:
|
|
def get(self):
|
|
raise ValueError("boom")
|
|
|
|
with pytest.raises(ValueError, match="boom"):
|
|
image_tool._wait_fal_result(_ErrHandler())
|
|
|
|
|
|
def test_upscale_interrupt_propagates(monkeypatch):
|
|
"""_upscale_image must NOT swallow the interrupt into a None fallback."""
|
|
|
|
monkeypatch.setattr(
|
|
image_tool, "_submit_fal_request", lambda *a, **k: _SlowHandler(30.0)
|
|
)
|
|
|
|
def _interrupt_soon(tid):
|
|
time.sleep(0.2)
|
|
set_interrupt(True, tid)
|
|
|
|
tid = threading.current_thread().ident
|
|
threading.Thread(target=_interrupt_soon, args=(tid,), daemon=True).start()
|
|
|
|
with pytest.raises(image_tool.ImageGenerationInterrupted):
|
|
image_tool._upscale_image("https://example.com/x.png", "prompt")
|