fix(tts): validate a plugin's stream rate once; keep nous off plugin routing

Independent-review findings on the plugin PCM streaming path:

- stream_sample_rate was checked, then re-read and int()-ed in the
  adapter: NaN/inf raised out of resolve_streaming_provider (the speaker
  then played nothing) and 0.5 became sample_rate=0. Validate once
  (finite, >= 1 Hz) and hand that int to _PluginPCMStreamer.
- `nous` is an alias of the openai path in _get_provider but not a
  reserved name, so a pinned `tts.streaming.provider: nous` could stream
  a plugin registered as `nous` while the sync path speaks OpenAI.
- Document that stream() must be thread-safe: the speaker prefetches up
  to 3 sentences, unlike the serialized synthesize() path.
This commit is contained in:
Teknium
2026-10-05 23:48:57 -07:00
parent 6fda0f3a03
commit f9798bcf9b
6 changed files with 29 additions and 20 deletions

View File

@@ -34,7 +34,10 @@ class TTSProvider(CatalogProviderBase):
and make ``stream(text, format="pcm", ...)`` yield raw int16 little-endian mono
PCM at that rate. Both are read (with :meth:`is_available`) each time a reply
resolves its streamer, so they may be properties reflecting live state; a
missing rate falls back to per-sentence :meth:`synthesize`.
missing rate falls back to per-sentence :meth:`synthesize`. Unlike
:meth:`synthesize`, ``stream()`` may run concurrently for consecutive
sentences of one reply (the speaker prefetches up to 3), so it must be
thread-safe.
"""
streams_pcm: bool = False

View File

@@ -268,6 +268,8 @@ def test_opted_in_plugin_streams_pcm_in_the_configured_voice(monkeypatch, stream
(_PCMPlugin(streams_pcm=False), {}),
(_PCMPlugin(rate=None), {}),
(_PCMPlugin(rate=0), {}),
(_PCMPlugin(rate=0.5), {}),
(_PCMPlugin(rate=float("nan")), {}),
(_PCMPlugin(available=False), {}),
(_PCMPlugin(), {"providers": {"fake-pcm": {"type": "command", "command": "say {input_path}"}}}),
])

View File

@@ -175,10 +175,9 @@ class _PluginPCMStreamer(StreamingTTSProvider):
"""A plugin ``TTSProvider`` that opted into raw PCM (``streams_pcm``; see
``tools.tts_tool_plugins._plugin_pcm_streaming_provider``) behind this ABC."""
def __init__(self, provider: Any, tts_config: Dict):
def __init__(self, provider: Any, sample_rate: int, tts_config: Dict):
super().__init__(tts_config, tts_config.get(provider.name) or {})
self._provider = provider
self.sample_rate = int(provider.stream_sample_rate)
self._provider, self.sample_rate = provider, sample_rate
@staticmethod
def available() -> bool:
@@ -191,8 +190,8 @@ class _PluginPCMStreamer(StreamingTTSProvider):
def _plugin_streamer(name: str, tts_config: Dict) -> Optional[StreamingTTSProvider]:
provider = _plugin_pcm_streaming_provider(name, tts_config)
return _PluginPCMStreamer(provider, tts_config) if provider is not None else None
found = _plugin_pcm_streaming_provider(name, tts_config)
return _PluginPCMStreamer(*found, tts_config) if found is not None else None
def resolve_streaming_provider(

View File

@@ -7,11 +7,13 @@ tool module stays importable without the plugin machinery).
from __future__ import annotations
import logging
from typing import Any, Dict, Optional
import math
from typing import Any, Dict, Optional, Tuple
from tools.tts_command_provider import (
BUILTIN_TTS_PROVIDERS, DEFAULT_COMMAND_TTS_OUTPUT_FORMAT, _get_named_provider_config,
_is_command_provider_config)
from tools.tool_backend_helpers import NOUS_MANAGED_PROVIDER
logger = logging.getLogger("tools.tts_tool")
@@ -35,7 +37,7 @@ def _plugin_route_key(provider: str, tts_config: Dict[str, Any]) -> Optional[str
"""Normalized *provider* when a plugin may service it, else None: built-in names never reach the
registry and a same-named ``type: command`` provider wins (config is more local than a plugin)."""
key = (provider or "").lower().strip()
if not key or key in BUILTIN_TTS_PROVIDERS:
if not key or key in BUILTIN_TTS_PROVIDERS or key == NOUS_MANAGED_PROVIDER: # nous = the openai path
return None
if _is_command_provider_config(_get_named_provider_config(tts_config, key)):
return None
@@ -52,12 +54,12 @@ def _plugin_voice_kwargs(tts_config: Dict[str, Any]) -> Dict[str, Any]:
"speed": float(speed) if isinstance(speed, (int, float)) else None}
def _plugin_pcm_streaming_provider(provider: str, tts_config: Dict[str, Any]):
"""The plugin that services *provider* when it opted into raw-PCM streaming, else None.
def _plugin_pcm_streaming_provider(provider: str, tts_config: Dict[str, Any]) -> Optional[Tuple[Any, int]]:
"""``(plugin, sample_rate)`` when the plugin servicing *provider* opted into raw-PCM streaming.
Opt-in = ``streams_pcm`` truthy AND a positive ``stream_sample_rate`` AND ``is_available()`` —
all read at resolve time, so per-profile state (key present, user opted out) belongs there.
A missing rate is refused loudly: PCM played at a guessed rate garbles speech.
Opt-in = ``streams_pcm`` truthy AND ``stream_sample_rate`` a finite number >= 1 Hz AND
``is_available()`` — read once per resolve, so per-profile state (key present, user opted out)
belongs there. A bad rate is refused loudly: PCM played at a guessed rate garbles speech.
"""
key = _plugin_route_key(provider, tts_config)
if key is None:
@@ -67,14 +69,14 @@ def _plugin_pcm_streaming_provider(provider: str, tts_config: Dict[str, Any]):
if plugin is None or not plugin.streams_pcm or not plugin.is_available():
return None
rate = plugin.stream_sample_rate
except Exception as exc: # noqa: BLE001 — a broken plugin falls back to per-sentence synthesis
logger.debug("plugin TTS streaming lookup failed for '%s': %s", key, exc)
except Exception: # third-party plugin code: a broken plugin falls back to per-sentence synthesis
logger.debug("plugin TTS streaming lookup failed for '%s'", key, exc_info=True)
return None
if isinstance(rate, bool) or not isinstance(rate, (int, float)) or rate <= 0:
logger.warning("TTS provider '%s' sets streams_pcm without a positive stream_sample_rate; "
"using per-sentence synthesis instead", key)
if isinstance(rate, bool) or not isinstance(rate, (int, float)) or not math.isfinite(rate) or rate < 1:
logger.warning("TTS provider '%s' sets streams_pcm without a valid stream_sample_rate (%r); "
"using per-sentence synthesis instead", key, rate)
return None
return plugin
return plugin, int(rate)
def _dispatch_to_plugin_provider(text: str, output_path: str, provider: str, tts_config: Dict[str, Any]) -> Optional[str]:

View File

@@ -98,6 +98,9 @@ plugin streams first. Under `tts.streaming.provider: auto` it is tried only afte
the built-in priority list. `streams_pcm`, `stream_sample_rate` and
`is_available()` are read every time a streamer is resolved. If the rate is
missing or the provider reports unavailable, Hermes keeps per-sentence synthesis.
As with the built-in streamers, the speaker pipeline prefetches up to three
sentences, so a plugin's `stream()` must tolerate concurrent calls (the sync
path serializes `synthesize()`; this path does not).
## Gateway streaming (platform adapters)

View File

@@ -469,7 +469,7 @@ Override these on your provider class for richer integration:
- `list_models()` → list of `{id, display, languages, max_text_length}` dicts.
- `get_setup_schema()` → return `{name, badge, tag, env_vars: [{key, prompt, url}]}` to power the picker row in `hermes tools` / `hermes setup`. Without this, the plugin still works but its row in the picker is minimal.
- `stream(text, *, voice, model, format, **extra)` → iterator yielding audio bytes (default raises `NotImplementedError`).
- `streams_pcm = True` + `stream_sample_rate` (Hz) → join the streaming voice path (CLI/TUI voice mode, desktop read-aloud, gateway streaming audio). Hermes then calls `stream(text, format="pcm", voice=..., model=..., speed=...)` with the same `tts.voice` / `tts.model` / `tts.speed` that `synthesize()` gets, and expects raw int16 little-endian mono PCM at that rate. Both attributes and `is_available()` are read each time a reply starts, so they can be properties that reflect live state. Without a positive `stream_sample_rate`, or when `is_available()` is `False`, Hermes keeps synthesizing one sentence at a time.
- `streams_pcm = True` + `stream_sample_rate` (Hz) → join the streaming voice path (CLI/TUI voice mode, desktop read-aloud, gateway streaming audio). Hermes then calls `stream(text, format="pcm", voice=..., model=..., speed=...)` with the same `tts.voice` / `tts.model` / `tts.speed` that `synthesize()` gets, and expects raw int16 little-endian mono PCM at that rate. Both attributes and `is_available()` are read each time a reply starts, so they can be properties that reflect live state. Unlike `synthesize()`, `stream()` can be called for up to three consecutive sentences at once while earlier audio plays, so it must be thread-safe. Without a positive `stream_sample_rate`, or when `is_available()` is `False`, Hermes keeps synthesizing one sentence at a time.
- `voice_compatible` property → set `True` if your output is Opus-compatible and the gateway should deliver it as a voice bubble (default `False` = regular audio attachment).
- `warm()` / `release()` → called when a surface toggles speech output on / when the last lease across surfaces is released, while your provider is the configured `tts.provider` — preload or unload a local model server here. Both default to no-ops; exceptions are logged at debug and never fail the toggle.