Files
hermes-agent/agent
Teknium 00adf69d92 feat(stt): live transcription while you speak on CLI, TUI and Desktop (stt.streaming)
Voice input was record-then-upload on every surface: nothing showed while the user
talked, and the transcript arrived only after the whole take uploaded. With
`stt.streaming: true` the openai, xai and elevenlabs providers now stream microphone
PCM to their realtime endpoints while the user speaks. Partial text renders live and
the final transcript is ready at end of speech.

- tools/transcription_streaming.py: one session API (push_audio / end_audio /
  finalize / cancel, an on_partial callback, a worker thread that owns the socket)
  with three wires, all verified live: the OpenAI Realtime `type: transcription`
  session (gpt-live-transcribe, 16->24 kHz resample since the endpoint refuses
  < 24 kHz), xAI wss /v1/stt with interim results, and ElevenLabs Scribe v2
  realtime. Plugin providers opt in through
  TranscriptionProvider.streaming_capable / open_stream_session.
- tools/voice_mode.py: AudioRecorder taps its capture callback into the session and
  parks it under the WAV path. transcribe_recording uses the live transcript and
  falls back to the file on any failure or empty result, so CLI push-to-talk, the
  continuous VAD loop and the TUI gateway gain streaming with no call-site changes.
- Classic CLI and Ink TUI show partial text in the input placeholder while recording
  (new `voice.partial` gateway event, contract regenerated). Desktop dictation
  streams PCM to the new /api/audio/transcribe-stream socket (provider keys stay on
  the host, and the host resamples) and shows partial text in the dictation pill.
  The recorded blob stays the fallback.

The plugin streaming contract (streaming_capable / open_stream_session /
TranscriptionStreamSession) and the transcribe-stream socket shape come from #103173.

Co-authored-by: Vinicius <timelord1025@gmail.com>
2026-10-06 12:50:31 -07:00
..
2026-09-11 09:31:13 -07:00