Speech-to-speech model
A speech-to-speech model takes spoken audio in and produces spoken audio out directly, without first converting the speech to text and back. It is used for live voice conversation and translation.
Also called: realtime voice, realtime API, S2S, full-duplex voice model
By Divya Sharma · Checked Nov 15, 2026 · How we review
The classic way to build a voice assistant is a pipeline: speech-to-text, then a language model, then text-to-speech. Each stage adds delay, and tone of voice is lost when speech is reduced to text. A speech-to-speech model replaces the pipeline with one model that listens and talks.
How it works
Kyutai's Moshi paper set out the problems with pipelines: delays of several seconds, lost emotion and non-speech sounds, and strict turn-taking that ignores interruptions. Moshi represents audio as tokens from a neural audio codec and models its own speech and the user's in parallel streams, so it can listen while it talks. You can run it from the Moshi repo. Hosted models such as GPT Live 1 are offered through realtime APIs that stream audio both ways.
Why builders care
Lower latency and the ability to interrupt make a voice agent feel more like a phone call. The model can also hear and produce emotion, laughter and emphasis.
Trade-offs
Pipelines are easier to inspect and control: you can log the transcript, swap the language model or apply text guardrails. Speech-to-speech models give less visibility into what happens between hearing and answering, and they tend to cost more per minute. Many products still use a pipeline for that reason.
Related terms
- Voice agent
- A voice agent is an AI agent that holds a spoken conversation in real time, listening to a person, deciding what to say or do, and answering in a synthetic voice, often over a phone line or in an app.
- Text-to-speech
- Text-to-speech (TTS) is software that turns written text into spoken audio. Modern TTS uses neural networks that predict the sound of speech from the text, then render it as a waveform.
- Speech-to-text
- Speech-to-text, also called automatic speech recognition (ASR), is software that converts spoken audio into written text. It powers transcription, captions, voice commands and the listening half of voice agents.
- Streaming (LLM output)
- Streaming sends an LLM's output to the client piece by piece as tokens are generated, instead of waiting for the full response. Users see text appear within moments, even when the complete answer takes much longer.
- Time to first token
- Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
- Guardrails (AI)
- Guardrails are runtime checks placed around a language model that inspect inputs and outputs and block, rewrite or redirect them, so an application stays on topic, avoids harmful content and follows rules the model alone would not reliably keep.