Voice agent
A voice agent is an AI agent that holds a spoken conversation in real time, listening to a person, deciding what to say or do, and answering in a synthetic voice, often over a phone line or in an app.
Also called: voice AI agent
By Divya Sharma · Checked Nov 15, 2026 · How we review
Voice agents answer support lines, book appointments, qualify sales leads and act as hands-free assistants. They are judged less on what they know than on how natural the conversation feels: quick replies, no talking over the caller, and graceful recovery when interrupted.
How it works
The common design is a pipeline of three models. Speech-to-text turns the caller's audio into words, a language model decides the reply and calls any tools, and text-to-speech speaks the answer. Each stage streams, so the voice can start before the full reply is written. The newer design uses a single speech-to-speech model that takes audio in and gives audio out, which can cut delay and keep tone of voice, at the cost of less control over each step.
Around the models sit the hard parts: turn detection to know when the caller has finished, handling barge-in when they interrupt, telephony, and tool calls to calendars or CRMs. Frameworks such as LiveKit Agents and platforms such as Vapi and Retell AI package these pieces.
Why latency matters
People notice pauses in speech far more than in text. Every stage adds delay, so builders track time to first token for the model and time to first audio for the voice.
Limits
Background noise, accents and crosstalk cause transcription errors that the rest of the pipeline inherits. Many places have rules on disclosing that a caller is speaking to an AI and on recording calls; check the law where you operate.
Related terms
- Speech-to-speech model
- A speech-to-speech model takes spoken audio in and produces spoken audio out directly, without first converting the speech to text and back. It is used for live voice conversation and translation.
- Text-to-speech
- Text-to-speech (TTS) is software that turns written text into spoken audio. Modern TTS uses neural networks that predict the sound of speech from the text, then render it as a waveform.
- Speech-to-text
- Speech-to-text, also called automatic speech recognition (ASR), is software that converts spoken audio into written text. It powers transcription, captions, voice commands and the listening half of voice agents.
- AI agent
- An AI agent is a system in which a language model decides its own next steps: it calls tools, reads the results and repeats until the task is done, instead of following a fixed script.
- Time to first token
- Time to first token (TTFT) is how long a user waits between sending a request to a language model and receiving the first token of its reply, covering queuing, prompt processing and network time.
- Voice cloning
- Voice cloning is the use of AI to make a text-to-speech system speak in a specific person's voice, learned from a recording of that person. Some systems need only a short sample.