Voice Agents (Speech-to-Text, LLM, Text-to-Speech)

What it is

A voice agent is a conversational system you talk to and that talks back. The common architecture is a cascade: a voice activity detector decides when you are speaking, speech-to-text turns audio into words, an LLM decides what to say, text-to-speech turns the reply into audio, and a transport (WebRTC in the browser, or a phone line) carries the audio both ways.

Every stage streams. The transcription arrives word by word, the LLM reply token by token, the speech sentence by sentence, so the user hears the start of the answer before the model has finished writing it. Without streaming the pauses are several seconds long and the conversation feels dead.

I built mine with Pipecat, an open-source Python framework that models the cascade as a pipeline of processors passing audio and text frames. Deepgram does speech-to-text, gpt-4o-mini is the brain, Deepgram Aura or Cartesia does text-to-speech, Silero VAD detects speech, and a peer-to-peer WebRTC transport connects the browser. I chose the cascade over an end-to-end speech model deliberately, because every stage can be measured, swapped and costed separately.

Why it's needed

Voice is the most natural interface and the least forgiving. Humans notice a 700-millisecond pause. They interrupt. They mumble and get transcribed wrong. A text chatbot can hide behind a typing indicator; a voice agent cannot. Building one forces you to deal with latency, streaming, interruption and cost at the same time, which is why it is such a strong demonstration of engineering depth.

Commercially, voice agents are replacing phone trees and first-line support, qualifying leads and booking appointments. The B1 Properties text chatbot I built lifted lead qualification; a voice version of the same thing is the obvious next step for that business.

How it works

• Transport: the browser captures the microphone and sends audio over WebRTC. Pipecat's SmallWebRTC transport does this without a third-party media server.
• VAD: Silero runs locally and emits 'user started speaking' and 'user stopped speaking' events. This drives turn-taking and interruption.
• STT: audio streams to Deepgram, which returns interim and final transcripts. The final transcript becomes the user message.
• LLM: the transcript goes into the same context, tools and system prompt as a text agent. Tokens stream out as they are generated.
• TTS: text is cut at sentence boundaries and each sentence is sent to the speech service as soon as it is complete, so audio starts early.
• Interruption: when VAD detects the user speaking mid-reply, the pipeline cancels the pending LLM and TTS frames and flushes the audio, so the agent stops talking and listens.

Code example

# Simplified Pipecat bot: the real file adds tools, cost guards and the UI.
import os
from pipecat.audio.vad.silero import SileroVADAnalyzer
from pipecat.pipeline.pipeline import Pipeline
from pipecat.pipeline.task import PipelineTask
from pipecat.pipeline.runner import PipelineRunner
from pipecat.processors.aggregators.openai_llm_context import OpenAILLMContext
from pipecat.services.deepgram.stt import DeepgramSTTService
from pipecat.services.deepgram.tts import DeepgramTTSService
from pipecat.services.openai.llm import OpenAILLMService

async def run_bot(transport):
    stt = DeepgramSTTService(api_key=os.environ["DEEPGRAM_API_KEY"])
    tts = DeepgramTTSService(api_key=os.environ["DEEPGRAM_API_KEY"])
    llm = OpenAILLMService(api_key=os.environ["OPENAI_API_KEY"], model="gpt-4o-mini")

    context = OpenAILLMContext([{"role": "system", "content":
        "You are the AI voice of Abdur. Keep replies to two short sentences."}])
    agg = llm.create_context_aggregator(context)

    pipeline = Pipeline([
        transport.input(),      # mic audio in (VAD attached to the transport)
        stt,                    # audio -> text
        agg.user(),             # add user turn to context
        llm,                    # text -> streamed reply
        tts,                    # reply -> audio, sentence by sentence
        transport.output(),     # audio out to the browser
        agg.assistant(),        # add assistant turn to context
    ])
    await PipelineRunner().run(PipelineTask(pipeline))

Where I used this

Use cases

• Phone and web concierges for real estate, clinics and hospitality
• Lead qualification calls that capture name, budget and intent
• Hands-free assistants for drivers and field workers
• Language-practice tutors
• Accessible interfaces for users who cannot type comfortably

Common mistakes

• Long LLM replies. A paragraph that reads fine in chat is twenty seconds of speech. The system prompt must demand short answers.
• Waiting for the full reply before speaking. Stream and cut at sentence boundaries or the latency is unbearable.
• No interruption handling. Users talk over the agent; if it keeps going, the conversation collapses.
• No cost guard. A voice session pays for STT, LLM and TTS every second. An open browser tab can run all night.
• Testing only with your own clear voice. Accents, background noise and short utterances break transcription; collect real sessions and check the transcripts.

Interview questions