Speech: ASR and TTS

The big idea

A dental clinic wants a phone assistant that books appointments. The caller says "Can I move my cleaning to Tuesday at ten fifteen?" and expects an answer as quickly as a receptionist would give one. Three jobs happen in that gap: turn the audio into text, decide what to say, and turn the reply back into audio.

ASRAutomatic speech recognition: turning recorded or live speech into text. Also called speech-to-text (STT) or transcription. does the first job and TTSText-to-speech: generating natural-sounding spoken audio from text. the last. A language model in the middle does the thinking. Each part is now good on its own. The hard part is time: people expect a reply about half a second after they stop talking, and a long pause feels broken.

Think of a simultaneous interpreter. They do not wait for the speaker to finish a whole speech, translate it on paper and then read it out. They listen, start translating while the sentence is still arriving, and begin speaking as soon as they have the first phrase. A voice agent that waits for each stage to finish before the next starts sounds like the paper version.

How it works

Speech recognition

Audio arrives as a waveform, thousands of numbers per second. Most modern ASR models first turn it into a mel spectrogramA picture of sound: how much energy each frequency band has in each short slice of time, with the frequency bands spaced the way human hearing perceives pitch., a grid of frequency energy over time, and then use a transformer to map that grid to text tokens. An open model from 2022, Whisper, is a clear example: it resamples audio to 16,000 samples per second, computes an 80-channel log-mel spectrogram on 25 millisecond windows every 10 milliseconds, and feeds 30-second segments to an encoder-decoder transformer. The decoder writes text tokens one at a time, like a language model, with special tokens for the language, the task (transcribe or translate) and timestamps.

Two properties matter in practice:

• Batch or streaming. Batch models see a whole recording at once and suit transcripts of meetings or calls after the fact. Voice agents need streaming ASRSpeech recognition that emits words while the person is still speaking, instead of waiting for the end of the recording., which emits partial text as audio arrives and revises it as more context comes in.
• Accuracy is measured as word error rate. Count the word substitutions, deletions and insertions needed to turn the transcript into a correct reference, and divide by the number of reference words.

One wrong word in a nine-word request is 1/9, about 11 percent. Whether that matters depends entirely on which word it was, as the code below shows.

Turn detection

The agent must decide when the caller has finished. The simplest rule treats a long enough pause as the end of the turn, using a small VADVoice activity detection: a small model that classifies short slices of audio as speech or not speech, used to find where speech starts and stops. model to tell speech from silence. Set the pause too short and the agent interrupts people who stop to think; too long and every reply starts late. Better systems add a model that listens for intonation and filler sounds, and let the language model judge whether the sentence sounds complete.

Text-to-speech

Neural TTS usually works in two stages: a model turns text into an acoustic representation such as a mel spectrogram, and a vocoderThe part of a speech synthesis system that turns an acoustic representation, such as a mel spectrogram, into an audio waveform you can play. turns that into a waveform. Tacotron 2, from 2017, used exactly this design and scored close to professionally recorded speech in listening tests (see below).

For an agent, the TTS numbers that matter are naturalness, time to the first audio byte, cost, language support, and whether it reports word-level timestampsTiming information saying when each word starts and ends in generated or recognised audio., which tell you exactly how much of the reply the caller heard before interrupting.

The latency budget

Voice-to-voice latency is the time from the moment the caller stops speaking to the first sound of the reply. A published breakdown for a well-built cascaded agent adds up as follows:

The same guide calls 1,500 ms an important target, compared with about 500 ms in normal human conversation. A telephone connection can add a few hundred milliseconds, which is enough to blow the budget. Notice that the language model's first token is the biggest single slice, and that the reply is spoken while it is still being written. That only works because every stage streams.

Try it

1. Start from the typical values. Add a 400 ms tool call to look up the calendar. Are you still under 1,500 ms? Which other stage would you shrink to make room?
2. Switch to "wait for the whole reply". How much does waiting for a 60-token answer cost, at the lab's assumed 60 tokens per second? Why is streaming the first sentence the default?
3. Set telephony to 300 ms and the language model to 1,000 ms, as a slower model with step-by-step reasoning might need. What is the total now, and which two changes bring it back under target?

Word error rate in Python. Four transcripts of the same request:

# Word error rate: the minimum number of word substitutions, deletions and
# insertions that turn the transcript into the reference, divided by the
# number of reference words. Classic edit distance, on words instead of letters.

def wer(reference, hypothesis):
    ref, hyp = reference.lower().split(), hypothesis.lower().split()
    # d[i][j] = edits to turn the first i reference words into the first j heard
    d = [[i + j if i * j == 0 else 0 for j in range(len(hyp) + 1)]
         for i in range(len(ref) + 1)]
    for i in range(1, len(ref) + 1):
        for j in range(1, len(hyp) + 1):
            same = ref[i - 1] == hyp[j - 1]
            d[i][j] = min(d[i - 1][j] + 1,                  # deletion
                          d[i][j - 1] + 1,                  # insertion
                          d[i - 1][j - 1] + (not same))     # substitution or match
    return d[-1][-1] / len(ref)

ref = "please move my appointment to tuesday at ten fifteen"
for heard in [
    "please move my appointment to tuesday at ten fifteen",
    "please move my appointment to tuesday at ten fifty",     # one word wrong
    "please move my appointment to thursday at ten fifteen",  # one word wrong
    "move my appointment to tuesday at ten fifteen please",   # same words, moved
]:
    print(f"WER {wer(ref, heard):5.1%}  {heard}")

# The reordered line scores worst yet means the same thing; the one-word
# errors score better but book the wrong slot. Also track the words that matter.

WER  0.0%  please move my appointment to tuesday at ten fifteen
WER 11.1%  please move my appointment to tuesday at ten fifty
WER 11.1%  please move my appointment to thursday at ten fifteen
WER 22.2%  move my appointment to tuesday at ten fifteen please

The reordered sentence has the worst score but means exactly the same thing. The two one-word errors have a better score and book the wrong appointment. Measure WER, but also check the words your task depends on: dates, times, names, numbers.

Going deeper

Make the language model robust to transcripts

Transcription errors are common in live streams because the ASR model has little context. The language model has the whole conversation, so tell it so: say in the system prompt that the input is a transcript of speech that may contain recognition errors, and that it should infer the intended words. Ask for short, plain sentences with no markdown, lists or symbols, because the output will be spoken. Spell out how to say numbers, dates and units if your TTS reads them badly.

Interruptions

People talk over agents, and agents must stop at once. Every stage has to be cancellable: stop generating, drop queued audio and stop playback on the device. Then fix the record: the language model probably generated more text than the caller heard, so use word-level timestamps to keep only what was actually played in the conversation history. False interruptions come from coughs, keyboard clicks, background voices and the agent's own voice leaking back through imperfect echo cancellation.

Speech-to-speech models

Some models take audio in and produce audio out directly, with no separate ASR or TTS. The promise is lower latency, better understanding of tone and more natural voices. A practitioner guide updated in 2026 reports that the voices really are more natural, but that these models still follow instructions and call tools less reliably than text models, are less configurable, and in practice were slower than a well-tuned cascade. The same guide notes that almost all production voice agents still use the cascade. Check the trade-off again for each new generation of models.

Trade-offs

Common mistakes

• Measuring only model inference time, not voice-to-voice latency from the caller's side, which also includes audio capture, network, end-of-turn waiting and playback.
• Waiting for the whole reply before starting TTS instead of streaming sentence by sentence.
• Letting the model answer in markdown, bullet lists or long paragraphs that sound wrong when read aloud.
• Keeping the full generated reply in the history after the caller interrupted, so the agent later refers to things the caller never heard.
• Judging ASR by overall word error rate alone, when one wrong date or digit can break the task.

In the wild

Check yourself

Recap