Capstone: voice assistant

The big idea

A dental clinic wants its phone line answered after hours. A caller says "Can you move my cleaning to... Tuesday at ten fifteen?" and the assistant should reply about as fast as a receptionist would. You will build that assistant: audio in, a transcript, a reply from a language model, audio out, plus the instruments to prove how fast it is.

Each piece already works well on its own. This capstone is about the seams between them. A voice assistant feels natural or broken because of time: how long it waits before deciding the caller has finished, how soon the first word of the reply plays, and whether it stops when the caller talks over it. None of that shows up when you test each model alone.

Think of a relay race. Four fast runners still lose if every handover is a standing start. The winning team starts the next runner moving before the baton arrives. In your pipeline the baton is text: speech recognition hands words to the model while the caller is still talking, the model hands sentences to speech synthesis while it is still writing, and audio starts playing while the rest of the reply is still being made.

How it works

What you need

• Python 3.10 or newer. The runnable core below simulates every service on a fake clock, so it runs anywhere and gives the same numbers every time.
• For the real build: a streaming speech recognition service or model, a language model API that streams tokens, a streaming TTS service, and a transport for live audio. An open-source framework such as Pipecat wires these together so you can swap providers.
• Twenty recorded test calls, including callers who pause mid-sentence, give dates and numbers, and interrupt. You will replay them after every change.

Checkpoint 1: plumbing and a stopwatch

Start with a pipeline that echoes the transcript back as speech, no model yet, and time it. Log a timestamp at every handover: last caller audio frame, end of turn decided, final transcript, first model token, first sentence complete, first TTS audio byte, first audio played. Put one turn ID on all of them, as you would on trace spans ( LLM observability and tracing ).

Voice-to-voice latencyThe time from the moment the caller stops speaking to the moment they hear the first sound of the reply, measured on the caller's side. is the number the caller feels. Server logs miss the microphone, network and playback, so also check it from outside: record a test call with both sides on one track and measure the silent gap between the end of the question and the start of the answer.

Done when: every turn prints its voice-to-voice latency and per-stage times, and the recorded gap agrees with your logs within about 100 ms.

Checkpoint 2: streaming recognition and turn detection

Use streaming ASR so the transcript is nearly finished when the caller stops. The partial transcripts are useful early: you can start fetching the caller's record before the turn ends.

The harder decision is when the turn ends. A pure silence rule forces a bad trade: a long pause threshold makes every reply late, a short one cuts off people who stop to think, as in "move my cleaning to... Tuesday". Keep a fast silence rule but add a second opinion about whether the sentence is finished. The runnable core uses a crude one, refusing to end a turn on words like "to" or "um". Real systems use a small audio model that hears intonation, or ask a language model whether the transcript sounds complete.

Done when: none of your test callers who pause mid-sentence is cut off, and the median end-of-turn wait is 300 ms or less.

Checkpoint 3: a model that talks

Text written to be read sounds wrong when spoken. Tell the model in the system prompt that its input is a transcript of speech that may contain recognition errors, and that its output will be read aloud: one or two short sentences, no markdown, no lists, numbers and times written as words.

Pick the model for its time to first token, not only its quality. The first token is usually the largest slice of the budget, so a smaller, faster model often beats a stronger, slower one for a phone assistant ( Routing, cascades and cost optimization ). Keep the long, stable part of the prompt first so prompt caching applies ( Latency and cost budgets ). When a tool call is needed, have the assistant say a short phrase such as "Let me check the calendar" while the tool runs, so the caller hears something.

Done when: replies are short and speakable, and median time to first token fits the budget you set in checkpoint 6.

Checkpoint 4: streaming speech out

Do not wait for the whole reply. A sentence splitter collects tokens until a sentence ends and sends that sentence to streaming TTS at once, so the caller hears "Sure." while the model is still writing the rest ( Streaming responses to users ). Normalize the text first so the voice reads "10:15" and "Dr." the way a person would.

Done when: the first audio plays before the model has finished generating, on every test call.

Checkpoint 5: interruptions

Callers talk over assistants, and an assistant that keeps talking feels rude. When the caller's voice is detected during playback: stop playback, cancel the model and TTS requests, drop queued audio. Then fix the history. The model wrote more than the caller heard, so use the TTS word timestamps to keep only the words actually played. Otherwise the next reply refers to things the caller never heard.

Done when: playback stops within about 200 ms of the caller speaking, and the stored history ends exactly where the audio stopped.

Checkpoint 6: tune against the budget

Write the budget down as a table, one row per stage, adding up to your target. A common target for voice-to-voice is about 1,500 ms, compared with roughly 500 ms between people in normal conversation ( Speech: ASR and TTS ). Then replay your test calls and look at the median and the 95th percentile, not the average: one slow reply in twenty is what callers remember. Change one thing at a time and keep the change only if the numbers improve without hurting answer quality.

Try it

The assistant core: streaming ASR with a pause rule and an optional unfinished-sentence check, a streamed reply, a sentence splitter, TTS with a playback queue, and barge-in that trims the history.

# Voice assistant core: streaming ASR -> LLM -> TTS on a simulated clock, with turn detection,
# voice-to-voice latency and barge-in. The services are fakes; their timings (ms) are assumptions.
import re

NET_IN, NET_OUT = 114, 89    # mic + encode + network in; network out + playback start
FILLERS = {"um", "uh", "and", "to", "the", "so"}  # a turn rarely ends on these words

def asr_stream(turn, per_word=280, final_ms=120, endpoint_ms=800, smart=False):
    """Yield (time, transcript, is_final). turn = words, '...' marks a 700 ms thinking pause."""
    t, said = 0, []
    for w in turn:
        if w == "...":  # silence: a pure pause rule ends the turn here if the pause is long enough
            if 700 >= endpoint_ms and not (smart and said[-1] in FILLERS):
                break
            t += 700
            continue
        t += per_word
        said.append(w)
        yield t, " ".join(said), False
    yield t + endpoint_ms + final_ms, " ".join(said), True, t  # final transcript + speech end

REPLIES = {"book": "Sure. Tuesday at ten fifteen is free. Shall I book it for you?",
           "move": "Sorry, I missed the end of that. Which appointment should I move?"}

def llm_stream(transcript, t0, ttft=650, tps=60):  # one word per token, for readability
    reply = REPLIES["book" if transcript.endswith("ten fifteen") else "move"]
    for i, tok in enumerate(reply.split()):
        yield t0 + ttft + round(i * 1000 / tps), tok

def sentences(tokens, stream=True):  # stream: hand each finished sentence to TTS at once
    buf, last = [], 0
    for t, tok in tokens:
        buf.append(tok)
        last = t
        if stream and re.search(r"[.?!]$", tok):
            yield t, " ".join(buf)
            buf = []
    if buf:
        yield last, " ".join(buf)

def speak(sents, ttfb=120, wps=2.6, batch_ms_per_word=0):  # TTS + playback: (start, end, text)
    free = 0
    for t, s in sents:  # a batch TTS call returns only after synthesizing the whole text
        start = max(t + ttfb + batch_ms_per_word * len(s.split()) + NET_OUT, free)
        free = start + round(len(s.split()) / wps * 1000)
        yield start, free, s

def turn(words, stream=True, barge_in_after=None, ttft=650, ttfb=120, **asr):
    *_, (t_final, text, _, speech_end) = asr_stream(words, **asr)
    t_final += NET_IN
    batch = 0 if stream else 40  # batch TTS: 40 ms of synthesis per word before any audio
    plays = list(speak(sentences(llm_stream(text, t_final, ttft), stream), ttfb, batch_ms_per_word=batch))
    interrupt_at = plays[0][0] + barge_in_after if barge_in_after is not None else None
    heard = []  # word-level timing: keep only what played before a barge-in
    for start, end, s in plays:
        ws = s.split()
        if interrupt_at is not None and interrupt_at < end:
            heard += ws[: max(0, (interrupt_at - start) * len(ws) // (end - start))]
            break
        heard += ws
    return {"transcript": text, "speech end": speech_end, "transcript final": t_final,
            "first audio": plays[0][0], "voice-to-voice": plays[0][0] - speech_end,
            "history": " ".join(heard)}

if __name__ == "__main__":
    ask = "can you move my cleaning to ... tuesday at ten fifteen".split()
    print("voice-to-voice latency, from the end of the caller's speech to the first audio:")
    for name, kw in [("batch ASR, LLM and TTS, 800 ms pause",
                      {"stream": False, "endpoint_ms": 800, "final_ms": 400}),
                     ("streaming, 800 ms pause rule", {"endpoint_ms": 800}),
                     ("streaming, 300 ms pause rule", {"endpoint_ms": 300}),
                     ("streaming, 300 ms + filler check", {"endpoint_ms": 300, "smart": True})]:
        r = turn(ask, **kw)
        print(f"{name:37} {r['voice-to-voice']:5} ms | heard: {r['transcript']!r}")
    r = turn(ask, endpoint_ms=300, smart=True, barge_in_after=1500)
    print(f"\ncaller talks 1500 ms into the reply (first audio at {r['first audio']} ms)")
    print(f"  history keeps only what was played: {r['history']!r}")

voice-to-voice latency, from the end of the caller's speech to the first audio:
batch ASR, LLM and TTS, 800 ms pause   2893 ms | heard: 'can you move my cleaning to tuesday at ten fifteen'
streaming, 800 ms pause rule           1893 ms | heard: 'can you move my cleaning to tuesday at ten fifteen'
streaming, 300 ms pause rule           1493 ms | heard: 'can you move my cleaning to'
streaming, 300 ms + filler check       1393 ms | heard: 'can you move my cleaning to tuesday at ten fifteen'

caller talks 1500 ms into the reply (first audio at 4893 ms)
  history keeps only what was played: 'Sure. Tuesday at'

And the budget runner: 200 simulated turns with realistic variation in each service, four designs, and the median per stage for the last one.

# Latency budget for the voice assistant: 200 simulated turns with jittered stage times,
# p50 and p95 voice-to-voice for each design, and where the time goes. Seeded, so repeatable.
import os, random, runpy

app = runpy.run_path(os.path.join(os.path.dirname(__file__), "AI-15.05-assistant.py"))
turn, NET_IN, NET_OUT = app["turn"], app["NET_IN"], app["NET_OUT"]
TURNS = ["can you move my cleaning to ... tuesday at ten fifteen".split(),
         "book a cleaning on tuesday at ten fifteen".split()]
TARGET = 1500  # ms, voice-to-voice

def sample(rng, ttft):  # real services vary from call to call; slow tails matter most
    return {"final_ms": round(120 * rng.lognormvariate(0, 0.3)),
            "ttft": round(ttft * rng.lognormvariate(0, 0.35)),
            "ttfb": round(120 * rng.lognormvariate(0, 0.3))}

DESIGNS = {  # name: (turn options, median LLM time to first token in ms, extra tool call in ms)
    "800 ms pause rule": ({"endpoint_ms": 800}, 650, 0),
    "300 ms pause + filler check": ({"endpoint_ms": 300, "smart": True}, 650, 0),
    "  + smaller, faster model": ({"endpoint_ms": 300, "smart": True}, 350, 0),
    "  + calendar tool call first": ({"endpoint_ms": 300, "smart": True}, 350, 400),
}

def pct(xs, p):
    return sorted(xs)[int(p * (len(xs) - 1))]

print(f"{'design':30} {'p50':>6} {'p95':>6}  under {TARGET} ms")
for name, (opts, ttft, tool) in DESIGNS.items():
    rng = random.Random(15)
    v2v, parts = [], []
    for i in range(200):
        s = sample(rng, ttft)
        r = turn(TURNS[i % 2], **opts, final_ms=s["final_ms"], ttft=s["ttft"] + tool, ttfb=s["ttfb"])
        v2v.append(r["voice-to-voice"])
        parts.append({"end-of-turn wait": opts["endpoint_ms"], "ASR final": s["final_ms"],
                      "network in + out": NET_IN + NET_OUT, "tool call": tool,
                      "LLM first token": s["ttft"], "TTS first audio": s["ttfb"]})
    ok = sum(v <= TARGET for v in v2v) / len(v2v)
    print(f"{name:30} {pct(v2v, .5):6} {pct(v2v, .95):6}  {ok:6.0%}")

print("\nwhere the time goes in the last design (p50 per stage, ms):")
for stage in parts[0]:
    print(f"  {stage:18} {pct([p[stage] for p in parts], .5):5}")

design                            p50    p95  under 1500 ms
800 ms pause rule                1913   2452      0%
300 ms pause + filler check      1413   1952     66%
  + smaller, faster model        1117   1397     98%
  + calendar tool call first     1517   1797     44%

where the time goes in the last design (p50 per stage, ms):
  end-of-turn wait     300
  ASR final            124
  network in + out     203
  tool call            400
  LLM first token      355
  TTS first audio      117

1. In the first output, the 300 ms pause rule is faster than the 800 ms rule but the assistant heard only "can you move my cleaning to". Trace why in asr_stream, then explain why the filler check fixes it without giving back the speed.
2. The batch design is about a second slower than streaming with the same pause rule. Using the code, list where that second goes: transcription, waiting for the whole reply, and synthesizing the whole reply.
3. The calendar tool call pushes the last design from 98 percent of turns under target to 44 percent. Name two fixes, one that hides the delay from the caller and one that removes it.

Going deeper

Where the real numbers come from

The simulation's stage times are assumptions in the range of the published breakdown in Speech: ASR and TTS . Your real numbers depend on providers, regions and load, so replace them with your own logs. The biggest structural win is usually location: every network hop between your server and a provider adds round trips, so run all services in one region close to your callers, or host the models yourself next to each other.

Telephone calls

The phone network adds delay you cannot remove, often a few hundred milliseconds, and narrowband audio makes recognition harder. If callers dial in, measure through the real phone path, not a browser, and give the phone hop its own row in the budget.

Evaluating the conversation

Latency is half the job. Also evaluate each turn: was the transcript right on the words that matter (dates, times, names), did the model choose the right action, and did the caller have to repeat themselves. Score these on your recorded calls with code checks where possible and an LLM judge for the rest ( LLM-as-judge ), and watch them in production ( Online evaluation and monitoring ).

Final checklist

• Timestamps at every handover, one turn ID, voice-to-voice per turn, checked against a recording.
• Streaming ASR and a turn detector that is fast and does not cut off callers who pause.
• A system prompt for speech: transcript-aware input, short spoken output, numbers as words.
• A model chosen for time to first token, with prompt caching and a spoken phrase during tool calls.
• Sentence-by-sentence streaming TTS with text normalization.
• Barge-in that stops audio fast and trims the history to what was heard.
• A written budget, and p50 and p95 voice-to-voice on recorded test calls before and after every change.

Trade-offs

Common mistakes

• Timing only model inference instead of voice-to-voice from the caller's side.
• Reporting average latency, which hides the slow replies callers remember.
• Waiting for the full reply, or the full audio, before playing anything.
• Ending turns on silence alone, so callers who pause mid-sentence are cut off.
• Keeping the whole generated reply in history after a barge-in.
• Leaving the caller in silence while a tool call runs.

In the wild

Check yourself

Recap