Latency, cost & where next

You have a working, streaming voice agent. This last chapter covers the two numbers that decide whether people enjoy using one, how long they wait and what each turn costs, and then the upgrades that separate a weekend project from a voice product.

Where the time goes

Every turn has a latency budget: the time from when you stop talking until the agent starts. With the streaming version it's made of three parts:

• Transcribing: about 0.3 seconds. That's what base.en took on a laptop CPU for a short question. It grows slowly with the length of what you said.
• Claude's first sentence: usually the biggest slice. This is the time until Claude starts writing, plus the time to finish sentence one. Network speed, model size and effort all change it. Keeping the first sentence short helps too, which is one more reason for the "short sentences" rule in the system prompt.
• Starting the voice: close to instant with the built-in voices, because they run locally.

Loading the Whisper model takes a few seconds too, but it happens once when the program starts, not on every turn. That's why whisper = WhisperModel(...) sits at the top of the file and not inside listen().

To see your own numbers, wrap each stage with a timer:

import time

t0 = time.perf_counter()
heard = listen()          # note: includes the time you spend talking
t1 = time.perf_counter()
# ... inside think_and_speak, record the moment the first sentence is queued ...
print(f"transcribe {t1 - t0:.2f}s")

If Claude's slice feels slow, you have three levers: keep effort at "low", keep the system prompt asking for short answers, or try a smaller, faster model such as claude-haiku-4-5 and listen to whether the answers are still good enough.

What a conversation costs

The ears and mouth are free. You pay only for Claude, by the token (roughly three quarters of a word), for the words going in and the words coming out. Any thinking Claude does is billed as output too.

The part that grows is the input. Every turn sends the whole conversation so far, so turn 20 carries turns 1 to 19 with it. Short spoken replies keep that history small: by turn 20 each request is still only a thousand or two tokens. As a rough estimate, a 20-turn chat on Claude Opus 5 comes to around ten cents in total. Check your real usage any time: reply.usage.input_tokens and reply.usage.output_tokens are on every message.

Upgrades that make it a product

Each of these is a real feature of commercial voice assistants, and each builds on the loop you've written.

• Hands-free listening. Instead of pressing Enter, notice when someone starts and stops talking. That's voice activity detection (VAD). faster-whisper has a small one built in: whisper.transcribe(audio, vad_filter=True) cuts silence out of a recording. An always-on agent runs a VAD on the live mic stream and ends the turn after about half a second of quiet.
• Barge-in. Let the user interrupt. Keep listening while the agent speaks, and if the user starts talking, stop the voice. subprocess.Popen instead of subprocess.run gives you a process you can .terminate(). Use headphones while testing, or the agent will hear itself and interrupt itself.
• A better voice. The built-in voices sound robotic. Neural text-to-speech sounds far more human, either locally (Piper is a free option) or from a cloud voice API that streams audio back as it's generated. Only speak() needs to change.
• Tools. A voice agent that can check your calendar or set a timer is far more useful than one that only chats. That's the same tool-use loop from the Build an AI Agent course, with the final answer spoken instead of printed.
• Speech-to-speech models. Some providers now offer models that take audio in and give audio out directly, skipping the separate ears and mouth. They can react faster and hear tone of voice, at the cost of less control over each stage. The three-stage design you built is still the most common and the easiest to debug.

The complete program

Here's all of voice_agent.py: about a hundred lines for an agent that hears you, thinks, and talks back.

import platform
import queue
import re
import subprocess
import threading

import numpy as np
import sounddevice as sd
from anthropic import Anthropic
from faster_whisper import WhisperModel

SAMPLE_RATE = 16_000  # Whisper expects 16 kHz mono audio

whisper = WhisperModel("base.en", device="cpu", compute_type="int8")
client = Anthropic()

SYSTEM = """You are a friendly voice assistant. Everything you write is read aloud by a
text-to-speech voice, so:
- Answer in one to three short sentences unless the user asks for more.
- Plain spoken sentences only: no markdown, lists, headings, code or emoji.
- Write numbers and symbols the way you would say them."""

messages = []


# ── Ears ──────────────────────────────────────────────────────────
def listen():
  input("\nPress Enter, then speak... ")
  chunks = []

  def on_audio(indata, frames, time, status):
      chunks.append(indata.copy())  # called ~every few ms with fresh audio

  with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, dtype="float32",
                      callback=on_audio):
      input("Listening... press Enter when you're done. ")

  if not chunks:
      return ""
  audio = np.concatenate(chunks)[:, 0]  # (n, 1) -> (n,)
  segments, _ = whisper.transcribe(audio, language="en")
  return " ".join(s.text.strip() for s in segments)


# ── Mouth ─────────────────────────────────────────────────────────
def speak(text):
  system = platform.system()
  if system == "Darwin":                      # macOS
      subprocess.run(["say", text])
  elif system == "Linux":
      subprocess.run(["espeak-ng", text])
  else:                                       # Windows
      import pyttsx3
      engine = pyttsx3.init()
      engine.say(text)
      engine.runAndWait()


def speaker(sentences):
  while (sentence := sentences.get()) is not None:
      speak(sentence)


# ── Brain (streaming) ─────────────────────────────────────────────
SENTENCE_END = re.compile(r"(?<=[.!?])\s+")


def think_and_speak(text):
  messages.append({"role": "user", "content": text})

  sentences = queue.Queue()
  voice = threading.Thread(target=speaker, args=(sentences,))
  voice.start()

  buffer = ""
  with client.messages.stream(
      model="claude-opus-5",
      max_tokens=16000,
      system=SYSTEM,
      output_config={"effort": "low"},
      messages=messages,
  ) as stream:
      for chunk in stream.text_stream:
          print(chunk, end="", flush=True)
          buffer += chunk
          *finished, buffer = SENTENCE_END.split(buffer)
          for sentence in finished:
              sentences.put(sentence)      # speak it while Claude keeps writing
      reply = stream.get_final_message()

  if buffer.strip():
      sentences.put(buffer)                # whatever's left after the last "."
  sentences.put(None)                      # tell the speaker we're done
  voice.join()
  print()

  messages.append({"role": "assistant", "content": reply.content})


# ── The loop ──────────────────────────────────────────────────────
if __name__ == "__main__":
  print("Voice agent ready. Say 'goodbye' to quit.")
  while True:
      heard = listen()
      if not heard:
          continue
      print(f"You: {heard}")
      if "goodbye" in heard.lower():
          speak("Goodbye!")
          break
      print("Claude: ", end="")
      think_and_speak(heard)

That's the whole course. You built each stage yourself: speech-to-text, a language model and text-to-speech, wired into a loop, and then made it feel live with streaming. Every voice AI product you use is built from those same parts.