Chapter 1 · Part 1

Ears, brain, mouth

In the next 30 minutes you'll build a program you can talk to. You press Enter, ask a question out loud, and it answers in a voice. It remembers the conversation and starts talking back in about a second.

It sounds like a big project, but every voice assistant, from phone support bots to the one in your car, is the same three programs wired into a loop:

📌The voice loop
  • Ears (speech-to-text). Turn the sound of your voice into written words.
  • Brain (a language model). Read those words and decide what to say back. We'll use Claude.
  • Mouth (text-to-speech). Turn the reply back into sound.

Claude only reads and writes text, so it can't hear or speak on its own. The ears and mouth are separate tools that sit on either side of it. Scroll to follow one question all the way round.

You speak. To the computer, that's just sound: a wobbly line of air pressure.

scroll↓

The hard part is the silence

Look at the meter under the loop. From the moment you stop talking to the moment the agent starts, you're sitting in silence. In a text chat a two-second pause is fine. In a spoken conversation, two seconds of dead air feels broken.

So the course has two halves. First we build the loop in the simplest possible way and get it working (chapters 2 to 4). Then we make it feel live by cutting that silence (chapter 5).

What you'll use

  • Microphone: sounddevice records the audio. Free.
  • Ears: faster-whisper, a fast local version of OpenAI's Whisper model. Free, and it runs on your CPU.
  • Brain: Claude, through the Anthropic Python SDK. A fraction of a cent per reply.
  • Mouth: your computer's built-in voice. Free.

To keep this to 30 minutes, we'll use push-to-talk: press Enter to start listening, Enter again when you're done. An always-listening agent that notices on its own when you stop is a fun upgrade, and chapter 6 shows how.

Set up (once)

You need Python 3.10 or newer, a microphone and speakers. Install everything in one go:

Terminal
pip install anthropic faster-whisper sounddevice numpy
export ANTHROPIC_API_KEY="sk-ant-..."   # from console.anthropic.com; never hard-code it

A few platform notes:

  • Linux: sounddevice needs PortAudio (sudo apt install libportaudio2), and the voice we'll use is espeak-ng (sudo apt install espeak-ng).
  • Windows: also run pip install pyttsx3 for the voice.
  • macOS: nothing extra. The first time you record, macOS asks whether your terminal may use the microphone. Say yes, or the recording comes back silent.

Check the key works with a one-line call to Claude:

hello.py
from anthropic import Anthropic

client = Anthropic()

reply = client.messages.create(
  model="claude-opus-5",
  max_tokens=1000,
  messages=[{"role": "user", "content": "Say hi in five words."}],
)
print(next(b.text for b in reply.content if b.type == "text"))

If it prints a greeting, you're ready. (Chapter 3 explains why we pick out the text block instead of just taking the first one.)

⚠️This course spends real money (a little)

The ears and mouth run on your machine for free, but every reply comes from Claude, which is a paid API. A spoken reply is short, so a whole conversation costs cents, not dollars. Still, set a spend limit in the Anthropic Console (Billing → Limits) before you start.

Next: give the agent ears.