Chapter 3 · Part 2

Brain: Claude, tuned for talking

The transcript is just text, so sending it to Claude is an ordinary API call. The interesting part is that a good spoken reply is different from a good written one, so we have to tell Claude that.

Write for the ear, not the eye

Ask Claude a question in a chat window and you might get a heading, three bullet points, a bold phrase and a follow-up question. That's great to read. Read aloud, it's a disaster: the voice says "asterisk asterisk", or ploughs through a list with no pauses, for forty seconds.

The fix is a system prompt, standing instructions that apply to every reply:

voice_agent.py (continued)

from anthropic import Anthropic

client = Anthropic()

SYSTEM = """You are a friendly voice assistant. Everything you write is read aloud by a
text-to-speech voice, so:
- Answer in one to three short sentences unless the user asks for more.
- Plain spoken sentences only: no markdown, lists, headings, code or emoji.
- Write numbers and symbols the way you would say them."""

Each line earns its place:

Short. Listening is slower than reading, and you can't skim speech. A short answer is also a fast answer, which matters a lot here.
No formatting. A voice can't show bold or bullets. Anything markdown-shaped either gets read out as symbols or lost.
Numbers as spoken. "3,776 m" might come out as "three comma seven seven six m". Asking for "about three thousand seven hundred seventy-six meters" gets it said properly.

Remember the conversation

The Claude API doesn't remember anything between calls. Each request has to carry the whole conversation so far. So we keep a messages list, add what the user said, and add Claude's reply after each turn:

voice_agent.py (continued)

messages = []

def think(text):
  messages.append({"role": "user", "content": text})
  reply = client.messages.create(
      model="claude-opus-5",
      max_tokens=16000,
      system=SYSTEM,
      output_config={"effort": "low"},   # quick answers for small talk
      messages=messages,
  )
  messages.append({"role": "assistant", "content": reply.content})
  return "".join(b.text for b in reply.content if b.type == "text")

That's what lets you say "how tall is it?" after asking about Mount Fuji, and have "it" mean the mountain.

Two lines here need explaining:

output_config={"effort": "low"}. Effort controls how hard Claude thinks before it answers. For a quick spoken question, deep reasoning mostly adds waiting, so we turn it down. If you ask it something tricky and the answers feel shallow, try "medium".
if b.type == "text". Claude can think before it answers, and that thinking comes back as its own block in reply.content, ahead of the text. So reply.content[0] isn't always the answer. Filtering for text blocks is the safe way to get only the words to speak.

📌Why append all of reply.content?

We store the full reply.content in the history, not just the text string. That keeps any thinking blocks in the conversation, which is what the API expects when you send the history back on the next turn.

Try the brain on its own

Wire the ears and the brain together and print the reply:

voice_agent.py (temporary test)

if __name__ == "__main__":
  heard = listen()
  print("You:", heard)
  print("Claude:", think(heard))

You: What's the tallest mountain in Japan, and how tall is it?
Claude: The tallest mountain in Japan is Mount Fuji, at about three thousand seven hundred
seventy-six meters. It's also an active volcano.

(Claude's exact wording will differ.) It's short, plain and speakable, which is just what the mouth needs.