Ears: speech to text

The ears have two jobs: record your voice, then transcribe it into text. We'll build each in a few lines, and they'll become the first part of voice_agent.py.

Record with push-to-talk

A microphone delivers audio as a stream of numbers, thousands per second, each one the air pressure at that instant. sounddevice hands them to a function you write, in small chunks, for as long as the stream is open.

So push-to-talk is simple: open the stream, collect every chunk, and close it when the user presses Enter.

import numpy as np
import sounddevice as sd

SAMPLE_RATE = 16_000  # Whisper expects 16 kHz mono audio

def record():
  input("\nPress Enter, then speak... ")
  chunks = []

  def on_audio(indata, frames, time, status):
      chunks.append(indata.copy())  # called every few ms with fresh audio

  with sd.InputStream(samplerate=SAMPLE_RATE, channels=1, dtype="float32",
                      callback=on_audio):
      input("Listening... press Enter when you're done. ")

  return np.concatenate(chunks)[:, 0] if chunks else None

Three details worth knowing:

• samplerate=16_000, channels=1. Whisper was trained on 16 kHz mono audio, so we record in exactly that format and skip any conversion.
• indata.copy(). sounddevice reuses the same buffer for the next chunk, so without the copy every chunk would end up holding the last bit of audio.
• [:, 0]. Each chunk has shape (frames, 1), one column per channel. We want a flat array, so we take column 0.

The second input() does the waiting. The stream records in the background until you press Enter, and then the with block closes it.

Transcribe with Whisper

Whisper is OpenAI's open-source speech recognition model. faster-whisper runs the same model several times faster on an ordinary CPU, and it runs on your machine. No audio leaves your computer and there's nothing to pay.

from faster_whisper import WhisperModel

whisper = WhisperModel("base.en", device="cpu", compute_type="int8")

def listen():
  audio = record()
  if audio is None:
      return ""
  segments, _ = whisper.transcribe(audio, language="en")
  return " ".join(s.text.strip() for s in segments)

if __name__ == "__main__":
  print("You said:", listen())

Run it, press Enter, ask something, press Enter again:

Press Enter, then speak...
Listening... press Enter when you're done.
You said: What's the tallest mountain in Japan, and how tall is it?

The first run downloads the model (about 145 MB), which takes a moment. After that it loads from disk in a few seconds, and transcribing a short question took about 0.3 seconds on a laptop CPU.

transcribe returns segments, chunks of speech with their own timestamps, because a long recording comes back in pieces. We only want the words, so we join the pieces into one string.

Your agent can hear. Next, give it something to say.