Chapter 6 · Part 3

Persist, scale & what's next

You have a working engine. Two things stand between it and something you'd actually run in production: not recomputing embeddings every time, and staying fast as the corpus grows. Both are easy to reason about.

Don't recompute — save the embeddings

Right now every run re-embeds the whole corpus. Embedding is the slow part, and your documents rarely change — so compute the matrix once, save it to disk, and load it instantly next time:

save.py
import numpy as np

# Run once, after building corpus_embeddings:
np.save("corpus_embeddings.npy", corpus_embeddings)

# In future runs, skip model.encode(corpus) entirely:
corpus_embeddings = np.load("corpus_embeddings.npy")

Now startup is instant, and you only ever embed a document again when you add or change one. The query still gets embedded live — it has to, since it's new every time.

When brute force stops being enough

Our search does one dot product per document. At 8 documents that's nothing; at 10,000 it's still a blink. But scoring every document against every query — a full scan — eventually gets slow. Where's the line?

  • Up to ~100k documents: the brute-force matrix multiply you already wrote is genuinely fine. Don't over-engineer.
  • Millions of documents: switch from "compare with everything" to approximate nearest neighbor search, which finds the closest vectors without scanning them all. You don't implement this yourself — you hand your embeddings to a tool built for it: FAISS (a local library), or a vector database like pgvector, Chroma, Pinecone or Weaviate. The concept — embed, then find nearest — is identical to what you built; they just make the "find nearest" step scale.

You built the engine behind RAG

Step back at what you made: text in, meaning out, ranked by closeness. That exact pipeline is the retrieval half of retrieval-augmented generation (RAG) — the technique that lets a chatbot answer from your documents. The recipe is one step longer than what you have:

  1. Retrieve — search(question) to pull the few most relevant passages. (You built this.)
  2. Augment — paste those passages into a prompt as context.
  3. Generate — hand that prompt to an LLM to write a grounded answer.

So this course is a direct on-ramp to building a RAG chatbot — the retriever is done.

Where to go next

  • Grow the corpus. Point it at something real — your notes, a product catalogue, the paragraphs of a PDF. The code doesn't change.
  • Understand why embeddings work. The visual companion, How Machines Understand Meaning, opens up the space you've been searching — why king − man + woman ≈ queen, and how these vectors are learned.
  • Close the RAG loop. Add the augment-and-generate steps above with any LLM to turn your retriever into a question-answering bot.

You didn't just read about semantic search — you built it, and it's the same engine running inside modern search and AI assistants.