Chapter 2 · Part 1

Why RAG?

A language model only knows what it saw during training. Ask it about your company's internal docs, a product it's never heard of, or anything newer than its cutoff, and it does the worst possible thing: it answers anyway, confidently, and makes it up. Let's watch that happen — it's the whole reason RAG exists.

Throughout this course we'll use a small, made-up knowledge base about a fictional note-taking app called Nimbus Notes. Because it's invented, the model has no way to know the real answers. Ask it a specific question:

hallucinate.py
import ollama

question = "How much does the Nimbus Notes Pro plan cost?"

reply = ollama.chat(
  model="llama3.2",
  messages=[{"role": "user", "content": question}],
)
print(reply["message"]["content"])
The Nimbus Notes Pro plan costs $9.99 per month, or $99/year if billed
annually, and includes unlimited storage and priority support.

Confident. Detailed. Completely invented — we never told it any of that, and (as you'll see next chapter) the real price is different. This is hallucination: the model fills the gap with something plausible because it has no source of truth to check against. You can't fix this by prompting harder. The model simply doesn't have the facts.

The one idea

RAG — retrieval-augmented generation — fixes this by giving the model the facts at question time, right in the prompt. Instead of asking the model to answer from memory, you:

  1. Retrieve — search your documents for the passages most relevant to the question. (This is the semantic search engine you already built.)
  2. Augment — paste those passages into the prompt as context, alongside the question.
  3. Generate — ask the model to answer using only that context.

The model stops being the source of knowledge and becomes the reasoner over knowledge you supply. It's the difference between a closed-book exam and an open-book one — and it's why a small local model can give expert answers about documents it has never seen.

Two of those three steps you've done before. Let's start assembling — beginning with the knowledge base the retriever searches.