Set up retrieval & a local LLM

This is a hands-on course. By the end you'll have a chatbot that answers questions from your own documents — and it runs entirely on your laptop: no API key, no cloud, no bill. It's the direct sequel to Build a Semantic Search Engine: there you built the retriever; here you add a language model to turn retrieved text into answers.

RAG has two halves, so setup has two parts: the retriever (embeddings, from last course) and a local LLM (new). Budget a bit longer here — the language model is a one-time download.

1 · Python + a virtual environment

You need Python 3.9+. Make an isolated environment and activate it:

python3 -m venv .venv
source .venv/bin/activate

python -m venv .venv
.venv\Scripts\Activate.ps1

2 · The retrieval half

Same library as the semantic-search course — it gives us the embedding model that turns text into vectors:

pip install sentence-transformers numpy

3 · The generation half — a local LLM with Ollama

Ollama runs open language models on your own machine. Install the app from ollama.com/download (macOS, Windows or Linux), then pull a small, capable model from your terminal:

ollama pull llama3.2

That's a ~2 GB one-time download. On a low-memory machine, use the smaller ollama pull llama3.2:1b instead. Once Ollama is installed it runs a local server in the background — the Python package just talks to it. Install that package too:

pip install ollama

4 · Verify both halves

Create check.py. This loads the embedder and sends the local model a message — if both print, you're ready:

from sentence_transformers import SentenceTransformer
import ollama

# Retrieval half
embedder = SentenceTransformer("all-MiniLM-L6-v2")
print("Embedder ready. Dim:", embedder.get_sentence_embedding_dimension())

# Generation half
reply = ollama.chat(
  model="llama3.2",
  messages=[{"role": "user", "content": "Reply with exactly: ready"}],
)
print("LLM says:", reply["message"]["content"])

python check.py

Embedder ready. Dim: 384
LLM says: ready

Both halves are alive on your machine. Before we wire them together, let's see why we need to — by watching the language model fail on its own.