The chatbot loop

You have a working answer() function. Three small additions turn it into a real chatbot: it should run in a loop, cite the sources it used, and refuse to answer when the documents don't contain the answer. That last one is what keeps RAG trustworthy.

Cite sources and keep the context

First, let answer return both the reply and the chunks it used, so the chatbot can show its receipts. Update the function in rag.py:

def answer(query):
  sources = retrieve(query)
  context = "\n".join(f"- {chunk}" for chunk in sources)

  prompt = f"""Answer the question using ONLY the context below.
If the answer is not in the context, say you don't know.

Context:
{context}

Question: {query}"""

  reply = ollama.chat(
      model="llama3.2",
      messages=[{"role": "user", "content": prompt}],
  )
  return reply["message"]["content"], sources

The loop

Now the chat interface — read a question, answer it, print the sources, repeat. Add this to the bottom of rag.py:

if __name__ == "__main__":
  print("Ask me about Nimbus Notes (type 'quit' to exit).")
  while True:
      question = input("\nYou: ")
      if question.lower() in {"quit", "exit"}:
          break

      reply, sources = answer(question)
      print(f"\nBot: {reply}")
      print("\nSources:")
      for chunk in sources:
          print(f"  - {chunk}")

python rag.py

Ask me about Nimbus Notes (type 'quit' to exit).

You: how do I get a refund?

Bot: All paid Nimbus Notes plans come with a 30-day money-back
guarantee, so you can request a refund within 30 days of purchase.

Sources:
- All paid Nimbus Notes plans come with a 30-day money-back guarantee.
- Nimbus Notes Pro costs $6 per month and adds unlimited notebooks, version history, and offline sync.
- Nimbus Notes support is available by email at [email protected] and replies within one business day.

The honesty test

The real test of a RAG bot isn't answering — it's not answering when it shouldn't. Ask something the documents don't cover:

You: who is the CEO of Nimbus Notes?

Bot: I don't know — that isn't covered in the provided documents.

Our knowledge base names the founder (Priya Raman) but never a CEO, so the grounding instruction does its job: the model declines instead of inventing a name. Compare that to Chapter 2, where it happily made up a price. That refusal is the difference between a demo and something you'd trust.

Where to go next

You've built the whole pattern — retrieve, augment, generate — with citations and a guardrail, running entirely on your machine. To take it further:

• Feed it real documents. Load your own files, split them into paragraph chunks, and drop them into documents. The pipeline doesn't change.
• Scale the retrieval. Past ~100k chunks, swap the brute-force search for a vector database (FAISS, pgvector, Chroma) — exactly the upgrade path from the semantic-search course.
• Add conversation memory. Keep past turns in the messages list so the bot handles follow-up questions like "and how much is that per year?"
• Understand the pieces. How Machines Understand Meaning explains the retrieval half; How ChatGPT Actually Works explains the generation half.

You didn't just read about RAG — you built the pattern behind most real-world AI assistants, and it runs on your laptop.