GenAI & Agentic AI Workshop
Day 1 · Session 4 · 2 hours

RAG: Retrieval-Augmented Generation

An LLM doesn't know your college's rules, timetable or syllabus. RAG fixes that: before answering, it looks up the right page in your documents and answers from it, with the source.

Open the slides Jump to the lab
What you will learn
  1. The problem: the model doesn't know your documents
  2. What RAG is: an open-book exam
  3. The RAG pipeline in six steps
  4. Step 1: Load
  5. Step 2: Chunk
  6. Step 3: Embed
  7. Step 4: Store in a vector database
  8. Step 5: Retrieve
  9. Step 6: Generate with citations
  10. When RAG goes wrong, and how to fix it
  11. How to test a RAG bot
  12. Lab: chat with the college FAQ
  13. Recap, quiz and glossary

1. The problem: the model doesn't know your documents

Ask any chatbot "What is the revaluation fee at our college?" and it can't know. Your college's rules were never in its training data, and even if similar text was, it might be out of date. So it either says "I don't know" or, worse, makes something up (Session 1: hallucination).

Without RAG, the LLM guesses the revaluation fee as 1,000 rupees, which is wrong and has no source. With RAG, the LLM is given the retrieved Exams page and answers 500 rupees per subject, with the source.

There are three ways to give a model your information:

OptionHowProblem
Paste everythingPut the whole document in every promptFine for 2 pages. For 500 pages: too many tokens, slow, expensive, may not fit the context window.
Fine-tuneTrain the model further on your documentsCostly, needs skills, and must be redone every time a rule changes. Still can hallucinate.
RAG ✅Search your documents, send only the few relevant piecesNeeds some setup, which is what this session is about.

2. What RAG is: an open-book exam

RAG stands for Retrieval-Augmented Generation:

Analogy: open-book exam Without RAG, the model writes a closed-book exam from memory. With RAG, it's an open-book exam: before each answer, it flips to the right page of the textbook and answers from it. And just like a good student, it can say which page the answer came from.

3. The RAG pipeline in six steps

RAG has two parts. Indexing prepares your documents once (and again whenever they change). Answering runs for every question.

Indexing: documents are chunked, embedded and stored in a vector database. Answering: the question is embedded, the closest chunks are retrieved from the database, and the LLM generates an answer with a source.
1. LoadRead PDFs, web pages, docs into text
2. ChunkCut into small pieces
3. EmbedTurn each piece into a vector
4. StoreSave in a vector database
5. RetrieveFind the pieces closest to the question
6. GenerateLLM answers from those pieces, with sources

4. Step 1: Load

First, get your documents into plain text. Common sources: PDFs (syllabus, rules), web pages (college website), Word files, Google Docs, or a database.

# Read a PDF page by page
from pypdf import PdfReader

reader = PdfReader("syllabus.pdf")
pages = [page.extract_text() for page in reader.pages]
print(len(pages), "pages")
print(pages[0][:300])
Garbage in, garbage out Check the extracted text! Tables often come out jumbled, headers and page numbers repeat on every page, and scanned PDFs (photos of paper) give no text at all; they need OCR first. A RAG bot can only be as good as the text you feed it.

5. Step 2: Chunk

We don't search the whole document as one block. We cut it into chunks: small pieces, each about one topic. Then we can send the model only the 2–3 chunks it needs.

A long document is cut into chunks of 120 words, each overlapping the previous by 20 words. Too-small chunks lose meaning, too-big chunks mix topics, one topic per chunk is just right.
def chunk_text(text, size=120, overlap=20):
    words = text.split()
    chunks, step = [], size - overlap
    for start in range(0, len(words), step):
        chunks.append(" ".join(words[start:start + size]))
        if start + size >= len(words):
            break
    return chunks

Libraries like LangChain and LlamaIndex have ready-made "text splitters" that do the same thing with more options. Writing your own once helps you understand what they do.

6. Step 3: Embed

Remember embeddings from Session 1: text turned into a list of numbers so that similar meanings are close together. We embed every chunk once.

from sentence_transformers import SentenceTransformer

embedder = SentenceTransformer("all-MiniLM-L6-v2")   # small, free, runs in Colab
vectors = embedder.encode(chunks, normalize_embeddings=True)
print(vectors.shape)   # (number of chunks, 384)
One rule you must not break Use the same embedding model for the chunks and the questions. Vectors from different models live in different "maps" and can't be compared.

Embedding model options: free local models (like all-MiniLM-L6-v2, or multilingual ones for Hindi, Kannada and Tamil), or API models from Google, OpenAI and others. Local is free and has no rate limits; API models are often more accurate.

7. Step 4: Store in a vector database

A vector database stores each chunk's vector together with its text and source, and can quickly find the vectors closest to a question, even among millions.

OptionWhat it isGood for
ChromaSimple open-source vector DB, runs inside PythonLearning, prototypes, small apps (we use it today)
FAISSFast similarity-search library from MetaLarge collections on your own machine
pgvectorVector search inside PostgreSQLApps that already use Postgres
ManagedPinecone, Weaviate, or cloud services (Vertex AI Vector Search, Azure AI Search, Amazon OpenSearch / Bedrock Knowledge Bases)Production: scaling, backups, security handled for you
import chromadb

db = chromadb.Client()
col = db.create_collection("college_faq")
col.add(
    ids=["Library-0", "Exams-0"],
    documents=[library_text, exams_text],
    metadatas=[{"source": "Library"}, {"source": "Exams"}],
    embeddings=embedder.encode([library_text, exams_text], normalize_embeddings=True).tolist(),
)

8. Step 5: Retrieve

When a question comes in, embed it with the same model and ask the database for the top-k closest chunks (usually k = 3 to 5).

q_vec = embedder.encode(["Is the library open on Sunday?"], normalize_embeddings=True).tolist()
res = col.query(query_embeddings=q_vec, n_results=3)
print(res["metadatas"][0])   # [{'source': 'Library'}, ...]
Example: what retrieval finds
QuestionTop chunk found
Is the library open on Sunday?[Library] "...closed on Sundays, except during exam months..."
Can I sit for more drives after a 7 LPA offer?[Placements] "...one-student-one-job policy..."
Who won the IPL in 2020?Nothing relevant: all similarity scores are low
The placement question doesn't contain the words "one-student-one-job", but the meaning matches. That's the power of embeddings.

Use a minimum similarity score: if even the best chunk scores low, the answer probably isn't in your documents, so don't send junk to the model.

9. Step 6: Generate with citations

Now we build a prompt (Session 2!) with three parts: rules, the retrieved chunks labelled with their source, and the question.

SYSTEM = """You are CampusBuddy, the helpdesk assistant for ABC Engineering College.
Answer the student's question using ONLY the context below.
- After each fact, cite the source in square brackets, like [Exams].
- If the answer is not in the context, say: "I couldn't find that in the
  college documents. Please contact office@abc.edu."
- Keep the answer under 80 words, in simple English."""

context = "\n\n".join(f"[{h['source']}] {h['text']}" for h in hits)
prompt = f"Context:\n{context}\n\nQuestion: {question}"

resp = client.models.generate_content(
    model=MODEL, contents=prompt,
    config=types.GenerateContentConfig(system_instruction=SYSTEM, temperature=0),
)
Result

Q: My attendance in DBMS is 70%. Can I still write the exam?

A: You need 75% to get a hall ticket [Exams]. With 70%, you can apply for condonation with a medical certificate and a ₹1,000 fee; the principal decides [Attendance].

Two sources, combined into one answer, and you can check both.

Why citations matter: students (and you) can check the answer, and it's easy to spot when the bot used the wrong chunk. "Only use the context" plus "say you couldn't find it" is your main defence against hallucination.

10. When RAG goes wrong, and how to fix it

Most RAG bugs are retrieval bugs, not LLM bugs. Always look at which chunks were retrieved first.

SymptomLikely causeFix
Answer is wrong or vagueThe right chunk wasn't retrievedPrint the retrieved chunks. Try different chunk size, more k, better splitting.
Says "not found" but it's in the documentText didn't load (scanned PDF, table), or score threshold too highCheck the extracted text. Lower the threshold.
Answers questions outside the documentsPrompt doesn't restrict it"ONLY use the context" + minimum score.
Old informationDocuments changed, index didn'tRe-index when documents are updated.
Exact codes or names not found (e.g. "CS301")Embeddings match meaning, not exact wordsHybrid search: combine keyword search with vector search.

Going further: re-ranking (a second model re-orders the top 20 chunks), query rewriting (rephrase the question before searching), and metadata filters (search only the 3rd-semester syllabus). You don't need these to start.

11. How to test a RAG bot

Don't just try two questions and call it done. Make a small test set:

  1. Write 10–20 real questions students ask, with the section where the answer is.
  2. Test retrieval: is the right section in the top 3? Count the hit rate.
  3. Test answers: is the answer correct, and is the citation right?
  4. Include 3–4 questions that are not in the documents. The bot should refuse them.
  5. Re-run the test set every time you change chunk size, k, the prompt or the model.

This is your first taste of evaluation, which we cover properly in Session 9.

12. Lab: chat with the college FAQ (45 min)

Build CampusBuddy-RAG over a sample FAQ for a fictional college: exams, attendance, fees, hostel, library, placements, scholarships and clubs. Then try it on your own college's PDF.

Open the lab in Colab View the sample FAQ Project 3 details
  1. Ask Gemini about the college with no documents. Watch it guess.
  2. Load the FAQ and chunk it (by word count, then by heading).
  3. Embed the chunks and search by meaning with NumPy: RAG from scratch.
  4. Store the chunks in Chroma and retrieve the top 3.
  5. Generate answers with citations, and refuse questions not in the FAQ.
  6. Measure the retrieval hit rate on 8 test questions.
  7. Chat with it.
Stretch goals Upload your own college PDF. Change chunk size, k and the minimum score and watch the hit rate. Add memory from Session 3 so follow-up questions work.

13. Recap

  1. LLMs don't know your private or recent documents, so they guess.
  2. RAG = look up the right pieces, add them to the prompt, then answer. An open-book exam.
  3. Indexing (once): load → chunk → embed → store.
  4. Answering (every question): embed the question → retrieve top-k → generate.
  5. Chunks of ~100–300 words with 10–20% overlap; split by section when you can; keep the source.
  6. Same embedding model for chunks and questions.
  7. Prompt: "use ONLY the context", cite sources, say when it's not found.
  8. Most RAG bugs are retrieval bugs. Test with a question set.

Quick quiz

1. What do the R, A and G in RAG stand for?Retrieval (find relevant chunks), Augmented (add them to the prompt), Generation (the LLM writes the answer).
2. Why not just paste the whole 500-page syllabus into every prompt?Too many tokens: expensive, slow, may not fit the context window, and models can miss details in very long inputs.
3. Why do chunks overlap?So that a sentence or idea cut at a chunk boundary still appears complete in at least one chunk.
4. You embedded the chunks with model A and the questions with model B. What happens?Retrieval breaks: the vectors are on different "maps" and the similarity scores are meaningless.
5. The bot says "not found" for a question whose answer is in a scanned PDF. Why?A scanned PDF is an image; pypdf extracts no text, so nothing was indexed. It needs OCR first.
6. Your bot gives a wrong answer. What's the first thing to check?Which chunks were retrieved. If the right chunk isn't there, fix retrieval (chunking, k, threshold) before touching the prompt or model.
7. Name two things you'd put in a RAG system prompt.Any two of: use only the context; cite the source for each fact; say "not found" if it isn't in the context; keep answers short.

Glossary

TermMeaning in one line
RAGRetrieve relevant document pieces, add them to the prompt, then generate the answer.
IndexingPreparing documents for search: load, chunk, embed, store.
ChunkA small piece of a document, usually one topic.
OverlapWords repeated between neighbouring chunks.
Embedding modelThe model that turns text into vectors.
Vector databaseA database that stores vectors and finds the closest ones fast.
Top-kThe k most similar chunks returned for a question.
Similarity scoreHow close two vectors are in meaning (higher = closer).
Citation / groundingLinking each fact in the answer to the chunk it came from.
Hybrid searchCombining keyword search with vector search.
Re-rankingRe-ordering retrieved chunks with a second, more careful model.
OCROptical Character Recognition: reading text from images or scans.

Learn more (free)

Instructor guide

Suggested plan for the 2 hours, with a short break. Press S in the slides to open speaker notes.

TimeSegmentWhat to do
0–10 minHookLive: ask a chatbot "What is the revaluation fee at ABC College?" It guesses. Show the with/without diagram. Ask: how would you fix this?
10–20 minThree options + what RAG isPaste vs fine-tune vs RAG. Open-book exam analogy. Pipeline diagram.
20–40 minLoad, chunk, embedShow a messy PDF extraction. Chunking diagram: ask the class to pick a chunk size for the syllabus. Recap embeddings from Session 1.
40–55 minStore, retrieve, generateVector DB options. Live retrieval in the notebook (Steps 4–6). Point at the citations.
55–65 minBreakStudents start the first install cell so it's ready after the break.
65–75 minProblems + testingSymptom table. "Print the retrieved chunks first." Show the evaluation cell.
75–115 minLabPairs work through the notebook. Fast pairs: upload a real college PDF (stretch 1). Collect the best hit rates.
115–120 minRecap + quizQuiz questions 1, 4 and 6. Preview Day 2: turning CampusBuddy into an agent that can use tools.
Before class Run the whole notebook once on Colab; the first install and model download take 1–2 minutes. Bring 2–3 real college PDFs (syllabus, placement brochure) on a pen drive or shared link for the stretch goal, and check they're text PDFs, not scans.