GenAI & Agentic AI Workshop · Day 1 · Session 4

RAG

Retrieval-Augmented Generation: make the AI answer from your documents

Press → to move · S for speaker notes · F for full screen

Ask it about our college 🏫

"What is the revaluation fee at ABC Engineering College, and how many days do I have to apply?"

It has never seen our rules.
So it guesses, or gives up.

Closed book vs open book

Without RAG the model guesses; with RAG it answers from the retrieved page with a source

Three ways to teach it your documents

📋 Paste everything

Fine for 2 pages.
500 pages? Too many tokens, slow, costly.

🏋️ Fine-tune

Retrain the model.
Costly, redo on every rule change, still hallucinates.

🔎 RAG

Search the documents.
Send only the 2–3 pieces that matter.

R · A · G

Retrieval

Find the pieces of your documents that match the question

Augmented

Add those pieces to the prompt

Generation

The LLM writes the answer from them, with sources

An open-book exam 📖

The RAG pipeline

Indexing: load, chunk, embed, store. Answering: embed question, retrieve, generate with source

Step 1: Load 📄

from pypdf import PdfReader
reader = PdfReader("syllabus.pdf")
pages = [page.extract_text() for page in reader.pages]

⚠️ Garbage in, garbage out: jumbled tables, repeated headers, and scanned PDFs give no text (need OCR).

Step 2: Chunk ✂️

Overlapping chunks; too small, just right, too big

Step 3: Embed 🔢

from sentence_transformers import SentenceTransformer
embedder = SentenceTransformer("all-MiniLM-L6-v2")      # free, runs in Colab
vectors = embedder.encode(chunks, normalize_embeddings=True)

🚨 Same embedding model for chunks and questions. Different models = different maps.

Step 4: Store in a vector database 🗄️

OptionGood for
ChromaLearning, prototypes (today)
FAISSBig collections on your own machine
pgvectorApps already on PostgreSQL
Managed / cloudProduction: Pinecone, Vertex AI, Azure AI Search, OpenSearch

Stores vector + text + source. Finds the closest vectors fast.

Step 5: Retrieve the top-k 🔎

QuestionTop chunk
Is the library open on Sunday?[Library] "...closed on Sundays, except exam months..."
More drives after a 7 LPA offer?[Placements] "...one-student-one-job policy..."
Who won the IPL in 2020?Nothing relevant: all scores low ❌

Use a minimum score: if nothing is close, don't send junk to the model.

Step 6: Generate with citations 📝

SYSTEM: Answer using ONLY the context below.
- After each fact, cite the source in [brackets].
- If it's not in the context, say "I couldn't find that in the college documents."

Context:
[Exams] ...hall ticket only with at least 75% attendance...
[Attendance] ...65% to 74%: condonation with medical certificate, fee ₹1,000...

Question: My attendance in DBMS is 70%. Can I still write the exam?

You need 75% for a hall ticket [Exams]. With 70%, apply for condonation with a medical certificate and ₹1,000 [Attendance].

When RAG goes wrong 🔧

SymptomFix
Wrong or vague answerPrint the retrieved chunks first. Fix chunking / k
"Not found" but it's thereCheck extracted text; lower the threshold
Answers off-topic questions"ONLY the context" + minimum score
Old informationRe-index when documents change
Can't find "CS301"Hybrid search: keywords + vectors

Most RAG bugs are retrieval bugs.

Test it like an engineer 🧪

  1. 10–20 real questions + the section that has the answer
  2. Retrieval: right section in the top 3? → hit rate
  3. Answer: correct and cited correctly?
  4. 3–4 questions not in the docs → should refuse
  5. Re-run after every change

Lab: chat with the college FAQ (45 min)

  1. Ask with no docs: watch it guess
  2. Load + chunk the FAQ
  3. Embed + NumPy search
  4. Store in Chroma
  1. Answer with citations
  2. Refuse what's not there
  3. Measure the hit rate
  4. Chat with it

Stretch: upload your college's PDF · tune chunk size, k, min score · add memory

Recap

  1. LLMs don't know your documents → they guess
  2. RAG = look it up, then answer (open book)
  3. Indexing: load → chunk → embed → store
  4. Answering: embed question → retrieve → generate
  5. Same embedding model · cite sources · "not found" is OK
  6. Test with a question set

Quick quiz 🧠

  1. R, A, G stand for? Retrieval, Augmented, Generation
  2. Why do chunks overlap? Ideas cut at the edge stay whole
  3. Chunks embedded with model A, questions with B? Retrieval breaks
  4. Wrong answer: check what first? The retrieved chunks

Up next · Session 5

Responsible AI

Hallucination, bias, privacy, prompt injection and guardrails. Then Day 2: agents!

Questions? 🙋