How LLMs Work
What happens inside ChatGPT, Gemini or Claude when you type a question? This session explains it in plain language. No heavy math.
0. Introduction: why this workshop, why now
You have probably seen the headlines: big tech companies and IT services firms cutting jobs, fresher hiring slowing down, and "AI will replace programmers" posts everywhere. Before we learn how LLMs work, let's be honest about what is happening and what you can do about it.
Why are there layoffs?
There is no single reason. Several things are happening at the same time:
- Correction after over-hiring. During 2020–22, demand for digital services exploded and companies hired very fast. When demand cooled, many of them cut back.
- Cost pressure. Higher interest rates and an uncertain economy mean companies and their clients cut "nice to have" projects first. Indian IT services feel this directly when overseas clients reduce spending.
- Money is moving to AI. Companies are spending heavily on GPUs, data centres and AI teams. To pay for it, they often shrink other teams. The budget didn't disappear; it moved.
- Routine tasks are being automated. AI tools now handle a lot of first-draft work: boilerplate code, simple test cases, basic support tickets, documentation, data entry. Roles made mostly of these tasks are hit hardest, and many of those are entry-level roles.
- Flatter teams. Many companies are removing layers of management and expecting smaller teams to deliver more, often with the help of AI tools.
Before: a fresher spends two days writing CRUD APIs and unit tests for a student-records module.
Now: an AI assistant writes the first draft in minutes. The valuable work is everything around it: understanding what the college actually needs, designing the data model, checking the AI's code for bugs and security issues, and deploying it reliably.
The person who can only type the code is easy to replace. The person who can do the rest is in demand.What to improve
| Skill | Why it matters more now |
|---|---|
| Fundamentals: DSA, OOP, DBMS/SQL, networking, operating systems | You can't judge whether AI-written code is correct, fast or secure if you don't understand it yourself. |
| Problem solving and system design | AI is good at writing a function. Deciding what to build and how the pieces fit together is still your job. |
| Reading and reviewing code | You will read far more code than you write. Spotting bugs in AI output is a core skill. |
| Communication: writing, explaining, asking good questions | Prompting is communication. So are requirements, documentation and interviews. |
| Ownership: finishing and shipping things | A deployed project that real people use beats ten unfinished tutorials. |
| Domain knowledge: finance, healthcare, education, manufacturing... | AI is generic. Knowing a business problem deeply is what makes your solution useful. |
What to learn
A practical path, roughly in order. This workshop covers the middle of it.
1. The big picture
People use "AI", "ML", "deep learning" and "GenAI" as if they mean the same thing. They don't. Each one sits inside the one before it:
Traditional programming vs machine learning
| Traditional programming | Machine learning | |
|---|---|---|
| You give it | Rules + data | Data + correct answers (examples) |
| It gives you | Answers | The rules (a "model") |
| Spam filter example | if "lottery" in email: spam. Spammers write "l0ttery" and it breaks. | Show it 10,000 emails labelled spam / not spam. It learns the patterns itself, including ones you never thought of. |
Deep learning
Deep learning uses neural networks: layers of simple maths units loosely inspired by brain neurons. "Deep" just means many layers. Each layer learns something a little more complex than the one before.
2. Transformers and attention
In 2017, researchers at Google published a paper called "Attention Is All You Need". It introduced the transformer, the design behind almost every modern LLM. The "T" in GPT stands for Transformer.
Older models read a sentence one word at a time, left to right, and often forgot the start of long sentences. A transformer looks at all the words at once and works out which words matter for each other. This is called attention.
"I went to the bank to deposit my scholarship money." → attention links bank with deposit and money, so it means a financial bank.
"We sat on the river bank and watched the sunset." → attention links bank with river, so it means the side of a river.
Same word, different meaning, decided by the words around it."The dog didn't cross the road because it was too busy." → it = the road.
You figured this out instantly. Attention is how a transformer does the same.
3. An LLM is a next-word predictor
A Large Language Model (LLM) is a huge transformer trained on a huge amount of text (books, websites, code, Wikipedia) to do one simple job: predict the next word.
The model scores every possible next word: cricket 92%, for 3%, in 2%, ... and picks one. Then it adds that word to the input and predicts the next one again. Word by word, it writes a whole answer.
"Large" refers to the number of parameters: the adjustable numbers inside the network that store what it learned. Modern LLMs have billions of them.
From predictor to helpful assistant
A model that only predicts the next word isn't a good chatbot yet. Ask it "What is the capital of France?" and it might continue with "What is the capital of Germany?" because that looks like a quiz list. So it is trained in stages:
4. Tokens
LLMs don't read words or letters. They read tokens: small chunks of text. A token can be a whole word, part of a word, a space plus a word, or punctuation.
Why tokens matter
- Cost: APIs charge per token, for both your input and the model's output.
- Limits: every model has a maximum number of tokens it can handle at once (see context window).
- Indian languages cost more: the same sentence in Hindi, Kannada or Tamil often takes several times more tokens than in English, because tokenizers saw less of that text in training. More tokens = more cost and less room.
- Strange mistakes: ask a model "How many r's are in strawberry?" and older models often said 2. They see tokens like
strawberry, not individual letters.
# Count tokens in Python (run in Google Colab)
!pip install tiktoken
import tiktoken
enc = tiktoken.get_encoding("cl100k_base")
text = "I love unbelievable biryani!"
tokens = enc.encode(text)
print(len(tokens), "tokens")
print([enc.decode([t]) for t in tokens])
5. Embeddings
Computers work with numbers, not meaning. An embedding turns a word, sentence or document into a list of numbers (a vector) so that things with similar meaning end up close together.
Student asks: "How do I apply for placements?"
| Sentence in the FAQ | Similarity (illustrative) |
|---|---|
| What is the process for campus recruitment? | 0.82 (very close) |
| Steps to register with the training & placement cell | 0.78 |
| Placement of furniture in the new hostel block | 0.21 (far) |
Where embeddings are used: semantic search, recommendations ("students who liked this course also liked..."), finding duplicate questions, and RAG, which we build in Session 4.
# Compare sentence meanings (run in Google Colab)
!pip install sentence-transformers
from sentence_transformers import SentenceTransformer, util
model = SentenceTransformer("all-MiniLM-L6-v2")
q = "How do I apply for placements?"
faqs = ["What is the process for campus recruitment?",
"Placement of furniture in the new hostel block"]
scores = util.cos_sim(model.encode(q), model.encode(faqs))
for faq, s in zip(faqs, scores[0]):
print(f"{s:.2f} {faq}")
6. Context window
The context window is the maximum number of tokens a model can look at in one go. It includes everything: the instructions, the whole conversation so far, any documents you pasted, and the answer it's writing.
- Sizes vary a lot: from a few thousand tokens in small models to hundreds of thousands, or even a million, in large ones. Check each model's documentation.
- When a chat gets longer than the window, the app drops or summarises older messages. That's why a chatbot sometimes "forgets" what you said at the start.
- LLMs have no memory between calls. Every request starts fresh. Chat apps feel like they remember because they re-send the whole conversation every time. You'll build this yourself in Session 3.
- Bigger isn't free: more tokens means higher cost and slower answers. Models can also miss details buried in the middle of very long inputs.
7. Temperature
Remember that the model gives a probability for every possible next token. Temperature controls how it picks from that list.
- Low temperature (0 – 0.3): almost always picks the top choice. Predictable and consistent.
- High temperature (0.8 – 1.5): gives less likely words a real chance. Creative and varied, but more likely to go off track.
| Task | Temperature | Why |
|---|---|---|
| Answer from the syllabus, extract data to JSON, write code | Low (0 – 0.3) | You want the same correct answer every time. |
| General chat, explanations | Medium (0.5 – 0.8) | Natural but still focused. |
| Names for the college tech fest, poems, brainstorming | High (0.9 – 1.2) | You want variety and surprise. |
You may also see top-p. It's a related setting: "only pick from the smallest group of words that together cover p% of the probability". Usually you tune one of the two, not both.
8. Why LLMs hallucinate
A hallucination is when a model gives an answer that sounds confident and fluent but is false or made up.
Prompt: "Summarise the 2019 research paper 'Quantum Biryani Optimization' by Dr. R. Sharma, IIT Bombay."
This paper doesn't exist. A model may still write a detailed summary, with methods and results, because that's what a summary of a paper looks like. Other common cases: fake citations, wrong dates, invented court cases, made-up library functions in code.
Why it happens
- It predicts plausible text, not true text. Its goal is "what words usually come next", not "what is a fact".
- It has no built-in fact database. What it learned is stored as patterns in its parameters, not as a list of facts it can check.
- Knowledge cutoff. It only knows what was in its training data up to a certain date. It doesn't know your college's latest timetable.
- It is rewarded for answering. Training often rewards a helpful-looking answer more than "I don't know", so it guesses.
How to reduce it
- Give it the source: paste the document, or use RAG to fetch the right text automatically (Session 4).
- Allow "I don't know": add "If the answer isn't in the text, say you don't know."
- Ask for citations, then actually check them.
- Lower the temperature for factual tasks.
- Give it tools like web search or a calculator (Day 2: agents).
- Keep a human in the loop for anything important: marks, money, medical, legal.
9. Open vs closed models, and how to choose
| Closed models | Open-weight models | |
|---|---|---|
| Examples | GPT (OpenAI), Claude (Anthropic), Gemini (Google) | Llama (Meta), Mistral, also Gemma (Google), Qwen (Alibaba), DeepSeek |
| How you use it | Through the company's app or API. You can't download the model. | Download and run it yourself: on a laptop (e.g. with Ollama), your own server, or a host like Groq. |
| Strengths | Usually the most capable. No hardware needed. Quick to start. | Full control. Data can stay on your machines. Can be customised (fine-tuned). No per-token fee if self-hosted. |
| Weaknesses | Pay per token. Your data goes to the provider. The provider can change or retire the model. | Need GPUs and skills to run well. Big models are expensive to host. Licences vary. |
"Open-weight" means the trained model is published. It doesn't always mean the training data and code are open too, and licences can restrict commercial use. Always read the licence.
How to choose: a simple checklist
- Quality: is it good enough at your task? Test with 10–20 real examples.
- Cost: how many requests per day × tokens per request × price per token?
- Speed: does the user need an answer in under a second (chat) or can it wait (overnight report)?
- Privacy: can this data leave your organisation?
- Language: does it handle Hindi, Kannada, Tamil, etc. well?
- Context length: do you need to fit long documents?
- Licence: are you allowed to use it commercially?
24-hour hackathon prototype
A closed model through an API, or a free tier (Gemini free tier, Groq). Fastest to start; quality matters more than cost here.Hospital summarising patient records; data must not leave the building
An open-weight model running on the hospital's own servers.Classifying 10 lakh customer reviews a day as positive / negative
A small, cheap, fast model. A giant model is overkill for a simple task at high volume.Customer support chatbot in Kannada
Test several models on real Kannada questions before choosing. Language quality varies a lot.10. Recap
- AI ⊃ ML ⊃ Deep Learning ⊃ LLMs. ML learns rules from examples.
- Transformers use attention to understand each word using the words around it.
- An LLM predicts the next token, again and again. Training stages turn it into an assistant.
- Tokens are the unit of input, cost and limits. Embeddings turn meaning into numbers.
- The context window is the model's short-term memory. LLMs don't remember between calls.
- Temperature: low for facts, high for creativity.
- LLMs hallucinate because they generate plausible text. Ground them with sources.
- Choose models by quality, cost, speed, privacy, language and licence, not by hype.
Quick quiz
1. An ML spam filter is given 10,000 labelled emails. What does it produce?
A model: rules/patterns it learned to decide whether a new email is spam.2. What does "attention" help a transformer do?
Decide which other words in the input matter for understanding each word (e.g. "bank" near "money" vs near "river").3. Roughly how many tokens is a 750-word English essay?
About 1,000 tokens (1 token ≈ ¾ word).4. Why does a chatbot "forget" the start of a very long conversation?
The conversation exceeded the context window, so older messages were dropped or summarised.5. You're extracting student names and marks into JSON. High or low temperature?
Low (around 0). You want consistent, exact output.6. Give two reasons LLMs hallucinate and one way to reduce it.
They predict plausible text, not verified facts; they have a knowledge cutoff; training rewards answering. Reduce it by giving source documents (RAG), allowing "I don't know", lowering temperature, or checking citations.7. A bank wants an internal assistant, and customer data can't leave its servers. Open or closed model?
An open-weight model hosted on the bank's own infrastructure (or a closed model only if the provider offers a deployment that meets the bank's data rules).Glossary
| Term | Meaning in one line |
|---|---|
| Model | The trained program that makes predictions. |
| Parameters | The numbers inside a model that store what it learned. |
| Neural network | Layers of simple maths units that learn patterns from data. |
| Transformer | The neural network design behind LLMs; uses attention. |
| Token | A chunk of text the model reads and writes. |
| Embedding | A list of numbers representing the meaning of text. |
| Context window | Max tokens the model can see at once (input + output). |
| Temperature | How random or creative the model's word choice is. |
| Hallucination | A confident but false answer. |
| Knowledge cutoff | The date after which the model has no training data. |
| Pre-training / fine-tuning | Learning from huge general text / then from smaller, focused examples. |
| RLHF | Reinforcement Learning from Human Feedback: humans rank answers to improve the model. |
| Open-weight model | A model whose trained parameters you can download and run. |
Learn more (free)
- 3Blue1Brown, Neural Networks video series on YouTube (includes visual explanations of transformers and attention)
- Andrej Karpathy, Intro to Large Language Models (1-hour talk on YouTube)
- Jay Alammar, The Illustrated Transformer (blog post)
- Vaswani et al., Attention Is All You Need (2017), the original transformer paper
Instructor guide
Suggested plan for the 60 minutes. Press S in the slides to open speaker notes.
| Time | Segment | What to do |
|---|---|---|
| 0–10 min | Intro + hook | Introduce yourself. Ask: who is worried about layoffs or AI taking jobs? Walk through why layoffs are happening, what to improve and what to learn. Keep it encouraging, not scary. Then: who used ChatGPT / Gemini / Claude this week? "How does it know?" |
| 10–18 min | AI → ML → DL → Transformers | Spam filter, placement prediction, face unlock, "bank" and "it" examples. |
| 18–22 min | Next-word prediction | Game: say "Virat Kohli is famous for playing..." and let the class finish it. Then "Once upon a..." The class is the LLM. |
| 22–35 min | Tokens, embeddings, context, temperature | Live tokenizer demo (English vs Hindi). Temperature slider demo. Desk analogy for context. |
| 35–45 min | Hallucination | Live: ask a chatbot about the fake "Quantum Biryani Optimization" paper, or about a fake event at your college. Discuss why. |
| 45–55 min | Open vs closed, choosing | Run the four scenarios as a class vote before revealing answers. If short on time, do two scenarios. |
| 55–60 min | Recap + quiz | Ask 3–4 quiz questions aloud. Preview Session 2: prompt engineering. |