The most valuable skill in AI engineering right now
Start With a Real Problem
Imagine you're a lawyer at a big firm.
Your firm has 10 years of case files — thousands of PDFs, contracts, legal documents, internal memos. All stored on your company's servers.
You get a new case. You need to find relevant precedents fast.
You open ChatGPT and ask:
"What similar cases has our firm handled involving
breach of contract in software agreements?"
ChatGPT's response:
"I don't have access to your firm's internal
case files. I can only provide general information
about breach of contract law..."
Useless. ChatGPT knows nothing about your firm's private documents.
Now you think — okay, let me just paste the documents in:
You try to paste 10 years of case files into ChatGPT
→ 10,000+ pages
→ Millions of tokens
→ Context window: 128,000 tokens max
→ Doesn't fit ❌
→ Even if it did — costs hundreds of dollars per query ❌
This is the problem RAG solves.
What RAG Stands For
R → Retrieval — find relevant information first
A → Augmented — add that information to the prompt
G → Generation — LLM generates answer using that information
RAG = find relevant docs first → inject into prompt → LLM answers from those docs.
The Core Idea — One Sentence
Instead of making the LLM memorize everything — give it only the relevant information it needs, right when it needs it.
Like an open book exam.
Closed book exam (normal LLM):
→ Student must memorize everything beforehand
→ If something wasn't memorized — student guesses
→ Guessing = hallucination
Open book exam (RAG):
→ Student has access to books during the exam
→ Looks up relevant pages for each question
→ Answers from what's written — no guessing needed
→ Answer is accurate and sourced
Why RAG Exists — Three Problems It Solves
Problem 1 — LLMs Have a Knowledge Cutoff
Every LLM was trained on data up to a certain date.
GPT-4 training cutoff: early 2024
User asks in July 2026: "What happened in the stock market last week?"
→ GPT-4 has no idea ❌
→ It might hallucinate an answer ❌
RAG solution:
Fetch last week's stock data → inject into prompt → LLM answers from it ✅
Problem 2 — LLMs Don't Know Your Private Data
Your company's internal data:
→ Employee handbook
→ Product documentation
→ Customer support history
→ Legal contracts
→ Financial reports
GPT-4 has never seen any of this.
It cannot answer questions about it. ❌
RAG solution:
Store your private docs in vector DB
→ Find relevant chunks for each question
→ Inject into prompt
→ LLM answers from your actual data ✅
Problem 3 — Hallucination
LLMs sometimes confidently generate wrong information. They don't know what they don't know.
User: "What is the dosage of MedicinX for children?"
LLM without RAG:
→ "The recommended dosage is 5mg twice daily..."
→ Completely made up ❌ — dangerous in medical context
LLM with RAG:
→ Retrieves actual MedicinX dosage guide from DB
→ "According to the MedicinX prescribing guide page 4:
dosage for children under 12 is 2.5mg once daily"
→ Sourced, accurate, verifiable ✅
RAG grounds the LLM in real information. It can't hallucinate about what's written in the document.
How RAG Works — The Complete Flow
RAG has two separate phases:
Phase A — Indexing (Done Once)
This is the setup phase. You do this when you first build the system or when documents are updated.
Your Documents (PDFs, Word files, websites, etc.)
↓
Load the documents
↓
Split into chunks
(each chunk = ~500 words)
↓
Embed each chunk
(OpenAI API → 1536 numbers)
↓
Store in Vector Database
(Chroma / Pinecone)
↓
Done — system is ready
Phase B — Querying (Every Time User Asks)
This happens in real time — every time a user asks a question.
User types a question
↓
Embed the question
(same OpenAI embedding model)
↓
Search Vector Database
(find top 3-5 most similar chunks)
↓
Get back relevant text chunks
↓
Build a prompt:
System: "Answer using only the context below"
Context: [chunk 1] [chunk 2] [chunk 3]
Question: [user's question]
↓
Send to LLM (GPT-4o)
↓
LLM reads the context and generates answer
↓
User gets accurate, sourced answer
Visualizing the Complete System
┌─────────────────────────────────────────────────────────┐
│ RAG SYSTEM │
│ │
│ INDEXING PIPELINE (runs once): │
│ │
│ [PDF] [Word] [Web] │
│ ↓ │
│ Document Loader │
│ ↓ │
│ Text Splitter → [chunk1] [chunk2] [chunk3] ... │
│ ↓ │
│ Embedding Model (OpenAI) │
│ ↓ │
│ Vector Database ← stores vectors + text + metadata │
│ │
│ ───────────────────────────────────────────────────── │
│ │
│ QUERY PIPELINE (runs every request): │
│ │
│ User Question │
│ ↓ │
│ Embedding Model (OpenAI) │
│ ↓ │
│ Vector Database → similarity search → top 5 chunks │
│ ↓ │
│ Prompt Builder │
│ ┌──────────────────────────────────┐ │
│ │ System: Answer from context only │ │
│ │ Context: [chunk1][chunk2][chunk3]│ │
│ │ Question: [user question] │ │
│ └──────────────────────────────────┘ │
│ ↓ │
│ LLM (GPT-4o) │
│ ↓ │
│ Answer (grounded in real documents) │
└─────────────────────────────────────────────────────────┘
RAG vs Fine-Tuning — Common Confusion
People often ask — why not just fine-tune the LLM on your data instead of using RAG?
Great question. Here's the difference:
FINE-TUNING:
→ Train the model further on your data
→ Model "memorizes" your data into its weights
→ Expensive — costs thousands of dollars
→ Takes days to train
→ When data changes — retrain again
→ Model can still hallucinate
→ Good for: teaching style, tone, format
RAG:
→ Keep base model as is
→ Retrieve relevant data at query time
→ Cheap — just API calls
→ Real time — works instantly
→ When data changes — just update vector DB
→ Model answers from actual retrieved text
→ Good for: knowledge, facts, private data
For most real applications — RAG is the right choice. Fine-tuning is for when you want to change HOW the model responds, not WHAT it knows.
Fine-tune when:
→ "I want the model to always respond like a pirate"
→ "I want the model to output only JSON"
→ "I want the model to match our brand voice"
RAG when:
→ "I want the model to know our company's documents"
→ "I want the model to answer from our product manual"
→ "I want the model to have up-to-date information"
Real World RAG Applications
RAG is everywhere right now:
Customer Support Bot:
→ Index: company FAQs + product docs + past tickets
→ Query: customer asks a question
→ Answer: from actual company documentation
Legal Research Tool:
→ Index: thousands of case files + contracts
→ Query: lawyer asks about similar cases
→ Answer: from actual firm documents
Medical Assistant:
→ Index: medical handbooks + drug guides + research papers
→ Query: doctor asks about drug interactions
→ Answer: from verified medical sources
Internal Company Chatbot:
→ Index: HR policies + engineering docs + meeting notes
→ Query: employee asks any internal question
→ Answer: from actual company knowledge base
Code Documentation Bot:
→ Index: your codebase + README files + API docs
→ Query: developer asks how something works
→ Answer: from your actual code and docs
Every single one of these is the same RAG pattern. Learn it once — build anything.
What You'll Build in Phase 5
By the end of this phase you will have built a complete RAG system:
Module 5.1 → What is RAG (this module)
Module 5.2 → Chunking — how to split documents properly
Module 5.3 → Retrieval + Context Injection
Module 5.4 → Hallucination and Grounding
Module 5.5 → Project — Full PDF Chatbot
Upload any PDF → Ask any question → Get accurate answers
The PDF chatbot will be a real, working application. Not a toy. Something you can actually show in a portfolio or use in a real project.
3-Line Summary
- RAG exists because LLMs have knowledge cutoffs, don't know your private data, and hallucinate — RAG solves all three by retrieving relevant documents first and injecting them into the prompt before the LLM generates an answer.
- RAG has two phases — indexing (split documents into chunks, embed them, store in vector DB — done once) and querying (embed the question, find similar chunks, inject into prompt, LLM answers — done every request).
- RAG is better than fine-tuning for most real applications — it's cheaper, faster to update, works instantly with new data, and grounds the LLM in actual retrieved text instead of memorized weights.
Module 5.1 — Complete ✅
Coming up — Module 5.2 — Chunking
This is more important than most people realize. How you split your documents directly determines how good your RAG system is. Split too small — you lose context. Split too large — you lose precision. We cover fixed chunking, semantic chunking, overlap, and exactly how to decide chunk size for different use cases.