The biggest problem in production AI — and exactly how to fight it
Start With a Scary Example
Imagine you've built a medical RAG chatbot for a hospital.
A doctor asks:
"What is the maximum daily dose of paracetamol for a
70kg adult with mild liver disease?"
Your RAG system fails to find relevant chunks — maybe the document wasn't indexed properly. So no context gets injected.
The LLM responds anyway:
"For a 70kg adult with mild liver disease, the maximum
daily dose of paracetamol is 3000mg, taken as 500mg
every 4 hours."
Sounds confident. Sounds medical. Sounds correct.
But it's wrong. The actual answer is much more nuanced — liver disease patients often need significantly reduced doses, sometimes as low as 1000-2000mg daily, and should always consult a specialist.
The doctor trusts the AI. Prescribes accordingly. Patient is harmed.
This is hallucination. And it kills people in high-stakes domains.
What Exactly is Hallucination?
Hallucination = when an LLM generates information that is confident, fluent, and wrong.
The word "hallucination" is perfect. A person who is hallucinating sees things that feel completely real to them — they're not lying, they genuinely believe what they're experiencing. Same with LLMs — the model isn't "trying" to lie. It's generating the most probable next token — and sometimes that probability leads to false information stated as fact.
Three types of hallucination:
Type 1 — Factual Hallucination
The model states a wrong fact confidently:
User: "Who invented the telephone?"
LLM: "The telephone was invented by Elisha Gray in 1876."
Reality: Alexander Graham Bell is credited with the patent,
though the history is complex. Elisha Gray also filed
a patent — but saying Gray "invented" it is wrong.
Type 2 — Source Hallucination
The model makes up references that don't exist:
User: "Give me research papers about RAG systems"
LLM: "Here are some key papers:
1. 'Retrieval Augmented Generation for Knowledge-Intensive NLP Tasks'
by Lewis et al., 2020 — NeurIPS
2. 'Advanced RAG Techniques' by Zhang et al., 2023 — ICML
3. 'Contextual Grounding in LLMs' by Patel et al., 2024 — ACL"
Reality: Paper 1 is real.
Papers 2 and 3 — completely made up.
But they look real. DOI numbers, conference names, author names.
Someone might actually try to find these papers.
Type 3 — Context Hallucination
The model ignores the provided context and generates from its own knowledge:
Context injected: "Our return policy allows returns within 14 days."
User: "How long do I have to return a product?"
LLM: "You have 30 days to return a product."
The LLM ignored the 14-day policy in the context and used
its training data about typical return policies instead.
This is extremely common and very dangerous.
Why Does Hallucination Happen?
Understanding the cause helps you prevent it.
Remember from Module 1.2 — an LLM generates text by predicting the most probable next token. It has no concept of "truth." It has no internal fact-checker. It just generates what seems most likely given the input.
Three root causes:
1. Training data had wrong information
→ Model learned wrong patterns
→ Confidently repeats them
2. Question is about something not in training data
→ Model has no pattern to follow
→ Fills the gap with plausible-sounding text
→ "Plausible" ≠ "true"
3. Context window has too much information
→ Model loses track of what's actually in context
→ Falls back on training data instead
→ Ignores the correct information you provided
What is Grounding?
Grounding is the opposite of hallucination.
A grounded response is one where every claim can be traced back to a specific source in the provided context.
Hallucinated response:
"Aspirin dosage for adults is 500mg every 6 hours."
→ Where did this come from? Nobody knows.
→ Could be right. Could be wrong.
→ Not verifiable.
Grounded response:
"According to medical_handbook.pdf page 23:
the recommended adult aspirin dose is 325-650mg every 4-6 hours."
→ Clear source
→ Verifiable
→ If it's wrong — you can find and fix the source document
Grounding = every answer has a traceable source.
How RAG Reduces Hallucination
RAG doesn't eliminate hallucination — but it dramatically reduces it when done correctly.
Here's why:
WITHOUT RAG:
LLM only has its training data
→ If it doesn't know → it guesses
→ Guesses sound confident
→ Hallucination
WITH RAG:
LLM gets real context injected
→ "Answer ONLY from this context"
→ If context has the answer → uses it correctly
→ If context doesn't have it → should say "I don't know"
→ Much less hallucination
But RAG is not magic. It only works if you do it right.
The Specific Ways RAG Can Still Fail
Even with RAG — hallucination can happen. Here's how:
Failure 1 — Wrong chunks retrieved:
User: "What is the aspirin dose for children?"
Retrieved chunk: About adult aspirin dosage
LLM: Answers about children using adult dose information
→ Wrong answer from wrong context
Failure 2 — Weak system prompt:
System prompt: "You are a helpful assistant. Use the context."
→ "Use the context" is too vague
→ LLM might use context + its own knowledge mixed together
→ Impossible to tell which parts are hallucinated
Failure 3 — Empty context:
No relevant chunks found → no context injected
LLM: Still answers from training data
→ Full hallucination — no grounding at all
Failure 4 — Context too long:
50 chunks injected → 40,000 tokens of context
LLM: Gets confused → loses track → starts hallucinating
→ More context is not always better
The Anti-Hallucination Toolkit
Here are the specific techniques — with code — that you use to fight hallucination.
Technique 1 — The "Only Use Context" System Prompt
This is the most important single thing you can do.
// ❌ WEAK system prompt — invites hallucination
const weakSystem = `You are a helpful assistant.
Answer the user's question using the provided context.`;
// Context is mentioned but LLM can still use other knowledge
// ✅ STRONG system prompt — forces grounding
const strongSystem = `You are a precise assistant that answers questions
STRICTLY from the provided context.
ABSOLUTE RULES:
1. You may ONLY use information explicitly stated in the context below
2. If the answer is not in the context — respond with exactly:
"I don't have that information in my knowledge base."
3. NEVER use your training knowledge to fill gaps
4. NEVER make assumptions beyond what's written
5. Always cite which source your answer comes from
6. If the context is partially relevant —
use what's there and say what's missing
Violating these rules causes real harm. Follow them exactly.`;
The difference between these two prompts is enormous in production.
Technique 2 — Empty Context Detection
Never send the LLM an empty context. Handle it before the LLM call:
async function safeRagQuery(question, chunks, openai) { // question = user question // chunks = retrieved chunks array
// Check if we have any context at all if (chunks.length === 0) { // No chunks found — do NOT call LLM // LLM would hallucinate an answer return { answer: "I don't have information about this topic in my knowledge base. " + "Please try rephrasing your question or contact support.", grounded: false, // flag that this answer is NOT from documents source: "no_context" }; }
// Check if best chunk is actually relevant const bestScore = chunks[0].similarity; if (bestScore < 0.5) { // Even best chunk is below 50% similar // Retrieved context is probably wrong topic return { answer: "I found some information but I'm not confident it answers " + "your specific question. Could you rephrase or be more specific?", grounded: false, source: "low_confidence", topScore: bestScore }; }
// Good context found — proceed with LLM call const { systemPrompt, userPrompt } = buildPrompt(question, chunks); const answer = await generateAnswer(systemPrompt, userPrompt, openai);
return { answer: answer, grounded: true, source: "knowledge_base", chunksUsed: chunks.length, topScore: bestScore }; }
Technique 3 — Confidence Scoring in the Prompt
Ask the LLM to tell you how confident it is:
function buildPromptWithConfidence(question, chunks) {const contextSection = chunks.map((chunk, i) => `[Source ${i + 1} — ${chunk.metadata.source} p.${chunk.metadata.page}]\n${chunk.text}`).join("\n\n---\n\n");const systemPrompt = `You are a precise assistant. Answer ONLY from the provided context.Never use knowledge outside the context.`;const userPrompt = `Context:${contextSection}Question: ${question}Answer in this exact format:CONFIDENCE: [HIGH/MEDIUM/LOW]REASON: [one sentence explaining your confidence level]ANSWER: [your answer here, citing sources]SOURCES_USED: [list which sources you used]If the answer is not in the context:CONFIDENCE: NONEREASON: Information not found in provided contextANSWER: I don't have that information in my knowledge base.SOURCES_USED: None`;return { systemPrompt, userPrompt };}
Example LLM output:
CONFIDENCE: HIGH
REASON: Source 1 directly addresses aspirin side effects with specific details.
ANSWER: According to Source 1 (medical_handbook.pdf p.23), common side
effects of aspirin include stomach irritation, nausea, and heartburn.
Long-term use may cause gastrointestinal bleeding (Source 2, p.7).
SOURCES_USED: Source 1, Source 2
Now your application knows HOW confident the answer is. You can show this to users or use it to decide whether to show the answer at all.
Technique 4 — Answer Verification
After getting the LLM answer — verify that claims in the answer actually appear in the context:
async function verifyAnswer(answer, chunks, openai) { // answer = LLM generated answer string // chunks = the chunks that were provided as context
const contextText = chunks.map(c => c.text).join("\n\n"); // combine all context into one string for verification
const verificationResponse = await openai.chat.completions.create({ model: "gpt-4o", temperature: 0, // temperature 0 = consistent, deterministic verification messages: [ { role: "system", content: `You are a fact-checker. Your job is to check if an answer is supported by the provided context. Be strict — only mark as supported if explicitly stated in context. Return JSON only.` }, { role: "user", content: `Context: ${contextText}
Answer to verify: ${answer}
Return this exact JSON: { "supported": true or false, "confidence": "high" or "medium" or "low", "unsupportedClaims": ["list any claims not found in context"], "supportedClaims": ["list claims that ARE in context"] }` } ] });
const raw = verificationResponse.choices[0].message.content; // raw = JSON string from LLM
try { return JSON.parse(raw); // parse JSON string into object // example: { supported: true, confidence: "high", ... } } catch (e) { return { supported: null, error: "Could not parse verification" }; } }
Usage:
const answer = await generateAnswer(systemPrompt, userPrompt, openai); const verification = await verifyAnswer(answer, chunks, openai);
if (!verification.supported) { console.log("⚠️ Answer may contain hallucinations:"); console.log("Unsupported claims:", verification.unsupportedClaims); // Either block the answer or flag it to the user }
This is called self-verification — using an LLM to check another LLM's output. Expensive but extremely effective for high-stakes applications.
Technique 5 — Citation Enforcement
Force the LLM to cite exactly where every claim comes from:
function buildCitationPrompt(question, chunks) {
const contextSection = chunks .map((chunk, i) => `[DOC_${i + 1}]: ${chunk.text}`) .join("\n\n"); // label each chunk with DOC_1, DOC_2, etc.
const systemPrompt = `You answer questions using ONLY the provided documents. Every single sentence in your answer MUST end with a citation in the format [DOC_X] where X is the document number. If you cannot cite a sentence — do not include that sentence.`;
const userPrompt = `Documents: ${contextSection}
Question: ${question}
Answer (every sentence must have a [DOC_X] citation):`;
return { systemPrompt, userPrompt }; }
Example output:
Aspirin is commonly used to reduce fever and pain [DOC_1].
Common side effects include stomach irritation and nausea [DOC_1].
Long-term use may cause gastrointestinal bleeding [DOC_2].
Aspirin is not recommended for children under 16 [DOC_3].
Every sentence cited. Nothing without a source. If the LLM tries to add something uncited — the instruction stops it.
Putting It All Together — Production Anti-Hallucination Pipeline
async function productionRagQuery(userQuestion, collection, openai) {
console.log(`\nProcessing: "${userQuestion}"`);
// Step 1 — Embed and retrieve const queryVector = await embedQuery(userQuestion, openai); const rawChunks = await retrieveChunks(queryVector, collection, { topK: 10, scoreThreshold: 0.45, });
// Step 2 — Empty context check if (rawChunks.length === 0) { return { answer: "I don't have information about this in my knowledge base.", hallucination_risk: "prevented — no context found", grounded: false }; }
// Step 3 — Take only top 5 chunks const chunks = rawChunks.slice(0, 5); // more than 5 adds noise — LLM gets confused
// Step 4 — Build prompt with citations const { systemPrompt, userPrompt } = buildCitationPrompt(userQuestion, chunks);
// Step 5 — Generate answer const answer = await generateAnswer(systemPrompt, userPrompt, openai);
// Step 6 — Verify answer (for high-stakes apps) const verification = await verifyAnswer(answer, chunks, openai);
if (!verification.supported) { // Answer not supported by context return { answer: "I found some relevant information but couldn't generate " + "a fully verified answer. Please consult the source documents directly.", hallucination_risk: "high — answer not supported by context", grounded: false, unsupportedClaims: verification.unsupportedClaims }; }
// Step 7 — Return grounded answer return { answer: answer, hallucination_risk: "low — answer verified against context", grounded: true, chunksUsed: chunks.length, topSimilarity: chunks[0].similarity, sources: chunks.map(c => `${c.metadata.source} p.${c.metadata.page}`) }; }
Hallucination Risk by Domain
Different applications need different levels of protection:
LOW RISK — basic protection enough:
→ Internal company FAQ bot
→ Product recommendation chatbot
→ General knowledge assistant
Techniques: Strong system prompt + empty context check
MEDIUM RISK — moderate protection needed:
→ Customer support bot
→ Legal document search
→ Financial report Q&A
Techniques: Above + citation enforcement + confidence scoring
HIGH RISK — maximum protection required:
→ Medical information systems
→ Safety-critical documentation
→ Compliance and regulatory apps
Techniques: All above + answer verification + human review
+ explicit disclaimer on every response
The Mental Model — RAG as a Safety Net
Without RAG:
LLM = person answering from memory only
→ Smart but can misremember
→ Fills gaps with confident guesses
→ No way to verify claims
With RAG (done right):
LLM = person answering with documents open in front of them
→ Reads from actual documents
→ Cites the page they're reading from
→ Says "it's not in here" when it's not
→ Every claim verifiable
Your job as developer:
→ Make sure the right documents are open
→ Make sure the LLM actually reads them
→ Make sure it cites what it reads
→ Handle the cases when nothing relevant is found
3-Line Summary
- Hallucination happens because LLMs predict probable tokens — not verified facts — so they confidently generate wrong information when they don't know the answer or when context is missing or ignored.
- RAG reduces hallucination by injecting real context — but only when combined with a strict system prompt that forbids using outside knowledge, empty context detection that blocks LLM calls when nothing relevant is found, and citation enforcement that requires every claim to reference a source.
- For production applications — match your anti-hallucination investment to your risk level — a general chatbot needs a strong system prompt, but a medical or legal application needs citation enforcement plus answer verification plus human review.
Module 5.4 — Complete ✅
Coming up — Module 5.5 — Project: Full PDF Chatbot
Everything we've learned — chunking, embeddings, vector database, retrieval, context injection, anti-hallucination — comes together in one complete working project. You'll build a real PDF chatbot that you can upload any PDF to and ask questions about. Real code, real output, portfolio-ready project.
No comments:
Post a Comment