Everything from Phase 1 to Phase 5 — built into one real working application
What You're Building
A complete PDF chatbot that:
1. Takes any PDF file
2. Extracts the text
3. Splits into chunks
4. Embeds each chunk with OpenAI
5. Stores in vector database
6. Accepts user questions
7. Retrieves relevant chunks
8. Generates grounded answers with sources
This is a real, working application. Not a toy. Something you can show in a portfolio, extend into a product, or use in actual projects.
Project Structure
pdf-chatbot/
├── .env ← API keys
├── package.json ← Node.js config
├── src/
│ ├── pdfLoader.js ← Extract text from PDF
│ ├── chunker.js ← Split text into chunks
│ ├── vectorStore.js ← Store and search chunks
│ ├── retriever.js ← Find relevant chunks
│ ├── generator.js ← Generate answers with LLM
│ └── chatbot.js ← Main pipeline — ties everything together
├── index.js ← Entry point — run the chatbot
└── sample.pdf ← Any PDF you want to test with
Modular structure — each file has one job. This is how real applications are built.
Step 1 — Project Setup
mkdir pdf-chatbot cd pdf-chatbot npm init -y
Update package.json:
{ "name": "pdf-chatbot", "version": "1.0.0", "type": "module", "scripts": { "start": "node index.js" } }
Install packages:
npm install openai pdf-parse dotenv readline
What each package does:
openai → OpenAI API for embeddings + chat
pdf-parse → Extract text from PDF files
dotenv → Read .env file
readline → Built-in Node.js — for terminal chat interface
Create .env:
OPENAI_API_KEY=sk-proj-your-key-here
Step 2 — PDF Loader
src/pdfLoader.js — extracts raw text from a PDF file:
import fs from "fs"; // fs = Node.js built-in file system module // used to read files from disk
import pdfParse from "pdf-parse"; // pdf-parse = npm package that reads PDF files // extracts text from each page
export async function loadPDF(filePath) { // filePath = path to the PDF file // example: "./sample.pdf" or "./documents/report.pdf"
console.log(`📄 Loading PDF: ${filePath}`);
// Check if file exists before trying to read it if (!fs.existsSync(filePath)) { throw new Error(`PDF file not found: ${filePath}`); // throw error so calling code can handle it }
const fileBuffer = fs.readFileSync(filePath); // readFileSync = read entire file into memory as a Buffer // Buffer = raw binary data (PDFs are binary files, not plain text) // example: <Buffer 25 50 44 46 2d 31 2e 34 ...>
const pdfData = await pdfParse(fileBuffer); // pdfParse = reads the binary buffer and extracts text // pdfData contains: // { // numpages: 10, ← how many pages in the PDF // text: "full text...", ← ALL text from all pages combined // info: { Title: "..." } ← PDF metadata // }
console.log(` ✅ Loaded ${pdfData.numpages} pages`); console.log(` Total characters: ${pdfData.text.length}`);
// Clean the extracted text const cleanedText = pdfData.text .replace(/\r\n/g, "\n") // normalize line endings // Windows uses \r\n, Unix uses \n — standardize to \n
.replace(/\n{3,}/g, "\n\n") // replace 3+ consecutive newlines with just 2 // PDFs often have lots of extra blank lines
.replace(/[^\S\n]+/g, " ") // replace multiple spaces/tabs with single space // but keep newlines (the \n is excluded from replacement)
.trim(); // remove leading and trailing whitespace
console.log(` After cleaning: ${cleanedText.length} characters\n`);
return { text: cleanedText, // the full cleaned text of the PDF // example: "Chapter 1: Introduction\n\nThis document covers..."
pageCount: pdfData.numpages, // total number of pages // example: 10
fileName: filePath.split("/").pop().split("\\").pop(), // just the filename without the path // example: "./documents/report.pdf" → "report.pdf" }; }
Step 3 — Chunker
src/chunker.js — splits the PDF text into chunks:
export function chunkText(text, fileName, options = {}) {
const { chunkSize = 800, overlap = 150, minChunkSize = 100, } = options;
console.log(`✂️ Chunking text...`); console.log(` Chunk size: ${chunkSize} chars`); console.log(` Overlap: ${overlap} chars`);
const chunks = []; let startIndex = 0; let chunkIndex = 0;
while (startIndex < text.length) {
let endIndex = Math.min(startIndex + chunkSize, text.length); // endIndex can never exceed text.length
// Try to find a natural boundary — only look FORWARD from 60% mark if (endIndex < text.length) { const searchArea = text.substring(startIndex, endIndex); const sixtyPercent = Math.floor(chunkSize * 0.6); // only look for boundary after 60% of chunk
const lastParagraph = searchArea.lastIndexOf("\n\n"); if (lastParagraph > sixtyPercent) { endIndex = startIndex + lastParagraph + 2; } else { const lastPeriod = Math.max( searchArea.lastIndexOf(". "), searchArea.lastIndexOf("? "), searchArea.lastIndexOf("! "), searchArea.lastIndexOf(".\n") ); if (lastPeriod > sixtyPercent) { endIndex = startIndex + lastPeriod + 2; } } }
// ── CRITICAL FIX ────────────────────────────────────── // endIndex MUST always be greater than startIndex // Without this — infinite loop when boundary adjustment fails if (endIndex <= startIndex) { endIndex = startIndex + chunkSize; // force move forward — ignore boundary preference } if (endIndex > text.length) { endIndex = text.length; } // ──────────────────────────────────────────────────────
const chunk = text.slice(startIndex, endIndex).trim();
if (chunk.length >= minChunkSize) { chunks.push({ id: `${fileName}_chunk_${chunkIndex}`, text: chunk, metadata: { source: fileName, chunkIndex: chunkIndex, startChar: startIndex, charCount: chunk.length, }, }); chunkIndex++; }
const nextStart = endIndex - overlap; // ── CRITICAL FIX ────────────────────────────────────── // nextStart MUST always be greater than current startIndex // This is what prevents infinite loop if (nextStart <= startIndex) { startIndex = startIndex + Math.floor(chunkSize / 2); // force jump forward by half chunk size } else { startIndex = nextStart; } // ────────────────────────────────────────────────────── }
console.log(` ✅ Created ${chunks.length} chunks\n`); return chunks; }
Step 4 — Vector Store
src/vectorStore.js — stores and searches chunks using embeddings:
import OpenAI from "openai"; import * as dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
// In-memory vector store // Works without any server — perfect for this project // For production — replace with Pinecone or Chroma const store = []; // store = array of objects // each object = { id, text, metadata, embedding }
// ───────────────────────────────────────── // Get embedding for one text // ─────────────────────────────────────────
async function getEmbedding(text) { // text = string to embed // example: "Aspirin reduces fever and pain..."
const response = await openai.embeddings.create({ model: "text-embedding-3-small", // 1536 dimensions, cheap, fast, good quality input: text.substring(0, 8000), // limit to 8000 chars — embedding model has token limit // 8000 chars ≈ 2000 tokens — well within limits });
return response.data[0].embedding; // array of 1536 numbers }
// ───────────────────────────────────────── // Cosine similarity between two vectors // ─────────────────────────────────────────
function cosineSimilarity(vecA, vecB) { const dot = vecA.reduce((sum, val, i) => sum + val * vecB[i], 0); // dot product = sum of products of matching elements
const magA = Math.sqrt(vecA.reduce((sum, val) => sum + val * val, 0)); const magB = Math.sqrt(vecB.reduce((sum, val) => sum + val * val, 0)); // magnitude = square root of sum of squares
return dot / (magA * magB); // cosine similarity = dot product / (magA × magB) // returns -1 to 1, higher = more similar }
// ───────────────────────────────────────── // Add chunks to vector store // ─────────────────────────────────────────
export async function addChunks(chunks) { // chunks = array of chunk objects from chunker.js // each has: id, text, metadata
console.log(`💾 Embedding and storing ${chunks.length} chunks...`); console.log(" (This calls OpenAI API once per chunk)\n");
// Clear existing store — fresh start for each PDF store.length = 0; // store.length = 0 empties the array without creating a new one
let successCount = 0;
for (let i = 0; i < chunks.length; i++) { const chunk = chunks[i];
try { const embedding = await getEmbedding(chunk.text); // call OpenAI to get 1536 numbers for this chunk's text
store.push({ id: chunk.id, // example: "report.pdf_chunk_0"
text: chunk.text, // original text
metadata: chunk.metadata, // source, chunkIndex, startChar, charCount
embedding: embedding, // [0.023, -0.089, ...] × 1536 });
successCount++;
// Progress indicator if ((i + 1) % 5 === 0 || i === chunks.length - 1) { // print progress every 5 chunks and at the end console.log(` Progress: ${i + 1}/${chunks.length} chunks embedded`); }
} catch (error) { console.log(` ⚠️ Failed to embed chunk ${i}: ${error.message}`); // skip this chunk and continue with the rest } }
console.log(`\n ✅ Stored ${successCount} chunks in vector store\n`); }
// ───────────────────────────────────────── // Search for similar chunks // ─────────────────────────────────────────
export async function searchChunks(query, topK = 5, threshold = 0.3) { // query = user question string // topK = how many results to return // threshold = minimum similarity score to include
if (store.length === 0) { throw new Error("Vector store is empty. Please load a PDF first."); }
// Embed the query const queryEmbedding = await getEmbedding(query); // queryEmbedding = [0.19, -0.82, ...] × 1536
// Calculate similarity against every stored chunk const scored = store.map((item) => ({ ...item, // spread all existing properties (id, text, metadata, embedding) score: cosineSimilarity(queryEmbedding, item.embedding), // add similarity score // example: 0.87 for very relevant chunk }));
// Sort by score — highest first scored.sort((a, b) => b.score - a.score);
// Filter by threshold and take topK const results = scored .filter((item) => item.score >= threshold) // only keep items above minimum similarity .slice(0, topK) // take top K results .map((item) => ({ id: item.id, text: item.text, metadata: item.metadata, score: item.score, // don't include embedding in results — not needed + takes memory }));
return results; // array of top matching chunks // each has: id, text, metadata, score }
// ───────────────────────────────────────── // Get store stats // ─────────────────────────────────────────
export function getStoreStats() { return { totalChunks: store.length, // how many chunks are stored isEmpty: store.length === 0, // true if nothing stored yet }; }
Step 5 — Retriever
src/retriever.js — finds relevant chunks for a question:
import { searchChunks } from "./vectorStore.js";
export async function retrieveContext(question, options = {}) { // question = user's question string // options = optional settings
const { topK = 5, // return top 5 chunks threshold = 0.3, // minimum similarity score } = options;
// Search vector store const chunks = await searchChunks(question, topK, threshold); // chunks = array of matching chunk objects with scores
// Assess quality of retrieval if (chunks.length === 0) { return { chunks: [], quality: "none", message: "No relevant content found for this question." }; }
const topScore = chunks[0].score; // highest similarity score
let quality; if (topScore >= 0.6) { quality = "high"; // clearly relevant content found } else if (topScore >= 0.4) { quality = "medium"; // possibly relevant content found } else { quality = "low"; // content found but not very similar }
return { chunks: chunks, // array of relevant chunk objects
quality: quality, // "high", "medium", or "low"
topScore: topScore, // best similarity score found // example: 0.87
message: `Found ${chunks.length} relevant sections (best match: ${(topScore * 100).toFixed(1)}%)` }; }
Step 6 — Generator
src/generator.js — generates grounded answers using the LLM:
import OpenAI from "openai"; import * as dotenv from "dotenv";
dotenv.config();
const openai = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
export async function generateAnswer(question, chunks) { // question = user's question // chunks = relevant chunks from retriever
// Handle case where no relevant chunks found if (chunks.length === 0) { return { answer: "I couldn't find relevant information in the document to answer your question. " + "Try rephrasing or asking about a different topic covered in the PDF.", sources: [], grounded: false, }; }
// Build context from chunks const contextSection = chunks .map((chunk, index) => { return `[Section ${index + 1}] Source: ${chunk.metadata.source} Relevance: ${(chunk.score * 100).toFixed(0)}%
${chunk.text}`; // format each chunk clearly // include source file and relevance score // example: // [Section 1] // Source: report.pdf // Relevance: 87% // // The financial results for Q3 showed... }) .join("\n\n" + "─".repeat(40) + "\n\n"); // clear separator between sections
// System prompt — forces grounding, prevents hallucination const systemPrompt = `You are a helpful assistant that answers questions about documents.
STRICT RULES: 1. Answer ONLY using information from the provided document sections below 2. If the answer is not in the sections — say "This information is not covered in the provided document sections" 3. Always mention which Section number your answer comes from 4. Never use outside knowledge — only what is in the sections 5. If sections are partially relevant — use what's there and note what's missing 6. Be concise and direct`;
// User prompt — context + question const userPrompt = `Here are the relevant sections from the document:
${"─".repeat(40)} ${contextSection} ${"─".repeat(40)}
Question: ${question}
Answer (cite Section numbers):`;
// Call GPT-4o const response = await openai.chat.completions.create({ model: "gpt-4o", temperature: 0.1, // low temperature = factual, consistent answers // 0.1 not 0 — allows slight natural language variation
max_tokens: 1000, // limit response length
messages: [ { role: "system", content: systemPrompt }, { role: "user", content: userPrompt }, ], });
const answer = response.choices[0].message.content; // the generated answer string
// Build sources list for display const sources = chunks.map((chunk) => ({ file: chunk.metadata.source, // example: "report.pdf" chunkIndex: chunk.metadata.chunkIndex, // example: 3 relevance: `${(chunk.score * 100).toFixed(0)}%`, // example: "87%" preview: chunk.text.substring(0, 80) + "...", // first 80 chars of the chunk text }));
return { answer: answer, // the LLM's response string
sources: sources, // array of source info objects
grounded: true, // we had context — answer is grounded
tokensUsed: response.usage.total_tokens, // how many tokens this call used // useful for cost tracking }; }
Step 7 — Chatbot Pipeline
src/chatbot.js — ties everything together:
import { loadPDF } from "./pdfLoader.js"; import { chunkText } from "./chunker.js"; import { addChunks, getStoreStats } from "./vectorStore.js"; import { retrieveContext } from "./retriever.js"; import { generateAnswer } from "./generator.js";
// ───────────────────────────────────────── // Load and index a PDF // Run this once when user uploads a PDF // ─────────────────────────────────────────
export async function indexPDF(filePath, chunkOptions = {}) { // filePath = path to PDF file // chunkOptions = optional: { chunkSize, overlap }
console.log("\n" + "=".repeat(55)); console.log("📚 INDEXING PDF"); console.log("=".repeat(55) + "\n");
// Step 1 — Load PDF text const pdfData = await loadPDF(filePath); // pdfData = { text, pageCount, fileName }
// Step 2 — Split into chunks const chunks = chunkText(pdfData.text, pdfData.fileName, chunkOptions); // chunks = array of { id, text, metadata }
// Step 3 — Embed and store chunks await addChunks(chunks); // embeds each chunk with OpenAI and stores in memory
const stats = getStoreStats(); // { totalChunks, isEmpty }
console.log("=".repeat(55)); console.log("✅ PDF indexed successfully!"); console.log(` File: ${pdfData.fileName}`); console.log(` Pages: ${pdfData.pageCount}`); console.log(` Chunks stored: ${stats.totalChunks}`); console.log("=".repeat(55) + "\n");
return { fileName: pdfData.fileName, pageCount: pdfData.pageCount, chunksCreated: stats.totalChunks, }; }
// ───────────────────────────────────────── // Answer a question about the indexed PDF // Run this for every user question // ─────────────────────────────────────────
export async function askQuestion(question) { // question = user's question string // example: "What are the main findings?"
console.log(`\n${"─".repeat(55)}`); console.log(`❓ Question: ${question}`); console.log("─".repeat(55));
// Step 1 — Check if PDF is indexed const stats = getStoreStats(); if (stats.isEmpty) { return { answer: "Please load a PDF first before asking questions.", sources: [], }; }
// Step 2 — Retrieve relevant chunks console.log("\n🔍 Searching document..."); const retrieval = await retrieveContext(question, { topK: 5, threshold: 0.25, // slightly lower threshold for PDFs // PDF text quality varies — be more permissive });
console.log(` ${retrieval.message}`);
// Step 3 — Generate answer console.log("\n🤖 Generating answer..."); const result = await generateAnswer(question, retrieval.chunks);
// Step 4 — Display answer console.log("\n💬 ANSWER:"); console.log("─".repeat(55)); console.log(result.answer); console.log("─".repeat(55));
// Display sources if (result.sources.length > 0) { console.log("\n📎 Sources used:"); result.sources.forEach((source, i) => { console.log(` ${i + 1}. ${source.file} (chunk ${source.chunkIndex}, relevance: ${source.relevance})`); console.log(` "${source.preview}"`); }); }
if (result.tokensUsed) { console.log(`\n 💰 Tokens used: ${result.tokensUsed}`); }
return result; }
Step 8 — Entry Point
index.js — runs the chatbot:
import * as readline from "readline"; // readline = Node.js built-in module for reading terminal input // lets us create an interactive chat loop
import { indexPDF, askQuestion } from "./src/chatbot.js"; import { getStoreStats } from "./src/vectorStore.js"; import * as dotenv from "dotenv";
dotenv.config();
// ───────────────────────────────────────── // Create terminal interface // ─────────────────────────────────────────
const rl = readline.createInterface({ input: process.stdin, // read from terminal keyboard output: process.stdout, // write to terminal screen });
function prompt(question) { // Wraps readline in a Promise so we can use await return new Promise((resolve) => { rl.question(question, (answer) => { resolve(answer); // resolve = return the user's typed input }); }); }
// ───────────────────────────────────────── // Main chat loop // ─────────────────────────────────────────
async function main() { console.log("\n" + "=".repeat(55)); console.log("🤖 PDF CHATBOT"); console.log(" Ask questions about any PDF document"); console.log("=".repeat(55)); console.log("\nCommands:"); console.log(" load <path> — load a PDF file"); console.log(" quit — exit the chatbot"); console.log(" Any other text — ask a question\n");
let isRunning = true;
while (isRunning) { // keep running until user types "quit"
const userInput = await prompt("You: "); // wait for user to type something and press Enter // userInput = whatever they typed
const trimmed = userInput.trim(); // remove leading/trailing whitespace
if (trimmed === "") continue; // skip empty input — ask again
if (trimmed.toLowerCase() === "quit") { // user wants to exit console.log("\n👋 Goodbye!\n"); isRunning = false; break; }
if (trimmed.toLowerCase().startsWith("load ")) { // user wants to load a PDF // example input: "load ./sample.pdf"
const filePath = trimmed.substring(5).trim(); // extract file path from command // "load ./sample.pdf" → "./sample.pdf"
try { await indexPDF(filePath); console.log("\n✅ Ready! Ask me anything about this PDF.\n"); } catch (error) { console.log(`\n❌ Error loading PDF: ${error.message}\n`); } continue; }
// Regular question — check if PDF is loaded first const stats = getStoreStats(); if (stats.isEmpty) { console.log('\n⚠️ No PDF loaded. Type "load <path>" to load a PDF first.\n'); continue; }
// Ask the question try { await askQuestion(trimmed); console.log(); // empty line for readability } catch (error) { console.log(`\n❌ Error: ${error.message}\n`); } }
rl.close(); // close the readline interface process.exit(0); // exit Node.js process }
main().catch(console.error);
Step 9 — Run It
First — make sure you have a PDF. Put any PDF in your project folder and name it sample.pdf.
Then run:
node index.js
You'll see:
=======================================================
🤖 PDF CHATBOT
Ask questions about any PDF document
=======================================================
Commands:
load <path> — load a PDF file
quit — exit the chatbot
Any other text — ask a question
You: load ./sample.pdf
Type:
load ./sample.pdf
Output:
=======================================================
📚 INDEXING PDF
=======================================================
📄 Loading PDF: ./sample.pdf
✅ Loaded 12 pages
Total characters: 28,450
After cleaning: 26,832 characters
✂️ Chunking text...
Chunk size: 800 chars
Overlap: 150 chars
✅ Created 38 chunks
💾 Embedding and storing 38 chunks...
Progress: 5/38 chunks embedded
Progress: 10/38 chunks embedded
Progress: 15/38 chunks embedded
Progress: 20/38 chunks embedded
Progress: 25/38 chunks embedded
Progress: 30/38 chunks embedded
Progress: 35/38 chunks embedded
Progress: 38/38 chunks embedded
✅ Stored 38 chunks in vector store
=======================================================
✅ PDF indexed successfully!
File: sample.pdf
Pages: 12
Chunks stored: 38
=======================================================
✅ Ready! Ask me anything about this PDF.
You:
Now ask a question:
You: What is this document about?
Output:
───────────────────────────────────────────────────────
❓ Question: What is this document about?
───────────────────────────────────────────────────────
🔍 Searching document...
Found 5 relevant sections (best match: 82.3%)
🤖 Generating answer...
💬 ANSWER:
───────────────────────────────────────────────────────
Based on Section 1 and Section 2, this document is a
comprehensive guide to machine learning fundamentals.
It covers the basic concepts of supervised and
unsupervised learning (Section 1), and provides
practical examples of neural network architectures
(Section 2).
───────────────────────────────────────────────────────
📎 Sources used:
1. sample.pdf (chunk 0, relevance: 82%)
"This guide provides a comprehensive introduction to..."
2. sample.pdf (chunk 1, relevance: 71%)
"Chapter 1 covers supervised learning including..."
💰 Tokens used: 847
You:
The Complete Flow — One More Time
User types: "load ./sample.pdf"
↓
pdfLoader.js → extracts text from PDF
↓
chunker.js → splits into 38 chunks
↓
vectorStore.js → embeds each chunk (38 OpenAI API calls)
→ stores { id, text, metadata, embedding }
↓
"✅ Ready!"
User types: "What are the main conclusions?"
↓
retriever.js → embeds the question (1 OpenAI API call)
→ searches all 38 stored embeddings
→ returns top 5 most similar chunks
↓
generator.js → builds prompt with context + question
→ calls GPT-4o (1 OpenAI API call)
→ returns grounded answer with sources
↓
User sees: Answer + which chunks it came from
Total API calls per question: 2 (1 embedding + 1 chat completion) Total API calls for indexing: N (one per chunk — 38 for a 12-page PDF)
Cost Estimate
For a typical 12-page PDF and 10 questions:
Indexing (38 chunks):
→ text-embedding-3-small
→ ~38 × 200 tokens = 7,600 tokens
→ Cost: ~$0.001 (less than 1 cent)
10 questions:
→ 10 embeddings: ~$0.0001
→ 10 GPT-4o calls: ~800 tokens each = 8,000 tokens
→ Cost: ~$0.05 (5 cents)
Total: ~$0.05 for loading + 10 questions
Basically free for development
What You've Built
✅ PDF text extraction
✅ Smart chunking with overlap
✅ OpenAI embeddings
✅ In-memory vector store with cosine similarity search
✅ Context retrieval with quality scoring
✅ Grounded answer generation with source citations
✅ Anti-hallucination system prompt
✅ Interactive terminal chat interface
✅ Error handling throughout
This is a complete, working RAG application.
Phase 5 — Complete 🎉
✅ Module 5.1 — What is RAG and why it exists
✅ Module 5.2 — Chunking strategies
✅ Module 5.3 — Retrieval and context injection
✅ Module 5.4 — Hallucination and grounding
✅ Module 5.5 — Full PDF Chatbot (this module)
You have now built the most in-demand AI engineering skill in the industry.
What's Coming — Phase 6: LangChain
Right now you built everything from scratch — your own chunker, your own vector store, your own retriever, your own generator. That's intentional. You understand every piece.
Phase 6 introduces LangChain — a framework that provides pre-built versions of all these pieces. Now that you understand what each piece does — LangChain will make complete sense instead of feeling like magic.
What you built manually → What LangChain provides
──────────────────────────────────────────────────
pdfLoader.js → PDFLoader
chunker.js → RecursiveCharacterTextSplitter
vectorStore.js → MemoryVectorStore / Chroma / Pinecone
retriever.js → VectorStoreRetriever
generator.js → ChatOpenAI + PromptTemplate
chatbot.js → RetrievalQAChain
Same concepts. Pre-built components. Much faster to build with.
No comments:
Post a Comment