CodeWithGagan | Programming Language and IT Lectures

Rebuilding the PDF chatbot in LangChain — see how much simpler it gets

What We're Doing

In Phase 5 you built a PDF chatbot manually — 7 files, ~370 lines.

Now we rebuild the exact same thing using LangChain components. Same functionality. Much less code. And because you built it manually first — you'll understand every line.

By the end of this module you'll have:

LangChain PDF Chatbot
├── .env
├── package.json
└── src/
    ├── 01_loaders.js      ← Document Loaders
    ├── 02_splitters.js    ← Text Splitters  
    ├── 03_vectorstores.js ← Vector Stores
    └── 04_rag_chain.js    ← Complete RAG in ~40 lines

Project Setup


    mkdir langchain-rag
    cd langchain-rag
    npm init -y

Update package.json:


    {
        "name": "langchain-rag",
        "version": "1.0.0",
        "type": "module",
        "scripts": {
            "start": "node src/04_rag_chain.js"
        }
    }

Install packages:

    npm install langchain @langchain/core @langchain/openai @langchain/classic dotenv

Create .env:


    OPENAI_API_KEY=sk-proj-your-key-here

Put any PDF in the project root — name it sample.pdf.

Part 1 — Document Loaders

What they do

Document loaders load content from different sources and return it as Document objects — the standard format LangChain uses everywhere.


    // Every loader returns this same structure:
    [
        {
            pageContent: "actual text from the source...",
            metadata: {
                source: "file.pdf",
                page: 1,
                // other source-specific info
            }
        }
    ]

Create src/01_loaders.js:


    import fs from "fs";

    import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
    // PDFLoader = loads PDF files
    // reads each page as a separate Document object

    import { TextLoader } from "@langchain/classic/document_loaders/fs/text";
    // TextLoader = loads plain .txt files

    import { CSVLoader } from "@langchain/community/document_loaders/fs/csv";
    // CSVLoader = loads CSV files
    // each row becomes a Document

    import { CheerioWebBaseLoader } from "@langchain/community/document_loaders/web/cheerio";
    // CheerioWebBaseLoader = loads web pages
    // scrapes HTML and extracts text

    import * as dotenv from "dotenv";
    dotenv.config();


    // ─────────────────────────────────────────
    // LOADER 1 — PDF Loader
    // Loads a PDF file — each page = one Document
    // ─────────────────────────────────────────

    async function loader1() {
        console.log("─".repeat(55));
        console.log("LOADER 1: PDFLoader");
        console.log("─".repeat(55));

        const loader = new PDFLoader("./sample.pdf", {
            // splitPages: true  = each page becomes separate Document (default)
            // splitPages: false = entire PDF becomes one Document
            splitPages: true,
        });

        const docs = await loader.load();
        // load() = reads the PDF and returns array of Documents
        // one Document per page

        console.log(`Pages loaded: ${docs.length}`);
        // example: 5 (one per page)

        // Show first document
        console.log("\nFirst page Document:");
        console.log("pageContent preview:", docs[0].pageContent.substring(0, 150) + "...");
        console.log("metadata:", docs[0].metadata);
        // metadata example:
        // { source: './sample.pdf', pdf: { version: '1.7', pages: 5 }, page: 0 }

        console.log("\nAll pages:");
        docs.forEach((doc, i) => {
            console.log(`  Page ${i + 1}: ${doc.pageContent.length} characters`);
        });

        console.log();
        return docs;
    }


    // ─────────────────────────────────────────
    // LOADER 2 — Text Loader
    // Loads a plain text file
    // ─────────────────────────────────────────

    async function loader2() {
        console.log("─".repeat(55));
        console.log("LOADER 2: TextLoader");
        console.log("─".repeat(55));

        // Create a sample text file first
        fs.writeFileSync("./sample.txt",
            `This is a sample text file.
            
        It contains information about AI engineering.

        LangChain makes it easy to build AI applications.
        You can load documents from many different sources.`
        );

        const loader = new TextLoader("./sample.txt");
        const docs = await loader.load();

        console.log(`Documents loaded: ${docs.length}`);
        // 1 — whole file as one Document

        console.log("Content:", docs[0].pageContent);
        console.log("Metadata:", docs[0].metadata);
        // { source: './sample.txt' }

        console.log();
        return docs;
    }


    // ─────────────────────────────────────────
    // LOADER 3 — Web Loader
    // Loads content from a web URL
    // ─────────────────────────────────────────

    async function loader3() {
        console.log("─".repeat(55));
        console.log("LOADER 3: CheerioWebBaseLoader");
        console.log("─".repeat(55));

        const loader = new CheerioWebBaseLoader(
            "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
            {
                selector: "p",
                // CSS selector — only extract <p> paragraph tags
                // ignores navigation, headers, footers
                // gives us clean article content
            }
        );

        try {
            const docs = await loader.load();

            console.log(`Documents loaded: ${docs.length}`);
            // Usually 1 — whole page as one Document

            console.log("Content preview:", docs[0].pageContent.substring(0, 200) + "...");
            console.log("Content length:", docs[0].pageContent.length, "characters");
            console.log("Metadata:", docs[0].metadata);

        } catch (error) {
            console.log("Web loader error (might be network issue):", error.message);
        }

        console.log();
    }


    // ─────────────────────────────────────────
    // COMPARE — What you built vs LangChain
    // ─────────────────────────────────────────

    function compareWithManual() {
        console.log("─".repeat(55));
        console.log("COMPARISON: Manual vs LangChain");
        console.log("─".repeat(55));

        console.log(`
        Your manual pdfLoader.js:          LangChain PDFLoader:
        ────────────────────────────       ────────────────────
        import fs from "fs"                import { PDFLoader }
        import pdfParse from "pdf-parse"   from "@langchain/community/..."
        import { createRequire }...
                                        const loader = new PDFLoader(path)
        export async function loadPDF(p) { const docs = await loader.load()
        check file exists...
        read buffer...
        parse PDF...
        clean text...
        return { text, pageCount, ... }  // Done — 2 lines
        }
        // ~40 lines                       // 2 lines
        `);
    }


    async function main() {
        console.log("\n📄 DOCUMENT LOADERS DEMO\n");
        await loader1();
        await loader2();
        await loader3();
        compareWithManual();
        console.log("✅ Loaders demo complete!");
    }

    main().catch(console.error);

Install the community package (has PDF and web loaders):


    npm install @langchain/community pdf-parse cheerio

Before run your package.json look like this:

  {
      "name": "langchain-rag",
      "version": "1.0.0",
      "main": "index.js",
      "type": "module",
      "scripts": {
          "start": "node src/04_rag_chain.js",
          "test": "echo \"Error: no test specified\" && exit 1"
      },
      "keywords": [],
      "author": "",
      "license": "ISC",
      "description": "",
      "dependencies": {
          "@langchain/classic": "^1.0.40",
          "@langchain/community": "^1.1.29",
          "@langchain/core": "^1.2.3",
          "@langchain/openai": "^1.5.5",
          "cheerio": "^1.2.0",
          "dotenv": "^16.4.5",
          "langchain": "^1.5.3",
          "pdf-parse": "^2.4.5"
      }
  }

Run:

node src/01_loaders.js

Part 2 — Text Splitters

What they do

The same job as your chunker.js — split large documents into smaller chunks.

LangChain's RecursiveCharacterTextSplitter is smarter than most manual implementations — it tries multiple separators in order.

Create src/02_splitters.js:


    import {
        RecursiveCharacterTextSplitter,
        CharacterTextSplitter,
    } from "@langchain/textsplitters";
    // RecursiveCharacterTextSplitter = tries multiple separators in order
    //   → first tries to split on \n\n (paragraphs)
    //   → then \n (lines)
    //   → then ". " (sentences)
    //   → then " " (words)
    //   → last resort: characters
    //   This gives much cleaner chunks than splitting blindly

    // CharacterTextSplitter = splits on one specific character only

    import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
    import * as dotenv from "dotenv";
    dotenv.config();


    // ─────────────────────────────────────────
    // SPLITTER 1 — RecursiveCharacterTextSplitter
    // The most commonly used splitter in LangChain
    // This is what replaces your chunker.js
    // ─────────────────────────────────────────

    async function splitter1() {
        console.log("─".repeat(55));
        console.log("SPLITTER 1: RecursiveCharacterTextSplitter");
        console.log("─".repeat(55));

        const splitter = new RecursiveCharacterTextSplitter({
            chunkSize: 800,
            // target size in characters per chunk
            // same as your chunker.js chunkSize

            chunkOverlap: 150,
            // characters repeated between consecutive chunks
            // same as your chunker.js overlap

            separators: ["\n\n", "\n", ". ", " ", ""],
            // tries these separators in ORDER
            // first tries paragraph breaks (\n\n) — cleanest split
            // then line breaks (\n)
            // then sentence ends (". ")
            // then word boundaries (" ")
            // last resort: split anywhere ("")
            // much smarter than your manual implementation
        });

        // Sample text to split
        const text = `
        Chapter 1: Introduction to RAG

        Retrieval Augmented Generation is a technique that combines the power of large language models with the ability to retrieve relevant information from a knowledge base.

        This approach solves two major problems with pure LLMs: knowledge cutoffs and hallucination. By retrieving real documents and injecting them into the prompt, RAG grounds the LLM's responses in actual information.

        Chapter 2: How RAG Works

        The RAG pipeline has two phases. The indexing phase processes documents: loading them, splitting them into chunks, converting to embeddings, and storing in a vector database.

        The query phase handles user questions: embedding the question, searching for similar chunks, injecting them into a prompt, and generating a grounded answer.

        Chapter 3: Implementation

        Building a RAG system requires choosing the right components: a document loader for your data source, a text splitter for chunking, an embedding model for vectorization, a vector store for storage and retrieval, and an LLM for generating answers.
        `;

        // Split plain text
        const chunks = await splitter.splitText(text);
        // splitText() = takes a string, returns array of strings

        console.log(`Original text: ${text.length} characters`);
        console.log(`Chunks created: ${chunks.length}`);

        chunks.forEach((chunk, i) => {
            console.log(`\nChunk ${i + 1} (${chunk.length} chars):`);
            console.log(chunk.substring(0, 100) + "...");
        });

        console.log();
    }


    // ─────────────────────────────────────────
    // SPLITTER 2 — Split Documents (not just text)
    // Works directly with Document objects from loaders
    // ─────────────────────────────────────────

    async function splitter2() {
        console.log("─".repeat(55));
        console.log("SPLITTER 2: splitDocuments — works with Document objects");
        console.log("─".repeat(55));

        // Load PDF
        const loader = new PDFLoader("./sample.pdf");
        const docs = await loader.load();
        // docs = array of Document objects (one per page)

        console.log(`PDF pages loaded: ${docs.length}`);

        const splitter = new RecursiveCharacterTextSplitter({
            chunkSize: 600,
            chunkOverlap: 100,
        });

        // Split Document objects — NOT just text strings
        const chunks = await splitter.splitDocuments(docs);
        // splitDocuments() = takes array of Documents
        // returns array of smaller Documents
        // AUTOMATICALLY preserves and updates metadata!

        console.log(`Chunks after splitting: ${chunks.length}`);

        // Show metadata preservation
        console.log("\nFirst chunk metadata:");
        console.log(chunks[0].metadata);
        // metadata still has source, page number — automatically preserved
        // {
        //   source: './sample.pdf',
        //   page: 0,         ← which page this chunk came from
        //   loc: { ... }     ← character location in original
        // }

        console.log("\nSample chunks:");
        chunks.slice(0, 3).forEach((chunk, i) => {
            console.log(`\nChunk ${i + 1}:`);
            console.log(`  Content: ${chunk.pageContent.substring(0, 80)}...`);
            console.log(`  Metadata: page=${chunk.metadata.page}, source=${chunk.metadata.source}`);
        });

        console.log();
        return chunks;
    }


    // ─────────────────────────────────────────
    // SPLITTER 3 — Comparing chunk sizes
    // See how different settings affect output
    // ─────────────────────────────────────────

    async function splitter3() {
        console.log("─".repeat(55));
        console.log("SPLITTER 3: Comparing different chunk sizes");
        console.log("─".repeat(55));

        const loader = new PDFLoader("./sample.pdf");
        const docs = await loader.load();

        const settings = [
            { chunkSize: 200, chunkOverlap: 50, label: "Small chunks" },
            { chunkSize: 600, chunkOverlap: 100, label: "Medium chunks (recommended)" },
            { chunkSize: 1200, chunkOverlap: 200, label: "Large chunks" },
        ];

        for (const setting of settings) {
            const splitter = new RecursiveCharacterTextSplitter({
                chunkSize: setting.chunkSize,
                chunkOverlap: setting.chunkOverlap,
            });

            const chunks = await splitter.splitDocuments(docs);

            console.log(`\n${setting.label} (size=${setting.chunkSize}, overlap=${setting.chunkOverlap}):`);
            console.log(`  Total chunks: ${chunks.length}`);
            console.log(`  Avg chunk size: ${Math.round(
                chunks.reduce((sum, c) => sum + c.pageContent.length, 0) / chunks.length
            )} chars`);
        }

        console.log("\nKey insight:");
        console.log("More chunks = more precise retrieval but more API calls to embed");
        console.log("Fewer chunks = cheaper but might miss specific details");
        console.log();
    }


    // ─────────────────────────────────────────
    // COMPARISON with your manual chunker
    // ─────────────────────────────────────────

    function compareWithManual() {
        console.log("─".repeat(55));
        console.log("COMPARISON: Manual vs LangChain");
        console.log("─".repeat(55));

        console.log(`
        Your manual chunker.js:              LangChain:
        ────────────────────────────         ──────────────────────────
        80 lines of while loop logic         const splitter = new
        handling boundaries, overlap,          RecursiveCharacterTextSplitter({
        edge cases, infinite loop bugs...      chunkSize: 800,
                                            chunkOverlap: 150
                                            })
                                            const chunks = await
                                            splitter.splitDocuments(docs)
                                            
                                            // 4 lines. Battle tested.
                                            // Handles all edge cases.
                                            // Never gets infinite loops.
        `);
    }


    async function main() {
        console.log("\n✂️  TEXT SPLITTERS DEMO\n");
        await splitter1();
        await splitter2();
        await splitter3();
        compareWithManual();
        console.log("✅ Splitters demo complete!");
    }

    main().catch(console.error);

Run:

node src/02_splitters.js

Part 3 — Vector Stores

What they do

Store embeddings and search for similar content — same as your vectorStore.js but pre-built.

Create src/03_vectorstores.js:


    import { OpenAIEmbeddings } from "@langchain/openai";
    // OpenAIEmbeddings = generates embeddings using OpenAI API
    // replaces your getEmbedding() function

    import { MemoryVectorStore } from "@langchain/classic/vectorstores/memory";
    // MemoryVectorStore = in-memory vector store
    // same as your store[] array — but built-in
    // no server needed — perfect for development

    import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
    import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
    import * as dotenv from "dotenv";
    dotenv.config();


    // ─────────────────────────────────────────
    // SETUP — Embeddings model
    // ─────────────────────────────────────────

    const embeddings = new OpenAIEmbeddings({
        model: "text-embedding-3-small",
        // same model you used manually
        // 1536 dimensions
    });


    // ─────────────────────────────────────────
    // EXAMPLE 1 — Create vector store from documents
    // ─────────────────────────────────────────

    async function example1() {
        console.log("─".repeat(55));
        console.log("EXAMPLE 1: Create vector store from documents");
        console.log("─".repeat(55));

        // Load and split PDF
        const loader = new PDFLoader("./sample.pdf");
        const docs = await loader.load();

        const splitter = new RecursiveCharacterTextSplitter({
            chunkSize: 800,
            chunkOverlap: 150,
        });
        const chunks = await splitter.splitDocuments(docs);

        console.log(`Chunks to embed: ${chunks.length}`);
        console.log("Embedding and storing... (calls OpenAI API)");

        // Create vector store from chunks
        const vectorStore = await MemoryVectorStore.fromDocuments(
            chunks,
            // documents to embed and store

            embeddings,
            // embedding model to use
            // LangChain calls OpenAI for each chunk automatically
        );
        // fromDocuments() does everything:
        // 1. Calls OpenAI to embed each chunk
        // 2. Stores embedding + text + metadata in memory
        // 3. Returns ready-to-search vector store

        console.log("✅ Vector store created!\n");
        return vectorStore;
    }


    // ─────────────────────────────────────────
    // EXAMPLE 2 — Similarity search
    // ─────────────────────────────────────────

    async function example2(vectorStore) {
        console.log("─".repeat(55));
        console.log("EXAMPLE 2: Similarity search");
        console.log("─".repeat(55));

        // Basic similarity search
        const results = await vectorStore.similaritySearch(
            "What is this document about?",
            // query string — LangChain embeds this automatically

            4,
            // top K results to return
        );
        // returns array of Document objects — most similar first

        console.log(`Found ${results.length} similar chunks\n`);

        results.forEach((doc, i) => {
            console.log(`Result ${i + 1}:`);
            console.log(`  Content: ${doc.pageContent.substring(0, 100)}...`);
            console.log(`  Source: ${doc.metadata.source}`);
            console.log(`  Page: ${doc.metadata.page}`);
            console.log();
        });
    }


    // ─────────────────────────────────────────
    // EXAMPLE 3 — Similarity search with scores
    // See the actual similarity scores
    // ─────────────────────────────────────────

    async function example3(vectorStore) {
        console.log("─".repeat(55));
        console.log("EXAMPLE 3: Similarity search with scores");
        console.log("─".repeat(55));

        const resultsWithScores = await vectorStore.similaritySearchWithScore(
            "main topic",
            // query

            5,
            // top 5 results
        );
        // returns array of [Document, score] pairs
        // score = similarity score (higher = more similar in MemoryVectorStore)

        console.log("Results with similarity scores:\n");

        resultsWithScores.forEach(([doc, score], i) => {
            // destructure [Document, score] pair
            const bar = "█".repeat(Math.round(score * 10));
            console.log(`${i + 1}. Score: ${score.toFixed(4)} ${bar}`);
            console.log(`   ${doc.pageContent.substring(0, 80)}...`);
            console.log();
        });
    }


    // ─────────────────────────────────────────
    // EXAMPLE 4 — Retriever interface
    // Cleaner way to use vector store in chains
    // ─────────────────────────────────────────

    async function example4(vectorStore) {
        console.log("─".repeat(55));
        console.log("EXAMPLE 4: Retriever interface");
        console.log("─".repeat(55));

        // Convert vector store to a retriever
        const retriever = vectorStore.asRetriever({
            k: 4,
            // return top 4 results

            searchType: "similarity",
            // "similarity" = cosine similarity search (default)
            // "mmr" = Maximum Marginal Relevance
            //         returns diverse results — avoids repetitive chunks
        });

        // Use retriever directly
        const docs = await retriever.invoke("What topics are covered?");
        // invoke() = same as similaritySearch() but cleaner interface

        console.log(`Retrieved ${docs.length} documents\n`);
        docs.forEach((doc, i) => {
            console.log(`Doc ${i + 1}: ${doc.pageContent.substring(0, 80)}...`);
        });

        console.log();
        return retriever;
    }


    // ─────────────────────────────────────────
    // EXAMPLE 5 — MMR search
    // Maximum Marginal Relevance
    // Returns diverse results — avoids duplicate content
    // ─────────────────────────────────────────

    async function example5(vectorStore) {
        console.log("─".repeat(55));
        console.log("EXAMPLE 5: MMR search (Maximum Marginal Relevance)");
        console.log("─".repeat(55));

        console.log("Regular similarity search — might return similar/duplicate chunks:");
        const regular = await vectorStore.similaritySearch("main topic", 3);
        regular.forEach((doc, i) => {
            console.log(`  ${i + 1}. ${doc.pageContent.substring(0, 60)}...`);
        });

        console.log("\nMMR search — returns diverse, non-redundant chunks:");
        const diverse = await vectorStore.maxMarginalRelevanceSearch(
            "main topic",
            // query

            {
                k: 3,
                // return 3 results

                fetchK: 10,
                // fetch top 10 candidates first
                // then pick 3 most diverse from those 10
                // more fetchK = more diverse but slower

                lambda: 0.5,
                // balance between relevance and diversity
                // 0 = pure diversity (ignore relevance)
                // 1 = pure relevance (same as regular search)
                // 0.5 = balanced (good default)
            }
        );

        diverse.forEach((doc, i) => {
            console.log(`  ${i + 1}. ${doc.pageContent.substring(0, 60)}...`);
        });

        console.log("\nMMR is better for RAG — avoids injecting the same");
        console.log("information multiple times into the LLM prompt.");
        console.log();
    }


    async function main() {
        console.log("\n🗄️  VECTOR STORES DEMO\n");

        const vectorStore = await example1();
        await example2(vectorStore);
        await example3(vectorStore);
        await example4(vectorStore);
        await example5(vectorStore);

        console.log("✅ Vector stores demo complete!");
    }

    main().catch(console.error);

Run:

node src/03_vectorstores.js

Part 4 — Complete RAG Chain

Now everything together — the full PDF chatbot in ~50 lines.

Create src/04_rag_chain.js:


    import { PDFLoader } from "@langchain/community/document_loaders/fs/pdf";
    import { RecursiveCharacterTextSplitter } from "@langchain/textsplitters";
    import { OpenAIEmbeddings, ChatOpenAI } from "@langchain/openai";
    import { MemoryVectorStore } from "@langchain/classic/vectorstores/memory";
    import { ChatPromptTemplate, MessagesPlaceholder } from "@langchain/core/prompts";
    import { StringOutputParser } from "@langchain/core/output_parsers";
    import { RunnablePassthrough, RunnableSequence } from "@langchain/core/runnables";
    import * as readline from "readline";
    import * as dotenv from "dotenv";
    dotenv.config();


    // ─────────────────────────────────────────
    // STEP 1 — Load PDF
    // ─────────────────────────────────────────

    async function loadAndSplitPDF(filePath) {
        console.log(`\n📄 Loading PDF: ${filePath}`);

        const loader = new PDFLoader(filePath);
        const docs = await loader.load();
        // docs = array of Documents (one per page)

        console.log(`   Loaded ${docs.length} pages`);

        const splitter = new RecursiveCharacterTextSplitter({
            chunkSize: 800,
            chunkOverlap: 150,
        });

        const chunks = await splitter.splitDocuments(docs);
        // chunks = array of smaller Documents

        console.log(`   Split into ${chunks.length} chunks`);
        return chunks;
    }


    // ─────────────────────────────────────────
    // STEP 2 — Create Vector Store
    // ─────────────────────────────────────────

    async function createVectorStore(chunks) {
        console.log("\n💾 Creating vector store...");
        console.log(`   Embedding ${chunks.length} chunks with OpenAI...`);

        const vectorStore = await MemoryVectorStore.fromDocuments(
            chunks,
            new OpenAIEmbeddings({ model: "text-embedding-3-small" })
        );

        console.log("   ✅ Vector store ready\n");
        return vectorStore;
    }


    // ─────────────────────────────────────────
    // STEP 3 — Build RAG Chain
    // This is the magic — all pieces connected
    // ─────────────────────────────────────────

    function buildRAGChain(vectorStore) {

        const retriever = vectorStore.asRetriever({
            k: 5,
            // retrieve top 5 most relevant chunks
            searchType: "mmr",
            // use MMR for diverse results
        });

        const llm = new ChatOpenAI({
            model: "gpt-4o",
            temperature: 0.1,
            // low temp = factual, consistent answers
        });

        // Format retrieved documents into a string
        function formatDocs(docs) {
            // docs = array of Document objects from retriever
            return docs
                .map((doc, index) =>
                    `[Source ${index + 1} — Page ${doc.metadata.page + 1}]\n${doc.pageContent}`
                )
                .join("\n\n" + "─".repeat(40) + "\n\n");
            // formats each chunk with source number and page
            // joins with clear separator
            // example output:
            // [Source 1 — Page 3]
            // text content here...
            // ────────────────────
            // [Source 2 — Page 5]
            // more content here...
        }

        // RAG prompt — forces grounded answers
        const ragPrompt = ChatPromptTemplate.fromMessages([
            [
                "system", `You are a helpful assistant that answers questions about documents.

                    STRICT RULES:
                    1. Answer ONLY using information from the context below
                    2. If answer is not in context — say "This is not covered in the document"
                    3. Always mention which Source number your answer comes from
                    4. Be concise and direct

                Context: {context}`
            ],
            ["human", "{question}"],
        ]);

        // Build the complete RAG chain using LCEL
        const ragChain = RunnableSequence.from([
            {
                context: retriever.pipe(
                    (docs) => formatDocs(docs)
                ),
                // retriever gets relevant docs → formatDocs converts to string
                // result stored as "context" variable for the prompt

                question: new RunnablePassthrough(),
                // RunnablePassthrough = pass the question through unchanged
                // result stored as "question" variable for the prompt
            },
            ragPrompt,
            // fills {context} and {question} in the template

            llm,
            // calls GPT-4o with the filled prompt

            new StringOutputParser(),
            // extracts string from AIMessage response
        ]);

        return ragChain;
    }


    // ─────────────────────────────────────────
    // STEP 4 — Interactive Chat
    // ─────────────────────────────────────────

    async function startChat(ragChain) {
        const rl = readline.createInterface({
            input: process.stdin,
            output: process.stdout,
        });

        function prompt(question) {
            return new Promise(resolve => rl.question(question, resolve));
        }

        console.log("=".repeat(55));
        console.log("✅ PDF Chatbot Ready!");
        console.log('   Type your question or "quit" to exit');
        console.log("=".repeat(55) + "\n");

        while (true) {
            const question = await prompt("You: ");
            const trimmed = question.trim();

            if (!trimmed) continue;
            // skip empty input

            if (trimmed.toLowerCase() === "quit") {
                console.log("\n👋 Goodbye!\n");
                rl.close();
                break;
            }

            try {
                console.log("\n🤖 Thinking...\n");

                // Stream the response for better UX
                const stream = await ragChain.stream(trimmed);
                // stream() = get response token by token

                process.stdout.write("Assistant: ");
                for await (const chunk of stream) {
                    process.stdout.write(chunk);
                    // print each token as it arrives
                }
                console.log("\n");

            } catch (error) {
                console.log(`Error: ${error.message}\n`);
            }
        }

        process.exit(0);
    }


    // ─────────────────────────────────────────
    // MAIN — Tie everything together
    // ─────────────────────────────────────────

    async function main() {
        console.log("\n" + "=".repeat(55));
        console.log("🤖 LANGCHAIN PDF CHATBOT");
        console.log("=".repeat(55));

        const pdfPath = process.argv[2] || "./sample.pdf";
        // process.argv[2] = first command line argument
        // example: node src/04_rag_chain.js ./my-document.pdf
        // if not provided — defaults to ./sample.pdf

        try {
            // Index the PDF
            const chunks = await loadAndSplitPDF(pdfPath);
            const vectorStore = await createVectorStore(chunks);
            const ragChain = buildRAGChain(vectorStore);

            // Start chatting
            await startChat(ragChain);

        } catch (error) {
            if (error.message.includes("not found")) {
                console.log(`\n❌ PDF not found: ${pdfPath}`);
                console.log("Usage: node src/04_rag_chain.js <path-to-pdf>");
            } else {
                console.log(`\n❌ Error: ${error.message}`);
            }
            process.exit(1);
        }
    }

    main().catch(console.error);

Run it:

node src/04_rag_chain.js ./sample.pdf

Or with a different PDF:

node src/04_rag_chain.js ./any-document.pdf

Manual vs LangChain — Final Comparison

YOUR MANUAL PDF CHATBOT:
────────────────────────
pdfLoader.js    →  40 lines
chunker.js      →  85 lines  
vectorStore.js  →  95 lines
retriever.js    →  45 lines
generator.js    →  65 lines
chatbot.js      →  70 lines
index.js        →  65 lines
────────────────────────────
Total:          → 465 lines
                   7 files

LANGCHAIN PDF CHATBOT:
──────────────────────
04_rag_chain.js → ~150 lines
                   1 file
────────────────────────────
Same functionality. 3x less code.
Battle-tested components.
No infinite loop bugs.
Streaming built in.
MMR search built in.

3-Line Summary

LangChain document loaders all return the same Document format — { pageContent, metadata } — so you can swap a PDFLoader for a WebLoader or CSVLoader without changing any downstream code.
RecursiveCharacterTextSplitter replaces your manual chunker — it tries multiple separators in order (paragraphs → lines → sentences → words) giving cleaner chunks without infinite loop risks, and splitDocuments() automatically preserves metadata.
MemoryVectorStore.fromDocuments() replaces your entire vectorStore.js — one line embeds all chunks, stores them, and returns a searchable store that supports similarity search, MMR search, and the retriever interface used in LCEL chains.

Module 6.4 — Complete ✅

Coming up — Module 6.5 — Retrievers & Agents in LangChain

The final module of Phase 6. We go deep on advanced retriever patterns — contextual compression, multi-query retrieval — and build a LangChain agent with tools that can reason and take actions autonomously.

CodeWithGagan | Programming Language and IT Lectures

Module 6.4 — Document Loaders, Text Splitters & Vector Stores

Rebuilding the PDF chatbot in LangChain — see how much simpler it gets

What We're Doing

Project Setup

Part 1 — Document Loaders

What they do

Part 2 — Text Splitters

What they do

Part 3 — Vector Stores

What they do

Part 4 — Complete RAG Chain

Manual vs LangChain — Final Comparison

3-Line Summary

Module 6.4 — Document Loaders, Text Splitters & Vector Stores