The solution to the problem — how vector databases make similarity search fast
Start With the Core Problem From Last Module
Last module we ended with this:
Naive vector search:
→ Compare question vector against every stored vector
→ 1 million documents = 1 million calculations
→ Each on 1536 numbers
→ Way too slow for real-time search 🐌
A vector database solves this with one key idea:
Don't search everything. Build a smart index that lets you skip most of the data and only check the promising areas.
This is called ANN — Approximate Nearest Neighbor search.
Let's understand it properly.
First — What is "Nearest Neighbor"?
Nearest neighbor simply means:
Find the vectors closest to my query vector.
Query vector (user question):
Q = [0.23, 0.87, 0.41, ...]
Stored vectors (document chunks):
A = [0.21, 0.85, 0.43, ...] ← very close to Q
B = [0.80, 0.12, 0.91, ...] ← far from Q
C = [0.22, 0.86, 0.40, ...] ← very close to Q
D = [0.11, 0.43, 0.77, ...] ← medium distance
Nearest neighbors of Q:
1st nearest → A (closest)
2nd nearest → C (second closest)
Finding the single closest vector = 1-Nearest Neighbor (1-NN) Finding the top 5 closest = 5-Nearest Neighbor (5-NN) Finding the top K closest = K-Nearest Neighbor (KNN)
In RAG — you typically search for top 3, 5, or 10 nearest neighbors. These become the document chunks you inject into your LLM prompt.
Exact KNN vs Approximate KNN
There are two approaches:
Exact KNN:
Check every single vector
Calculate similarity with all of them
Sort all results
Return top K
Result: 100% accurate
Speed: Very slow at scale 🐌
Approximate KNN (ANN):
Use a smart index to skip most vectors
Only check the "promising" areas
Return top K from those areas
Result: 95-99% accurate (might miss rare edge cases)
Speed: Milliseconds even at billions of vectors ⚡
The trade-off is tiny. You lose maybe 1-5% accuracy. You gain 1000x speed.
For RAG applications — this trade-off is completely worth it. Missing one slightly relevant document chunk almost never matters when you're returning 5-10 results anyway.
How ANN Indexing Works — The HNSW Algorithm
The most popular ANN algorithm used in vector databases is called HNSW — Hierarchical Navigable Small World.
The name sounds scary. The concept is actually beautiful.
Let me explain it with a real life analogy first.
The Analogy — Finding a Restaurant in a New City
Imagine you arrive in a new city and want to find the best pizza place near your hotel.
Naive approach (Exact KNN):
Visit every single restaurant in the city
Try the pizza at each one
Compare all of them
Pick the best
→ Accurate but takes weeks 🐌
Smart approach (HNSW):
Step 1 — Start at the city level (zoom out)
→ You know pizza places cluster in the city center
→ Head toward the city center generally
Step 2 — Zoom into neighborhood level
→ Now you're in the right area
→ Ask locals "where's the good pizza around here?"
→ They point to the Italian neighborhood
Step 3 — Zoom into street level
→ You're on the right street
→ Walk and compare the few pizza places here
→ Pick the best one
→ Fast AND finds a great result ⚡
You didn't visit every restaurant. You navigated intelligently — going from broad to specific.
HNSW does exactly this — but with vectors in multi-dimensional space.
How HNSW Actually Works
HNSW builds a multi-layer graph of your vectors.
LAYER 2 (top — few vectors, long connections)
A ─────────────────── E
LAYER 1 (middle — more vectors, medium connections)
A ──── B ──── C ──── E
│
D
LAYER 0 (bottom — all vectors, short connections)
A ─ F ─ B ─ G ─ C ─ H ─ D ─ I ─ E ─ J
Layer 2 — a small number of vectors with long-range connections. Good for big jumps across the space.
Layer 1 — more vectors, medium connections. Good for getting close.
Layer 0 — all vectors, short connections. Good for fine-tuning the final result.
Search process:
Query comes in: Q = [0.23, 0.87, ...]
Step 1 — Start at Layer 2 (top)
→ Start at a random entry point
→ Move to whichever neighbor is closest to Q
→ Keep moving until no neighbor is closer
→ You've found the approximate area in Layer 2
Step 2 — Drop to Layer 1
→ Use your Layer 2 position as starting point
→ Navigate again — move to closest neighbors
→ Zoom in further
Step 3 — Drop to Layer 0 (bottom)
→ Now you're very close to the target area
→ Check all nearby vectors
→ Return the closest ones
Result: Found nearest neighbors in milliseconds
without checking 99% of the data
The Numbers — Why This is So Fast
Let's make this concrete with real scale:
Without ANN index (naive):
1,000,000 vectors
Each comparison: ~0.001ms
Total: 1,000,000 × 0.001ms = 1,000ms = 1 second per query 🐌
With HNSW index:
1,000,000 vectors
HNSW checks roughly: log(1,000,000) ≈ 20 vectors per layer
Total comparisons: maybe 200-500 vectors
Time: ~1-5ms per query ⚡
Speed improvement: 200-1000x faster
Accuracy loss: less than 1%
At 1 billion vectors — the difference becomes even more dramatic.
Anatomy of a Vector Database
A vector database is not just a search engine. It stores and manages several things together:
┌────────────────────────────────────────────────────┐
│ VECTOR DATABASE │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ VECTOR INDEX (HNSW or similar) │ │
│ │ The fast search structure │ │
│ │ Stores: relationships between vectors │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ VECTORS │ │
│ │ The actual embedding arrays │ │
│ │ [0.23, -0.87, 0.41, ...] × 1536 │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ METADATA │ │
│ │ Extra information about each vector │ │
│ │ { source: "doc1.pdf", page: 3, │ │
│ │ category: "medical", date: "2024-01" } │ │
│ └─────────────────────────────────────────────┘ │
│ │
│ ┌─────────────────────────────────────────────┐ │
│ │ ORIGINAL TEXT (optional) │ │
│ │ The actual text chunk that was embedded │ │
│ │ "Aspirin can cause stomach bleeding..." │ │
│ └─────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────┘
When you search — you get back not just vectors but the full record including the original text and metadata. This is what you inject into your LLM prompt.
Metadata Filtering — The Power Feature
This is where vector databases become really powerful for real applications.
Pure vector search gives you:
"Find the 10 most similar document chunks to this question"
Metadata filtering gives you:
"Find the 10 most similar document chunks to this question
WHERE source = 'medical_handbook.pdf'
AND date > '2023-01-01'
AND language = 'english'"
You combine semantic similarity with structured filters — just like SQL WHERE clauses — but alongside vector search.
Real world examples:
// E-commerce: similar products but only in stock search(queryVector, { filter: { inStock: true, category: "electronics" } })
// Legal: similar cases but only from specific jurisdiction search(queryVector, { filter: { jurisdiction: "California", year: { gte: 2020 } } })
// Medical: similar symptoms but only pediatric cases search(queryVector, { filter: { patientAge: { lte: 18 }, verified: true } })
This combination of vector similarity + metadata filtering is what makes vector databases production-ready.
Popular Vector Databases
There are several good options. Here are the main ones you'll hear about:
Pinecone
Type: Cloud-only (managed service)
Best for: Production apps, teams who don't want to manage infra
Pricing: Free tier available, then pay per usage
Setup: Very easy — just an API
Scaling: Handles billions of vectors automatically
Our course: We'll use this for cloud deployment
Chroma
Type: Open source, runs locally
Best for: Development, learning, small projects
Pricing: Free (self-hosted)
Setup: npm install, runs on your machine
Scaling: Good for small to medium scale
Our course: We'll use this first — perfect for learning
Qdrant
Type: Open source, can self-host or use cloud
Best for: High performance production systems
Pricing: Free self-hosted, paid cloud option
Setup: Medium complexity
Scaling: Excellent — very high performance
Weaviate
Type: Open source, can self-host or use cloud
Best for: Complex RAG with built-in ML features
Pricing: Free self-hosted, paid cloud option
Setup: More complex
Scaling: Excellent
Milvus
Type: Open source, self-hosted
Best for: Very large scale enterprise systems
Pricing: Free self-hosted
Setup: Complex
Scaling: Best-in-class for massive scale
pgvector (PostgreSQL extension)
Type: Extension to PostgreSQL
Best for: Teams already using PostgreSQL, smaller scale
Pricing: Free
Setup: Easy if you already have PostgreSQL
Scaling: Good up to ~10 million vectors
Which One For This Course?
Phase 4 learning → Chroma (local, no setup, free)
Phase 5 RAG project → Chroma first, then Pinecone
Production projects → Pinecone or Qdrant
We start with Chroma because:
- Runs entirely on your machine
- No API key needed
- Perfect for learning
- JavaScript support is great
- You can see exactly what's happening
The Core Operations — What You Do With a Vector DB
Every vector database supports these basic operations:
1. Upsert — Store a vector
// Store a document chunk with its embedding await vectorDB.upsert({ id: "doc1_chunk3", // unique ID for this chunk // example: "medical_handbook_page_5_chunk_2"
vector: [0.23, -0.87, 0.41, ...], // the embedding — 1536 numbers // generated by OpenAI embedding model
metadata: { source: "medical_handbook.pdf", page: 5, category: "side_effects" }, // any extra info you want to store alongside the vector
text: "Aspirin can cause stomach bleeding..." // the original text chunk (optional but useful) });
2. Query — Find similar vectors
// Find top 5 most similar chunks to a question const results = await vectorDB.query({ vector: [0.19, -0.82, 0.38, ...], // embedding of the user's question
topK: 5, // return top 5 most similar results
filter: { category: "side_effects" }, // optional metadata filter
includeMetadata: true, // return the metadata alongside results });
// results looks like: // [ // { id: "doc1_chunk3", score: 0.94, text: "Aspirin can cause...", metadata: {...} }, // { id: "doc2_chunk7", score: 0.87, text: "Common side effects...", metadata: {...} }, // ... // ]
3. Delete — Remove a vector
// Remove a specific chunk (e.g. document was updated) await vectorDB.delete({ id: "doc1_chunk3" });
4. Fetch — Get a specific vector by ID
// Get a specific stored vector const item = await vectorDB.fetch({ id: "doc1_chunk3" });
These four operations — upsert, query, delete, fetch — are 90% of what you'll do with a vector database.
The Complete Flow — Putting It All Together
Here is the full picture of how a RAG system uses a vector database:
SETUP PHASE — done once when you build the system:
PDF / Documents
↓
Split into chunks (e.g. 500 words each)
↓
For each chunk:
→ Call OpenAI embeddings API
→ Get vector [1536 numbers]
→ Store in Vector DB with metadata
↓
Vector DB builds HNSW index
↓
System is ready
QUERY PHASE — happens every time a user asks something:
User question: "What are aspirin side effects?"
↓
Call OpenAI embeddings API on the question
↓
Get question vector [1536 numbers]
↓
Query Vector DB with that vector (topK: 5)
↓
HNSW index finds top 5 similar chunks in ~2ms
↓
Get back 5 text chunks + their metadata
↓
Build LLM prompt:
"Answer this question using only the context below:
Context:
[chunk 1 text]
[chunk 2 text]
[chunk 3 text]
Question: What are aspirin side effects?"
↓
Send to GPT-4
↓
Get accurate, grounded answer
↓
Show to user
This is RAG. And vector database is the heart of it.
A Real Life Analogy — The Smart Library
Think of a vector database like a smart library — but not organized by title or author.
This library organizes books by topic similarity.
When you add a book:
→ Librarian reads it
→ Understands what it's about
→ Places it near books about similar topics
→ Books about space travel cluster together
→ Books about cooking cluster together
→ The shelves arrange themselves by meaning
When you search:
→ You describe what you're looking for
→ Librarian understands the meaning of your description
→ Goes directly to the right area of the library
→ Returns the closest matching books
You don't need to know the title, author, or exact words.
You just describe what you need — and meaning does the matching.
3-Line Summary
- A vector database uses ANN indexing (like HNSW) to find similar vectors in milliseconds — instead of comparing against every single vector, it navigates a smart multi-layer graph to zoom in on the right area.
- Vector databases store three things together — the vector itself, metadata (source, date, category), and optionally the original text — metadata filtering lets you combine semantic search with structured filters just like SQL WHERE clauses.
- The four core operations are upsert (store), query (find similar), delete (remove), and fetch (get by ID) — query with topK is what powers RAG, returning the most relevant document chunks to inject into your LLM prompt.
Module 4.2 — Complete ✅
Coming up — Module 4.3 — ANN Search, Top-K & Metadata Filtering in Depth
We go deeper on the search mechanics. How exactly does Top-K work in practice? What happens when you combine vector search with metadata filters? What are the parameters that control search quality? And what tradeoffs do you make in production? All of that — next.
No comments:
Post a Comment