The Hardware Physics of Vector Databases
Transcript
Made automatically from the audio, so it may contain mistakes. Click a line to play from there.
You know, usually when a developer ships a new database feature, the failure modes, they they just make logical sense. Right.
Yeah.
Like you miss an index, your query slows down. You write a bad join, you get a timeout.
Exactly.
You look at the stack trace and you fix it. But when you step into the world of retrieval augmented generation, when you start treating human languages like high-dimensional geometry, the failure modes get incredibly weird.
Oh, they get completely bizarre.
Yeah. And picture this scenario. Because I mean it is happening in production every single day right now. A team builds a shiny new AI search feature. They take their user's query, convert it to a dense vector, and run a standard approximate nearest neighbor search against their database.
Standard rag pipeline.
Right. Then on the application side, they apply a simple post filter. Just a basic SQL clause like where tenant ID equals small customer.
Yep.
The query executes and it returns zero rows. The small customer gets a completely blank screen. And why? Because out of the top twenty most semantically similar vectors in the entire global database, every single one of them happened to belong to a massive enterprise client.
Wow.
Yeah. The small customer is perfectly good matches. We're sitting at, you know, position 21 and beyond. The application literally threw away the only results the database returned.
And it is a terrifying bug to debug too, because it doesn't throw a 500 error. No. The CPU isn't spiking or anything. The system is behaving exactly as the math dictates, but the user experience is just it's entirely broken. It's a silent structural failure of architecture.
It really is. And if the zero row bug doesn't keep you up at night, the cloud bill absolutely will.
Oh, the vector tax.
The vector tax. I talked to a data engineering team recently that decided to index 30 million open AI embeddings. Just standard 1536 dimensional float 32 vectors.
Okay, yeah.
They figured, hey, 30 million rows. That's a rounding error for a modern and Postgres database. So they spun up their infrastructure, built an HNSW index, and at the end of the month they got slammed with an AWS bill that practically made their CFO's hair fall out.
I can imagine. Just for 30 million rows.
Yes, you are just bleeding cash because of this. So today we are tearing the engine apart to find out why your vector database is eating your RAM, why your filters are failing, and how to actually fix it.
I love it.
This is our deep dive into the internal mechanics of storing and searching embeddings. We're getting past the hype and looking at the actual hardware physics.
And just to set the operational boundaries for this discussion, we are entirely skipping the you know introductory tutorials on what an LLM is or how to prompt Chat GPT.
Right. You guys already know that.
Exactly. We assume you already know what a vector embedding is conceptually. We also assume you understand how standard Postgres heaps and GIN indexes work from the first deep dive in the series.
Yep.
And that you understand change data capture CDC synchronization pipelines from our fourth deep dive. Today is purely about systems engineering.
The heavy lifting.
Right. We are looking at the cold hard hardware physics of HNSW, IVF, and disk ANN, and drawing the architectural lines between using Postgres PG vector versus a dedicated engine like Coodrint, Milvis, WeV8, or Pinecone.
Okay, so I want to start with a 220 gigabyte memory problem.
Mm-hmm.
Because I I feel like we can't solve it until we we kind of establish why finding these vectors is mathematically so expensive in the first place.
That's a fair point.
Because I I mean, if I if I got a database of like, say 30 million user profiles, right? And I want to find everyone named Smith, a standard B tree index finds that in like milliseconds, using almost no memory, right?
Yeah, B trees have literally run the global economy for 40 years.
Right. So so why does why does a standard B tree just completely kind of shatter when we try to put embeddings into it?
Well it breaks because an embedding is a physical array of floating point numbers. It's usually anywhere from 768 to 3072 dimensions, depending on the model you're using.
Okay.
A B tree is phenomenal for one-dimensional data. If you are sorting alphabetical strings or integers, a B tree recursively divides the data space into predictable ordered partitions.
Like checking your dictionary.
Exactly. You ask for Smith, the tree checks the root node, says S is in the right half, goes down a branch, and in O log n time, you have your record. But vectors are high dimensional. When you try to build a tree structure in 1,536 dimensions, you run headfirst into a geometric nightmare known as the curse of dimensionality.
I hear that phrase thrown around at AI conferences constantly, but break down the actual geometry of it for me. Like what is a fifteen hundred thirty-six dimensional space actually look like to a computer?
Let's do a thought experiment to visualize the math. Imagine a two-dimensional space, like a square piece of paper.
Okay, got my paper.
Draw a circle inside that square, so it touches all four edges. The circle takes up roughly seventy-eight percent of the area of the square. The corners of the square where the circle doesn't reach, they're relatively small.
Right. Makes sense.
Now let's move to three dimensions. Imagine a cube and place a sphere inside it, so it touches all the faces. The sphere now only takes up about 52% of the cube's volume. The corners are getting larger.
Oh, I see. The sphere is shrinking relative to its container.
Yes. Now mathematically extrapolate that out to a hundred dimensions. If you put a 100 dimensional hypersphere inside a 100 dimensional hypercube, the volume of the sphere becomes practically zero.
Wait, really?
Yeah, it's something insane like point zero zero zero zero zero zero zero zero zero zero zero zero one percent of the cube's volume. Almost a hundred percent of the volume of high dimensional space is concentrated out in the corners.
That is wild. So if my data points are scattered in this fifteen hundred and thirty-six dimensional cube, they're all sitting out in these bizarre infinite corners.
Precisely. And because they are all pushed out into the corners, the distance between any two randomly chosen points in that space tends to converge.
What do you mean by converge?
Mathematically, they all start looking roughly the exact same distance apart from each other. The boundaries between what is considered close and what is far become incredibly fuzzy.
Okay.
So if you try to partition this space with a B tree, almost every branch of the tree overlaps with the region you need to search. The tree cannot eliminate any branches. It ends up having to visit almost every single node anyway.
So it just degrades into a linear scan.
Exactly. But worse, it's a linear scan with the added computational overhead of navigating a completely useless tree structure.
Wow. Okay, so if the tree can't tell what's close, how do we we have to rely on specific distance metrics, right?
We do. We rely on raw mathematical comparisons between the arrays. The three you'll generally see in production are cosine similarity, which measures the angle between two vectors while ignoring their magnitude. Right. Then there's dot product or inner product, which measures both the angle and the magnitude. And finally, L2 distance, which is just the straight line physical Euclidean distance between the points.
Got it.
And a really crucial architectural detail here for anyone building this. If you are using open AI embeddings or any models that output normalized vectors, meaning the math is constrained so every vector has a length of exactly one, you should always configure your database to use dot product, not cosine.
Let me guess. If the length is always one, the magnitude doesn't matter. So cosine and dot product will rank the results in the exact same order.
Yep.
But dot product requires fewer CPU cycles because it doesn't have to calculate the denominator of the cosine equation.
That is precisely the mechanism. In PG vector, for example, you should configure your index using the inner product operator, that's the less than hash greater than symbol, rather than the cosine operator if your vectors are normalized.
That makes total sense.
Because at scale, avoiding that division operation across billions of dimensions saves a massive amount of CPU time.
Before we get into the complex indexes that speed this up, I actually want to challenge the premise for a second.
Sure.
If high dimensional indexes are so memory hungry and complex, why build an index at all? What is the mathematical limit of just brute forcing the search, like just scanning every single row?
So you're talking about an exact flat scan.
Yeah.
The mathematical complexity there is O of N times D, number of vectors multiplied by the number of dimensions. For every single user query, the CPU literally calculates the distance between the query vector and every single vector in the database, sorts the millions of results, and returns the top K.
Which on paper sounds like an impossible bottleneck. Sounds like it would take 10 minutes per query.
It does sound like that. But modern hardware has a trick up its sleeve called SIMD instructions. Single instruction, multiple data.
Oh, I've seen that acronym. How does it work?
SIMD allows a CPU core to perform the exact same mathematical operation on a large batch of numbers simultaneously in a single clock cycle.
Oh wow.
Yeah. So when you do a flat scan, the CPU loads chunks of your vectors into its L1 cache and fires off these SIMD instructions. A standard modern CPU core can chew through millions of floating point multiplications in milliseconds.
So if I have, say, a multi-tenant SaaS application, and my users only ever search within their own workspace. And a workspace has like 50,000 vectors.
You don't need a vector index at all.
Really?
None at all. You absolutely do not. This actually gives us our first hard scale rule. If your search scope is consistently under roughly 100,000 vectors per tenant, an exact flat scan often beats a complex index.
That's fascinating.
It guarantees perfect 100% recall. You never miss the true nearest neighbor, and it requires absolutely zero RAM overhead for maintaining a graph structure. It is pure brute force hardware efficiency.
But I'm guessing that hardware efficiency hits a brick wall eventually. I mean I can't brute force 10 million vectors in a multi-tenant workspace, let alone a billion globally, and expect a 50 millisecond response time.
No, you can't. The SIMD instructions are fast, but they can't outrun O of N times D forever. Once your search scope crosses that 100,000 to 500,000 threshold, you must trade perfect recall for speed.
You have to compromise.
Exactly. You must accept that you might occasionally miss the absolute mathematically closest vector as long as you get a very good vector in under 50 milliseconds. That is the definition of an approximate nearest neighbor or ANN index.
Okay, so let's crack open the internal mechanics of these ANN structures. Because this is where the RAM explosion from our cold open actually happens.
Right.
Let's start with IVF, the inwarded file index. How does it organize this high-dimensional space if it can't use a tree?
IVF, specifically IVF flat, relies on clustering. During the index build phase, the database runs a K-means clustering algorithm over your entire data set. It groups your vectors into clusters based on their proximity.
Okay, so just bunching them up.
Yeah, and each cluster has a mathematical center point called a centroid. These centroids divide your vector space into what are geometrically known as Voronoi cells.
Okay, I always struggle to visualize Vornoi cells in high dimensions. If I drop it down to two dimensions, is it sort of like a map of cell phone towers?
That's a great analogy, actually.
Where the boundary of a cell is just the exact halfway line between two towers. So if you are standing anywhere inside a specific polygon, you are guaranteed to be closest to that specific tower?
That is a highly accurate mental model. The entire space is partitioned into these polygons. In an IVF index, the number of these towers or centroids you create is controlled by a parameter usually called n list.
N list, okay.
If you set n list to a thousand, you have partitioned your entire database into a thousand distinct Voronoi cells.
Okay, so a query comes in from a user. Does the database check every cell?
No, because that would defeat the purpose. The query vector is compared only against the 1000 centroids. It finds the handful of centroids that are closest to the query.
How many is a handful?
Well, the number of centroids it is allowed to explore is controlled by a query time parameter called n probe.
Got it.
So if n probe is three, the database identifies the three closest cell towers, completely ignores the other 997, and does an exact flat scan only on the vectors residing inside those three specific Voronoi cells.
I see the speed advantage immediately. You are skipping like 99.7% of the math. But I also see the massive flaw here.
The boundary problem.
Yeah. What if my query vector lands just inches inside the boundary of cell A? The algorithm searches cell A. But what if the absolute best, most semantically perfect matching document in the entire database is sitting literally one millimeter across the border inside cell B.
Yep.
If cell B wasn't included in my n probe limit, I missed the perfect match entirely.
And that is the exact nature of the approximation trade-off. IVF is notorious for this boundary problem. You trade accuracy at the edges of the Voronoi cells for the sheer speed of ignoring the rest of the map.
Which I assume explains why the industry largely moved away from IVF as the default and shifted almost entirely to HNSW.
Yes.
Hierarchical, navigable small world. It feels like every single modern vector database uses this under the hood.
Pretty much.
And it doesn't use clustering, right? It builds a graph.
That's right. HNSW abandons the Voronoi cells entirely and builds a multi-layer proximity graph. The foundational concept is based on a skip list, but instead of visualizing an airport hierarchy, think of it more like the concept of six degrees of Kevin Bacon or a massive global social network.
Okay, map that out for me.
At the very bottom layer of the graph, we'll call it layer zero. Every single vector in your database is a person in the social network, and everyone is connected to a few of their closest friends via edges.
Okay, simple enough.
If you want to find someone across the globe by only asking people their friends know, it would take you millions of hops.
Right, taking local roads only. Exactly. It takes forever.
So HNSW introduces higher layers. Layer one contains a random subset of those people, maybe 10% of them. Layer two contains 10% of layer one. This goes up until the top layer has only a handful of entry points. Think of them as global celebrities who know a lot of different groups.
Okay, I'd follow.
When a query comes in, the search starts at the top layer. It asks the celebrity, who do you know that is closest to this query? It hops to that node, then drops down a layer, it asks that person the same question, finding a closer match, and drops down again.
So by using these long-range celebrity connections at the top, I can jump across the entire data space in two or three hops, and then do the fine-grained meticulous searching among the close friends down at layer zero.
The routing efficiency is staggering. You traverse massive geometric distances almost instantly. Now the structure of this graph is dictated by three critical parameters you will see in every vector database configuration.
I always see these in the docs and just leave them as default. What are they?
First is M. This dictates the maximum number of edges or friends any single node can have at each year. A higher M creates a denser social network with more pathways.
Which means the algorithm is less likely to hit a dead end, so recall improves. But I'm guessing the database has to store all those extra connections in memory, inflating the footprint.
Exactly. You're trading memory for accuracy. The second parameter is EF construction.
What does that do?
This determines how many candidate nodes the algorithm considers when it is inserting a new vector and trying to figure out who its best friend should be. A higher EF construction builds a beautifully optimized, highly connected graph, but the CPU cost during the index build phase is brutal. It will literally lock up your writer nodes.
Okay, so that's for building. And the third parameter is for when the user actually hits enter on the search bar.
Yep. EF search. Or sometimes just EF. This controls the size of the dynamic candidate list during the search phase. It defines how broadly the algorithm explores the neighborhood around the target once it reaches the bottom layer.
So increasing GF search boosts your recall because you check more local neighbors, but it linearly increases latency.
Spot on. So HNSW gives us this incredible routing mechanism. Low latency, high recall. But here is the poison pill. Let's walk through the actual math problem of storing this graph, going back to the cloud bill nightmare from our cold open. Let's assume a mid-size enterprise data set, say 25 million documents.
Okay, let's do the math. 25 million documents.
We embed them using OpenAI's text embedding 3-small or the standard 80002, giving us 1,536 dimensions. We are storing these as standard 32-bit floating point numbers of P32.
Right. So 1,536 dimensions multiplied by four bytes per float. That's exactly 6,144 bytes per vector, just over six kilobytes.
The arithmetic is unyielding. 25 million vectors multiplied by 6,144 bytes per vector gives us roughly 153.6 gigabytes of raw uncompressed vector data.
153.6 gigabytes, just for the arrays themselves. But we have to build this social network graph on top of it. We have to store the edges.
Right. The memory overhead for the graph depends heavily on your M parameter, but a standard production configuration typically adds about 35% overhead to your raw data size just to store the node pointers and layer structures.
So 153.6 gigs plus 35%. That puts us squarely at 207 gigabytes of total index size.
And here is the fatal architectural requirement of standard HNSW. For that graph traversal to be fast, it absolutely requires random access memory. RAM. Yes. Think about the social network hop. You jump from a celebrity in layer three down to a regional node in layer two. Geometrically, those two vectors might be nowhere near each other on a physical disk. The algorithm is constantly bouncing erratically around the data set.
Right. It's not a sequential read at all.
Exactly. If those vectors are sitting on a storage disk, even a top-tier NVMe SSD, the disk IO latency will completely bottleneck the CPU. Standard HNSW demands that the entire 207 gigabyte graph and all the underlying vectors be pinned continuously in RAM.
207 gigabytes of DRAM for just 25 million vectors. I mean, if you want to run that in production, you are renting massive memory optimized instances. If you scale to 500 million vectors, you are looking at terabytes of RAM. Yep. That is financially ruinous for most engineering teams. There has to be a way to put this on disk without the I.O. latency destroying the search.
There is. And it represents a major divergence in how vector databases are built. It brings us to an architecture called disk ANN.
Disk A N N. Yeah.
Okay. Yeah, which relies on a completely different graph structure called the Vimanograph. If you look at an engine like Milvis, they integrate disk A and N principles to solve this exact RAM explosion.
So if Vimana is a graph, how does it avoid the random access disk penalty that kills HNSW?
Well, HNSW builds multiple hierarchical layers to facilitate those long jumps, right? Vamana flattens the architecture entirely. It is a single layer graph.
Just one layer.
Just one layer. But it is constructed using a very specific two-pass algorithm. It actually starts by connecting all the points randomly.
Wait. If Vimana starts with a completely random web of connections, isn't that just mathematical chaos? If I'm traversing a random web, I'll be bouncing all over the SSD reading garbage. How does it organize that?
The random graph is just the starting state to ensure the entire data set is reachable. The magic happens during the two optimization passes. In the first pass, the algorithm ruthlessly prunes redundant local edges and tries to connect close neighbors, much like layer zero of HNSW.
Okay, so it cleans up the neighborhood.
Right. But in the second pass, it introduces a parameter called alpha. If you set alpha greater than one, say 1.2 or 1.5, the algorithm deliberately seeks out and creates long-range edges. It mathematically forces connections between nodes that are quite far apart in the vector space.
Oh, I see. So instead of building a whole separate celebrity layer above the data, it just builds massive superhighways straight through the local neighborhoods.
Exactly. These long range edges act as shortcuts. By weaving them directly into the single layer, Vimana keeps the total diameter of the graph incredibly small. You can get from any point A to any point B in very few hops.
Which is critical for a disk-based system because every hop requires an I.O. read. By minimizing the hops, you minimize the disk latency.
You got it.
But still, even with fewer hops, waiting for an SSD read mid-search is still agonizingly slow compared to CPU cache speeds. How does disk N actually execute the search fast enough for production?
Disk ANN employs a brilliant hybrid memory architecture. It does not put the entire system on the SSD.
Okay, what stays in RAM?
It takes the massive 1536-dimensional FP32 vectors and the full VMANA graph and stores them on the fast NVMe SSD. But it generates a microscopic, heavily compressed representation of those vectors using product quantization, which we'll break down shortly, and pins only those tiny codes in RAM.
So the RAM holds a highly compressed map, and the SSD holds the actual heavy data.
Correct. When a query arrives, the search algorithm, the beam search, which is controlled by a parameter called beam width ratio, navigates the graph entirely in RAM using the compressed codes.
So it's fast.
Super fast. It figures out the rough neighborhood of candidates instantly without touching the disk. Only after it has identified the target neighborhood does the system reach out to the NVMe SSD. Asynchronously fetch the full precision FP32 vectors for just those few candidates and recalculate the exact distances to re-rank the final results.
That's amazing. The traffic hop lives in RAM, but the heavy lifting happens asynchronously on disk. That fundamentally changes the economics. You can serve a billion vectors from a single machine with a standard 64 gigabyte RAM allocation, whereas HNSW would require a sprawling distributed cluster.
Exactly. But, and this is a big butt, disk ANN requires a purpose-built storage engine designed around asynchronous I/O and SSD block sizes. Right. What if you aren't using a dedicated engine? What if you are running PG vector inside Postgres QL, which is designed around in-memory page caching?
Yeah, Postgres doesn't do async IO like that.
Right. If you have to use in-memory HNSW, you must find a way to shrink that 207 gigabyte monster.
Which brings us to the quantization ladder. If we can't push it to disk, we have to crush the data itself. We have to throw away precision to save memory and just you know, pray we don't destroy the semantic meaning of the vectors in the process.
Let's walk down the ladder.
Let's do it. We are starting at 32-bit floats.
The first rung down is scalar quantization, specifically precision reduction. You convert your FP32 numbers to FP16, 16-bit floats. Okay. In Postgres QL, this is implemented as the half vec data type. You are literally cutting the precision of every single dimension in half. Your 207 gigabyte footprint instantly drops to about 103 gigabytes.
And mathematically, how badly does chopping the precision in half hurt my search relevance? Like am I going to start returning the wrong documents?
The cost to recall is practically zero.
Really? Zero.
Pretty much. You have to understand that the neural networks generating these embeddings are incredibly robust, but they aren't magic. An embedding model rarely encodes meaningful semantic variants in the seventh decimal place of a floating point number.
Oh. So that deep precision is mostly noise, anyway.
Exactly. Dropping the half vec throws away the noise. It is essentially a free 50% memory reduction. Unless you have a bizarre mathematically hypersensitive edge case, you should always be using half vec over FP32 in production.
Okay. So 103 gigabytes is better, but it's still heavy. Let's push further down the ladder. What about int 8 scalar quantization?
Here we compress the floating point numbers into 8-bit integers. The algorithm finds the minimum and maximum values across a dimension in your dataset, maps those extremes to 0 and 255, and bins all the floating point numbers into those 256 discrete integer buckets.
So a complicated decimal value like point one two three four five just becomes like fourteen?
Exactly. This yields a 4x compression ratio compared to FP32. Your 207 gigabyte footprint drops to roughly 52 gigabytes.
That's a huge drop. What about accuracy?
The accuracy loss is slightly higher than half vec, but usually less than a 1% drop in top 10 recall for high-dimensional models like OpenAIs. Furthermore, modern CPUs possess SIMD instructions explicitly optimized for 8-bit integer math.
Oh nice.
Yeah, so your footprint shrinks by 75%, and your distance calculations actually execute faster.
Okay. 52 gigabytes is totally manageable on a standard cloud instance. But I know there is an even more extreme tier. Binary quantization.
Oh yeah.
One-bit vectors. We take a floating point number with all its subtle infinite nuance, and we crush it into a single bit, a one or a zero. Usually, if the original float is greater than zero, it becomes a one. If it's less than zero, it becomes a zero.
Yes. One bit quantization. It reduces your memory footprint by a staggering factor of 32. That 207 gigabyte HNSW index, it compresses down to about six gigabytes of RAM.
I I have to push back here.
Go for it.
If you drop 31 bits of data per dimension, you're literally throwing away the magnitude entirely. You just have a string of signs, plus or minus. How can you possibly retain semantic meaning when you've reduced a complex multi-dimensional space to a coin flip?
You do lose a massive amount of geometric precision. There's no getting around that. But you gain a computational superpower. Which is when vectors are binary strings, you no longer use cosine or dot product to calculate distance. You use Hamming distance.
Hamming distance. Okay.
Hamming distance measures the difference between two binary strings using a simple logical XOR CPU operation.
Oh. And a standard floating point multiplication takes multiple clock cycles. But an XOR operation on a 64-bit integer compares 64 dimensions simultaneously in a single CPU cycle.
And it is violent speed.
You can scan millions of binary vectors in a fraction of a millisecond.
That's insane. But to address your pushback, yes, the raw results of a Hamming distance search on one bit vectors are extremely noisy. They are not accurate enough to serve directly to a user. That is why one bit binary quantization mandates an architectural pattern called oversampling and rescoring.
So how does that Okay walk me through that that how that pattern recovers the accuracy.
Okay, so well, because the Hamming distance scan on the one-bit index is so unbelievably fast, you don't ask it for your top ten results. You ask it for your top 200 or 300 results.
Are you oversample?
You oversample by a factor of 20 or 30. The one-bit index acts as a very coarse, very fast sieve. It hands you 300 document IDs that are roughly in the correct semantic neighborhood.
Then what? I can't show the user 300 roughly related documents.
Then you take those 300 IDs, go to your slower storage tier, like your main database heap or an SSD, and fetch the original uncompressed FP32 vectors for only those 300 documents. You recalculate the exact cosine distance in memory, re-rank them perfectly, and return the true top 10 to the user.
That is brilliant. It's conceptually similar to the disk AN architecture, but purely through quantization logic. You use the six gigabyte one-bit index to rapidly find the neighborhood, and then do the heavy, expensive math on just a tiny handful of rows.
It is a phenomenal design pattern for massive scale. Now there is one final quantization technique we must cover because it operates on a different axis entirely. Product quantization or PQ.
This was the one used in disk AN, right?
Exactly. Scalar and binary quantization truncate the numbers themselves. PQ chunks the vector structure.
Break down the exact process of PQ for me. Like if I have my vector array 1.2, 0.5, negative 0.3, and so on. What is PQ doing to it?
Let's take your 1,536 dimensional vector. Okay. PQ will slice that array into, say, eight equal segments.
Okay, cutting it up.
You now have eight subvectors, each with 192 dimensions. During the index build, the database runs a k-means clustering algorithm independently on all the first chunks across your entire database, finding 256 cluster centroids just for chunk number one. Then it does the same for chunk two, chunk three, all the way up to eight.
Why specifically 256 clusters?
Because 256 unique cluster IDs can be represented perfectly by a single 8-bit byte.
Oh, that's clever.
Very. So for every vector in your database, the PQ algorithm looks at its first chunk, compares it to the 256 learned centroids, finds the closest one, say centroid number 42, and replaces those 192 floating point numbers with the single byte 42.
Wow.
It repeats this for all eight chunks. Your massive 1536 dimension float array is mathematically replaced by an eight-byte proxy code.
Then it's up to 64x compression.
But but wait.
No, you can't.
You use a technique called asymmetric distance computation, or ADC. This is vital. When the user's query vector comes in, you do not compress it. You leave the query vector uncompressed.
Okay, so query is full precision, database is compressed.
Exactly. Before you scan the database, you pre-calculate the distance between the uncompressed queries first chunk and all two hundred and fifty six possible centroids and store those distances in a tiny lookup table in the CPU cache.
Oh, I see where this is going.
During the scan, when the database sees the byte 42, it doesn't do any math. It just looks up the precalculated distance for Centroid 42 in the table. You balance extreme memory compression with highly efficient lookup calculations.
Okay. We have covered the physics we understand how the graph structures navigate the void and we have quantized the vectors to fit the footprint into RAM. Our system is humming.
Everything is looking good.
And now we run straight into the exact bug I mentioned in the cold open the filterable ANN trap. We run a search we want to filter the results by tenant ID and the small customer gets zero rows. Let's dissect this.
They fell victim to the mathematical failure of naive post filtering. The system executed exactly as instructed but the architecture was fundamentally flawed.
The application ran the HNSW search first. It asked for the top 20 semantic matches globally. The index traversed the graph, found the 20 absolute best matches across all tenants and handed them to the Postgres execution engine.
Yes. And then Postgres applied the SQL where clause where tenant ID equals small customer. But all twenty of those top results belonged to the massive enterprise client.
So they all failed the check.
The filter stripped every single one of them out, leaving zero rows. The application filtered the output rather than guiding the search.
So if I'm a developer seeing this, my immediate reaction is, okay, post filtering is stupid, I will pre-filter.
Right. Natural response.
I will apply the SQLWare clause first, isolate only the rows belonging to the small customer, and then I will tell the HNSW index to run its search on whatever's left.
And that developer has just introduced an even more catastrophic architectural failure. Naive pre-filtering physically breaks the HNSW graph.
Why does isolating the rows break the graph?
Remember the social network analogy we used. HNSW relies on connections between nodes to navigate. If you apply a strict pre-filter, say filtering down to just one percent of the database, you are effectively telling the search algorithm that 99% of the nodes in the graph are invalid and cannot be traversed.
Oh. It's like trying to play six degrees of Kevin Bacon, but I tell you that you are legally forbidden from talking to anyone who has no left-handed dentist.
Precisely. You completely sever the connectivity of the graph. The search algorithm lands at an entry node, evaluates its friends, and realizes every single friend violates the SQL filter. They are off limits.
So it's just stuck.
The algorithm hits a dead end immediately. It cannot navigate toward the target. It gets stranded in the first neighborhood and returns terrible results or fails entirely.
Post-filtering gives you zero results because it blindly filters the top K, and prefiltering breaks the graph so you can't navigate. It's a lose loose trap. How do modern engines actually escape this?
This is where we see the most significant divergence in engineering approaches between standalone vector engines and Postgres's PG vector. Let's look at the standalone engine approach first. Systems like QDRI and Weave It.
Okay, how do they handle it?
They solve this using a concept called filterable HNSW, heavily augmented by roaring bitmaps.
Explain how a standalone engine integrates the filter into the graph.
When engines like QDrant build the HNSW graph, they do not just store the vectors, they index the payload metadata, like the tenant ID, directly into the graph's internal structure. They create additional specialized edges that take these filters into account, but the real magic happens at query time using the roaring bitmap.
I've seen roaring bitmaps mentioned in a dozen white papers. I know it's a way to compress a list of integers, but how does it actually work under the hood? Like, if I have a billion rows and I need a bit for every row to say yes or no to a filter, a billion bits is still a lot of memory.
Roaring bitmaps don't just store a flat array of ones and zeros. They dynamically analyze the density of the integers they are storing and segment them into different container types.
What do you mean by container types?
If a block of IDs is incredibly dense, say every row from ID 1000 to 5000 matches the filter, it uses run length encoding. It just stores 1,000 through 5,000 rather than 4,000 individual bits.
I see.
If it's sparse, it uses an array container. It can compress massive lists of matching document IDs into kilobytes.
So before the graph search even starts, QDrant evaluates the filter and generates this tiny highly compressed roaring bitmap containing the IDs of every valid node.
Yes. And as the algorithm traverses the HNSW graph, it constantly checks its neighbors against this bitmap. If a neighbor is not in the bitmap, the algorithm doesn't just hit a dead end. It uses those extended edges and the bitmap as a guide to dynamically route around the invalid nodes, bypassing them without ever losing connectivity to the rest of the graph.
It maintains the speed of HNSW while guaranteeing the filter is applied flawlessly.
Exactly.
That is incredibly elegant. But what about PostgresQL? PG Vector is an extension living inside a relational database. Postgres owns the where clauses, the execution planner, the memory bounds. The vector index doesn't control the metadata. How does PGVector escape the trap?
PG Vector had to solve this natively within the constraints of Postgres. In version 0.8, they introduced a mechanism called iterative index scans. It effectively turns the HNSW search into a continuous pipeline between the index and the Postgres executor.
Visualize that pipeline for me.
When a query arrives with a strict SQL filter, PGVector does not just ask the index for 20 results. It asks the index to start yielding candidates one by one in order of semantic proximity.
Okay.
The index hands the best candidate to the Postgres executor. Postgres checks that specific row against the SQL where clause. If it fails, Postgres discards it and asks the index for the next candidate.
And it just keeps going.
The index continuously scans deeper and wider into the graph, emitting a stream of candidates until Postgres finally says, stop. I have collected 20 valid rows that match both the semantic distance and my SQL filters.
It just keeps digging until it hits water. But wait, if the filter is highly restrictive, like matching only one row in a million, won't Postgres force the index to scan the entire graph, completely locking up the CPU?
That is the exact risk of an iterative scan. To prevent a runaway query from destroying your database performance, PG Vector implements a hardcap using a parameter called hnsw.max scan tuples.
A safety valve.
Exactly. If the pipeline evaluates, say 20,000 candidates and still hasn't satisfied the Postgres where clause, it forcefully terminates the search and returns whatever it found. But this iterative approach cleanly separates the vector traversal from the relational filtering logic, allowing them to cooperate without breaking the graph.
Alright, so we've solved the RAM problem, we solve the filter trap, we can reliably find semantic matches belonging to a specific user. But sometimes semantic search, dense embeddings, cosine similarity, it all completely fails, even without a metadata filter. Let's talk about the blind spots of dense vectors.
The classic failure mode of purely dense retrieval occurs when the user is searching for exact, highly specific lexical matches. Dense embodings capture latent meaning, but they inherently smudge exact characters.
Right. If a user in an e-commerce app searches for a highly specific SKU number like AB 1234X, or an engineer searches a documentation base for a specific obscure error code like ERR7781. A dense embedding model might map that query to the general concept of product identifiers or system errors.
Yep, it completely loses the exact alphanumeric string.
It returns documents that are semantically similar, maybe other error codes, but entirely wrong.
This is why lexical search, traditional keyword search is absolutely still mandatory in modern architectures. Dense vectors are for semantic meaning, sparse vectors are for exact term matching.
When you say sparse vectors, you mean traditional BM25, right? Term frequency inverse document frequency.
BM25 is the classic standard, yes. It creates a vector where each dimension represents a specific word in the vocabulary. And the value is based on how often that word appears in the document versus the whole database. Right. Because most documents only contain a tiny fraction of the total vocabulary, the vector is mostly zeros, hence sparse. There are also more advanced sparse models now like splade, which are sparse lexical and expansion models.
How does splade differ from BM25?
Splade uses neural networks to generate sparse vectors, but it injects synonyms and related terms into the vector, bridging the gap between exact match and semantic meaning.
Which brings us to hybrid search, combining the dense and the sparse. A modern system executes a dense vector search to get the top semantic results, and a BM25 or splade search to get the top exact keyword results.
But merging those two result sets mathematically is extraordinarily difficult. The raw scores they output are completely incompatible.
Because they live on different scales.
A cosine similarity score from a normalized dense vector is strictly bounded. It will always fall between negative one and one or zero and one, depending on how the engine scales it. But a BM25 score is unbounded.
Right. A BM25 score could be five or fifty or eight hundred, depending on the rarity of the word and the length of the document.
Exactly. You cannot just average a cosine score of 0.85 with a BM25 score of 124. The BM25 score will completely dominate the arithmetic. The dense results will be buried.
So how do you merge them fairly?
The industry standard solution is reciprocal rank fusion or RRF. RRF completely ignores the raw mathematical scores. It throws them away entirely.
It ignores the scores.
Yep. It looks exclusively at the rank or the position of the document in each respective list.
Okay. I know the formula is one over K plus rank, where K is usually a smoothing constant like 60. But formulas and audio are terrible. Walk me through a tangible mock search competition to show how RRF balances this.
Let's do it. Assume a user searches for blue running shoes model X12. We run the dense search and we run the sparse BM25 search. Let's track two specific documents. Document A is a blog post talking generally about great blue athletic footwear.
Okay.
It is an absolute perfect semantic match. It is rank one in the dense search. But it never mentions Model X12, so it is rank 100 in the sparse search.
Okay. Document A, dense rank 1, sparse rank 100.
Now document B. Document B is the actual product page for the X12 shoe. It has the exact string model X12, but maybe the description is short, so it's only rank 5 in the sparse search. And because it lacks flowery semantic language, it's only rank 10 in the dense search.
Document B, dense rank 10, sparse rank 5. Who wins?
Let's calculate the RRF score for document A. For its dense rank of 1, it gets 1 over 60 plus 1, which is roughly 0.016. For its sparse rank of 100, it gets 1 over 60 plus 100, which is roughly 0.006. We sum those up. Document A gets a total RRF score of 0.022.
Now let's calculate document B. For its dense rank of 10, it gets 1 over 60 plus 10, which is roughly 0.014. For its sparse rank of 5, it gets 1 over 60 plus 5, which is roughly 0.015. Sum them up. Document B gets a total RRF score of 0.029.
Document B wins.
Document B wins. And this perfectly illustrates the philosophy of RRF. Document A was the absolute number one semantic match, but because it bombed the keyword search, it was penalized. Document B wasn't the absolute best in either category, but it performed very well in both categories.
So RRF mathematically rewards consistent high performance across multiple retrieval algorithms, regardless of the wild variants in their original RAW scores.
It's an incredibly robust way to build a hybrid pipeline without having to manually tune weights every time your data changes. Alright. This brings us to the final massive architectural decision. We have covered the physics, the geometry of high dimensions, the graph structures, the quantization methods, the filter traps, and hybrid search.
We've covered a lot.
Now, as an engineering lead, you have to choose a system to deploy to production tomorrow. Do you put this inside your existing Postgres database using PG Vactor? Or do you take the operational hit of standing up a dedicated cluster using Kudrant, Milvis, Weave It, or Pinecone? Let's talk trade-offs and production failure modes.
This is arguably the most consequential architectural decision a data team will make in the AI era, and it hinges entirely on operational maturity, security requirements, and shear scale. Let's build the case for PG vector first. Why put the vectors directly inside your relational database?
I mean, the single most compelling argument for PG vector is acid compliant permission joints.
Unquestionably. In modern B2B SAS applications, data security and row level permissions are strictly non-negotiable.
Absolutely.
If your security access control lists, your complex organization hierarchies, and your vector embeddings all live in the exact same Postgres transaction space, you get perfect bulletproof consistency. You can execute a standard SQL join between your vector search and your permission tables.
And tying this back to our fourth deep dive, if you keep it all in Postgres, you have zero CDC lag.
Exactly. If you use a standalone vector database, you have to synchronize your primary Postgres database with that external vector database. You must build a change data capture pipeline.
Which introduces latency.
Yes. CDC pipelines inherently have lag. It might be 50 milliseconds, it might be three seconds, or if the Kafka cube backs up, it could be 10 minutes.
Which means if a user in the highly regulated industry hits delete account in your application, the main Postgres database deletes them immediately. But their sensitive embedded documents might still be fully searchable in the external quadrant or pine cone cluster for several minutes until the CDC pipeline catches up.
Which is a huge problem.
That is an eventual consistency leak. For a SOC2 audit or HIPAA compliance, that can be a massive liability.
Furthermore, with PG Vector, you get the operational simplicity of shared backups. When you take a Postgres snapshot, you are backing up your relational data and your vector indexes at the exact same point in time. Point in time recovery is mathematically guaranteed to be consistent.
PG vector sounds like the holy grail. Zero sync lag, acid joins, easy backups. So why would anyone ever use anything else? Where is the ceiling?
The ceiling for PG vector typically manifests around the tens of millions of vectors, highly dependent on your hardware and indexing strategy. The primary failure mode is operational bloat on the writer node.
Because of the HNSW builds.
Yes. HNSW index builds require massive amounts of CPU and memory. If you attempt to build a 20 million row HNSW index on your primary Postgres writer, it can consume all the working memory and CPU cycles, severely degrading the performance of your regular OLTP transactional workloads.
Right. You do not want your basic user login queries or payment processing queries timing out just because someone is rebuilding a vector index in the background.
Exactly.
You can mitigate this somewhat by forcing the vector queries onto read replicas, but you still have to build the index on the primary to replicate the wall logs. Additionally, sharing a single massive approximate index between thousands of multi-tenant users can degrade recall unless you heavily utilize Postgres table partitioning to create separate physical indexes for each tenant.
And managing thousands of partitions becomes an administrative nightmare.
It really does.
So that is the ceiling for Postgres. Now, let's make the case for the standalone clusters, the purpose-built engines.
What's the main draw there?
The argument for dedicated engines is pure unadulterated scale and specialized architecture. Systems like Quadrant, Milvis, and WeV8 were designed from their inception to do exactly this math. They have native integration with architectures like DiskN, utilizing memory mapped files and map and NVMe asynchronous I.O. tuning right out of the box.
So they handle the disk-based stuff we talked about way better.
Yes. And they handle massive distributed sharding natively. If you need to search a billion vectors, Milvis will shard that across a Kubernetes cluster transparently. Postgres native sharding is still notoriously complex.
And they push the boundaries of quantization much faster than Postgres extensions can, right?
Yes. Standalone engines are always going to be ahead of Postgres in raw vector capabilities. Quandrant, for example, has features optimizing extreme quantization methods. They offer native multi-vector support, allowing you to seamlessly map multiple distinct embeddings to a single document and search across them concurrently.
But the cost of that specialized power is the dual store sync lag. You have to maintain the CDC pipeline. And you vastly increase your operational surface area.
Yep.
You are no longer just maintaining a highly tuned Postgres instance. You were maintaining a distributed Postgres cluster and a distributed vector database cluster with two different sets of monitors, backup strategies, and failure modes.
It is a massive operational tax. You need infrastructure engineers who deeply understand how to tune and monitor two entirely different distributed systems. You do not adopt a dedicated vector database lightly.
We have covered a tremendous amount of ground. Let's distill this technical exploration into three concrete hard rules. Decision boundaries for you to take back to your engineering team tomorrow morning.
Sounds good.
Rule number one.
Rule number one if your users only need exact lexical matches, if they are exclusively searching for SKU numbers, internal IDs, or highly specific technical jargon, do not incur the complexity of embeddings and ANN indexes. Use Postgres native TS vector and GIN full text search. Do not force semantic AI infrastructure into a purely lexical problem.
Rule number two, the PG vector sweet spot.
If you have anywhere from a hundred thousand to roughly 10 to 50 million vectors, and ACID joins and strict row level permissions are critical to your application's security posture, PG vector inside your existing Postgres database is your absolute sweet spot.
Yep.
Utilize half vec quantization to keep the memory footprint manageable and rely on iterative index scans to handle your filters safely. Keep your architecture as simple as possible until sheer scale forces your hand.
And rule number three, the dedicated scale boundary.
If you are crossing 100 million into a billion plus vectors, or if your application requires extreme queries per second on complex, highly filtered ANN searches across massive global data sets, you have officially outgrown the relational paradigm.
You have to bite the bullet.
You must accept the operational tax, build the CDC pipelines, and deploy a dedicated engine like QDrint, Milvis, Huiviate, or Pinecone to leverage native sharding and disk ANN architectures.
That is a phenomenal framework to operate by. And it leads perfectly into our next discussion. Next up in the modern data store series for deep dive number six, we are pivoting to running data stores in production. We will tackle high availability, split brain scenarios, disaster recovery, and the grim operational reality of keeping these massive distributed systems alive during a 3 a.m. pay troop alert.
I am really looking forward to that discussion. And to leave you with a final thought to mull over regarding everything we unpacked today.
What have you got?
Well, we spend so much time in the tech industry talking about the magic of AI models. We obsess over the billions of parameters and the latest transformer networks and the nuances of prompt engineering. But when you look closely at architectures like disk ANN, or the mechanics of one-bit binary quantization using Hamming distance, you realize a profound truth about this era.
Yeah.
The actual physical bottleneck of the AI revolution wasn't just the theoretical math of neural networks. It was hardware physics. The unsung heroes of modern AI search aren't just the researchers writing theoretical papers. They are the SIMD CPU instructions executing thousands of floating point operations per cycle. And the NVMe SSDs moving gigabytes of quantized graph data with sub-millisecond latency.
It all comes back to the metal. And hopefully, with this deep dive, you won't be the developer staring at a zero row result screen or a 220 gigabyte cloud bill, wondering why the engine is stalling. You now know exactly how the mechanics of vector databases actually flow beneath the abstraction. Thanks for diving deep with us.