The Internal Mechanics of NoSQL Databases
Transcript
Made automatically from the audio, so it may contain mistakes. Click a line to play from there.
So it is 2 a.m. on Black Friday. You've provisioned your dynamo DB table for I don't know, massive 50,000 write capacity units per second.
Right. You basically paid for the absolute Ferrari of database infrastructure.
Exactly. But suddenly PagerDuty is just screaming at you. You look at your dashboards, and your database is throttling at just two percent capacity.
Which is terrifying.
Yeah. And the rest of the cluster is sitting there completely idle while your whole checkout application just burns to the ground. Why? Well, because every single incoming write was hitting one viral tenant ID partition.
Yeah, instantly blowing past Dynamo DB's hard physical limit of 1,000 right capacity units per partition.
It's like the ultimate distributed systems nightmare. You paid for global scale, but you got bottlenecked by a single piece of hardware.
And you know, here is outage nightmare number two. Think about a Cassandra Q table.
Okay.
You're executing a basic select query, like the simplest thing in the world. But the query times out. Then the entire application heap crashes.
Wait, just from a select?
Just a select. Because the storage engine had to silently scan past a hundred thousand delete tombstones. Oh wow. Yeah, literally just invisible markers saying data used to be here, just to find a single piece of live data. And the garbage collection tail spike alone took down the node.
That is brutal. It's the hidden operational tax of immutable storage engines. I mean, if you don't understand how the engine physically interacts with a disk, it will absolutely destroy your uptime.
It really will.
Well, welcome back to the deep dive, everyone. We are super glad you are here with us. Just to give you a quick recap, in our second deep dive, we ventured out into the world of analytical lake houses.
Right.
But today, we're shifting our focus right back to the operational data tier. You know, the beating heart of your live applications.
And let's set some ground rules right up front because you already know your weight around the block.
Yeah. We are assuming you're comfortable with OLTP B trees. You know the basics of log structured merge trees or LSMs. And well, you're well acquainted with Postgres, JSON B and GIN indexes from our very first deep dive.
Exactly. So we are absolutely not going to waste your time re-explaining basic SQL joins or like third normal form today.
No. Our mission for this deep dive is actually to dismantle two very dangerous industry myths. The myth of the schemaless database and the marketing claim that NoSQL is just inherently faster than SQL.
And to do that, we are going to unpack the internal mechanical realities of the four major NoSQL families. Key value, document, wide column, and graph.
Right. We're going to explore exactly why NoSQL deliberately drops relational joints and forces you into this highly rigid paradigm called query first data modeling.
So let's start right at the bare metal foundation, key value stores.
Yeah. If we want to talk about key value, I mean we have to start with the undisputed king of latency, right?
Oh, absolutely, Redis.
Usually for like transient state or leaderboards. But mechanically, how does it consistently beat a traditional database on speed?
Well, the secret is actually its simplicity. Redis operates entirely in RAM. Okay. So it bypasses disk I.O. completely for its primary read and write paths. But more importantly, a Redis instance is fundamentally single threaded for command execution.
See, I've always found that counterintuitive. I mean, we put 64 core processors in our servers, and Redis only uses one of them for processing commands. Like how is that fast?
Because it completely eliminates context switching and lock contention. Oh, right. Yeah. In a multi-threaded database, threads are constantly fighting for mutex locks over shared memory spaces. And that coordination, well, it takes CPU cycles.
It's expensive.
Very. Redis avoids it entirely. It uses an event loop utilizing the system calls like ePoll or KQ to multiplex thousands of client connections on a single thread. Wow. So when it executes a command, it just reads or writes directly to a memory pointer. It takes fractions of a microsecond.
Okay, so I was looking at the distributed architecture for Redis cluster, and the documentation mentions it uses exactly 16,384 hash slots to distribute data across multiple nodes.
Yeah, that specific number.
Right. Why that number? And how does it actually route the data?
So that number is a very deliberate design choice regarding cluster metadata. Redis cluster doesn't use standard consistent hashing. It uses an algorithmic partition map.
Okay. What does that look like in practice?
Well, when your application wants to write a key, say user 1000, the Redis client runs that keystring through a hashing algorithm called CRC16.
Right.
And it takes the output of that hash and performs a modulo operation by 16,384.
So it's basically doing division and taking the remainder, which means the result is always going to be a number between zero and sixteen thousand three eighty-three.
Exactly that.
Okay.
And that resulting integer is the hash slot. Now every master node in a Redis cluster is responsible for a contiguous subset of those slots. Okay. So node A might own slots zero to fifty five hundred. Node B owns the next chunk, and so on.
But how does my web application know which node has which chunk? Like if I have a cluster of 10 nodes, does my application just guess and send the request to a random node?
It could, but that would be super inefficient. A Redis cluster node doesn't proxy requests for you.
Oh does it?
No. If your application asks node A for a key that actually hashes to a slot owned by node B, node A immediately replies with a moved redirection error.
So it's basically saying I don't own this, go ask them.
Right. It says I don't own the slot, but node B does. Here is node B's IP address, go ask them.
It feels like a lot of network back and forth though. Doesn't that kill the sub millisecond latency you just talked about?
It absolutely would if it happened on every single query. But smart Redis clients maintain a local map of the hash slots in their own memory.
I see.
Yeah when the cluster first spins up the client fetches the slot map. From then on the client does the CRC16 math locally, realizes the key belongs to node B, and opens a TCP connection directly to node B.
Okay, so the routing intelligence is heavily pushed to the client side. The moved error is really just a safety mechanism for when the cluster topology changes like if you're doing a scale up or there's a failover.
Precisely.
Which brings up a critical point right failovers and durability. Because while RAM is fast, RAM is completely volatile. If a server loses power, that data's just gone.
Poof, gone.
I know Redis offers two main persistence options RDB snapshots and AOF logs. Let's break down the snapshotting first, because I read that it uses a system call called fork. What is actually happening at the operating system level when Redis takes a snapshot?
Oh, this is a beautiful piece of operating system engineering. Think about it. If Redis stopped processing commands to write gigabytes of data to disk, your application would stall completely.
Right, would just hang.
Exactly. To avoid this, Redis issues a fork system call to the Linux kernel. And the kernel creates a child process that is an almost exact duplicate of the Redis parent process. Okay. The child process then takes over the task of serializing the dataset to a temporary RDB file on the disk.
But wait, if I have 30 gigabytes of data in Redis and the OS creates a duplicate process, doesn't that require 60 gigabytes of RAM? That seems like an instant out-of-memory crash is waiting to happen.
That is the intuitive assumption, yeah.
But it's not what happens thanks to modern memory management. And specifically, copy on write semantics.
Copy on write.
Yeah.
When the kernel forks the process, it doesn't actually copy the 30 gigabytes of physical RAM. It just duplicates the virtual memory page tables. Both the parent and the child process point to the exact same physical memory pages.
So they're sharing the memory. But what if the parent process, which is, you know, still serving live traffic, needs to update a key while the child is busy writing the snapshot?
That is where the copy on write magic kicks in. The kernel marks all those shared memory pages as read-only. Okay. So if the parent process tries to modify a piece of data, it triggers a hardware exception, a page fault.
Oh, interesting.
The CPU traps into the kernel, the kernel intercepts the modification, and at that exact microsecond, it makes a physical copy of just that specific four kilobyte memory page.
I get it. So the parent modifies the new copy, and the child process just continues reading from the untouched original page.
Yes.
That is mechanically brilliant. It only consumes extra RAM for the specific data that actually changes during the exact window of the snapshot.
Yes, exactly. But you know, while RDB is incredibly efficient, snapshotting isn't perfect durability.
Right, because if your server crashes 10 minutes after the last snapshot, you lose 10 minutes of data.
Which is fundamentally unacceptable for a lot of operational workloads. And that's where AOF, the append only file, comes into play.
I imagine AOF is more like a traditional database transaction log.
Very similar. AOF logs every single write operation. So every set, every increment command received by the server into a sequential file.
Okay, so it just keeps a running list.
Exactly. If the server crashes upon restart, it literally reads the AOF file from top to bottom, replaying every single command to reconstruct the exact state of the data set in memory.
But appending to a file on disk on every single write command, doesn't that completely defeat the purpose of an in-memory database?
That's the tension. You have to tune the f sync policy. In Linux, when you write to a file, it usually goes into an OS buffer in memory first, not directly to the physical SSD. Right. To guarantee the data is on the SSD, you have to call Fsync. Redis lets you configure this. You can F sync on every single command, which is perfectly durable, but extremely slow.
Probably too slow for most use cases.
Yeah. So the most common compromise is you configure Redis to F sync the AOF file every one second in a background thread.
So the absolute worst case scenario is losing just one second of data during a hard power failure.
Correct. You get the blistering speed of RAM for reads, asynchronous writes to the OS buffer, and a maximum of one second of vulnerability.
That makes a lot of sense. Now, for anyone wanting to dive deeper into how Redis manages memory when it's completely full, specifically the approximated LRU caching algorithm and eviction policies. We actually covered that mechanic back in Web Architecture Season 2, Episode 7.
Yeah, that was a good one.
We talked about how Redis samples random keys to find eviction candidates instead of maintaining a strict memory heavy linked list. So I highly recommend cross-referencing that deep dive. But today we are talking about persistence and scale.
Yes, which leads us directly to the descendants of Amazon's original dynamo paper. Specifically, DynamoDB.
Okay, so Dynamo DB.
Dynamo DB is where we take this key value paradigm and push it to massive multi-region disk-based scale. Unlike Redis, where you have to worry about hash slots and fork memory overhead, DynamoDB abstracts the underlying nodes entirely away from you.
Because it's a fully managed service.
Exactly. You provision a table, you define your throughput, and AWS handles all the physical hardware under the hood.
But they don't abstract the physics. I mean, you still have to understand the mechanical reality of how they partition your data across their hardware, especially because of that 2 a.m. Black Friday failure mode we opened the show with.
The 50,000 WCU Ferrari that hit a brick wall.
Yeah, exactly. Let's break down how DynamoDB actually stores data, starting with partition keys and sort keys.
So in Dynamo DB, every item you write must have a primary key. This can be a simple partition key or a composite key made up of a partition key and a sort key.
Right.
The partition key is fed into an internal hashing algorithm. Just like our Redis CRC16 example, that hash output determines which physical underlying server, like which physical storage node your data is stored on.
And what does the sort key do?
The sort key determines the clustered index order of the items within that specific physical partition. It physically groups related data together on the SSD sorted by the sort key value, allowing for very fast range queries.
Okay, I was reading the AWS documentation, and there is a very specific hard mathematical limit that causes so much pain.
Oh, I know the one.
Yeah. Every single physical partition in DynamoDB is hardcapped at delivering a maximum of 3,000 read capacity units or RCUs and 1,000 write capacity units or WCUs per second. How do those units actually translate to real-world data payloads?
Okay, so one read capacity unit represents one strongly consistent read per second for an item up to four kilobytes in size. Okay. And one write capacity unit is one write per second for an item up to one kilobyte in size.
Let's do the math on a real world scenario then. Imagine an e-commerce company writing user clickstream profiles. Every single payload they write is 20 kilobytes.
Okay.
According to that math, writing a single 20 kilobyte profile consumes 20 write capacity units.
That math is exactly right. And this is where the trap closes. Because a single physical partition is hardcapped at 1,000 WCUs, you can only write 50 of those 20 kilobyte profiles per second to any single partition key.
Wait, 50. That's incredibly small.
It is.
So if my partition key is something like tenant ID, and I have a tenant that suddenly goes viral, maybe they drop a highly anticipated sneaker on Black Friday, thousands of users are generating clickstream data for that one tenant.
Yep.
All of those rights hash to the exact same physical partition.
And you hit that 1,000 WCU physical ceiling on that one specific SSD. The storage node literally cannot process the IO any faster. AWS throttles your requests and your application starts throwing HTTP 400 provisioned throughput exceeded errors.
But wait. If I went into the AWS console and provisioned 50,000 WCUs for the table globally, why am I getting throttled? I paid for 50,000.
You paid for 50,000 across the entire table, which AWS distributed across 50 different physical partitions under the hood.
I see.
But because all your traffic is hitting one specific tenant ID, one partition is completely overwhelmed, and the other 49 partitions are sitting there completely empty, basically twiddling their thumbs. You are throttled not by your wallet, but by the physics of the hash distribution.
That is a brutal reality check. So if I have a known hot key, like a massive tenant or a viral product, how do I mechanically engineer around that bottleneck?
You have to implement a technique called write sharding. Your application code has to deliberately interfere with the partition key.
Interfere how?
Instead of just writing tenant123, your code appends a random suffix, like a number from one to one hundred to the end of the partition key. So the rights become tenant123 slash one, tenant123 slash forty seven, tenant123 slash ninety nine.
And because the string is different, the hashing algorithm spits out a different physical partition. You force the data to spread evenly across all 50 underlying servers.
Exactly. You eliminate the hot partition, but you introduce a new massive problem on the read path.
Right, because when I want to read all the data for that tenant, I can't just query tenant one two three anymore. The data doesn't live there.
Exactly.
My application has to issue a scatter gather query, has to fire off a hundred parallel queries for all 100 possible suffixes, wait for the network responses from all of them, and then stitch the results back together in the application memory.
Yes, and that is a fundamental paradigm shift. It forces you into what the industry calls query first data modeling.
Okay, let's get into that.
Because no SQL databases like DynamoDB deliberately drop the concept of third normal form relational joins. You cannot just normalize your data into cleanly separated entity tables and figure out your queries later using SQL.
Right.
You have to design your schema backward from the exact access patterns your application needs on day one.
So mechanically, what does that look like? Because in Postgres, if I have users and orders, I have a users table and an orders table. I join them on user ID when I need a dashboard. Simple, elegant, flexible.
Very flexible.
But in DynamoDB's best practices, specifically the single table design philosophy, I am putting users and orders into the exact same table. How does the database even distinguish them?
You use a concept called overloaded keys. You stop naming your columns user ID or order ID. Your partition key is just generically named PK, and your sort key is generically named SK.
Okay.
For a user item, the PK might be the string user hashtag 123, and the SK might be the string hashtag metadata.
Okay, and for their orders.
For an order belonging to that user, the PK is still user hashtag 123, but the SK is order hashtag 456. Right. Because they share the exact same partition keystring, the dynamo DB hash algorithm guarantees they are stored physically adjacent on the exact same SSD.
Oh wow.
When your application queries for PK equals user hashtag 123, the storage node reads the user profile and all of their orders in a single blazing fast disk seek.
You've effectively pre-computed the join.
You've materialized the relationship on disk. That explains the speed. But what if the business requirements change?
They always do. What if I need to query those same orders by date across all users instead of by user ID?
Then you create a global secondary index or GSI. You tell DynamoDB to asynchronously replicate the data into a brand new physical partition layout behind the scenes using the order date as the new partition key for the GSI.
I see the efficiency, but this query first design feels incredibly brittle. It's like packing a highly specific suitcase for a beach trip, right?
Yeah, that's a good way to put it.
You pack your swimsuit, your towel, your sunscreen perfectly into these little optimized compartments. It's incredibly efficient, right up until the product manager asks to go skiing six months later.
Which happens constantly.
Right. If they ask for a new access pattern, like filtering orders by a newly introduced payment status column, and I didn't originally model the GSI for it. Well, I can't just write a new SQL query with a where clause.
No, you can't. That is the hidden operational tax of key value noSQL. If you need a completely new access pattern, you often have to run a background map reduced job.
Wait, let's unpack mapReduce briefly for the listener. Mechanically, what is that job doing?
So a mapReduce job in this context means spinning up a distributed compute cluster like AWS EMR or Apache Spark to physically scan the entire Dynamo DB table row by row.
Ouch.
Yeah. It reads millions of items, computes the new attribute, writes the new attribute back to the live table, and then you spin up a new GSI on that new attribute. It is an enormous, expensive operational headache just to add a filter column.
So the trade-off is stark. You are trading the schema flexibility of a relational database for guaranteed single-digit millisecond latency at any conceivable scale.
Precisely. And because that rigidity is so painful for rapidly evolving agile teams, we see engineering departments constantly fleeing key value stores. They want the scale, but they don't want the pain of single table design.
So they get lured in by the dangerous promise of a schemaless database.
Yes.
Which brings us to the second major NoSQL family, the document store, championed heavily by MongoDB. Let's dismantle the schemaless myth.
Let's do it.
If key value is too rigid because the value is just an opaque blob to the database, document stores try to solve that. They said, what if the value is a richly structured JSON document? Or technically BSON, binary JSON. Mechanically, why Bson instead of just plain text JSON?
Because parsing text JSON is incredibly CPU intensive for a database. How so? Well, to find a nested field in a text.json blob, the CPU has to read the string byte by byte, looking for quotation marks and colons. Exactly. BSON is a binary serialization format. It stores length prefixes for elements.
Oh.
This allows the database storage engine to physically jump to a specific byte offset in the document, instantly extracting a nested array without parsing the entire file. It is vastly more efficient for indexing and querying nested structures.
Under the hood, modern MongoDB uses the Wired Tiger storage engine. I found it fascinating that Wiretiger actually uses B trees for storage, very similar to Postgres.
It does.
But it handles concurrency, specifically how multiple users write to the database at the same time. Very differently than older databases.
That's a crucial distinction. Historically, early versions of MongoDB had infamous database level or collection level locking.
Oh yeah, I remember those days.
If one user was writing to a collection, the entire collection was locked. No one else could write. It was an enormous bottleneck. But Wired Tiger introduced document-level concurrency control using optimistic execution and intent locks.
Okay. Let's ELI5 and intent lock.
Imagine a hotel.
What is that mechanically?
In the old system, if you wanted to change the sheets in room 101, you lock the front door of the entire hotel. No one else could enter.
Very inefficient.
Right. An intent lock is like putting a sign on the hotel's front door saying, I intend to modify a room inside, and putting another sign on the hallway saying I intend to modify a room in this hall. But you only actually lock the door of room 101.
Okay, so this allows another main to simultaneously change the sheets in room 102.
Exactly.
And the optimistic part?
Optimistic concurrency means the storage engine assumes conflicts are rare. It doesn't lock room 101 immediately. It goes in, changes the sheets, and right at the moment of saving the changes, it checks if anyone else modified room 101 while it was working.
And if they did.
If they did, it throws out its work and silently retries the operation from scratch. It trades locking overhead for a retry overhead, which is highly efficient for most workloads.
The major pitch for MongoDB is the flexibility of document modeling. In relational databases, we normalize. In MongoDB, the fundamental architectural decision is embedding versus referencing. Let's explore that.
Okay, so if we have a blog post and its comments, in Postgres, that requires two tables and a foreign key join.
Right.
In MongoDB, you can embed the comments directly inside the blog post document as a nested array. When your application fetches the blog post, it retrieves the comments for free in one single disk read.
It is a localized pre-computed join, much like the Dynamo DB single table design, but nested instead of flat.
Yes.
But there is a massive production failure mode here that teams hit all the time. The strict 16 megabyte document limit. Why exactly 16 megabytes?
It stems from the historical overhead of BSON serialization and network transport. When a document is loaded into memory, the database has to allocate a contiguous block of RAM.
Okay.
If documents could be unlimited in size, you would cause massive memory fragmentation and network congestion, just transferring a single document over a TCP socket.
So if I embed an array, say an IoT sensor logging and temperature reading every single second into a daily document, and that array grows unbounded, I will eventually hit that 16 megabyte ceiling. What happens then?
The database will literally reject the write. Your application crashes.
Yikes.
But the damage starts long before you hit 16 megabytes. Because of how B trees allocate space on disk, if you constantly append to an array, the document outgrows its allocated physical space on the SSD.
Right.
Wired Tiger has to pick up the entire multimegabyte document, find a larger contiguous block of free space on the disk, and rewrite the whole thing. Exactly. Moving a 10 megabyte document just to append a 10-byte temperature reading is catastrophic for disk I.O.
So if you can't embed forever, you are forced to reference. You put the comments in a separate MongoDB collection and store the post ID on the comment. But wait, MongoDB doesn't natively enforce referential integrity across collections.
No, it doesn't.
You've basically just recreated a relational structure, but without the safety in it of foreign keys.
Exactly. And this leads to the sharding dilemma. When you outgrow a single server, MongoDB allows you to shard the data across multiple servers. You pick a shard key and the system chunks the data.
Okay.
You have two main strategies here: ranged sharding and hashed sharding.
Let's define the mechanics. Ranged sharding divides data based on continuous values. User IDs one to a thousand go to shard A, 1001 to 2000 go to shard B. Hashed sharding runs the key through an algorithm, guaranteeing a random, perfectly even distribution across all servers, exactly like DynamoDB.
Yes. And the trade-off between the two is the source of many, many outages. Hashed sharding gives you perfectly even write distribution, completely eliminating the hot partition problem.
Right.
But if you need to run a range query, say, give me all users who signed up this week, hashed sharding has scattered those users completely randomly across every server in the cluster.
Which triggers the classic MongoDB scatter-gather broadcast nightmare.
Yes, it does. If your query does not include the specific hashed shard key, the query router, a process called Mongo S, has absolutely no idea which physical shard holds the requested data. It is flying blind. So it performs a cluster wide broadcast operation. It opens parallel network connections to every single shard in the cluster. It sends the query to all of them, waits for dozens of servers to execute their own B-tree searches. Sounds waits for all the network responses, pulls all that data into the MongoS router's local memory, sorts it, and then returns it to the client.
If you have a 50 shard cluster, a simple query from one user just consumes CPU, RAM, and network I.O. on 50 different servers simultaneously. The memory pressure on the query router alone could crash it.
Easily.
This completely obliterates the faster than SQL myth.
It does. And teams try to solve this by moving to multi-document transactions. They try to update documents on different shards transactionally to keep things in sync. The overhead is immense.
Simple.
In a distributed MongoDB cluster, a multi-document transaction requires a two-phase commit protocol. The coordinator has to network with the config servers, ping the primary nodes of multiple replica sets, ensure they all secure locks, wait for network acknowledgements, write to multiple journals, and then send a final commit message across the network. Very severe. It should be used as a rare exception, not as a crutch for avoiding proper document modeling.
You know what Martin Fowler warned the industry about implicit schema debt years ago? I want to challenge the premise of document stores for a minute.
Sure.
If I'm a startup and I'm just tossing flexible JSON into a database because I don't know my schema yet, and my Postgres JSON B column can already handle that natively with GIN indexing. Is a document store actually giving me a performance edge, or is it just giving me application-sized scatter-gather bugs?
You're right to point out the debt, but we have to remember why startups make this trade-off. In week one of a startup, schema migrations in SQL are a massive friction point to developer velocity. Document stores let you iterate incredibly fast, and you just dump the object into the database. Right. The problem isn't the database technology itself. The problem is that teams never refactor that debt when the product matures. Schema-less is a marketing term. The reality is that the schema always exists.
Right. If it's not strictly enforced by a Postgres table definition, it is enforced by your application code. Your Python or node code has to handle the if-else logic for parsing a user document written in 2021 versus a completely different document structure written in 2026.
That is a massive hidden operational tax. Every developer has to know the historical quirks of the data. You should only move from Postgres to a document store if your data is inherently hierarchical, naturally self-contained, meaning it doesn't need joins. Right. And you strictly require transparent horizontal sharding because you've outgrown the biggest SSD you can put in a single Postgres server.
Okay. So key value is too rigid when product requirements change, and document stores can trap you in unbounded arrays, write amplification, and broadcast queries. But what if your problem isn't flexible JSON at all?
What if it's just pure volume?
Exactly. What if your problem is just an absolute avalanche of flat data? Say you were building an IoT ingest pipeline for millions of smart meters or a global chat application, and you need to sustain 200,000 writes per second constantly, 24-7. Both MongoDB's B trees and Postgres will completely choke on that volume of concurrent disk I.O.
They absolutely will.
For this, we turn to the wide column right juggernauts. We are talking about the direct descendants of Google's big table and Amazon's Dynamo paper. Cassandra and its high performance C cousin, ScyllaDB.
To understand how these systems ingest 200,000 writes per second without melting the disks, we have to look at the distributed architecture. They completely abandon the primary secondary replication model used by Postgres or Mongo.
Okay. What do they use instead?
They use a masterless, consistent hashing token ring.
We talked about hashing in Redis and Dynamo DB, but the token ring concept is structurally different. How does it work?
Imagine the entire hash space from zero to a massive number, like two to the 127th power, visualized as a continuous circle, a ring. Every physical server node in the cluster is assigned a position on this ring.
Okay.
When a piece of data is hashed, it falls somewhere on the ring, and it is stored by the node located immediately clockwise on the ring.
I read about the early implementations of consistent hashing, and there was a fatal flaw regarding cluster expansion. If I have a perfectly balanced ring of three nodes, and I add a fourth node to increase capacity, that new node only takes load from its immediate clockwise neighbor.
Exactly.
It leaves the other two nodes completely untouched. The cluster instantly becomes unbalanced. How do modern wide column stores fix that?
The sources explicitly detail how Cassandra solves this with virtual nodes or V nodes. Instead of a physical server owning one massive contiguous slice of the ring, it is assigned hundreds of tiny randomly distributed tokens, V nodes, scattered completely randomly across the entire ring.
Oh, clever.
When you add a new physical node to the cluster, it takes a tiny sliver of data from every other node in the cluster simultaneously, keeping the load and disk utilization perfectly balanced.
And because the architecture is masterless, there is no single point of failure. Any node in the cluster can accept a read or write requests from the client and act as the coordinator.
Right.
It hashes your partition key, figures out which nodes on the ring own the replicas, and routes the network requests to them. This routing flexibility introduces the concept of tunable consistency using quorums.
This is where we get into the famous R plus W is greater than N equation, which dictates how the database navigates the cap theorem. N is your replication factor.
Okay, so let's say you configure the cluster to store three copies of your data across three different nodes, so n equals three.
Exactly.
And W and R.
W is your write consistency level. How many nodes must acknowledge a write before the coordinator tells the client success? R is your read consistency level. How many nodes the coordinator must query to resolve the latest version of the data?
Let's give a concrete example. If I am building a financial ledger and I want strict consistency, ensuring I always read the latest write without fail.
Then the number of nodes you write to, W, plus the number of nodes you read from, R, must be mathematically greater than your total replicas n.
So if n equals three, you could require writes to be acknowledged by two nodes, W equals two, and reads to be acknowledged by two nodes, R equals two. Two plus two is four, which is greater than three.
Exactly. And because of the overlap, the math guarantees that at least one of the nodes I read from will absolutely contain the latest write.
But let's say you are building an IoT sensor ingest pipeline. You don't care about strict consistency. You care about maximum availability and never dropping a write, even if half the network is offline.
In that case, you can set W equals one and R equals one. The system becomes eventually consistent. You might occasionally read stale data, but your database will survive massive network partitions. This decoupling of data redundancy from availability is an incredibly powerful architectural lever.
Okay, so the network routing and consistency layers are masterless and highly available, but that still doesn't explain the disk I/O. Mechanically, how does the actual storage engine on a single node physically absorb 200,000 writes per second? Standard B trees have to do random disk seeks to update indexes, and that takes milliseconds.
They achieve this by completely abandoning B trees in favor of log structured merge trees or LSMs.
Okay, the LSMs again.
Yes. The write path is optimized purely for sequential I.O., which is what mechanical hard drives and modern SSDs are best at. When a write payload comes into a Cassandra or SkillaDB node, two things happen immediately.
First step.
First, the payload is appended to a commit log on disk. This is strictly sequential I.O. The disk head never moves. It just writes in a straight line. This guarantees durability in case of a power failure.
Okay, in the second step?
Second, the exact same data is inserted into an in-memory, self-balancing tree structure called a mem table.
Okay, and because it's just writing to RAM and appending to a sequential log, there are no random disk seeks, it acknowledges the right back to the coordinator in fractions of a millisecond. That is how it ingests a fire hose of data.
Exactly. But RAM is finite. When the mem table hits a memory threshold, say 128 megabytes, it flushes that entire sorted structure to disk as a brand new immutable file called an SS table, a sorted string table.
Okay.
Once written to disk, an SS table is absolutely never modified.
But if the files are never modified, how do I read a specific row? Over a month, an IoT sensor might have written data that is now scattered across hundreds of different SS table files. Opening and searching hundreds of files for a single read query would take seconds.
A bloom filter is a highly space efficient, probabilistic data structure kept entirely in RAM. Right. When a read query comes in for a specific key, the engine checks the Bloom filter for every SS table. The mathematics of a Bloom filter guarantee that it can tell you with 100% certainty if a key is not in an SS table.
Oh wow. So it instantly eliminates 99% of the files from the search.
Right. If the Bloom filter says the key might be there, only then does the engine perform a disk seek to check the file.
This sounds incredibly fast for writes, but continually accumulating thousands of immutable SS tables will eventually degrade read performance even with Bloom filters.
Which is why the database constantly runs compaction processes in the background. Compaction reads multiple small SS tables, merges them together, sorts them, and writes out a single larger SS table while deleting the old ones.
And choosing the correct compaction strategy is critical to survival. The sources highlight three main types. Let's start with size tiered compaction strategy or STCS.
Okay, so STCS merges SS tables of similar sizes. For small files, merge into one medium file. For medium files, merge into one large file.
Makes sense.
It's the default, and it's great for heavy write workloads, but it has a massive, often fatal drawback. Temporary space amplification.
How bad is it?
In the worst case scenario, when merging the largest tier of files, you need 50% of your disk space to be completely empty just to have enough room to write the new merged file before deleting the old ones.
Wow. That's a staggering operational cost. You are paying AWS for terabytes of SST disk space that you're literally not allowed to use just to keep the database from locking up during compaction.
It is. Then there is leveled compaction strategy, LCS. It organizes small fixed-size SS tables into tiered levels. It guarantees much better read performance and uses far less temporary disk space, only about 10% overhead. But but it causes massive write amplification. A single piece of data might be rewritten to disk 10 times as it moves through the levels. This will burn out SSD lifespans faster and consume massive CPU.
And for time series data, like our IoT sensor example, where data naturally expires.
You use time window compaction strategy, TWCS. It groups SS tables strictly by time windows, say one day. It only compacts data within the same day.
Oh, that's smart.
Once a day closes, those files are sealed and never compacted again. It is incredibly efficient for time series workloads because you can just drop the entire file when the data reaches its time to live.
Speaking of data expiring and deleting data, let's revisit the cold open. The tombstone failure that crashed the application heap. If SS tables are completely immutable, how do you actually execute a delete command? You can't modify the file.
You don't delete it. You write a brand new record called a tombstone that acts as a deletion marker. The tombstone has a newer timestamp than the original data.
Okay.
When a read query scans the disk and merges the data from different SS tables in memory, the tombstone technically eclipses the old data so the database knows to filter it out and not return it to the client.
This is where it gets incredibly dangerous. Deleting data actually takes up more disk space because you are writing a new tombstone record. And it slows down reads. It's like trying to clean your messy room by putting in ignore this piece of trash, sticky note, on every single item until the room is completely full of sticky notes.
That is the perfect analogy for the mechanism. Now imagine you run a select query that scans a partition looking for live data. If you have deleted a lot of data, the database has to physically read 100,000 of those sticky notes off the disk. Load all 100,000 of them into the application's RAM to process the timestamps, just to find one valid live item at the end of the scan.
And because Cassandra is written in Java, holding 100,000 tombstones in memory instantly bloats the Java heap.
Exactly. The Java virtual machine panics and triggers a stop the world garbage collection pause to try and clear the memory.
Stop the world.
Yeah. The database node literally freezes for several seconds. It stops responding to network pings. The other nodes in the cluster think it's dead. Gossip protocols trigger, read repairs initiate, and your tail latencies go from milliseconds to timeouts.
And the only way to actually clear the tombstones off the disk is to wait for a major compaction to physically merge the files and discard the eclipse data, which is governed by a setting called GC Grace seconds, usually said to 10 days to allow deleted markers to replicate across down nodes.
Yes. And this specific JVM garbage collection nightmare is exactly where CelaDB enters the picture to fundamentally rewrite the architectural bottlenecks of Cassandra.
I saw SkylyDB uses a thread per core architecture and something called DPDK to fix this. But mechanically, how does pinning a thread to a CPU core actually eliminate the latency spikes?
Well, SkylaDB is a complete rewrite of Cassandra in C using an asynchronous framework called C Star. In a traditional database, you have a pool of hundreds of threads handling requests. Right. These threads constantly context switch on the CPU, fighting over global memory and using mutex locks to ensure they don't corrupt each other's data.
And locking takes time.
Right. SkylarDB eliminates this entirely using a shared nothing architecture. It pins exactly one thread to one CPU core. It preallocates a specific isolated chunk of RAM entirely to that core.
So core zero only touches core zero's memory.
Core zero only touches core zero's memory, core one only touches core One's memory. Because no other thread ever touches that memory, you completely eliminate the need for mutex locks or semaphores.
And what about the network layer? How does DPDK factor into this?
DPDK stands for Data Plane Development Kit. It is a technology for kernel bypass. Normally, when a network packet arrives at the network interface card, it triggers a hardware interrupt.
Okay.
The CPU drops what it's doing, switches to kernel mode, copies the packet from kernel memory space to user memory space, and hands it to the database. That context switch is expensive.
So DPDK skips the operating system kernel entirely.
Completely. SCLADB allocates a subset of the network card's cues directly to specific CPU cores. Core zero constantly pulls its own queue directly from user space. Zero hardware interrupts, zero kernel context switches, zero memory copying.
That is wild.
A query comes into the NIC, is pulled directly by the specific CPU core that owns that piece of the hash token ring, processes the LSM tree without any locks, and sends the response back out.
That is mechanical sympathy at its finest. Just pure unadulterated exploitation of modern hardware. It allows Skeletb to handle millions of operations per second with microsecond tail latencies, completely bypassing the JVMGC pauses.
It's impressive.
But whether you use Cassandra or Skeletb, you still have to deal with the modeling rules. Just like Dynamo DB, you cannot do relational joins. The golden rule of wide column stores is one table per query.
Yes, the query first paradigm strikes again. If you have a user profile and you want to query it by user ID, that's one table. If the product manager says we also need a login screen to query by email address, you do not create a secondary index. You create a second, entirely separate table where email is the partition key and your application writes the data twice.
Because disk space is cheap, but CPU joins and network scatter gather queries are catastrophically expensive at this scale. But what if I try to cheat the system? What if I use Cassandra's lightweight transactions or LWTs to ensure a user's email is unique across the cluster before I write it?
You will pay a heavy price. Cassandra uses a variant of the Paxos consensus protocol for LWTs. To guarantee a linearizable compare and set operation, you know, checking if the email exists before writing, the coordinator node has to execute four separate network round trips to the replica nodes to achieve consensus.
So a single write command turns into four network hops.
Exactly. If you misuse LWTs for regular high volume writes, your 200,000 writes per second will instantly bottleneck on network consensus overhead. LWTs are for rare critical state changes, not for bypassing proper data modeling.
Okay, so wide column stores are absolute juggernauts for massive flat append-heavy workloads. But what if your data isn't flat? What if it is deeply intensely interconnected?
Like a graph.
Yeah. Let's say you were building a real-time recommendation engine or investigating a financial fraud ring. You need to follow the money. Person A transferred money to account B, which is owned by person C, who shares an IP address with person D. That is a four-hop traversal. Y column stores are completely useless for this.
And in a traditional SQL database, while you can technically do this using recursive common table expressions or CTEs, the performance will exponentially degrade at every single hop.
Why? I mean, if I have indexes on my foreign keys, shouldn't joins be fast?
A single join is fast.
Okay.
In a relational database, traversing a relationship requires a B tree index lookup. A B tree search is an O log n operation. The database has to traverse the tree structure to find the foreign key.
So if table B has a billion rows, O log n means it takes about 30 operations to find the record. That takes a fraction of a millisecond.
Right. But if you do a six-hop traversal, the database is executing that O log n index lookup over and over and over again. And worse, the intermediate result sets multiply exponentially.
Oh, I see.
Hop one finds ten accounts, hop two finds a hundred accounts, hop three finds ten thousand accounts. By hop six, the database is trying to execute millions of O log N B tree searches simultaneously in memory. A query that takes two milliseconds for one hop will literally take hours for six hops.
Which requires the final family in our deep dive, the graph traversers. Specifically property graphs like Neo4j. How do they mechanically solve the multi-hop problem?
Enter the magic of index-free adjacency. In Neo4J, data is stored as nodes, like a person or an account, and edges, the relationships, like transferred to.
Okay.
Index-free adjacency means that every single node contains direct physical memory pointers to all of its connected edges.
Wait, mechanically, what is a memory pointer look like in this context?
Under the hood, a node is essentially a struct in memory. It contains a linked list of memory addresses. When you traverse from node A to node B, the database engine does not execute a search. It does not look at an index.
It just jumps.
It simply reads the memory address of node B directly from node A's edge list. Say 0x7FFF. And the CPU dereferences that pointer, jumping instantly to that location in RAM.
It's an O of one operation. Constant time complexity.
Exactly. It completely bypasses the need for global index lookups during the traversal. Whether your graph database has 10 nodes or 10 billion nodes, a six-hop traversal takes the exact same microscopic amount of time because following a physical memory pointer takes nanoseconds.
It's like reading a choose your own adventure book. In a relational database, when you reach the end of page 10, it tells you to go to the index at the back of the book, search alphabetically for the next chapter, find the page number, and turn there.
I love that analogy.
In a BRAF database with index-free adjacency, the bottom of page 10 literally hands you page 42 directly. It is a direct physical link.
That is a phenomenal way to visualize it. And because the architecture is designed this way, you use a specialized query language like Cypher, which uses ASCII art style syntax to intuitively pattern match these physical paths. You draw the path you want. Arrow transferred to arrow account.
But just like every other NoSQL family we've discussed today, there has to be a catastrophic production failure mode waiting for you if you model it wrong. What is the equivalent of the Dynamo DB hot partition or the MongoDB scatter-gatherer for a graph database?
In graph databases, it is the celebrity supernode fan-out explosion.
A super node. So let's explain that. Say we are modeling a social network like Twitter in Neo4J. Millions of normal users are nodes, and they have maybe 500 edges pointing to the people they follow. 500 memory pointers is nothing.
Nothing at all.
But Taylor Swift is a node, and she has a hundred million follower edges pointing to her. She is a super node.
Exactly. Now imagine you are running a real-time fraud traversal query. Find me all paths between user A and user B. The graph engine starts traversing the memory pointers. If your query accidentally steps onto the Taylor Swift node, the engine suddenly has to load and traverse 100 million physical memory pointers into RAM simultaneously.
It's a massive fan out bottleneck.
The traversal grinds to an absolute halt. The memory spikes, the CPU pegs at 100%, and the query times out entirely. How do you architect around that? Because you obviously can't just delete Taylor Swift from the data set.
No, you can't. You have to explicitly model your graph to handle dense nodes. Often you isolate them. If a node has too many connections, you flag it in the application logic as a supernode and explicitly exclude it from deep automated traversals.
Makes sense.
Or you architecturally refactor the graph structure itself. Instead of 100 million users pointing directly to the Taylor Swift node, you create intermediate nodes. They might point to a fan club node, which points to a demographic segment node, which finally points to her. You group the edges to reduce the fan out multiplier at any single hop. It forces you to realize that a graph database isn't just about drawing lines between every piece of data you have, it's about deliberately modeling the specific paths you intend to traverse safely.
Precisely. And that brings us to the synthesis of this entire conversation. We've mapped the internals, the architectures, and the fatal traps of all four NoSQL families. Key value, document, wide column, and graph.
Yes, we have.
Now we must synthesize this into concrete, actionable rules for you, the engineering leader listening right now to make architectural decisions for your next project.
The most important rule we can leave you with is defining your baseline. Unless you have a specific quantified, mathematically proven reason not to, you should start and stay on PostgreSQL. Use normalized relational tables for your core business entities, and use Postgres JSON B with GIN indexes for your flexible semi-structured data. Modern Postgres running on NVMe SSDs can handle tens of thousands of transactions per second and easily manage terabytes of data.
Do not adopt No SQL just to avoid writing schema migrations. You adopt these systems because you have hit the physical hardware limitations of B trees and relational joins. Here are the four explicit quantified triggers that justified taking on the immense complexity of No SQL.
Trigger number one. You need guaranteed sub-millisecond read and write latency for transient state, rate limiting, or session caching, and the data fits entirely in RAM. That is the trigger to introduce Redis.
Trigger number two.
You need masterless, multi-region active-active availability with guaranteed single-digit millisecond latency at any scale. And you are willing to lock into AWS infrastructure and absorb the operational pain of query-first single-table design, that is the trigger for DynamoDB.
Trigger number three.
You have a relentless append-heavy workload like IoT sensors, telemetry, or distributed tracing logs, sustaining well over 50,000 partitioned writes per second. You need absolute tuning control over compaction strategies and read-write consistency quorums. That is the trigger for Cassandra, or preferably ScyllaDB, if you want to avoid JVM tuning and leverage thread per core hardware optimization.
And finally, trigger number four.
Your core business logic fundamentally requires variable depth traversals of five hops or more, like real-time fraud detection, recommendation engines, or complex identity resolution where relational B tree joins exponentially degrade. That is the trigger to leverage the index-free adjacency of Neo4J. So if we pull back and look at the big picture of what we've discussed today, well, we've essentially shattered our nicely normalized relational data into these highly specialized, denormalized, pre-computed NoSQL query tables, all just to get read speed and massive write scale. But data changes. A user updates their email address. In a normalized Postgres database, you update exactly one row in one table, and it is instantly consistent everywhere.
That is the beauty of their normal form.
But in our new highly optimized NoSQL Utopia, that single email address might be denormalized into a Dynamo DB global secondary index. It might be embedded in a MongoDB document array. It might be written as a separate query first partition in ScyllaDB, and it might be a node in Neo4J. Have we just optimized our database storage layer so aggressively that we've created a massive distributed synchronization nightmare in our application layer?
You've hit the nail on the head. That is the fundamental architectural trade-off of the modern era. Keeping these highly specialized, denormalized stores in sync without introducing massive data anomalies, like a user having one email in the billing service and a different email on the recommendation service is the next great challenge. You absolutely cannot rely on distributed transactions or two-phase commits across different database families. The latency would destroy your system.
Which is exactly why the industry moved toward immutable event logs in stream processing. And that perfectly tees up our next deep dive on streaming and event-driven data. We are going to dive deep into technologies like Kafka and Red Panda and unpack how event sourcing acts as the central nervous system to keep all these denormalized NoSQL limbs perfectly in sync.
It's the only way to manage the massive complexity we've just discussed without breaking the application logic.
Thank you so much for joining us on this deep dive. Remember, true engineering isn't about skipping schemas because they are inconvenient for your agile sprint. It's about deliberately choosing your trade-offs based on the internal mechanical realities of the hardware. As you go back to your architecture whiteboards this week, I want you to ponder that data synchronization dilemma. If you denormalize your data for read speed today, how are you going to mathematically guarantee data consistency tomorrow? Think about it. We'll see you in our next deep dive.