Modern Data Stores
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's dive straight into this explainer and build a proper mental map of the modern data landscape. If you're a software engineer or a tech lead, I know you've already got basic SQL under your belt, but let's be honest, today's architecture demands way more than just chucking a single relational database into your back end.
In this series, we're going to completely deconstruct the modern data stack so you can confidently choose and actually run the exact right data store for your specific engineering needs. Now, what's really fascinating about this is why we can't just rely on a single monolithic SQL database anymore when we're operating at scale.
This brilliant quote from the foundational Amazon Dynamo paper absolutely hits the nail on the head. It highlights that when you reach a certain scale, hardware and network failures aren't just edge cases.
They are your standard mode of operation. You literally have to treat failure handling as normal. And that fundamentally changes how you must store, replicate and serve data. The old way, it simply doesn't survive in this environment.
So, here is our roadmap for deconstructing all of this chaos. One, the polyglot persistence problem. Two, categorizing the data landscape. Three, a polyglot architecture example, four, avoiding common database mistakes, five, choosing the right data store, and six, looking ahead to OLTP versus OLAP.
Let's kick things off with section one, the polyglot persistence problem, and breaking down that paralysis of choice. Let's define the immediate pain point most tech leads face today. Modern systems handle incredibly diverse data, right?
You've got user transactions, event streams, JSON documents, telemetry, binary assets, and analytical facts. Since a single data store rarely satisfies all those different access patterns efficiently, we inevitably end up needing multiple highly specialized systems to handle those diverse workloads.
This is exactly what we call polyglot persistence. You're no longer just using one database, you're orchestrating a combination of them, precisely tailored to your application's unique needs. Categorising the data landscape and drawing the boundaries.
So the absolutely crucial point here is knowing how to categorize these systems by their trade-offs, starting with their core workload type. This table brilliantly illustrates the massive divide between operational and analytical systems.
On the operational side, OLTP, you're dealing with consistent real-time transactional operations. Think of tiny precise updates, like a user adding an item to a shopping cart. The data is usually normalized, and you'd use robust products like PostgreSQL or Media SQLite.
But over on the analytical side, OLAP, you're processing massive historical aggregations. We're talking petabytes of data, where the shape shifts to columna formats like Parquet to optimise those read speeds.
For that, you'll be looking at systems like Asia Data Explorer or Databricks. But you know, resting data is only half the story. We also have to categorize how data actually moves. We need to weigh the trade offs between batch processing and streaming processing.
Batch systems, using tools like Hadoop or HBase, compute over static files or row groups. It's incredibly efficient, but comes with high latency. We're talking minutes or even hours.
Streaming, on the other hand, deals with continuous events in milliseconds, using systems like Apache Kafka or Samsung. And just look at the failure recovery trade off here. Batch uses periodic global snapshots, whereas advanced streaming requires lightweight asynchronous barrier snapshotting so you don't store your ingestion pipeline.
So, based on those boundaries, these are the specific data store families we're going to explore over the next seven parts of this explainer. We'll be looking at relational, key value, document, column family, search and vector, streaming, and finally analytics and lakehouses.
Now, a quick note, we're always going to use real products as concrete examples of these architectures, like Redis or Cassandra, but never as blind endorsements. Our goal is to understand the underlying mechanics, so you know exactly what is happening under the hood of these families.
Right, section three, a polyglot architecture example or the inside out database. Let's look at a worked example to see how this builds into what Apache Samsung co-creator Martin Kletman calls the inside out database. Instead of a monolithic black box, we literally unbundle the database.
Step one, your user performs an action, and that transaction hits a highly consistent operational database, like Postgres. Step two, we use change data capture to stream that change as an immutable event into a distributed log, like Kafka.
And step three, that event eventually lands in a lake house environment, saved in a columnar format like Parquet, completely ready for heavy analytical queries. By doing this, we process transactions operationally, stream events reliably, and build materialized views purely for analytics.
Avoiding common database mistakes and the pitfalls in the modern data stack. Of course, stringing all these diverse systems together introduces a lot of complexity. And the biggest mistake I see engineers make is stubbornly clinging to the expectation that one database fits all workloads.
If you try to force all your analytical reporting into an operational staging area, your business applications are going to absolutely suffer from stale data and severe latency bottlenecks. Analytical queries are incredibly resource hungry.
If they share compute with your operational transactions, your customer-facing app will just slow to a crawl, and your analysts will be looking at data that's hours out of date. And there's another major pitfall: safety.
As the famous Jepson analyses show, when you decouple systems, you introduce network boundaries and eventual consistency. You absolutely must be acutely aware of distributed safety properties and consistency models to prevent dirty reads across your decoupled systems.
Take the aborted read anomaly, for example, or G1A. This happens when a part of your system reads data from a transaction that ultimately got rolled back. If you aren't mapping your dependencies properly and understanding those consistency guarantees, you will easily corrupt your system's data state.
Okay, onto section 5. Choosing the right data store, because it always depends on your workload. So how do you actually choose between these NoSQL and relational options? Well, my answer is always it depends.
It depends entirely on things like whether your workload is read-heavy or write-heavy, or whether you need strict ACID transaction guarantees. If you need a flexible polymorphic schema, a document store like MongoDB is brilliant, but you have to rely on optimistic locking.
If you need massive write throughput for something like IoT telemetry, a column family store like Cassandra is basically unparalleled. But that depends on you designing your row keys perfectly up front. And if you just need lightning fast point lookups, key value stores like Redis are fantastic.
But remember, the database sees the value as completely opaque. It can't query inside it. To really drive this it depends point home, just look at the number 281. That is the number of terabytes that a single SQLite database can seamlessly handle.
281 terabytes. That's mind-blowing for a local file-based database. But choosing it over a traditional client-server model depends heavily on whether your application requires high concurrent writers over a network.
If it does, SQLite is simply the wrong choice, regardless of its massive storage capacity, because it only supports one writer at a time. Your trade-offs dictate absolutely everything. Finally, section 6.
Next up, OLTP versus OLAP in part E01. We've established our taxonomy and mapped out our architectural flows, but to truly master polyglot persistence, we need to dig deeper into the foundations. We need to look at the bare metal disk level.
But how exactly do the physical guts of an operational database actually differ from an analytical one? Could the very structure of your data on disk be the hidden reason your application is bottlenecking?
Join us in the next explainer, part E01, where we'll unpack B trees, log structured merge trees, and columnar compression to find out. I'll see you there.