Modern Data Stores: Ep 0
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome to our architectural blueprint of the modern data storage landscape. If you've ever looked at the sheer number of database technologies out there today and wondered, why on earth do we have so many and when should I actually use them? Well, you are in exactly the right place.
In this explainer, we're going to unpack the foundations of modern data stores. We'll map out how operational, analytical, streaming, NoSQL, and vector stores all fit together. By the end, you'll have the concrete tools you need to make the right architectural decisions.
Alright, let's jump straight into this foundational concept from software architecture pioneer Martin Fowler. He actually calls this the database Thor. You see, for decades, the relational database was the undisputed king.
But today, modern applications are practically forced to abandon that single relational model. We are gearing up for a massive shift to what is known as polyglot persistence. Basically, it's an environment where any decent sized enterprise uses a whole variety of different data storage technologies, each carefully chosen for how their data is uniquely manipulated.
Here is our roadmap for the explainer. We're covering one, baseline and polyglot persistence, two, physical access, rows versus columns, three, the non-relational landscape, four, streaming and immutable logs, five, a master comparison matrix, and finally, six, up next, OLTP versus OLAP.
Let's get into it. Baseline and polyglot persistence. We're going to start with the established orthodoxy, the traditional relational database, and the very real operational tax you pay when you decide to leave it behind.
Now, while Fowler strongly advocates for using the right tool for the job, he explicitly warns us about the polyglot persistence tax, and this is huge. Bringing in specialized non-relational stores just isn't free. Every single new database introduces a new interface for your developers to learn.
It introduces, frankly, terrifying operational burdens, like dual write bugs, where data successfully writes to your main database, but silently fails to update your cache or search index. It increases the risk of your backups drifting completely out of sync, but most painfully, it massively expands the on-call surface area for your engineering teams.
So the absolute crucial takeaway is this. A relational database remains your baseline until specific, quantified thresholds literally force your architecture to evolve. Do not pay that polyglot tax prematurely.
Take SQLite, for example. People often dismiss it as just a toy, but it can comfortably handle a hundred thousand hits a day on a single local file, and it's capable of supporting databases up to a staggering 281 terabytes.
Unless you are hitting quantified barriers in queries per second, dealing with highly irregular payload shapes, or demanding massive analytical scan volumes, that relational orthodoxy is actually your best friend. Physical access, rows versus columns.
But what happens when we do hit those breaking points? Well, when we need to scale, we have to drop right down to the physical disk level to really understand the ultimate fork in the road. This perfectly illustrates the great divide driving the entire data landscape.
Think about operational databases, like Postgres or SQL Lite. They are built to manipulate 8 kilobyte row pages, making them fantastic for extremely fast point lockups and multi-row transactional consistency. But then think about analytical systems.
If you try to run a billion row analytics query on an operational database, the CPU has to painstakingly pull out every single 8 kilobyte row just to read a single column. It's horribly inefficient.
Analytical systems instead execute vectorized scans across immutable columnless segments, like parquet files, reading only the specific data they actually need. As the creators of DuckDB put it, replacing that sequential row by row grind with columnar vectorized batches fundamentally changes the CPU overhead.
It pulls a large batch of values from a single column, literally a vector, and processes them all in one rapid operation. This is exactly why analytical queries on systems like DuckDB, ClickHouse, or Snowflake are exponentially faster than trying to run that exact same query on a traditional relational database.
Mapping the non-relational landscape. Having understood that physical split, let's explore what actually happens when those operational constraints finally force us out of the relational model. It's super helpful to categorize this fragmented landscape by access patterns.
Depending on your query shape, you're essentially trading relational acid constraints for something else. Got evolving JSON payloads, trade for a document store like MongoDB. Need extreme low latency caching, reach for a key value store like Redis.
Dealing with high throughput, sparse telemetry data, you'd use a column family store like Cassandra. And then of course you have graph stores for deep relationship traversals, or vector databases for approximate nearest neighbor similarity searches in modern AI applications.
To really put this into perspective, take a look at Amazon Dynamo DB's 99.9%ile latencies over a 24 hour period. You see, you don't choose DynamoDB for complex SQL joins, you choose it specifically to maintain an ironclad latency budget.
Notice how the buffered writes brilliantly smooth out the massive latency spikes you get with direct writes. When your application's survival absolutely depends on retrieving a session key in single digit milliseconds 99.9% of the time, this is the highly specialized engine you reach for.
Streaming and immutable locks. Okay, so we've got operational relational databases, analytical columna engines, and highly specialized NoSQL stores. With all these different databases in play, we desperately need a unifying nervous system to connect them.
We need to rethink the database entirely. If we're paying that polyglot tax by unbundling our monolith into all these separate modular components, how do we stop our architecture from descending into total chaos?
How do we cleanly separate our writes from our reads across all these distributed systems without constantly fighting dual write bugs? The solution is Kletman's unbundled architecture. Step one, you stop writing directly to your read databases.
Instead, your absolute source of truth becomes an append-only partitioned log of immutable events. Think Apache Kafka. Step two, rather than relying solely on bounded batch processing over static files, you process these unbounded events using asynchronous streaming frameworks like Flink or Samsung.
Finally, those streams continuously feed and pre-compute read-optimized materialized views, caches, and search indexes. You're effectively taking the chaotic, shared mutable state of a typical database and turning it into an elegant, unbounded event stream that safely updates all your polyglot stores.
The master comparison matrix. Let's pull this all together and synthesize these physical and operational concepts into a definitive cheat sheet for the modern data ecosystem. This master matrix right here contrasts the primary systems we'll be covering in this series.
We've got relational OLTP like Postgres, prioritizing row level access and strict acid transactions. We have Columna OLAP like Click House, demanding massive batch scans over immutable parquet files.
We have NoSQL families tuning their consistency models, balancing those tricky pack elk trade-offs for specific key or document access, and underpinning the whole thing event logs like Kafka, handling unbounded, append-only streams. It all comes down to choosing the right tool for the right access pattern with the exact right latency budget.
And what's really exciting is how all of this builds into new hybrid patterns, like the lake house architecture. Historically, you had cheap data lakes full of unmanaged parquet files and expensive data warehouses with powerful management features.
But the Lake House sits a metadata and caching layer directly on top of the open data lake. It brings transaction management, governance, and indexing directly to your low-cost object storage, effectively bridging the gap between raw files and a structured database.
Up next, OLTP versus OLAP. We've covered a massive amount of ground today, mapping this foundational blueprint. Let's take a quick look at exactly where we're heading next. In our next explainer, we are diving straight into the absolute core rivalry in data engineering, OLTP versus OLAP.
We'll unpack the fundamental tension between protecting transactional integrity on a per row basis and demanding analytical throughput across byrions of columns. We're going to explain the severe quantifiable performance penalties that occur when you try to mix operational rights and analytical scans in a single database, and crucially, how to architect a solution that gives you the best of both worlds.
But before we wrap up, I want to leave you with a provocative question to think about regarding your own systems. If polyglot persistence really is our architectural reality now, do you know the exact quantified threshold, whether it's queries per second, latency, or data size, where your current single relational database actually breaks and becomes a liability?
Understanding that specific number is step one in modernizing your data architecture. Thanks for joining me on this explainer, and I'll catch you in the next one.