Data Store Architecture
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome back to the series 5 finale of our explainer. Following on from our deep dive into episode 6's rather brutal production operational constraints, today we're synthesizing everything we've learned into one concrete architectural decision framework for your data stores.
Let's kick things off with a quick recap of our core operational constraint. As Google SRE principles remind us, the rule here is absolute. You can't just freeze your code base, but you absolutely cannot introduce chaos into your critical state either.
So production demands agility, but it has to be perfectly mathematically balanced against absolute stability. To strike that balance today, we're going to explore the tension between boring and purpose-built tech, define our seven workload intake questions, build out a master architecture decision table, examine the polyglot tax, and finish up with some real-world production scaling case studies.
Right, let's jump straight into section one. Boring versus purpose-built, the core architectural tension. This debate really is the heavyweight clash of modern architecture. On one side, you've got Dan McKinley, advocating for choose boring technology, things like Postgres, strictly because its failure modes are well known and predictable.
But then, on the exact opposite end of the spectrum, we have Werner Vogels. He argues that monolithic, one-size-fits-all databases just fundamentally fail when they hit diverse internet-scale access patterns. He says you absolutely need purpose-built data stores.
Now, McKinley has this brilliant concept. He argues that every company only gets about three innovation tokens to spend on shiny new technology before the sheer operational complexity completely crushes your team.
And make no mistake, choosing a brand new database, that immediately burns one of those precious tokens. Because here's the thing: when you're managing critical state, boring is actually a highly desirable trait.
You want predictability. You need to avoid accidental complexity at all costs. Unproven, exciting tech? Well, that's exactly what leads to those dreaded 3am pager alerts.
Surprises in production are quite literally the absolute nemesis of site reliability engineering. Seven workload intake questions. Bridging the divide.
So how do we actually bridge this divide? We filter our architectural decisions through these seven concrete workload intake questions. You really have to quantify your exact needs before adopting new tech.
You need to ask yourself, what exactly is your P99 latency budget? What are your strict PASE elk consistency invariants or your RPO and RTO targets? And what does your payload shape and data relationship structure actually look like in the real world?
To put that into perspective, just look at your dataset growth curve. Knowing this is absolutely vital, because even a monolithic powerhouse like PostgresQL has a hard 32TB limit per table. If you run the numbers and confidently project that you'll exceed that limit rapidly, well, a traditional relational approach might just mathematically fail you from day one.
The master architecture decision table. Categorising the ecosystem. This is where we draw our explicit engineering thresholds. Look at embedded stores, for instance.
SQL Lite or DuckDB are perfect for device local data, or maybe sub-100,000 hit web traffic. But you are instantly disqualified if you have high concurrent writers. Then you've got relational OLTP stores like Postgres or MySQL.
These are your defaults for acid transactions. But if you require extreme horizontal scale, you'll need to graduate to distributed SQL, like cockroach DB or Vitesse. And meanwhile, for in-memory key-value stores, Redis is your go-to entry for microsecond latency, but it completely falls over if you require complex querying.
Now, as we look at purpose built non-relational stores, the rule is simple. You only spend an innovation token here when your payload shape or QPS fundamentally breaches those relational limits. So you might choose a document store like Dynamo DB or MongoDB for evolving aggregates.
Or a wide column store like Cassandra or CeilerDB for high-right IoT telemetry. Or maybe a graph database like Neptune or Neo4J if you need deep relationship traversals. And of course, for event-driven architectures, this is exactly where streaming logs like Kafka enter the fold.
And we can't forget about analytical workloads. As you can see here, open lake house data formats like Apache Iceberg, especially when paired with engines like ClickHouse, now offer highly competitive query durations.
They are seriously challenging those traditional closed world data warehouses like Snowflake or BigQuery. It's a massive shift in the landscape. Let's move on to section 4.
Polyglot taxes and evolution, the cost of complexity. Okay, so adopting multiple databases sounds great in theory, but embracing polyglot persistence incurs a brutal database per service tax. Your management effort multiplies and distributed cross-service transactions become incredibly difficult.
To survive this, you really have to abandon risky dual rights. Instead, you'll want to adopt the transactional outbox pattern paired with Debesium CDC for safe syncing, and when you're migrating live traffic, you do it incrementally using the Strangler fig pattern.
Even just adding an external cache like Redis isn't free. As we covered extensively back in Web Architecture Series 2 episode 7, it introduces severe taxes. You're suddenly dealing with cash coherence issues across your fleets.
Plus, as we discussed in episode 6 on resilience, you introduce the very real fatal risk of taking down your primary database with a cold start thundering herd if that cache ever reboots. So how do we safely mitigate these taxes?
You have to implement resilient patterns. A great example is request coalescing. Think about how Discord implemented Rust-based request coalescing to handle trillions of messages, ensuring that only one single request actually hits the database during a cache miss.
You pair that with a soft and hard time to live fallback strategy, which allows you to gracefully serve stale cached data and preserve overall availability during downstream brownouts. Finally, section 5, production scaling case studies.
Theory in practice. Let's ground all this theory in reality. Notice how companies like Figma and Notion didn't just immediately rewrite everything into NoSQL the second things got tough.
Instead, they completely exhausted their boring Postgres runway first. They scale vertically, maxing out massive R5.24X large CPU and NVMe instances. Then they added read replicas, and they squeezed every last drop of performance out of the system using PG Bouncer for connection pooling.
It was only when Postgres's autovacuum physically stalled and they hit hard CPU limits at a hundred X scale that they finally executed horizontal application sharding, often utilizing tools like Vitesse. Notion is a brilliant example here.
They meticulously split their data model into exactly 480 logical shards across 32 physical databases. And they did this strictly to keep their tables under a 500 gigabyte limit, choosing that operational path over rewriting everything into NoSQL.
The exact same logic applies to modern AI data. Keep it simple. Start with PG Vector, right inside Postgres for your similarity searches, so you can maintain acid compliance and complex joins.
You should graduate to dedicated vector stores like Qrant or Milvus only when your scale absolutely demands it. And similarly, if you find you must use NoSQL, measure your bottlenecks first. Discord famously evolved their chat storage from MongoDB to Cassandra, and finally to ScyllaDB, specifically to escape JVM garbage collection pauses and hot partition stalls.
It just goes to show that purpose built evolution must be strictly data driven. True engineering excellence isn't about rushing to the newest, shiniest tool. It's really about choosing the least worst option to safely keep the company running.
So take a hard look at your current architecture roadmap. Ask yourself honestly are you migrating to a new data store because your quantified metrics strictly demand it, or are you just eager to spend an innovation token?