Modern Data Stores Ep 4
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's jump straight into this explainer and build on exactly what we learned about keeping distributed data perfectly synchronized. You know, when you're managing modern systems, data rarely just sits in one place, does it?
And ensuring every single component actually agrees on the current state of the world, well, that's arguably one of the absolute hardest problems in software engineering. You'll probably remember from episode three's look at denormalized NoSQL views, that we critically need a way to reliably sync our operational databases to our derived stores.
I mean, we can't just cross our fingers and hope our search indexes and caches stay fresh, right? We need a genuinely bulletproof mechanism to make it happen. And this right here perfectly illustrates what we call the dual write anti-pattern.
Think about it. If your application writes to Postgres and then tries to publish straight to Kafka or Redis directly in the application code, you are literally walking into a trap. Say your application suffers a partial crash, maybe just two milliseconds before publishing the event.
Well, that data's already committed to the database, but the event itself, lost forever. So now your Redis caches and analytics are permanently out of sync with your primary database, completely corrupting your data state.
It's a total disaster. So to fix this, here is our roadmap for mastering complex event-driven data systems today. We're going to look at Kafka's log mechanics, the Outbox and CDC pipeline, event patterns and streams, then we'll bust the exactly once myth, and finish up with a solid engineering checklist.
Let's kick things off with section one, Kafka's log mechanics. Basically, we are turning the database inside out. Now what's really fascinating here is the mechanical sympathy of Kafka's append-only partitioned log.
You see, Kafka isn't just a traditional message broker. It leverages immutable on-disk segment files paired directly with the OS page cache and send file zero copy network transfers. That is how it achieves such incredible throughput.
Plus, it uses consumer groups for offset tracking and cooperative rebalancing, which is fantastic, because it allows multiple independent systems to just replay the data at their own pace. Oh, and it handles long-term storage beautifully through log compaction, using tombstone markers to basically clean up deleted records.
But there is a vital trade-off here, and you really need to keep this in mind. Kafka provides strict ordering only within one specific partition. Global ordering across the entire topic? That simply does not exist.
So if you absolutely need a user's events to be processed in the exact order they occurred, you have to ensure you hash their user ID as the partition key. That way, all of their events route to that one specific partition. The outbox and CDC pipeline.
This is how we actually fix that dual write disaster we talked about earlier. So by using the transactional outbox pattern paired with Debezium's logical decoding of the Postgres writer headlog or wall, we solve the problem completely. Here's how it works.
The application commits both the business data and an event record to an outbox table in one single atomic Postgres transaction. Then Debesium tails that wall via the PG output plugin, and those committed changes stream reliably straight into Kafka.
Finally, independent consumers pick up those events to seamlessly update those Reddish derived caches we built back in Web Architecture Season 2 episode 7. The result? No rate conditions and absolutely no dropped events.
Alright, section three. Event patterns and streams. Let's dig into the architectural complexity and the trade-offs involved. So this breaks down Martin Fowler's event-driven patterns.
Event notification is pretty simple, it just sends an ID. Super lightweight, sure, but it does require the consumer to query back for details. Then you've got event-carried state transfer, which passes the full state, naturally reducing coupling.
But take a look at event sourcing and CQRS. These are powerful because they grant complete temporal auditing. That means you can literally rebuild your database state for any point in history. Sounds amazing, right?
But here's the catch. They charge a massive architectural tax when evolving your schemas, often requiring these really complex upcasters to translate old event formats. It's an absolutely brilliant pattern for something like a financial ledger, but trust me, it can be incredibly painful for your standard CRUD applications.
And it's also worth noting the stark contrast between simple cues and these heavy-duty stateful stream processing engines. Like RabbitMQ is great for managing stateless message acknowledgements. Kafka handles your consumer offsets.
But when you start needing complex joins or time-windowed aggregations, that's when you reach for things like Apache Flink or Kafka streams. These engines are beasts. They handle out-of-order network data using event time watermarks, and they manage immense amount of local state using embedded ROXDB and Chandy Lampore asynchronous checkpoint barriers, just to guarantee fault tolerance.
It's some seriously impressive engineering. The exactly once myth. It is definitely time to deconstruct some of this industry marketing hype. So you'll hear a lot about exactly once delivery.
Now, Kafka's KIP98 absolutely guarantees exactly once processing, but specifically during a consume transform produce loop. It achieves this using an Idenpunt producer, PID, sequence numbers to drop duplicates, a two phase transaction coordinator, and read committed isolation levels.
Essentially, this ensures that a stream processing app reading from Kafka and writing back to Kafka will never process a message twice, even if the application crashes mid-flight. But, and this is a huge butt, this protection stops strictly at the broker boundary.
The moment you trigger external side effects, like charging a stripe card, sending an email, or updating a non-transactional sync, you instantly fall back to at least once delivery. Think about it.
If your Flink job hits an API and then crashes before committing its checkpoint, it's going to retry that API call the second it restarts. So what do you do? You absolutely must use the idempotent retries and unique request keys that we established back in Web Architecture Season 2 episode 6 to protect those external systems.
There's just no way around it. Which brings us nicely to our final section, section 5, our engineering checklist, where we cover rules and next steps. I want you to take these quantified engineering rules right back to your teams.
First, never dual write. Always rely on Outbox and CDC. Second, keep event sourcing far away from simple CRUD systems. It'll save you from absolute schema evolution nightmares.
Third, size your Kafka partitions carefully. You have to balance your need for strict key ordering against your need for consumer parallelism. Fourth, build idem potency into literally all external syncs.
And finally, this is absolutely critical, monitor your Postgres logical replication slots. Because if a Debesium connector stalls, it will hold that slot open, which prevents the database from recycling WAL segments.
If that happens, your primary database disk will fill up to 100% and crash your entire system. And nobody wants that kind of 2am wake-up call, right? I'll leave you with a final thought to ponder.
Now that our data streams are perfectly robust, completely immune to dual write corruption, and capable of exactly once transformations, how do we actually pipe this massive scale of event data into vector databases to feed generative AI models? Well, that is the next frontier of modern data infrastructure, and it's exactly what we'll be tackling next time in episode 5.