Streaming and Event-Driven Data
Podcast · Streaming and Event-Driven Data

Solving Dual Writes with Change Data Capture

by ZumiLabs · 56m
0:00 56m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

Picture this scenario. It is Black Friday, right?

Host 2

Oh, the classic nightmare.

Host 1

Exactly. Traffic is just spiking vertically on your monitoring dashboards. And your engineering team just deployed this brand new inventory microservice to handle the load.

Host 2

Sounds dangerous already.

Host 1

Right. So to keep the downstream systems synchronized, the devs implemented a classic try-catch block in the application code. The logic seems, you know, straightforward enough.

Host 2

It always does.

Host 1

Yeah, they just dual write the customer order to a primary Postgres database and then immediately publish that exact same order event to a Kafka topic.

Host 2

I can see where this is going.

Host 1

Right. Because right under peak load, the infrastructure hits this transient network partition. The app successfully commits the transaction to Postgres.

Host 2

But it crashes.

Host 1

Yes. It hard crashes exactly two milliseconds before it can execute the network published call to the Kafka broker.

Host 2

Ouch.

Host 1

And the result is just catastrophic. Thousands of phantom inventory updates leak into the architecture. I mean, your analytics pipelines, your Elasticsearch indexes, your Redis caches.

Host 2

All of them.

Host 1

Yeah, all of them rely on that Kafka topic. And now they are permanently and silently out of sync with the primary database.

Host 2

And you know, the insidious part of that failure mode is the silence.

Host 1

Right. Because the application layer threw an exception that was just lost in the container crash.

Host 2

Exactly. So your standard alerting doesn't even catch the data drift. You only figure out there's a massive inconsistency like days later.

Host 1

When the customer support tickets start flooding in.

Host 2

Yeah, about out of stock items being sold or when financial reconciliation jobs just completely fail at the end of the month.

Host 1

It's a disaster. So okay. A team tries to fix that specific dual right race condition by moving to change data capture, right?

Host 2

Yeah, logical next step.

Host 1

They deploy a Debezium connector to tail the Postgres database, completely removing the application from the publishing path. And it runs flawlessly for months.

Host 2

Until it doesn't.

Host 1

Until a downstream consumer service dies over a weekend. That causes the Debezium connector to stall out and stop acknowledging messages.

Host 2

The replication slot issue.

Host 1

Yep. That stall connector sits there, silently holding open a Postgres PG output replication slot. And because the consumer is unresponsive, the database can't safely recycle its right-ahead log segments.

Host 2

So they just pile up.

Host 1

Exactly. Those wall files start piling up on the primary database disk. Slowly but surely, monitoring shows disk utilization creeping to 80%, then 90%, and finally, 100%.

Host 2

And boom.

Host 1

Boom. The primary production database hard crashes, taking the entire e-commerce platform down with it.

Host 2

I mean, we are looking at two entirely different architectural approaches to streaming data here. And both end in, well, catastrophic resume generating events.

Host 1

Yeah, these aren't just theoretical edge cases.

Host 2

Not at all. These are the actual operational clips that distributed systems engineers fall off when they, you know, misunderstand the mechanics of stream processing.

Host 1

Well, welcome to part four of our modern data stores series. If you've been following along, you know we've been building up these architectural primitives.

Host 2

Step by step.

Host 1

Just to quickly recap our previous deep dive, we spent our time heavily focused on denormalized NoSQL read views and how to structure data models for just massive read scalability.

Host 2

Right. And building on that foundation, plus considering we already covered right ahead log durability back in part one.

Host 1

And the analytical lake houses in part two.

Host 2

Exactly. So we are completely bypassing the foundational concepts today. We are operating under the assumption that you, the listener, are an experienced tech lead.

Host 1

Yeah, we won't be defining JSON payloads or basic pub sub here.

Host 2

No, we are going straight into the deep end of the engine room.

Host 1

So the mission for today's deep dive is exploring the unifying commit log abstraction. This is an architectural concept championed heavily by engineers like Jay Kreps and Martin Kleppin.

Host 2

It's a game changer.

Host 1

It really is. We need to figure out how to keep operational databases, derived caches and complex search indexes perfectly synchronized without tightly coupling the producers to the consumers.

Host 2

Which is the holy grail.

Host 1

Right. So we are unpacking Kafka style append only logs, log-based change data capture, stateful stream processing, and this is a big one. We are putting a massive, unforgiving magnifying glass on vendor marketing claims surrounding exactly one's processing semantics.

Host 2

I love that we are tackling that. Because to understand why that application level dual right disaster in the cold open actually occurred, we really have to look at the fundamental physics of the append only commit log first.

Host 1

Which people often get wrong.

Host 2

They do. The industry constantly conflates a system like Kafka with traditional message brokers like you know, RabbitMQ or Amazon SQS, but they operate on fundamentally different storage paradigms.

Host 1

Right, because a traditional queue is essentially just a transient transport mechanism.

Host 2

Exactly.

Host 1

Like when a consumer reads a message, that message is typically acknowledged and then destructively removed from the queue. I mean, a queue's ideal state is empty.

Host 2

Yes. But Kafka models data as a durable partitioned commit log.

Host 1

It's a completely different beast.

Host 2

It really is. When an app publishes an event to a broker, it isn't just held in memory waiting to be popped. It is appended to an immutable sequential on disk segment file.

Host 1

And that immutability is absolute, right?

Host 2

Absolutely. You cannot go back and edit a previous event in the middle of the file. You only append new events to the tail end.

Host 1

And because the operation is strictly append only, the disk I.O. performance changes dramatically.

Host 2

No, it's night and day.

Host 1

We are talking about purely sequential writes. I mean, even on traditional spinning hard drives, sequential writes can hit hundreds of megabytes per second.

Host 2

Yeah, because you completely avoid that severe latency penalty associated with random disk seeks.

Host 1

Right. But the sequential nature of the disk access is really just one piece of the performance profile, isn't it?

Host 2

Yeah. The deeper architectural advantage lies in how Kafka interacts with the operating system's memory management.

Host 1

The zero copy I.O. pathway.

Host 2

Exactly. Kafka relies heavily on the OS page cache combined with the send file system call.

Host 1

I want to break down zero copy I.O. for a second because it is just a brilliant piece of system level engineering. Let's trace a standard I.O. operation first.

Host 2

Good idea.

Host 1

Normally, if an app wants to send data from a file over a network socket, it requires multiple context switches and data copies. Like the OS has to copy the data from the disk hardware into a kernel space buffer.

Host 2

Right. Step one.

Host 1

Then it copies that data from the kernel space up into the application's user space memory.

Host 2

Step two.

Host 1

The application then copies the data back down into the kernel space, specifically into the socket buffer.

Host 2

Step three.

Host 1

And finally, the kernel copies it to the network interface controller, the NIC. I mean, that is a massive amount of CPU cycles burned just shuffling bytes around between memory boundaries.

Host 2

And the CPU overhead of those contact switches between user mode and kernel mode becomes the primary bottleneck in high-throughput systems.

Host 1

Way before the network bandwidth is even saturated.

Host 2

Exactly. The processor is basically doing clerical busy work.

Host 1

So a good analogy here for the send file system call is like a library return system. Standard I.O. is like a patron returning a book by walking into the library, handing the book to the librarian at the front desk.

Host 2

Who has to do the context switching?

Host 1

Right. The librarian logs it in their computer and then walks it to the back room to place it on a sorting cart. It requires active handling by the application layer.

Host 2

But the send file approach is entirely different.

Host 1

It's like installing a return chute on the outside of the building that drops the books directly onto the sorting cart in the back room.

Host 2

Bypassing the front desk entirely.

Host 1

Exactly. The data goes straight from the disk buffer in the OS page cache directly to the network socket via direct memory access or DMA.

Host 2

The Kafka application just issues a command to the operating system saying, you know, take the bytes from this file descriptor and transfer them to the socket descriptor.

Host 1

So Kafka acts merely as a traffic director.

Host 2

Yeah. Because it serves immutable files to consumers without needing to inspect or manipulate the payload bytes in user space. It doesn't need to pull the data into its own application heap.

Host 1

Which is wild.

Host 2

It allows a Kafka broker to completely saturate a 10 gigabit network link while the CPU utilization sits comfortably in the single digits.

Host 1

The bottleneck is pushed entirely off the processor and onto the network hardware.

Host 2

Which is incredibly efficient.

Host 1

But pushing the bottleneck to the network and relying on sequential files introduces some very specific architectural trade-offs.

Host 2

Oh, definitely.

Host 1

Especially regarding how the data is organized. This is where development teams often introduce fatal bugs when they're migrating from traditional cues.

Host 2

Yeah, let's talk about partitions and the illusion of global ordering.

Host 1

This represents the non-negotiable rule of distributed logs. Strict offset ordering exists exclusively within a single partition.

Host 2

Let me emphasize that. There is no global ordering across an entire topic.

Host 1

Because the topic is purely a logical abstraction, right?

Host 2

Exactly. Under the hood, that topic is split into multiple physical partitions, which are individual segment files distributed across different broker nodes in the cluster.

Host 1

So when a producer application sends a message, it typically provides a key, like for instance, a customer ID.

Host 2

Right. And the producer's client library uses a hashing algorithm, often something like murmur two to hash that key.

Host 1

And the result of that hash determines which specific partition the message is routed to.

Host 2

Yes. Because the same string key always yields the same hash, all events related to that specific customer ID will be appended to the same physical log file.

Host 1

In the exact order they were received by the broker.

Host 2

Exactly.

Host 1

So if a specific customer updates their profile and then later deletes their account, those two events are guaranteed to be sequenced correctly for downstream consumers.

Host 2

Because they share a partition, right.

Host 1

But what if customer A updates their profile and customer B creates a new account?

Host 2

Well, those events will likely hash to completely different partitions. So if a downstream consumer app is reading from the topic as a whole, consuming from multiple partitions concurrently, it might process customer B's event before customer A's event.

Host 1

Regardless of which event actually occurred first in absolute wall clock time.

Host 2

Exactly. Gotcha. And if the downstream business logic attempts to enforce causal ordering across different entities, the partition log architecture will completely break the application state.

Host 1

So partition key design is not just a performance optimization.

Host 2

No, it is a fundamental correctness requirement. You have to define the partition key based on the exact entity boundary that requires strict causal ordering.

Host 1

That makes total sense.

Host 2

So once the data is safely partitioned and ordered, the consumption model shifts, doesn't it?

Host 1

It does. Because the log is persistent and immutable, the storage layer is completely decoupled from the consumers.

Host 2

Right, because in a traditional queue, reading is a destructive act.

Host 1

Exactly.

Host 2

But in Kafka, consumers are organized into consumer groups. A consumer group doesn't modify the data.

Host 1

It simply maintains a logical pointer.

Host 2

An offset.

Host 1

Right. An offset indicating its current position in the log for each partition.

Host 2

And because of that, multiple independent consumer groups can traverse the same log at entirely different velocities without interfering with one another. Sure. A real-time fraud detection service can read at the head of the log, processing events milliseconds after they hit the broker. Concurrently, you could have a data warehouse batch ingestion job that wakes up once an hour.

Host 1

And it queries its last committed offset and just chews through a massive block of historical data.

Host 2

Right. They share the same underlying segment files via the OS page cache, but they share no state other than the immutable data itself.

Host 1

And as those consumer groups scale out for higher throughput, the framework handles the partition assignment through cooperative rebalancing protocols.

Host 2

Yes. So if a topic has 10 partitions and a consumer group spins up five worker instances, the group coordinator assigns two partitions to each worker.

Host 1

And if one of those worker instances crashes.

Host 2

The group coordinator detects the missed heartbeats, revokes the partitions from the dead worker, and reassigns them to the surviving instances.

Host 1

Aaron Powell Which then query the consumer offset's internal topic to find the last committed offset and resume processing.

Host 2

Without dropping data.

Host 1

But this persistent architecture has to introduce a long-term storage challenge. I mean, if the log is append only and consumers are merely advancing pointers, the disk storage is eventually going to hit capacity.

Host 2

Oh, absolutely. Administrators typically configure time-based or size-based retention policies. You know, dropping segment files older than seven days, for example.

Host 1

But certain architectural patterns require retaining the log indefinitely as the definitive source of truth.

Host 2

Yes, and if we need an infinite retention window without infinite disk space, we rely on log compaction.

Host 1

Well, compaction. So that's an asynchronous background process running on the brokers.

Host 2

Right. It evaluates the log segments and guarantees that it retains at least the most recent value for any given key.

Host 1

Okay, so consider a topic capturing user profile updates. If a user modifies their shipping address five times over a three-year period, downstream systems attempting to rebuild current state don't necessarily need the historical sequence of address changes.

Host 2

Exactly. They only need the final current state.

Host 1

So the compaction cleaner thread scans the inactive segment files, identifies duplicate keys, and discards the older records.

Host 2

It collapses the history, keeping only the latest event for each key.

Host 1

But what if that user deletes their account entirely? I mean, we need a way to signal to the compaction thread that the key itself should be purged.

Host 2

Right. And that's where the producer application emits a message containing the user's key, but with a null payload. A tombstone. When the background compaction thread processes the segment files, it observes that tombstone marker, recognizes the entity has been deleted, and eventually purges all historical occurrences of that key from the disk.

Host 1

That's brilliant. So compaction ensures the physical size of the log grows proportionally to the number of unique active keys in the data set.

Host 2

Rather than growing infinitely based on the total volume of historical mutations.

Host 1

Okay, so we've mapped out the internal storage mechanics, the zero copy I/O paths, and the compaction processes. But now we really need to elevate the discussion to the application architecture layer.

Host 2

We need to categorize the semantic meaning of the payloads traversing these pipes.

Host 1

Right. Because the industry is just saturated with theoretical definitions surrounding event-driven architecture.

Host 2

But Martin Fowler categorized these architectures into four distinct patterns that dictate system coupling.

Host 1

Event notification, event-carried state transfer, event sourcing, and CQRS, command query responsibility segregation.

Host 2

And these aren't just academic distinctions.

Host 1

Yeah.

Host 2

I mean, choosing the wrong pattern leads directly to cascading microservice failures.

Host 1

Wait. Aren't all of these basically just different flavors of PubSub when you get down to it?

Host 2

Not at all. And let's dissect the evolution of these patterns, because many teams start with event notification. It's minimal.

Host 1

Right. A microservice executes a local transaction and broadcasts a lightweight signal. Like an order was created, the ID is 4099.

Host 2

Exactly. It acts merely as a trigger. But when the downstream consumer receives that trigger, it lacks the context to do anything meaningful.

Host 1

So it has to make a synchronous REST or GRPC callback to the originating service just to fetch the full order details.

Host 2

And that synchronous callback introduces severe runtime coupling.

Host 1

Oh, I see. Because if a batch process suddenly generates a massive spike in notifications.

Host 2

The downstream service reacts by firing thousands of simultaneous API requests back at the upstream system.

Host 1

You've essentially built a distributed denial of service attack against your own database.

Host 2

Exactly. The thundering herd effect just overwhelms the upstream connection pools.

Host 1

So to break that coupling, architectures usually evolve to event carried state transfer.

Host 2

Right. In this pattern, the payload contains all the relevant mutated state.

Host 1

An order was created, and here is the fully hydrated JSON object. The customer ID, the line items, the shipping address, total price.

Host 2

Yep. The downstream service can now consume the event and update its own local materialized view or search index without ever querying the upstream system.

Host 1

The network dependency is severed.

Host 2

It is. But notice that in both event notification and event-carried state transfer, the primary source of truth remains a traditional relational database.

Host 1

Events are just byproducts of state changes happening in Postgres or MySQL.

Host 2

Right. The architecture shifts radically when we move to the third pattern, full event sourcing.

Host 1

Okay. Event sourcing is heavily hyped, but I want to push back on its practical application for standard business domains.

Host 2

Oh, absolutely.

Host 1

Because the core premise is that the event log itself becomes the primary database. You no longer store the current mutable state of an entity.

Host 2

You store an immutable ledger of state mutating events.

Host 1

So if you want a user's current account balance, you don't query a balance integer column. You read their entire history of deposits and withdrawals from the log and calculate the sum dynamically in the application layer.

Host 2

The theoretical benefits are compelling.

Host 1

I mean, it provides a mathematically perfect audit trail and allows engineers to execute point-in-time queries by replaying the log up to a specific microsecond. However, imposing event sourcing on a standard CRUD application introduces a crushing architectural burden.

Host 2

It really does.

Host 1

First is the read performance problem. If an entity has tens of thousands of historical events, replaying that ledger on every read request introduces unacceptable latency. So you have to build snapshotting mechanisms. The system must periodically calculate the current state and persist it as a materialized view.

Host 2

So subsequent queries only need to replay events that occurred after the snapshot timestamp.

Host 1

But even worse than the performance overhead is the schema evolution tax.

Host 2

This is where event sourcing architectures often collapse under their own weight. Business requirements change.

Host 1

Yeah. Assume two years into a project, a new regulatory requirement forces you to add a tax calculation field to every order.

Host 2

Well, your immutable event log now contains two years of historical events serialized in version one of the schema, which lacks the new field.

Host 1

And because the log is immutable, you can't write a SQL migration script to run an alter table and backfill the old records. The historical bytes are frozen on disk.

Host 2

Exactly. So the engineering team must write and maintain event schema upcasters.

Host 1

Upcasters.

Host 2

Yeah. These are complex translation layers embedded in the application code. When the application reads the log, the upcaster intercepts the binary payload, deserializes it, identifies it as a version one event, and maps it in memory to a version two object structure.

Host 1

Injecting default values for the missing tax field before passing it to the domain logic.

Host 2

Alright. And as the system evolves to version five or version six, these upcasters become a tangled web of technical debt, making the system incredibly fragile to reason about.

Host 1

Because querying an append-only log by anything other than the entity key is functionally impossible, event sourcing forces you into the fourth pattern. CQRS.

Host 2

You have to strictly separate your write models from your read models.

Host 1

The write path appends to the log. But the read path requires asynchronous worker processes to consume those events and project them into separate, highly indexed relational or noSQL databases.

Host 2

It's a massive distributed systems overhead for a simple web application.

Host 1

Which brings us back to the pragmatic reality of modern engineering. Most teams are building microservices backed by traditional relational databases like Postgres. Because SQL is highly flexible for evolving queries.

Host 2

But they still need to synchronize that data out to external systems. Populating an Elasticsearch cluster, updating a Redis caching layer, or feeding a Snowflake data warehouse via Kafka.

Host 1

And this circles back perfectly to our cold open. Knowing the dangers of event sourcing, how do we safely extract data from a standard CRUD database without falling into that dual write trap?

Host 2

Because we know that attempting a dual write, executing a Postgres insert and a Kafka produce call within the same application method creates a race condition that leads to phantom data on partial crashes. Right.

Host 1

So what is the solution?

Host 2

The industry standard architecture for this problem is the transactional outbox pattern implemented alongside log-based change data capture.

Host 1

And the premier open source tool for this CDC layer is Debesium.

Host 2

Exactly.

Host 1

Let's trace the exact system level flow of the outbox pattern. Say we have an application creating a new user order.

Host 2

Okay. Instead of trying to coordinate a distributed transaction between Postgres and Kafka, the application only communicates with Postgres.

Host 1

Right. It initiates a local database transaction. It executes an insert statement against the ordered domain table.

Host 2

But crucially, within that exact same local transaction, it executes a second insert statement.

Host 1

Writing a serialized JSON or Avro representation of that business event into a dedicated outbox table.

Host 2

And finally, it commits the transaction.

Host 1

Because both inserts occur within a single database transaction. The operation relies on the database's acid properties.

Host 2

Atomicity guarantees that either both the domain data and the outbox event are durably persisted to the database disk, or neither are.

Host 1

There is mathematically no possibility of a partial failure state where the business data exists but the event is lost.

Host 2

The application's responsibility ends the moment the transaction commits.

Host 1

Now we introduce the change data capture infrastructure. Debesium is typically deployed within a Kafka Connect cluster, running entirely outside the application microservice.

Host 2

And it establishes a connection to the Postgres database, but not via standard SQL queries.

Host 1

It connects using the replication protocol, right? Impersonating a standby database replica.

Host 2

Yes. Debezium leverages Postgres's logical decoding feature, specifically utilizing the PG output plugin.

Host 1

So when the local application transaction commits, Postgres writes the low-level binary changes to its write-ahead log.

Host 2

And the PG output plugin intercepts those binary WAL records and decodes them into a logical format, capturing the exact row level mutations.

Host 1

Debezium streams this logical replication feed in real time, filters for the outbox table, extracts the event payloads, and safely publishes them to the destination Kafka topic.

Host 2

Exactly.

Host 1

We have successfully eliminated the application level dual write. The database's own transaction log acts as the highly durable local queue.

Host 2

But deploying CDC introduces a complex bootstrapping problem, doesn't it?

Host 1

It does.

Host 2

If we deploy Debezium on an existing database that already contains five million historical orders, Debezium cannot simply tail the WAL.

Host 1

Because the WAL only retains the most recent transactions.

Host 2

Right. It must perform an initial historical snapshot to sync the current state into Kafka.

Host 1

And snapshotting a live database without taking application downtime is a notoriously difficult concurrency problem.

Host 2

Oh, it's a nightmare. If Debezium executes a massive select query across a table with millions of rows, that query might take hours to complete.

Host 1

And during those hours, production traffic is continuously updating rows that have already been scanned or updating rows that haven't been scanned yet.

Host 2

Exactly. So if Debezium isn't careful, it could emit a stale read event from the slow snapshot that overwrites a newer live update event streaming from the wall.

Host 1

Which would permanently corrupt the downstream systems.

Host 2

Yes. So Debezium resolves this race condition using a highly sophisticated snapshot window algorithm combined with in-memory deduplication.

Host 1

How exactly does that window mechanism work?

Host 2

The process operates in strict phases. Before the select query begins, Debezium connects to the replication slot, notes its starting log sequence number, and begins actively tailing the wall.

Host 1

But it doesn't publish these incoming live events to Kafka yet, right?

Host 2

No. Instead, it buffers them locally in memory.

Host 1

It essentially sets up a holding pen for all the live production changes.

Host 2

Exactly.

Host 1

Once the buffering begins, Debesium opens the snapshot window and starts paginating through the historical table chunks.

Host 2

Generating a read event for every row it scans.

Host 1

Right.

Host 2

But before emitting that read event to Kafka, Debezium interrogates its in-memory buffer. It compares the primary key of the scanned row against the primary keys of all the buffered update and delete events that have occurred since the snapshot started.

Host 1

So if it finds a match, it knows that the row it just scanned from the table has already been modified by live traffic.

Host 2

Exactly. If there is a key collision, Debesium silently drops the stale read event generated by the snapshot. It allows the newer update event from the wall buffer to take precedence.

Host 1

So it performs continuous live deduplication, guaranteeing a mathematically consistent initial state transfer without ever acquiring table locks or halting production database rights.

Host 2

Yep. Once the historical scan completes, the snapshot window closes, the memory buffer is fleshed to Kafka, and Debezium transitions seamlessly to purely tailing the wall.

Host 1

The transactional outbox pattern paired with Debessium CDC provides just a bulletproof pipeline for safely extracting data into Kafka.

Host 2

It really does. But raw streams are essentially raw fire hoses of granular events.

Host 1

Right. To build the materialized views we need to power low latency API caches or complex search indexes. We need more than just transport. We need stateful stream processing.

Host 2

And this requires a conceptual shift in how engineers model data. In the stream processing paradigm formalized by Kafka's creators, there is a foundational principle known as stream table duality.

Host 1

Duality implies they are two representations of the exact same underlying concept. Let's break down how a stream relates to a table.

Host 2

A stream represents the change log of a table. It is an immutable, chronologically ordered history of every mutation, every insert, update, and delete.

Host 1

Conversely, a table is simply a materialized snapshot of a stream at a specific point in time.

Host 2

Exactly. If you take a continuous stream of events and apply them sequentially, accumulating the state, you construct a table.

Host 1

And if you take a static table and monitor every modification made to its rows, you emit a stream.

Host 2

In frameworks like Kafka streams, this duality is exposed directly in the API through two primary interfaces. The K-stream representing the raw stateless event flow, and the K-table representing the materialized, continuously updating aggregation.

Host 1

But stateful operations like windowed aggregations or stream-to-stream joins require the processor to maintain memory of previous events.

Host 2

Right. And because these stream processors run as distributed applications across multiple Kubernetes pods, they cannot rely on querying a remote database for every single event to fetch state.

Host 1

The network latency would just destroy the throughput guarantees.

Host 2

Exactly. They require embedded local state stores.

Host 1

And the industry standard for this embedded storage is ROXDB. It's an incredibly high performance key value store optimized for fast storage media.

Host 2

Right. But we need to understand how it operates within the container. RocksDB runs directly within the memory space and local disk volume of the stream processing application.

Host 1

Utilizing a log structured merge tree architecture.

Host 2

Yes. When a Kafka streams application processes an event and updates a K-table aggregation, it writes that new value into RocksDB's in-memory mem table.

Host 1

And as that memory fills up, RocksDB flushes it down to immutable SS tables on the local SSD.

Host 2

But relying on local container storage introduces a massive reliability question. Kubernetes pods are ephemeral by design.

Host 1

Right. If the underlying node hardware fails and the pod crashes, the local SSD and therefore the entire aggregated ROXDB state is destroyed.

Host 2

So how does the system recover the state without starting the aggregations from scratch?

Host 1

The recovery mechanism relies directly on stream table duality, doesn't it?

Host 2

It does. Every time the stream processing application updates its local ROXDB instance, it simultaneously emits that specific state change to a hidden, heavily compacted Kafka topic known as a changelog topic.

Host 1

The Kafka broker acts as the durable backup.

Host 2

Exactly. If a pod crashes and the orchestrator spins up a replacement container on a new node, the new instance initializes an empty ROXDB database.

Host 1

And before it resumes processing the main input streams, it subscribes to its specific partition of the changelog topic and replays the compacted history.

Host 2

Reconstructing the exact local state by applying the change log, completely restoring the RocksDB instance to the moment before the crash.

Host 1

But stateful aggregations introduce another layer of immense complexity, and that is the concept of time.

Host 2

Flink strictly forces developers to define the temporal domain they are operating in. You have to choose between processing time or event time.

Host 1

Now let's unpack those. Processing time refers to the wall clock time of the physical server executing the Flink operator.

Host 2

Right. It provides extremely low latency because the system doesn't wait for anything. It just groups events based on when the CPU encounters them.

Host 1

But it is fundamentally non-deterministic, right?

Host 2

Absolutely. If a network switch fails or a Kafka broker undergoes a leader election, causing a brief partition outage, events will be delayed in transit.

Host 1

So when they finally arrive, they will be grouped into incorrect time windows based on the delayed processing moment, completely skewing the aggregations.

Host 2

Exactly. For any domain requiring accuracy, financial transaction rollups, billing meters, precise telemetry, processing time is unacceptable. We have to use event time.

Host 1

Event time relies on the chronological timestamp embedded inside the event payload itself, like the exact millisecond the event was recorded on the originating source.

Host 2

A mobile device or a point of sale terminal, yeah.

Host 1

Event time solves the determinism problem, but it introduces the chaos of distributed systems latency. Events do not arrive at the stream processor in neat chronological order.

Host 2

No, they don't. A mobile phone might lose cellular service, cache 50 events locally, and then transmit them in a single burst an hour later when it reconnects.

Host 1

So if a flink window operator is attempting to calculate the total sales for the 1 p.m. to 2 p.m. interval, how does it know when it is mathematically safe to close that window and emit the final result?

Host 2

Right, because it cannot hold the window open infinitely, waiting for delayed network packets.

Host 1

Does specific synchronization problem solve by watermarks, isn't it?

Host 2

Yes. A watermark is a specialized control message generated by the framework that flows sequentially in line with the data streams.

Host 1

It carries a specific timestamp, let's call it T.

Host 2

Right.

Host 1

When a flink operator receives a watermark carrying timestamp T, it serves as a formal declaration by the system that it believes no further events will arrive, possessing an embedded timestamp older than T.

Host 2

It is essentially a heuristic boundary.

Host 1

The system is drawing a line in the sand and stating based on our configured network lag tolerance, anything older than this watermark is officially classified as late data.

Host 2

Precisely. Watermarks bound the out-of-order delay. As the watermarks flow through the system and cross the threshold of an active time window, for example, a watermark arrives indicating 201 p.m.

Host 1

The flink window operator recognizes that the 1 p.m. to 2 p.m. window can safely be closed.

Host 2

Exactly. It triggers the final aggregation function and emits the result downstream.

Host 1

But even with watermarks managing the temporal windows, these stream processors are maintaining massive amounts of in-flight state across thousands of operators. We are talking about window contents, partial aggregations, buffered early events.

Host 2

It's a lot of state.

Host 1

So if a flink node crashes, how does the system recover the global in-flight state across the entire distributed cluster without dropping data or double counting records?

Host 2

The recovery relies on a brilliant distributed systems algorithm. Flink's implementation of asynchronous barrier snapshotting, which is derived directly from the Chandy Lamport algorithm.

Host 1

To really understand Chandy Lamport, we need an analogy for capturing distributed state. Because if you want to back up a running stream processor, you cannot execute a global stop the world pause, freeze all the network pipes, copy gigabytes of RAM to Amazon S3, and then unpause.

Host 2

No, the latency spike would cripple the upstream producers.

Host 1

Right. You are required to capture a globally consistent snapshot of the system's state while the data is actively flowing through the operators.

Host 2

So how do you visualize that?

Host 1

Think of taking a panoramic photograph of a crowded busy room where people are constantly moving. If you pan the camera slowly from left to right, a person might run behind you and appear in the photograph twice, ruining the consistency of the image.

Host 2

That makes sense.

Host 1

To capture a truly consistent snapshot without freezing time, you need a mechanism to mark exactly where everyone was at one specific relative moment.

Host 2

And in Flink, that marking mechanism is the checkpoint barrier.

Host 1

The centralized job manager periodically injects a special barrier control record into all the source data streams at the very beginning of the pipeline.

Host 2

And these barriers flow strictly in line with the data records. They do not jump ahead or overtake the data.

Host 1

The barriers physically segment the stream into two distinct eras. Every day record positioned in front of the barrier belongs to the current state snapshot.

Host 2

And every record positioned behind the barrier belongs to the next future snapshot.

Host 1

Right. As these barriers traverse the distributed execution graph, they arrive at the individual Flink operators.

Host 2

When an operator receives a barrier from an input channel, a complex alignment process begins.

Host 1

Because many operators consume from multiple input channels.

Host 2

Exactly. The operator must briefly block the processing of the fast channel that delivered the barrier, buffering its incoming data, while it waits for the corresponding barriers to arrive on the slower input channels. Yes. Once the operator has received the barrier from all of its incoming channels, it performs a local snapshot.

Host 1

Capturing its current state, the contents of its RocksDB stores, internal counters, window buffers, and asynchronously begins writing that state data to a highly durable distributed file system like S3.

Host 2

And crucially, the moment the state capture begins in memory, the operator immediately broadcasts the barrier forward to its downstream consumer operators and resumes processing the data streams.

Host 1

So the actual disk IO to S3 happens in the background without blocking the data flow.

Host 2

Exactly. The snapshot process ripples dynamically across the entire cluster. When the final sync operators at the very end of the pipeline report to the job manager that they have successfully flushed their state to S3, the job manager declares the global checkpoint complete. And if a node fails, the entire cluster restarts, pulls the last completed global snapshot from S3 into the operators, and rewinds the Kafka consumer offsets to the exact positions associated with that specific barrier.

Host 1

Which introduces the most heavily scrutinized, mathematically controversial phrase in data engineering. Exactly once semantics.

Host 2

Yes. They really do.

Host 1

But we need to deconstruct the reality of how it functions at the network layer, specifically looking at the mechanics of Kafka's transactional API introduced under KIP98.

Host 2

Before KIP98 streaming applications were plagued by the retry problem. Right. If a producer dispatched a batch of messages to a broker, the broker appended them to the disk, but the network connection dropped before the broker could transmit the acknowledgement packet.

Host 1

The producer receiving a timeout assumed the right field and retried the transmission. And the broker appended the retry batch, resulting in duplicate records in the log. That constitutes at least once delivery.

Host 2

Right. So to eliminate the duplication, KIP ninety-eight introduced the concept of IDOPOTEN producers.

Host 1

How's that work?

Host 2

When a producer initializes, the broker assigns it a unique, mathematically rigorous identifier called a producer ID or PID, along with an epoch number to fence off zombie processes.

Host 1

Furthermore, the producer library assigns a strictly incrementing sequence number to every individual batch of messages it transmits.

Host 2

Exactly. The broker maintains a small cache, tracking the highest sequence number it has successfully committed for each specific PID and partition combination.

Host 1

So if a network timeout occurs and the producer retries the transmission, it sends the batch with the exact same sequence number.

Host 2

And the broker inspects its cache, sees that it has already committed that sequence number for that PID, and silently drops the incoming duplicate batch while returning a success acknowledgement.

Host 1

The deduplication is handled entirely at the storage layer. That's IDOPotents.

Host 2

Yes. But idempotence only prevents duplication for a single producer writing to a single partition. Complex stream processing involves a loop.

Host 1

Consume from an input topic, transform the state, and produce to an output topic.

Host 2

Right. If the stream processor crashes in the middle of that loop, how does the system mathematically guarantee that it doesn't process the input record twice and emit the output record twice upon restart?

Host 1

That requirement necessitates distributed transactions.

Host 2

KIP98 introduced a dedicated broker component called the transaction coordinator, which manages a specialized two-phase commit protocol.

Host 1

So when a flink application processes an event, it initiates a transaction with the coordinator. As it computes the result, it writes the output records to the destination data partitions.

Host 2

However, it also writes its own consumer offset, the pointer proving it successfully read the input record to the internal consumer offsets topic as part of that exact same transaction group.

Host 1

Meaning the outputs and the consumer progress pointer are cryptographically bound together in the transaction.

Host 2

Exactly. When the computation finishes, the application issues a commit request to the transaction coordinator. The coordinator executes the two-phase commit.

Host 1

First, it writes a prepare state to the internal transaction log.

Host 2

Second, it writes physical specialized commit marker control records directly into all the involved partitions, both the output data partitions and the offset partition.

Host 1

And to make this work, the downstream consuming applications must be configured with a specific isolation level, right? Read committed.

Host 2

Yes. These consumers will fetch data from the broker, but they buffer the uncommitted messages in their local memory. They will not advance their last stable offset or expose those messages to the application logic until they physically encounter the commit marker in the log stream.

Host 1

And if the upstream application crashes and the transaction is aborted.

Host 2

The transaction coordinator writes an abort marker and the downstream consumers instantly discard the buffered messages.

Host 1

Through this mechanism, we can prove that a consume transform produce pipeline, operating strictly inside the Kafka cluster functions atomically. The state changes and the offsets are locked together.

Host 2

Inside the cluster being the keyword.

Host 1

Right. This is where we must draw a hard architectural line. Because what happens when the stream processor must leave the Kafka ecosystem? What if the Flink Jobs final operation is to execute a REST API call to charge a credit card through Stripe, or to send a transactional email through SendGrid, or to execute an upsert against an external Postgres database?

Host 2

This is the critical operational boundary that marketing literature completely ignores. The mathematical guarantees of exactly once semantics terminate precisely at the network boundary of the Kafka brokers.

Host 1

Because the transaction coordinator cannot reach across the internet.

Host 2

Exactly. If the stream processor initiates a transaction, calls the Stripe API to charge the user's credit card, and Stripe returns a 200 okay, but the Flink container crashes a microsecond later before it can commit the Kafka transaction, the system is in an inconsistent state.

Host 1

When Flink restarts, it will replay the uncommitted event and it will execute the Stripe API call a second time.

Host 2

And the Kafka transaction coordinator possesses no mechanism to issue a rollback command to Stripe's financial infrastructure. External side effects fundamentally operate under at least once delivery semantics.

Host 1

We established strict rules for this exact scenario in part six and seven of our web architecture series. When mutating external syncs, engineers cannot rely on the messaging framework for protection. You must engineer the external systems to be item potent.

Host 2

Absolutely. If you are interacting with Stripe, the engineering team must generate a deterministic item potency key. For example, a cryptographic hash of the Kafka topic, partition, and message offset, and inject that key into the HTTP header of the request.

Host 1

So Stripe's edge infrastructure will cache that key. If Flink crashes and retries the request, Stripe will recognize the duplicate key and return the previous success response without hitting the credit card a second time.

Host 2

Exactly. Similarly, if the stream processor is pushing data into a Postgres materialized view, the SQL statements cannot be simple insert commands.

Host 1

They must be designed as insert on conflict due update, an upsert operation.

Host 2

Ensuring that applying the exact same event multiple times results in the exact same final row state.

Host 1

Exactly once processing is mathematically sound, but only as long as you reside within the walled garden of the cluster. The moment you traverse the network to an external sync, you must manually engineer defenses against retries.

Host 2

It's mandatory.

Host 1

Let's transition from theoretical semantics to physical production failures. We really need to dissect the Debezium PG output failure from the cold open.

Host 2

The one that took down the entire e-commerce platform?

Host 1

Yes. The scenario was a consumer died. The Debesium connector stalled, and it eventually triggered a 100% disk utilization crash on the primary Postgres database. Let's trace the mechanics of that failure.

Host 2

The root cause lies in the contractual obligations of Postgres replication slots.

Host 1

Because Debezium utilizes a logical replication slot to tail the right-ahead log.

Host 2

Right. And the fundamental architectural contract of a replication slot is absolute preservation. Postgres guarantees that it will never, under any circumstances, delete or recycle a wall segment file if any registered replication slot still requires it to stream changes.

Host 1

Postgres literally pins the files on the storage volume until the downstream consumer explicitly acknowledges receipt.

Host 2

Exactly. The Debezium connector continuously transmits a log sequence number or LSN back to the database. This LSN acts as a high watermark.

Host 1

Essentially stating, I have successfully streamed and processed all binary changes up to this exact byte offset.

Host 2

Right. Upon receiving this LSN update, Postgres consults its internal state or recognizes that no other replication slots need the older data and safely garbage collects the obsolete WAL segments to free up disk space.

Host 1

However, if the downstream Kafka Connect cluster loses network connectivity, or if Debezium crashes due to an unhandled exception, it ceases transmitting those LSN acknowledgements.

Host 2

And Postgres adheres strictly to its contract. It assumes Debezium is temporarily disconnected and will return. So it begins hoarding every single WAL file generated by the continuous production traffic.

Host 1

Which on a heavily utilized database executing thousands of transactions per second means those unrecycled WAL files will consume gigabytes of storage capacity extraordinarily fast.

Host 2

Very fast.

Host 1

If the infrastructure lacks proper alerting, the primary volume hits 100% utilization, and the Postgres engine panics and shuts down immediately to protect the database pages from corruption. To prevent this catastrophic failure, database administrators must implement strict infrastructure monitoring against the PG replication slot system view. And if that lag exceeds a defined threshold, or if the disk trajectory indicates rapid filling, an aggressive pager alert must fire.

Host 2

But you know, there is a more insidious silent variant of this replication slot bug that occurs even when the network is perfectly healthy.

Host 1

Oh, the low traffic schema edge case.

Host 2

Yes. What happens in a multi-tenant database environment where Debezium is only capturing a very low traffic table?

Host 1

So consider a Postgres instance hosting multiple databases or schemas. Debezium is configured to capture changes exclusively from a configuration table that updates perhaps once a week.

Host 2

BZM's architecture dictates that it only transmits an LSN acknowledgement back to Postgres after it successfully processes an event.

Host 1

If no rows change in the configuration table for seven days, Debezium has no events to process, and therefore it sends no LSN updates to the database.

Host 2

However, the write-ahead log and Postgres QL is instance-wide. It is shared globally across all schemas.

Host 1

So high volume transactional traffic in completely unrelated schemas is rapidly generating gigabytes of wall files.

Host 2

And because Postgres has not received an updated LSN from the idle Debesium slot, it refuses to delete the globally shared wall files. It's waiting for Debesium to acknowledge a change that isn't coming.

Host 1

How do engineers mitigate that?

Host 2

The mitigation requires forcing Debesium to transmit its LSN continuously, regardless of table activity. You configure a specific property in the connector called heartbeat.action.query.

Host 1

Okay, so the engineering team creates a dummy table in the database and configures Debesium to execute a physical update on a row in that table at regular intervals, for instance, every 10 seconds.

Host 2

Right. This artificial write generates a physical wall record that Debesium is configured to capture.

Host 1

It processes the heartbeat event and immediately transmits the current LSN back to Postgres.

Host 2

This acts as a pulse, explicitly signaling to the database engine the connector is alive and we have advanced past this global byte offset. The database can then safely garbage collect the shared wall segments.

Host 1

That's a crucial configuration. Now let's examine another devastating distributed systems failure mode, the poison pill, and how it triggers cascading consumer lag.

Host 2

A poison pill is an event residing on the Kafka partition that is structurally or schematically invalid.

Host 1

A rogue upstream producer might bypass the schema registry and inject a malformed JSON string or a corrupted Avro binary payload.

Host 2

Right. When the stream processor consumer pulls that batch of messages and attempts to deserialize the payload, the parsing library encounters the corruption and throws a fatal exception.

Host 1

In a batch processing paradigm, a failure might just fail the batch. But in stream processing, we cannot simply skip the corrupt record and move forward.

Host 2

No, because if the framework automatically swallowed the error and advanced the offset, it might be discarding a critical financial transaction or a legitimate security audit event. Skipping equates to silent data loss.

Host 1

Because the default behavior of most frameworks prioritizes zero data loss, the system allows the exception to bubble up, crashing the specific consumer thread or task.

Host 2

And the cluster orchestrator detects the failure and restarts the task. It hits the exact same poison pill, throws the exact same exception and crashes again. And while it is trapped in this cycle, millions of healthy events are queuing up behind the poison pill on that specific partition. Consumer lag spikes exponentially.

Host 1

Because Kafka guarantees ordering within a partition, any customer whose user ID hashes to that stalled partition is completely locked out of the downstream systems until the issue is resolved.

Host 2

Exactly. To break the cycle without losing data or halting the pipeline, engineers must implement a dead letter cue.

Host 1

Right. You configure the consumer framework's deserialization logic to catch the parsing exceptions.

Host 2

When a poison pill is encountered, instead of crashing the thread, the application wraps the raw, unparsed binary payload along with the stack trace and offset metadata.

Host 1

And publishes that entire package to a dedicated separate Kafka topic, the dead letter cue or DLQ.

Host 2

After successfully publishing to the DLQ, the consumer acknowledges the offset of the bad message and continues processing the healthy records behind it.

Host 1

The engineering team can then monitor the DLQ, manually inspect the corrupted bytes to determine why the upstream producer violated the schema contract and eventually replay the corrective messages back into the main pipeline.

Host 2

It's the only safe way to handle it.

Host 1

Let's tackle one final operational hurdle before we close. We explored the Chandy Lamport algorithm and how Flink injects checkpoint barriers to capture a state snapshot. What happens to that snapshot process when a downstream system bottlenecks and triggers severe TCP back pressure?

Host 2

TCP backpressure is a fundamental flow control mechanism.

Host 1

If a Flink operator is performing a complex external database join and begins to slow down, its incoming network buffers fill up.

Host 2

The TCP window size drops to zero, signaling the upstream Flink operators to stop transmitting data. The network pipes physically clog, and the data velocity grinds to a halt.

Host 1

And because the checkpoint barriers flow sequentially in line with the data, if the data stops moving, the barriers stop moving.

Host 2

Exactly. The barriers become trapped in the network buffers. And because the operators cannot receive the barriers, they cannot initiate their local snapshots.

Host 1

The checkpoints begin timing out and failing.

Host 2

And if the global checkpoint fails, the system loses its recovery points. If a catastrophic node failure occurs during a period of heavy back pressure, the cluster might have to rewind its state to a checkpoint from three hours ago rather than three minutes ago.

Host 1

Which compounds the crisis because restarting with a three-hour backlog of data creates an even worse back pressure scenario. How does the architecture prevent the barriers from getting stuck in traffic?

Host 2

Flink engineered a solution known as unaligned checkpoints. When this mode is enabled, the system does not force the checkpoint barrier to wait sequentially in the TCP buffers.

Host 1

Instead, the barrier is allowed to overtake the buffered data, essentially jumping to the front of the network queue to reach the operator immediately.

Host 2

Right.

Host 1

But wait. If the barrier bypasses the data, the fundamental premise of the Chandelamport snapshot is broken. The data that was logically positioned in front of the barrier, and therefore should have been processed and included in the snapshot is now trapped in the network buffers behind the operator.

Host 2

To maintain mathematical consistency, Flink expands the scope of the snapshot. When the barrier overtakes the data, the operator doesn't just snapshot its internal ROXDB state.

Host 1

It actively captures all the raw, unprocessed data residing in the input and output network buffers and serializes those raw buffers to S3 as part of the state snapshot.

Host 2

The system is literally persisting the traffic jam itself to disk.

Host 1

That is wild.

Host 2

It increases the byte size of the checkpoint payloads significantly, I imagine.

Host 1

It does, but it entirely decouples the snapshot duration from the throughput velocity of the pipeline. Even under crushing back pressure, the barriers propagate instantly. The network buffers are dumped to S3, and the checkpoints complete in milliseconds. Guaranteeing the system always maintains a recent, safe recovery point.

Host 2

Exactly.

Host 1

We have dismantled the internal mechanics of the append-only commit log, trace the complexities of change data capture, and navigated the operational cliffs of stream processing. To conclude this deep dive, we need to distill these architectures into three concrete engineering rules for the architectural review board.

Host 2

Let's do it. Rule number one: never execute dual rights in application code.

Host 1

Wrapping a database commit and a network publish within a tri-catch block guarantees race conditions.

Host 2

And partial failures and silent data corruption across your distributed systems.

Host 1

If your architecture demands the synchronization of primary databases with derived stores like search indexes or caches, mandate the use of the transactional outbox pattern paired with a robust CDC tool like Debesium.

Host 2

Force the database's right-ahead log to act as the single source of truth.

Host 1

Always.

Host 2

Rule number two: avoid full event sourcing for standard cred applications.

Host 1

Right. Well, treating the event log as the primary source of truth offers theoretical purity, the operational burden of maintaining state snapshots, and the crippling technical debt introduced by schema upcasting when business models evolve rarely justify the cost.

Host 2

Exactly. Reserve full event sourcing architectures for strict temporal or audit-heavy domains, financial ledgers, medical records, or shopping cart histories.

Host 1

Places where the chronological sequence of how a state was reached provides tangible business value over simply knowing the current state.

Host 2

Keep the relational database as the primary state store for standard applications. And rule number three: size your Kafka partition counts with deliberate mathematical intent.

Host 1

The partition count dictates a permanent architectural trade-off.

Host 2

Yes. Fewer partitions enforce wider, stricter key ordering across a broader set of entities, but artificially limit your maximum downstream consumer parallelism.

Host 1

And expanding the partition count unlocks massive throughput and scaling potential, but drastically increases the risk of subtle ordering bugs if the hashing keys are not designed to encapsulate causal boundaries.

Host 2

Calculate your network throughput and processing latency requirements before establishing the topic topology.

Host 1

Before we wrap up part four of our deep dive series, I want to leave you, the listener, with a provocative architectural thought. We've seen how frameworks like Flank and Kafka streams have become extraordinarily robust at maintaining materialized views in local ROX DB instances.

Host 2

We've seen how they reconstruct perfectly consistent state from the change log.

Host 1

So, at what point in the evolution of this technology does the stream processor stop being a secondary analytics tool and actually become the primary database?

Host 2

It represents a fundamental blurring of the boundary between compute and storage.

Host 1

If the stream processor handles the distributed rights via the commit log and locally serves sub-millisecond reads from its embedded LSM trees, do we eventually render traditional relational database management systems obsolete for massive read-heavy workloads?

Host 2

Are we simply witnessing the database being turned inside out?

Host 1

It's a fascinating paradigm shift to consider. Well, we have spent this session mastering the mechanics of streaming, capturing, and aggregating operational data in real time. But the next frontier for distributed architectures is figuring out how to feed this rich, constantly mutating context into AI systems.

Host 2

How do you prevent a large language model from hallucinating by grounding it with up to the millisecond streaming facts?

Host 1

Looking ahead to our next deep dive, we will be tackling vector databases, focusing extensively on engineering rag architectures and embedding pipelines with live data streams. Until then, keep a close eye on your replication slots and ensure your checkpoints will remain aligned.

Save to playlist

Manage playlists →

Share