Idempotency & "Exactly-Once" Delivery
Podcast · Idempotency & "Exactly-Once" Delivery

The Myth of Exactly Once Delivery

by ZumiLabs · 55m
0:00 55m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

So imagine this. You are in the back of an Uber, you're running like incredibly late for a flight, the city traffic is a total nightmare, and you are just staring at your phone screen with white knuckles.

Host 2

Oh yeah. The classic panic mode.

Host 1

Right. And you finally reach this tunnel, and right as the car plunges into the absolute darkness of the underpass, you tap confirm order for this massive non-refundable purchase.

Host 2

Naturally. Worst possible timing.

Host 1

Exactly. Instantly, the bars on your cell signal drop to absolute zero. The little loading spinner on the app does its, you know, anxious looping dance for about thirty seconds. And then the screen flashes, network error, request timed out.

Host 2

Ugh. That is a uniquely modern feeling of dread right there. Just that exact moment of complete suspension.

Host 1

It is terrifying. Because right there in the dark of that tunnel, you are trapped in a state of unresolvable uncertainty. You're sitting there wondering, like, did that tiny network packet die on a cell tower somewhere between my phone and the server?

Host 2

Right.

Host 1

Or did the payment gateway already capture my money and it was only the confirmation receipt that got lost on its way back to my phone. You simply cannot know.

Host 2

You really can't. And you know, this isn't just some frustrating user experience design flaw. What you're describing sitting in that Uber is actually a physical real-world manifestation of one of the most famous unsolvable paradoxes in computer science. Wait, really? Yeah. It's a flawless everyday example of the two generals problem.

Host 1

Oh, I love when our mundane everyday anxieties have a proper dramatic academic name. Break down the two generals problem for us, because I know this ties directly into the Tyler Treaton analysis of distributed systems from our reading stack today.

Host 2

For sure. So the two generals problem is this foundational thought experiment. Imagine two Allied armies commanded by two generals, and they are preparing to attack a fortified city.

Host 1

Okay. I'm picturing it.

Host 2

They're camped on opposite hills. Now, to win, they have to attack at the exact same time. If only one attacks, their single army will just be slaughtered.

Host 1

So extremely high stakes.

Host 2

It just like hitting confirm on a non-refundable flight upgrade.

Host 1

The stakes are literally everything.

Host 2

So the only way they can communicate to coordinate this attack time is by sending human messengers through the valley that separates them. But that valley happens to be crawling with enemy troops who might capture the messenger.

Host 1

Right, so the network is hostile.

Host 2

Exactly. The valley is the internet. So General A sends a messenger down into the valley saying, let's attack at dawn. Okay. If General B actually gets that message, General B now has to send a messenger back saying, I received your message, we will attack at dawn.

Host 1

Makes sense.

Host 2

But General B immediately realizes a fatal flaw in the plan. What if my acknowledgement messenger is captured on the way back? General A won't attack, but B will, and B's army gets destroyed.

Host 1

Wow. So General B needs an acknowledgement of the acknowledgement. Like General A has to confirm they got the receipt.

Host 2

Precisely. So General A sends a third messenger back saying, I got your acknowledgement. But what if that third messenger gets captured?

Host 1

Then A thinks everything is fine and prepares to attack, but B is paralyzed with doubt, just waiting for the third messenger.

Host 2

Yes. And the mathematical proof here, and it is an absolute ironclad logical proof established decades ago, is that no finite chain of acknowledgments over an unreliable link can ever create shared certainty.

Host 1

Wait, no finite chain.

Host 2

None. If you send 100 messengers, you still need the 101st to confirm the hundredth arrived safely. You can never truly know if the server got your message unless you get the receipt. And the server can never know if you got the receipt.

Host 1

It is the ultimate cliffhanger.

Host 2

Yeah.

Host 1

And honestly, that brings us perfectly to the mission for this deep dive. Because in our last deep dive, SO8EO1 on randomness and entropy, we saw exactly how operating system cryptographic random number generators meant those beautiful collision-free random UUIDs.

Host 2

Right, the V4 UUIDs.

Host 1

Yeah. So today in episode SO8E02, item potency and exactly once delivery, we are going to explore why those exact, perfectly random identifiers are literally the only thing standing between your flaky Wi-Fi connection and a double charge credit card.

Host 2

That is the mission. We are completely dismantling the myth of exactly once delivery today.

Host 1

I'm ready.

Host 2

We are unpacking the architectural patterns, the deep database constraints, and the state machines that actually save us from complete network chaos. And just as a quick reminder to you listening, we are setting some strict ground rules for this deep dive.

Host 1

Yes, no holding back.

Host 2

We are skipping the basics. We're not re-explaining rest semantics from scratch. We are assuming you already understand UUID entropy from the last deep dive and you know what a basic database transaction is. We are jumping straight into the deep end of distributed systems and edge case disasters.

Host 1

Which is exactly where we like to be. So to start dismantling this myth, let's talk about the marketing.

Host 2

Oh boy, the marketing.

Host 1

Yeah. Because if you browse the landing pages of various tech platforms, enterprise message brokers, cloud providers, you will see these bold, massive fonts promising exactly once network delivery.

Host 2

Yeah, it's everywhere.

Host 1

It sounds so comforting. It sounds like they somehow solved the two generals problem and cured network partitions.

Host 2

I mean, it is entirely a mirage. To understand why those marketing pages are selling a mathematical impossibility, we have to look at the actual physics of distributed systems.

Host 1

Physics.

Host 2

Yeah, I mean physics. Almost literally the constraints of space, time, speed of light delays, independent hardware failures. We are drawing heavy insights here from Tyler Treat's essay You Cannot Have Exactly Once Delivery. Pat Helen's brilliantly titled paper, Item Potence is not a medical condition.

Host 1

Such a great title.

Host 2

Right. And also the rigorous work of Malcolm Fettenby and Mark Brooker over at the AWS Builders Library. They all point back to the same foundational truth, which is the FLP impossibility result.

Host 1

The FLP result.

Host 2

Yep.

Host 1

This is Fisher Lynch and Patterson from their landmark paper in the 1980s. But like what does impossibility actually mean in this context? Because my bank seems to process my transfers correctly most of the time.

Host 2

Yeah, most of the time being the operative phrase there. In 1985, Fisher Lynch and Patterson published a mathematical proof demonstrating that in a purely asynchronous distributed system where messages can be delayed arbitrarily by network congestion and even just one single process might crash or pause. It is mathematically impossible for the system to reliably agree on a decision in a bounded amount of time.

Host 1

So no guarantees.

Host 2

None. It's impossible to achieve perfect consensus with absolute certainty if you are dealing with independent nodes that can fail silently. When you translate that FLP reality to sending data over a network, it means you practically only have two choices when you send a message over a wire.

Host 1

Okay, let's look at the menu. What are my actual options when I tap that confirm order button in the Uber?

Host 2

Optional one is at most once delivery.

Host 1

Okay.

Host 2

This means your phone fires the network packet off into the void through the cell tower, and you just hope it arrives. You never retry. The primary benefit of at most once is that you will never ever double charge the customer.

Host 1

Because you only ever send it once.

Host 2

Exactly. You'll never accidentally trigger the action twice. But the massive fatal risk is data loss. The message drops because a router hiccups, the order is never placed, and the user is left waiting for a confirmation screen that will never load.

Host 1

I mean, that sounds terrible for business. You're just dropping revenue on the floor because a cell tower blinked. No commercial application can survive if they just shrug when a packet drops.

Host 2

They really can't.

Host 1

Yeah.

Host 2

Which is why almost no commercial system uses at most once for critical financial transactions or state mutations. So that leaves the entire industry with option two, which is at least once delivery.

Host 1

At least once.

Host 2

Right. This means the client, your phone, your browser, or a back end microservice, keep sending the message, retrying over and over again until it successfully receives a definitive acknowledgement from the server.

Host 1

You know, I've always visualized at least once delivery, like trying to communicate by shouting across an incredibly crowded, noisy room.

Host 2

Well, like that.

Host 1

Yeah, like let's say I need to tell you to order a pizza for the group. I yell across the room, order the pizza. But the music is blaring. I can't see your face to know if you heard me.

Host 2

Right. I'm facing the other way.

Host 1

Exactly. So a few seconds later, I cut my hands and yell it again. Order the pizza. I am gonna keep yelling it until I see you physically turn around and give me a thumbs up.

Host 2

Yeah.

Host 1

But because of the delay and the noise, you might have actually heard me three separate times before you finally had the chance to turn around and signal back.

Host 2

That is a brilliant analogy, honestly, because it highlights the exact trade-off. You guaranteed that I heard the message at least once. But you introduce a new critical danger, which is duplication. If I am not incredibly smart about how I process your shouts, I might turn to my phone and order three separate pizzas.

Host 1

Which would be great for us, but bad for my wallet.

Host 2

Right. In the digital world, that means draining a customer's bank account with three identical charges. If we accept that the physics of the network can only ever give us at least once delivery, then the burden of preventing duplicates shifts entirely away from the network and onto the server's application logic.

Host 1

The server has to figure it out.

Host 2

And the server has to be smart enough to realize that those three shouts were all part of the exact same intent.

Host 1

And that concept, the ability for the server to recognize the intent and safely absorb those duplicated shouts without blowing up the state of the system. That is the definition of idopotency.

Host 2

That's the definition. Which brings us to the official HTTP semantics. Because we really need to define what item potency actually means in the context of an API. It is one of the most widely misunderstood terms in web development.

Host 1

It really is.

Host 2

If we look at RFC 9110, which officially governs HTTP semantics, it defines an item potent method not by what the server returns to the client, but by the intended effect on the server's state.

Host 1

Let's use a vivid mental model, like the classic record 42 scenario.

Host 2

Let's do it.

Host 1

Let's say I send an HTTP delete request to a back-end API to remove a user account, user ID42. The request travels over the wire, the server successfully deletes record 42 from its database, and the server sends back a 200 okay status code.

Host 2

A perfectly successful textbook interaction.

Host 1

But wait, I'm in that Uber tunnel again.

Host 2

Uh-oh.

Host 1

The 200 OK response packet gets dropped by the failing cell tower. My phone's network library hits a 10-second timeout. Because we are operating in an at least once paradigm, my phone's retry helper automatically fires that exact same delete request a second time.

Host 2

Okay, so let's track the execution here. What happens on the server when that second request hits?

Host 1

Well, the server gets the second delete request for user ID42. It looks in the database to find record 42, but it's already gone. We literally just deleted it a second ago.

Host 2

Right.

Host 1

So the database returns nothing, and the server replies to my phone with a 404 not found. My phone finally gets this response out of the tunnel and sees an error. This is where the confusion happens. The first request on the server side returned a 200 OK. The retry returned a 404 not found.

Host 2

Yep.

Host 1

The HTTP responses were completely different. So the big question, is that operation still itempotent?

Host 2

Yes, it is perfectly itempotent.

Host 1

Even with the 404.

Host 2

Even with the 404. This is a crucial fundamental mental shift that developers have to make. Item potency does not mean the server's response code is exactly the same every time. It means the end state of the universe, or at least the database is the same.

Host 1

The state of the universe.

Host 2

Right. The intended effect of your request was record 42 must not exist. After the first request, record 42 did not exist. After the second request, record 42 still did not exist. The state on the server is identical, regardless of whether you called that delete method once or a thousand times.

Host 1

That makes so much sense.

Host 2

The 404 not found is just the server accurately reporting the current weather, so to speak. But the state mutation, the actual deletion of data only happened once.

Host 1

It's about the state of the data, not the status code of the HTTP response. If the house is already demolished, sending a second demolition crew who reports back, there's no house here. Doesn't change the fact that the intent was fulfilled.

Host 2

Exactly.

Host 1

And this concept of protecting state brings up another tool heavily mentioned in the RFC, which is optimistic locking.

Host 2

Optimistic locking is a vital safeguard, especially when you have multiple independent clients trying to mutate the exact same state concurrently. The RFC details conditional requests using headers like if match.

Host 1

How does that work?

Host 2

The mechanism works like this. You tag a resource on the server with a version number or an entity tag, an e-tag. When a client wants to update that resource, it sends a payload saying, I want to change this data, but only if the current version on the server matches the version I am looking at right now.

Host 1

Oh, so it prevents the classic lost update problem. Let's use like a wiki page analogy.

Host 2

Perfect.

Host 1

You and I are both editing a wiki page at the exact same time. I download version one of the page, I spend 10 minutes making my edits, and I hit save. The server updates it, increments the tag, and it is now version two.

Host 2

Yep.

Host 1

If you also downloaded version one, spent 15 minutes making completely different edits, and then you hit save, what happens? Your HTTP request will have an if match version one header. But the server sees the database is now sitting in version two.

Host 2

Exactly. The server physically compares the e-tags. It pulls the current row from the database, looks at the version column, sees a two, compares it to the one in your header, and realizes a collision has occurred. It violently rejects your request with a 412 preconditioned failed status code.

Host 1

I love the 412 preconditioned failed. It is the server forcing you to fetch the latest state, read my changes, and manually merge them, rather than allowing your delayed request to silently overwrite 10 minutes of my work.

Host 2

It is a phenomenal mechanism.

Host 1

It essentially turns a standard mutation into a conditional item potent operation based on a known starting state. But, and this is a big butt. Optimistic locking is strictly used for updating existing resources.

Host 2

Right.

Host 1

You need something to exist so you can put an e-tag on it. What happens when we are creating something entirely new out of thin air, like a brand new financial charge or a new ride share booking?

Host 2

There's nothing to check.

Host 1

There's nothing to check.

Host 2

Exactly. There is no existing resource. There is no version one.

Host 1

And this is where we graduate from basic HTTP REST semantics and step into the big leagues of distributed architecture. We need to talk about the Stripe Item Potency Key Architecture.

Host 2

Yes.

Host 1

This is based on a truly phenomenal deep dive by Brandur Leach, who worked on Stripe's core API experience. And it also directly aligns with the IETF draft specifications on the item potency key HTTP header.

Host 2

The core problem Stripe had to solve is intent. If you send a post request to a charges endpoint for $20, and your phone's connection drops, and your phone retries that exact same post request, how does the Stripe backend know the second request is a retry of the first $20 charge, and not a legitimate brand new request from you wanting to charge your card $20 again?

Host 1

Because I might actually want to buy two identical $20 items in a row. Like maybe I'm buying two movie tickets one after the other. The server cannot just blindly guess based on the dollar amount.

Host 2

The server cannot guess. The client has to explicitly state its intent. And the client does this by generating a unique identifier. This is where the UUID V4 from our last episode comes in, and sending it in an HTTP header called item potency key. The IETF draft specifies that this key must be strictly scoped to the user's logical checkout intent.

Host 1

Okay. Let's paint a detailed spoken picture of how the database physically handles this. Because Brander's Postgres model is just an absolute masterclass in backend design. We want to avoid reading raw SQL syntax here. So I want you to imagine we are looking at the schema for a Postgres table named Item Potency Keys.

Host 2

Okay, I have the mental whiteboard ready.

Host 1

First, every single row in this table needs to represent a unique intent. So the absolute most critical part of this table is a compound unique constraint. We lock down two columns together at the database level, the user ID and the item potency key.

Host 2

Right. And I want to pause here and explain why that is a brilliant structural decision. By making the unique constraint a combination of both the user and the key, rather than just the key alone, you solve two massive problems simultaneously.

Host 1

Which are?

Host 2

First, it prevents cross-user collisions. If by some astronomical chance, or more likely a terrible bug in a random number generator, two completely different users generate the exact same UUID, the database will not reject the second one. The combination of user A plus UUID is different from user B plus UUID.

Host 1

Oh, that's smart.

Host 2

Second, it ensures that a single user cannot reuse an old key for a new distinct action.

Host 1

But how does Postgres actually enforce that? When I say unique constraint, what does the database actually doing under the hood to guarantee that?

Host 2

When you define a unique constraint, Postgres automatically creates a B tree index on those specific columns behind the scenes. When a new insert request comes in, the database traverses that B tree index from the root node down to the leaf nodes.

Host 1

It physically searches.

Host 2

It physically searches the stored data on disk and in memory to see if that exact combination of user ID and item potency key already exists. If it finds a match, the database core violently rejects the insert operation before it ever writes the main table heap. It throws a constraint violation error back to the application. The database's atomic uniqueness guarantee becomes our primary unbreakable shield against duplicates.

Host 1

So the flow is like I tap confirm order, my phone generates a UUID, sends the POST request, and the very first thing the back end API does is attempt to insert that UUID and my user ID into the item potency keys table.

Host 2

Yes.

Host 1

If the database accepts the insert, the back end knows it has the right of way to process the charge. But here's where a really nasty edge case pops up. What if a user or a buggy client application submits the exact same UUID but modifies the payload of the request?

Host 2

Give me a scenario.

Host 1

Okay. Let's say the first request was for a $10 shopping cart. The network drops, the request fails. While offline, the user adds another item to their cart, bringing the total to $100.

Host 2

Okay.

Host 1

The app comes back online and the buggy client retries the network request, but it accidentally uses the same original item potency key for the new $100 payload.

Host 2

Ouch. This is what backend engineers call the 422 trap. If the server just blindly trusted the item potency key without verifying the context, it would look at the database, say, I recognize this key. I already processed this, and return the cached success response for the $10 charge.

Host 1

Wait, so the user thinks they bought $100 of stuff.

Host 2

The client thinks it successfully bought $100 worth of goods, but the backend only charged them $10. Or worse, the back end processes the $100 charge but associates it with the $10 key's lifecycle, deeply corrupting the financial state of the application.

Host 1

That sounds like a fast way to lose a lot of money. So how do enterprise systems like Stripe or tools like AWS Lambda Power Tools actually solve the 422 trap?

Host 2

They hash the request payload. When that very first request comes in before it does anything else, the server takes the entire JSON body. The dollar amount, the currency code, the item IDs, everything, and runs it through a cryptographic hash function, like SHA256.

Host 1

Oh wow.

Host 2

It takes that resulting hash string and saves it in the item potency keys table right next to the UID. So it creates a cryptographic fingerprint of the exact intent. When the delayed retry comes in with the modified $100 cart, exactly. Exactly. The database says, I recognize this UUID, but let me pull the hash of the payload you sent last time and compare it to the hash of the payload you are sending me right now. The hashes do not match.

Host 1

Boom.

Host 2

The server immediately throws its hands up and fails loud. It returns a 422 unprocessable content error. It effectively tells the client you are violating the API contract. You cannot change the terms of a request while attempting to use the same item potency key. Generate a new key for this new intent.

Host 1

That is so elegant. It perfectly enforces the rule that a single key represents a single immutable intent. Okay, so that handles a delayed retry with modified data. But what about concurrent duplicates?

Host 2

Like racing conditions.

Host 1

Yeah. Like I'm impatient, my app is lagging, and I literally mash the pay button three times in half a second. Three identical network requests launch from my phone at the exact same millisecond and hit the back end load balancer simultaneously.

Host 2

This is a classic race condition, and it's where the locked at state comes in. In our mental Postgres table, we have a timestamp column called locked at. When those three identical requests hit the back end simultaneously, they all try to insert or update that specific row.

Host 1

Okay.

Host 2

But because we wrapped this initial check in an atomic serialized database transaction, Postgres acts as the ultimate bouncer.

Host 1

I'm going to drill down into that. How does Postgres act as a bouncer? If three requests hit the CPU at the exact same time, how does the software decide who wins?

Host 2

It comes down to row level locks and internal page latches. When a transaction attempts to modify a row, often using select for update command, the database engine physically flags that row in memory. It tells the CPU this specific memory address is locked by transaction A.

Host 1

So it's completely claimed.

Host 2

Yes. When transaction B and transaction C arrive a microsecond later and try to touch that same row, the database engine forces their execution threads to pause. They literally have to wait in a queue. Only one of those requests wins the race to claim the lock and set the locked at timestamp.

Host 1

What happens to the losers? Do they just wait forever?

Host 2

No, and you don't want them to wait because that ties up resources. In a well-designed item potency system, the other two requests see that the row exists and that the locked at timestamp is actively set. This means another process is currently working on this exact key.

Host 1

Right, someone else is on it.

Host 2

Instead of waiting around or trying to execute the charge in parallel, they immediately abort their transactions and return a 409 conflict status code to the client.

Host 1

A 409 conflict? This is essentially the server saying, I'm already working on this exact problem right now. Please back off.

Host 2

Precisely. It forces the client to back off, usually implementing an exponential jittered delay algorithm and try again a few seconds later. When the client eventually retries, the first request will likely have finished. The locked at flag will be cleared, the final HTTP response will be saved in the database, and the server can safely return that cached final result to the client without ever touching the payment gateway a second time.

Host 1

Okay. I want to talk about one more crucial column in this brilliant Postgres schema, the recovery point column. I like to think of this as a video game save point.

Host 2

That's an excellent way to visualize it. Walk us through how a save point applies to an API request.

Host 1

So a complex API endpoint isn't just one single action. It is a multi-step state machine. If I'm booking a ride share, the back end has to do several things in sequence. Step one, create a ride record in the local database. Step two, call the payment gateway over the network to authorize the credit card. Step three, query a different microservice to assign a driver.

Host 2

Each taking time, yeah.

Host 1

Each of these steps takes time. If the back end server's power cord gets abruptly unplugged right after step one finishes, what happens?

Host 2

Well, historically, without recovery points, the system would be left in an inconsistent, corrupted state. When the client's phone retries the request, a brand new server picks it up. It might try to execute step one again, creating the ride record, fail because the database says the write already exists and completely blow up.

Host 1

Or what it just starts over?

Host 2

Or it might just start entirely from scratch, yeah, losing the context of the original attempt.

Host 1

But with the recovery point column, every time a logical step finishes, we run a quick update on the item potency keys table. We save the game. We change the status from started to ride created. Then after the payment successfully hits, we change it to charge authorized.

Host 2

Right. So let's revisit the server crash. The power cord is pulled after step one. The user's phone inevitably retries the network request with the exact same item potency key. A new server process picks up the request, queries the database for that key, and says, I see the recovery point is set to ride created. I will completely skip step one, load the game from ride created, and proceed directly to step two, calling the payment gateway.

Host 1

It just resumes.

Host 2

Yes. The impossible transaction. This is where many naive distributed architectures go to die.

Host 1

Let's set up the dilemma. You have to save a user's order in your local Postgres database. But you also have to send a message to a distributed message broker, like Apache Kafka, to tell the completely separate fulfillment microservice to actually put the item in a box and ship it.

Host 2

Right. Two completely separate systems. Postgres on one server, Kafka on a cluster of different servers.

Host 1

Exactly. And you cannot wrap a database commit and a network call to Kafka in a single ACID transaction. I want to pause here actually. We use the term ACID all the time. Atomicity, consistency, isolation, durability. But what does it really mean in this context?

Host 2

Atomicity is the key here. It means all or nothing. In a local database, you can update five different tables, and if anything goes wrong, the database rolls back all the changes as if they never happened.

Host 1

It's a safety net.

Host 2

A local one, yes. But you cannot extend that atomicity across a network to a completely different system like Kafka. It defies the laws of physics and the FLP impossibility result we talked about earlier. If you commit the data to Postgres first, and then the network call to Kafka fails, your database thinks the order is placed, but the fulfillment center never hears about it.

Host 1

The user gets charged, but the physical box never ships.

Host 2

Exactly.

Host 1

And what if you try to reverse the order? Like if you send the message to Kafka first and then attempt to commit the local database transaction, but the database disk is full and it fails.

Host 2

Then the fulfillment center receives a valid message and ships a box for an order that technically does not exist in your primary database. You have just given away free inventory, and your finance team will have a nightmare trying to reconcile the books. Your system is fundamentally irreparably out of sync. You have a dual right problem.

Host 1

Why doesn't that solve it?

Host 2

So two-phase commit or two PC is a protocol where a coordinator asks both systems to prepare to commit, and only if both say yes, it tells both to commit. The problem is that it requires holding locks across the network.

Host 1

Which we know is bad.

Host 2

Terribly bad. If the coordinator crashes mid-commit, your database tables are locked indefinitely. It destroys performance and availability. Modern microservices entirely reject 2PC across network boundaries for this reason.

Host 1

So if we can't do a two-phase commit, what is the industry standard solution?

Host 2

This is a profound architectural pain point. And the solution is a pattern called the transactional outbox paired with an item potent inbox. We're drawing heavily from the microservices.io patterns established by Chris Richardson and the official documentation for Debezium here.

Host 1

Okay, let's break down the transactional outbox. How does it fix the impossible transaction?

Host 2

By entirely removing the network call to Kafka from the critical path of the user's request. Instead of trying to talk to Postgres and Kafka at the same time, your application only talks to Postgres. You do what Postgres is phenomenally good at, which is a local atomic transaction. In the exact same transaction where you save the order details to the orders table, you also construct the JSON message payload you intend to send to Kafka, and you insert that payload into a separate dedicated table called the Outbox table.

Host 1

So it's an all or nothing local database commit. Either the order data and the intent to send a message are both saved to disk simultaneously, or neither is. There's absolutely zero chance of them being out of sync.

Host 2

Exactly. The application finishes its job, commits the transaction, and immediately returns a success response to the user. The database is perfectly consistent. Then a completely separate asynchronous background process takes over. This is usually a change data capture system or CDC, like Debesium. Debezium acts as a relentless message relay.

Host 1

How does Debezium actually know a new message is there? Is it just constantly running a select query on the outbox? Because that sounds like a polling nightmare.

Host 2

No, and that's the beauty of it. Postgres maintains an internal file called the write ahead log or WAL. Physically, this is an append-only binary file on the database's hard drive where Postgres records every single change before it even updates the main tables.

Host 1

Oh, that's clever.

Host 2

Yeah. Debezium connects to Postgres and pretends to be a replica database. It physically reads that binary wall stream in real time. When it sees a transaction that inserted a row into the outbox table, it extracts that row. And Debezium handles the messy business of sending it over the network to Kafka. If the network drops, Debezium just keeps retrying. It guarantees at least once delivery to the broker.

Host 1

I love that. Yeah. It's like a mail carrier is sitting at your desk watching you put a letter in your physical outbox tray and immediately grabbing it to take it to the post office.

Host 2

Great analogy.

Host 1

But wait a minute. If Debezium guarantees at least once delivery to Kafka, that means it might accidentally send the exact same message to Kafka twice if a network timeout occurs during the handoff. We are right back to the shouting in a crowded room problem.

Host 2

We absolutely are. The network is still the network. Which is why the transactional outbox is only half the solution. The receiving microservice, the fulfillment center application pulling messages off Kafka, must implement an item-potent inbox.

Host 1

The inbox.

Host 2

Right. When it pulls a message off the queue, it has to extract a unique message ID and record it in its own local database before it processes the shipping label. If it checks its database and sees a message ID it has already processed, it silently discards the duplicate. The outbox guarantees the message gets there. The inbox guarantees it is only acted upon once.

Host 1

It is a beautiful asynchronous choreography of state. But, and this brings up a massive point of contention in the engineering community. I am going to push back on you here. Because I know there are senior backend developers listening who are screaming this ethospeakers right now.

Host 2

Let's hear it.

Host 1

Wait a minute. Apache Kafka literally has a built-in feature called exactly once semantics. It was introduced in KIP-98. AWSSQS has FIFO cues that advertise exactly once processing. Why on earth are we building outboxes and inboxes and manual IDA potency keys if the multi-million dollar infrastructure broker does it for us out of the box?

Host 2

I'm so glad you brought this up because this is one of the most dangerous, widely misunderstood pieces of marketing in modern software engineering. It traps so many developers into building catastrophic systems.

Host 1

Tell me why.

Host 2

When Kafka says they support exactly once semantics, and don't get me wrong, KIP98 is a stunning engineering achievement. They are talking about a mathematical guarantee that exists entirely and exclusively within Kafka's own closed ecosystem.

Host 1

What does a closed ecosystem mean in practice? Give me an example of where it works.

Host 2

It works perfectly if you are reading data from a Kafka topic, doing a pure mathematical transformation on that data in memory, and writing the result back to another Kafka topic. Kafka's internal transactional coordinator uses a variant of two phase commit internally across its own partitions. Because it controls both the source of the data and the destination of the data, it can guarantee that the read, the processing and the write all happen atomically.

Host 1

Okay, so if I am just shuffling data around inside the Kafka bubble, I am safe. But what if I need to do something in the real world? What if I need to actually use that data?

Host 2

That is the hard boundary. The magical exactly once guarantee completely vanishes, the microsecond your consumer application reaches out to the real world.

Host 1

The microsecond.

Host 2

Yes. If your stream processor reads a message from Kafka and then fires off a welcome email via the send grid API or hits a third party webhook or calls the Stripe API to process a credit card charge. Kafka cannot reach out across the public internet and magically wrap Sendgrid's email servers in its internal transaction coordinator. It is physically impossible.

Host 1

Because FLP impossibility still rules the universe.

Host 2

Exactly. The physics of the network have not changed. For external side effects, the moment you touch an external API, you're immediately, inescapably back to at least once delivery. Imagine your server reads a message, successfully calls the Stripe API to charge a card, but right before it can commit the offset back to Kafka to say I'm done, your server's power supply explodes.

Host 1

Kafka doesn't know the charge succeeded.

Host 2

Exactly. Kafka just sees that your consumer died before acknowledging the message. So Kafka correctly redelivers that exact same message to a new consumer on a different server. That new consumer runs the code and calls Stripe again. If you didn't pass an item potency key to Stripe, you just double charged your user.

Host 1

Wow.

Host 2

Kafka's exactly once semantics cannot save you from side effect duplication. You must rely on item potency keys at the application level.

Host 1

That is a harsh, necessary reality check. The boundary of exactly once is the edge of the broker's closed system. Okay. Let's transition. We have talked about the deep mechanics, the grand theory and the heroic architectures that save us. Now, I want to talk about the disasters.

Host 2

The fun stuff.

Host 1

Because knowing the theory is one thing, but watching it fail in practice is where the real learning happens. So the database can protect us if we pass it the right information. But that assumes the client is behaving rationally. What happens when a developer trusts the front end too much? Let's look at the vibe coder trap.

Host 2

I love this section. This is where the rubber meets the road. We are moving from theory to the exact mistakes developers make every single day.

Host 1

So in our pre-show notes, we talked about the illusion of front end security. I call this the gray button fallacy. So this is the exact trap that AI coding agents generate constantly if you don't prompt them correctly.

Host 2

Oh, constantly.

Host 1

So let's say you ask an AI to build a checkout form in React. So let's say well, the AI generates a beautiful looking slick UI. So let's you type in your credit card, you click the submit button, and then instantly the button turns gray, right? It gets it gets disabled. And then there's a little loading spinner that appears, and you physically cannot click again. Okay. Great. So you as a developer, you're watching this in your browser and you feel incredibly secure. You think, great, I have completely prevented multiple double clicks, right? The user cannot accidentally submit the form twice.

Host 2

I mean, that's it is a very satisfying user interface.

Host 1

Sure is.

Host 2

But I mean, from from a systems architecture perspective, it's fundamentally a lie.

Host 1

Why is it a lie? The button is disabled.

Host 2

Because it operates under the dangerous illusion that the front-end UI controls the network. Disabling a graphical button in a web browser does absolutely nothing to protect the back end API.

Host 1

Oh, right.

Host 2

Step out of the happy path. What if a network blip causes the browser's internal fetch library or a service worker to silently retry the connection behind the scenes? The button is still gray. The user hasn't even touched their mouse, but the browser just fired a second post request.

Host 1

Or, you know, what if the user is on a subway, the connection drops, they get frustrated, they hit refresh on the browser, they get that annoying native browser warning about resubmitting form data, and they hit yes resubmit.

Host 2

Exactly. Or step outside the browser entirely. What if a malicious user completely bypasses your beautiful React UI, opens up their terminal, and just hammers your API endpoint using a curl script in a loop?

Host 1

The front end doesn't matter anymore.

Host 2

The front end reality does not matter to the back end. The back end post route is sitting there on the server completely unguarded. If it receives two requests, it will blindly execute two charges because it lacks true database-backed item potency. Relying on front-end UI state to enforce back-end data integrity is like putting a screen door on a submarine. It looks nice, but it won't stop the pressure when things go wrong.

Host 1

A screen door and a submarine. I love that. The back end has to assume the front end is actively hostile or entirely compromised, which leads us to the next disaster, the retry helper disaster.

Host 2

This is a bad one.

Host 1

This one keeps me up at night because the code looks so close to being correct. Let's say our developer realizes the front end isn't enough. They read an article about item potency keys, so they ask their AI agent to write a robust network client in Python or TypeScript that automatically handles retries and includes an item potency key header.

Host 2

A noble, well intentioned goal.

Host 1

The AI writes a function. It sets up a standard while loop. If the network call fails, it catches the exception, waits two seconds, and tries again. And inside that loop, right before it makes the actual HTTP fetch call, it calls UUID.v4 to generate a brand new item potency key, attaches it to the header, and fires the request.

Host 2

Oh no.

Host 1

This is one of the most common catastrophic mistakes I see in code reviews.

Host 2

Walk us through the consequence of that AI generated code. Why is that so dangerous?

Host 1

By generating a fresh UUID inside the retry loop itself on every single iteration of the loop, the code completely defeats the entire architectural purpose of the item potency key. Remember the Stripe model. The key is supposed to represent a single immutable user intent across multiple network attempts.

Host 2

So if the first request goes into the tunnel and times out.

Host 1

The loop catches the timeout error.

Host 2

It loops back around, it generates a brand new, completely different UUID, and sends the second request. When that second request hits the back end, the Postgres database looks at the item potency keys table.

Host 1

And it hasn't seen it before.

Host 2

It has never seen this new UUID before. It treats it as a brand new, completely distinct intent. Now what if the first request actually succeeded on the server, but it was just the response that got dropped in the tunnel? The server will now happily execute the charge a second time.

Host 1

Oh my god.

Host 2

A simple timeout on a flaky cell connection turns a helpful retry loop into a rapid fire weapon that systematically drains a user's bank account with unique, seemingly valid charges.

Host 1

It's essentially a recursive bankruptcy function.

Host 2

So what is the fix? Where does that UUID generation physically belong in the code base?

Host 1

The UUID must be generated exactly once, at the absolute highest level of user intent. If we go back to the React front end, the UUID should be generated the exact millisecond the user clicks the submit button, or perhaps even when the checkout page first loads.

Host 2

Way up top.

Host 1

Yes. That specific string must be stored in a constant variable and passed down into the retry loop. The loop must use that exact same string for every single network attempt related to that specific checkout action. Generate the intent once, retry the network execution infinitely.

Host 2

That is the mantra.

Host 1

Generate the intent once, retry the execution infinitely. Okay. I want to look at one more classic edge case disaster. You mentioned this in the notes, and it's a deep cut. We are diving into the miter common weakness enumeration database for this one, CWE367, time of check, time of use, race conditions, or TOCTOU.

Host 2

This is a subtle and devastating vulnerability. It is incredibly common when dealing with webhooks from external providers.

Host 1

Let's set the stage. You are building an integration with Stripe. Stripe processes a payment on their end and fires a webhook over the internet to your server to say, payment successful, grant this user access to the premium content.

Host 2

Okay.

Host 1

But distributed systems are chaotic. Sometimes Stripe might accidentally fire that exact same webhook twice, mere milliseconds apart. Or a mobile app on a spotty connection might retry a request, sending two identical packets that hit your load balancer almost simultaneously.

Host 2

Right. So we have two identical network requests. Let's call them request A and request B, hitting your server's application code at nearly the exact same time.

Host 1

A vibe coding developer or an AI agent might rate the logic in node or Python like this. First, I'll run a select query to check if the order exists in the database. If it doesn't exist, I'll run an insert to create the order, call the API to provision the premium content, and then mark it as complete.

Host 2

This is what we call the check then act anti-pattern.

Host 1

Exactly. So let's slow down time and break down the precise low-level mechanics of why this fails under default database conditions. Let's assume we are using Postgres with its default transaction isolation level. I know you wanted to touch on this. What is the default isolation level and what does it actually mean?

Host 2

The default isolation level in Postgres is called read committed. What this means is that when a query runs inside a transaction, it only sees data that was successfully committed by other transactions right before that specific query began. It does not lock the table against other people reading it at the same time.

Host 1

Okay. So request A arrives at T equals zero milliseconds. Request B arrives at T equals two milliseconds. What happens next?

Host 2

Under read committed isolation, request A executes its select query. Does this order ID exist? The database looks at the disk, sees nothing, and answers, no, it does not.

Host 1

But right at that moment, before request A's thread can actually move to the next line of code and execute the insert.

Host 2

Request B's thread executes its own select query. Does this order ID exist? Because request A hasn't committed anything yet, it hasn't even run the insert. The database looks at the exact same empty state and answers request B, no, it does not.

Host 1

Da da. The explosion.

Host 2

Yes. Both requests have independently checked the state of the world and both receive the green light to proceed. Because they checked the state before either could act, the check is completely invalidated by the time they reach the action phase.

Host 1

They both think they're first.

Host 2

Exactly. Request A proceeds to execute the side effect, provisioning the premium content, sending the welcome email, and request B, holding its own green light, proceeds to execute the exact same side effects. You just provision two accounts, sent two welcome emails, and corrupted your business logic.

Host 1

The times of check is fundamentally divorced from the time of use. It's like it's like two people trying to book the exact same seat on an airplane in the same millisecond.

Host 2

Yep.

Host 1

The reservation system checks seat 14B for person A and says it's empty. Then before person A's credit card processes, it checks seat 14B for person B and says, it's empty. Then it prints a ticket for both of them. They both checked reality, but reality changed the millisecond before they hit print.

Host 2

That is a perfect metaphor. The state of the world changed while your eyes were closed, processing the next line of code. To survive a TOCTU vulnerability in a distributed environment, you simply cannot rely on application level check than act logic. You must push the concurrency control down into the database's atomic layer.

Host 1

How do we actually do that in code? We go back to Brander's Postgres model, right?

Host 2

Yes. You completely skip the select query.

Host 1

Skip it completely.

Host 2

Yes. Do not ask the database if the row exists. You boldly attempt to insert the item potency key first. You let the database's hard-coded physical unique constraint violently reject the second request. If request A and request B attempt to execute an insert simultaneously, Postgres guarantees through those deep, low-level page latches we discussed earlier that one will succeed and the other will fail with a constraint violation error.

Host 1

And then what?

Host 2

You catch that specific error in your application code and you immediately halt the second request before any external API is touched. The database constraint must be the absolute first gatekeeper.

Host 1

Okay. This brings us perfectly to the practical application of everything we've discussed. We've seen how easily things can go wrong, especially when relying on AI code generation that tends to write naive, happy path check then act logic. We need a playbook. We need a strict prompt contract that we can put into our claw.md or agents.md system prompts to force these AI tools and to remind ourselves to build robust systems.

Host 2

This is critical for modern development. If you are using AI agents to scaffold your back-end microservices, you must establish the rules of engagement for state machines and external network calls up front, or it will generate vulnerable code.

Host 1

Let's lay out the specific spoken state machine directives for this playbook. Rule number one, which we just established, always perform an atomic database unique claim first. Never check then act. Rely on database constraint violations to detect duplicates.

Host 2

Agreed. That is the foundational rule. Rule number two, never hold database row locks open across slow external network calls.

Host 1

I want to spend a minute on this one because this isn't just a data integrity issue. This is a catastrophic infrastructure failure waiting to happen. Why is holding a lock over a network call so dangerous?

Host 2

It is a massive performance killer. Let's say your code starts a database transaction, locks a row, and then makes a five-second HTTP call to a third-party API like Stripe. For those entire five seconds, that database connection and that specific row lock are held hostage.

Host 1

Okay.

Host 2

Now imagine your marketing team launches a huge campaign and you get a spike in traffic. Suddenly you have 500 concurrent requests, all making five-second HTTP calls, all holding 500 database connections open. You will instantly exhaust your database connection pool.

Host 1

Everything just stops.

Host 2

No other queries can run. You starve your entire application, causing a massive cascading outage across your whole system.

Host 1

So the rule is claim the lock, commit the state, release the database connection back to the pool, and then make the external network call. When the network call finishes, open a brand new brief transaction to update the final state.

Host 2

Exactly. Keep your database transactions as short as physically possible. Okay, rule number three. Deduplicate every single incoming webhook based on the provider's unique event ID. Do not trust the provider to only send it once.

Host 1

Yes. Every major provider, Stripe, Twilio, GitHub includes a unique event ID in their JSON webhook payloads. Your webhook handler must treat that specific event ID as an idempotency key. Attempt to insert it into a processed events table atomically before you execute any side effects using the exact same unique constraint pattern we discussed.

Host 2

And finally, rule number four, which is the most important one for verifying our architecture. Always write continuous integration tests that explicitly fire concurrent duplicate requests.

Host 1

This is how you prove the system works. You cannot prove item potency with a sequential test where you send request one, wait for it to finish, and then send request two. That doesn't test the race condition.

Host 2

Exactly. Your CI pipeline must literally spin up a script that blasts the exact same API endpoint with three identical payloads at the exact same millisecond. If the endpoint doesn't return exactly one success response and two 409 conflicts or whatever your chosen back-off signal is, the test must fail the build. You have to prove the item potency holds up under extreme concurrency pressure.

Host 1

That is a rock solid prompt contract. But in the spirit of balance, we also need to talk about strict YAGNE boundaries. You aren't gonna need it. Because once developers learned about the transactional outbox, item potency keys and concurrent TOC TOU race conditions, there's a dangerous almost magnetic tendency to overengineer everything.

Host 2

Oh, it's the classic architectural pendulum swing. You get burned by a duplicate charge once, so you try to wrap your entire application in a massive Kafka-back temporal workflow. It is incredibly important to know when to stop and evaluate the actual requirements of the feature.

Host 1

Exactly. So let's set those boundaries. First, do not build custom item potency tables for HTTP GET requests.

Host 2

Get requests are inherently safe and item potent by definition. They read state, they do not mutate it. If a GET request fails in a tunnel, the client just retries. The server doesn't need to track anything because querying a database doesn't change the database. Adding item potency keys and state machines to a read-only endpoint is pure wasted computational overhead.

Host 1

What about database upsert operations? Like insert on conflict due update in Postgres?

Host 2

Natural database upserts are often inherently itempotent all on their own. If you are just syncing a user's profile data, say, set this user's email address to X, and then the client sends that exact same command three times due to network retries, the database just overwrites X with X three times.

Host 1

The end result is the same.

Host 2

The end state of the row is exactly the same. You do not need a complex item potency key state machine for simple, naturally item potent state overwrites. The heavy complexity is only required when the operation triggers non-item potent side effects, like sending an email, charging a credit card, or appending a new transaction to a financial ledger.

Host 1

And finally, the infrastructure trap. Please do not drag Apache Kafka, Debesium CDC or temporal into a simple CRUD application just because you read a Medium article about the dual write problem.

Host 2

I cannot stress this enough. Distributed infrastructure carries a massive, compounding operational tax. You have to monitor it, patch it, scale it, and debug it. If you are building a simple e-commerce site, doing a few thousand transactions a day, a single Postgres item potency keys table handles that volume effortlessly.

Host 1

That's all you need.

Host 2

You can implement a simple background worker using a lightweight tool like GraphLorker or even a basic cron job to sweep unfinished tasks in the outbox. You do not need a multinode Kafka cluster and a zookeeper ensemble to prevent double charges on a mid-sized application. Architectural complexity is a liability. You should only spend that liability when your scale absolutely demands it.

Host 1

Keep it simple until the pain of simplicity forces you to scale. Okay, we have covered a massive amount of ground today, from the physical realities of the two generals problem to the intricate mechanics of Postgres constraint violations and internal page latches. And it is fascinating how much of our job as software engineers is just managing the inevitable consequences of hardware and network failure.

Host 2

It really is. The network is fundamentally hostile, and our job is to build fortresses of logic to protect the user's data from that hostility.

Host 1

Speaking of things spiraling out of control today, we talked a lot about runaway network retry loops, where a client just keeps hammering a server over and over again because of a timeout.

Host 2

A vicious cycle that can take down a whole system if you aren't careful.

Host 1

Well, next time we are bringing that concept of endless loops much, much closer to home. We are transitioning from runaway network loops over the internet to functions calling themselves inside a single processor. In our next deep dive, episode S08E03, we are tackling recursion and tail call optimization.

Host 2

Oh, that's gonna be good.

Host 1

We are gonna look at the mathematical beauty of recursive algorithms, and then we are gonna see exactly what happens at the hardware level when recursion gets out of control and physical call stack RAM literally explodes. It is gonna be a fascinating look at the physical limits of memory.

Host 2

I'm looking forward to it.

Host 1

Stack overflows are a rite of passage for every developer. But before we go, I want to leave you with one final provocative thought, based on everything we explored today. We've established that we can never truly trust the network. We've seen that the only way to achieve anything close to exactly once processing is by anchoring our application logic to the hard atomic constraints of a single database like Postgres. The database is our ultimate source of truth, the bedrock that stops the duplicates.

Host 2

It is the anchor in the chaotic storm of the network.

Host 1

But here's the lingering question. What happens when the database itself gets partitioned? As our systems grow to a massive global scale, we can't always rely on a single monolithic Postgres instance running on one machine. We start sharding databases, replicating them across continents, using complex distributed consensus algorithms like Paxos or RAFT. We fragment the anchor itself. It is something to ponder as you look at the fragile, fascinating illusion of certainty in a distributed world.

Save to playlist

Manage playlists →

Share