Idempotency & Exactly-Once
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's get straight into it. Last time on The Explainer, we uncovered exactly how operating systems' pseudo-random number generators mint those unguessable UUID4 identifiers. Today, we're putting those very keys to work against the unforgiving chaos of lossy networks.
Look, we're skipping the basic HTTP stuff today to immediately confront a pretty harsh mathematical reality about distributed systems. Tyler Treat hit the nail on the head when he said, you cannot have exactly once message delivery.
Whether it's a web browser talking to a server or a server talking to a message queue, pure network certainty is just an illusion. We literally have to design our systems accepting this fundamental limitation. Section one.
Picture this. A mobile client fires off a post request to charge $99, but it hits a five-second timeout. Well, what just happened?
Did the packet drop before it even reached the server? Did the server crash right in the middle of executing? Or worst case scenario, was the card actually charged on Stripe side, but the 200OK acknowledgement just vanished on dodgy Wi-Fi.
You know, this is the classic two generals problem in action. Over a lossy network, these three realities are completely indistinguishable to your client. And honestly, this is exactly why pure transport-level exactly once delivery is a total myth.
It violates the FLP impossibility result, which proves you can't have deterministic consensus in an asynchronous network where nodes can fail. So, in the real world, production systems don't even try to force exactly once delivery over the wire.
Instead, we pair at least once retries with highly stateful idempotent receivers. That is how we create the illusion of exactly once processing. Alright, section two. Retries, jitter, and protocol mechanics.
So we have to retry, right? But think about it: if thousands of clients hit a network blip and instantly retry at the exact same millisecond, they're gonna absolutely crush a recovering server. It's that classic thundering herd problem.
To avoid taking ourselves down, we use an exponential back-off algorithm, waiting 100 milliseconds, then 200, 400, and so on. But crucially, we combine that with randomized full jitter. Jitter smears those retry spikes out over time, which gives your back end the breathing room it desperately needs to recover.
Now, if we look at the protocol layer, HTTP semantics, specifically RFC 9110, dictate that get, put and delete natively guarantee idem potency. In math terms, f of f of x equals f of x. Applying it twice gives you the exact same server state as applying it once.
Post and patch, though, they absolutely do not have those guarantees. If you just blindly retry a POST request, you run a massive risk of charging that user twice. We definitely don't want that.
Now for your simpler, non-financial updates, we can just rely on optimistic concurrency using conditional requests with e-tags. Basically, a client sends an update with an if-match header, holding the resource version it last saw.
If another client has already modified that state, the server just kicks back a 412 precondition failed. It's incredibly elegant, very lightweight, and perfectly solves the lost update problem.
Moving on to section three, Stripe's Postgres State Machine. But what about those heavy, non-idempotent financial mutations? Well, here is the IETF draft definition for an idem potency key. It's essentially a client-provided unique identifier, like a UUID V4.
If a duplicate key hits our server, we have to handle it safely. The draft says we return a 409 conflict if there's a concurrent in-flight execution already processing that exact key. Or, and this is vital, we return a 422 unprocessable content if a rogue client tries to reuse an old key with altered parameters, which we can catch by comparing a SHA 256 fingerprint of the request payload.
To see this in action, we can look at Brandor Leach's brilliant stripe-like architecture. He implements this by dropping the key right into Postgresle and advancing the request through atomic phases inside a directed acyclic state machine.
So the request moves step by step. Started, ride created, charge created, and finally finished. And the database mechanics behind this are just fantastic. By using atomic transactions with an insert into on conflict do nothing statement, the system uniquely locks the row.
It essentially logs the recovery point, meaning if the server crashes mid flight, a retry just picks up exactly where it left off. Once it's all finished, it caches the response payload for replay, usually keeping it on hand for a 24-hour cleanup window.
Right, section 4. Q allusions and outbox patterns. We need to talk about something incredibly common and incredibly dangerous. The dual write trap.
Look, writing to your local Postgres database and emitting an event to a Kafka queue at the exact same time create a massive race condition. They are absolutely not atomic operations. Your database write could succeed while the Kafka publish fails, or vice versa.
Either way, you're left with a fractured, hopelessly inconsistent system. So how do we fix it? We use the transactional outbox pattern. By utilising local ACID commits and a change data capture tool, like Debezium, reading the write-ahead log sequence numbers paired with an idempotent consumer inbox, you absolutely prevent lost or duplicate events.
On the flip side, if you rely on Q level exactly once illusions, you are doing so at your own peril. Things like Kafka KIP98 or AWS SQS FIFO five-minute deduplication windows, they instantly lose their magic the very second your worker invokes an external HTTP API like Stripe or Sendgrid, or exceeds a visibility timeout.
Which brings us to section 5, the Vibe Coder trap. Take a look at the sheer disaster waiting to happen here. Just disabling a React Submit button, or worse, regenerating a new UUID inside a front-end retry loop creates a nasty time of check time of use race condition, also known as CWE367.
If your code simply says, if order doesn't exist, then charge, under a standard Postgres read committed isolation level, two concurrent network retries will blow right past that check and double charge your user just five milliseconds apart. It's a nightmare.
To stop this from happening, we use a YAGNI, you aren't gonna need it, rubric. Always, always enforce database unique constraints alongside your idem potency key. Use the outbox pattern for external mutations. But listen, only enforce that heavy Postgres state machine we talked about for critical financial or external mutations.
Skip them completely, in favour of simple, optimistic e tag concurrency for read only data, basic puts, or approximate telemetry counters. Don't overengineer a massive solution where a simple protocol feature will do. So, we've thoroughly secured our external network loops against the impossibility of exact delivery.
But how exactly does self-reference factor into building robust, infinitely repeating internal execution without completely blowing up the stack? Well, that is exactly what we'll be tackling in the next explainer. Until then, ask yourself: how many dual rights are quietly hiding in your code base right now?
Catch you next time.