Architecture of Resilience
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome to this explainer on advanced web architecture. We're diving straight into tuning high TPS services for absolute production resilience. We're bypassing the introductory SQL versus NoSQL debates today.
Instead, we're looking strictly at how you keep a high throughput backend alive when traffic spikes and your database is literally screaming for mercy. 1. Caching topologies and traps.
2. Stampedes and cascading failures. 3. Upstream connection pooling.
4. Mitigating system overload. And finally, 5. Tuning cache eviction.
Section 1. Caching topologies and traps. The illusion of safety. Take a look at this quote from the AWS Builder Center.
It hits the nail on the head, pointing out a pretty terrifying reality. Services often become completely addicted to their cache. You build it, your downstream dependencies shrink their fleets because latency drops, and then suddenly your cache isn't just an optimization anymore, it's a critical point of failure.
A simple cold start can instantly plunge the whole system into a catastrophic outage. So we have to look at the massive trade-offs in caching topologies. Think about a local in-process heap cache.
It gives you zero network round trip time. It's blisteringly fast. But you're gonna suffer from fleet staleness and cold starts because every new instance hits the database.
Now, on the other hand, distributed external caches, like a Redis cluster, solve that shared state issue. But they introduce totally new availability risks and network overhead. And when we compare caching patterns, an inline cache, like your standard read or write-through, gives you a nice uniform API.
It's transparent to the client. But if that cache goes down, the whole system fails. Contrast that with a side cache or cache aside pattern. Sure, lazy population can introduce race conditions, but crucially it leaves your application in control.
It allows you to gracefully fall back or shed load if the external cache cluster suddenly vanishes. Section two, stampedes and cascading failures, the anatomy of a meltdown. Right, so a cascading failure is essentially a domino effect.
Picture a cache stampede, what we often call a thundering herd. Let's say a highly popular hotkey expires. Suddenly, under a load of, say, 5,000 requests per second, 5,000 threads all miss the cache at the exact same time.
They rush the database to recompute that one identical key. This positive feedback loop crushes your backend in absolute milliseconds. The really crucial point here is how rapidly this failure compounds. This sequence maps out the notorious GC death spiral.
A simple CPU bottleneck slows down your downstream requests. Those slower requests mean more concurrent requests stay in flight, eating up your active memory. This memory pressure triggers aggressive garbage collection, which restricts your CPU even further.
That drops your cash hit rates, sending a lethal wave of traffic straight to the back ends until they just melt down completely. Similarly, naive retries are incredibly dangerous. A client hits a load limit, gets rejected, and immediately retries.
That amplified retry traffic completely overwhelms the backend queue. You end up in a perpetual crash loop where you're essentially DDoSing yourself. Your servers are burning 100% of their CPU while performing exactly zero useful work.
Section three, upstream connection pooling. The architectural shield. To survive all of this, reverse proxies like Envoy act as a critical shield for upstream hosts.
Look at HTTP 1.1. You've got a rigid one to one-to-one mapping between requests and connections. This is exactly why spinning up thousands of ephemeral serverless containers exhausts Postgres backend processes so fast.
You just run out of sockets without transaction level multiplexing tools like PG Bouncer or RDS proxy. But HTTP2 and HTTP3 actually multiplex thousands of requests over a single robust connection, which drastically reduces that connection churn.
Envoy goes even further to protect systems. It uses intelligent features like Happy Eyeballs for rapid IPv4 and IPv6 failover. It also dedicates entirely segregated connection pools for routing priorities. This means that if background sync traffic suddenly spikes, it gets throttled in its own pool, ensuring your critical interactive user traffic never starves.
Section 4. Mitigating system overload. Tactical defenses. Now this is the ultimate engineering question during an incident.
The pages are screaming at 3am. How do we survive a downstream brownout when the database simply cannot keep up with thousands of transactions per second? Well, we mitigate this using a combination of tactics.
To block those cache stampedes we just talked about, we use request coalescing, often implemented via single-flight mutex locking, so only one thread queries the database while the others wait. We employ soft and hard TTLs, maybe even jittered TTLs, to serve slightly stale data rather than failing completely.
We enforce strict deadline propagation to cancel doomed RPC work early. And when all else fails, we gracefully degrade by returning cheaper to compute results. Section 5.
Tuning cache eviction, the micro level. When max memory is finally reached, Redis has to evict data. And you've got several tools to control this. All keys LRU tracks recency, perfect for a standard 8020 Pareto traffic distribution.
All keys LFU tracks frequency using Morris counters, which is great when specific keys just have to stay cached. And the newer All Keys LRLM protects heavily red data by only updating timestamps upon write modifications.
But a quick warning: beware of single-threaded event loop blocking traps here. Things like giant hashes, slow Lua scripts, or running keys asterisk will stall eviction entirely. Now here's a brilliant, somewhat surprising reality about LRU under the hood.
True least recently used caching actually costs way too much memory to track across millions of keys. So Redis smartly approximates it. It randomly samples a handful of keys and evicts the oldest one in that specific sample.
And the magic number here is 5. By simply configuring max memory samples to 5 in your Redis config, Redis achieves an incredibly accurate approximation of true LRU, without the massive RAM overhead. It's just a fantastic engineering trade-off.
Finally, I want to leave you with this highly practical mantra from Google SRE. If at first you don't succeed, back off exponentially. To prevent cascading failures, never retry blindly. Always use jittered exponential back-off, because at the end of the day, protecting the system's survival is always more important than serving every single individual request.
So I'll leave you with this. Take a hard look at your own architecture today. Is your cache a resilient shield? Or is it a ticking time bomb just waiting for a single cold start?