CDNs & HTTP Caching
Podcast · CDNs & HTTP Caching

How Cache Addiction Melts Origin Servers

by ZumiLabs · 56m
0:00 56m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

Imagine your origin server is just humming along beautifully. You know, you are the lead architect, you're looking at your dashboards, and everything is perfectly green.

Host 2

Right. The absolute dream scenario.

Host 1

Exactly. You provisioned your database, your worker nodes, your memory, all based on a very comfortable, very standard assumption in modern web architecture, which is a 98% CDN hit ratio.

Host 2

Yep. Meaning out of every 1,000 requests hitting your domain from users out in the wild, only 20 actually make it all the way back to your origin database.

Host 1

Right. So the edge is handling the rest, you are sleeping well at night.

Host 2

It is a comfortable place to be. I mean, your infrastructure is heavily shielded and your cloud bill is super predictable. But that comfort actually breeds a very specific kind of vulnerability.

Host 1

And that vulnerability usually gets exposed on like a random Tuesday morning. Let's say a junior developer deploys a tiny bug at a cache header. Or maybe a well-meaning SRE triggers a global hard purge because of a minor content update.

Host 2

Oh yeah. We've all been there.

Host 1

Suddenly that 98% hit ratio drops. It doesn't drop to zero, just drops to 80%. Now intuitively, going from a 98 out of 100 score to an 80 out of 100 feels like an 18% degradation.

Host 2

Right. It sounds like you're just 18% worse off.

Host 1

Exactly. But let's break down the brutal math of what is actually happening to your origin server in that exact moment.

Host 2

The math here is terrifying, honestly, because the relationship between cash hit ratio and origin load is completely nonlinear. The moment you drop from 98 to 80, you aren't seeing an 18% increase in traffic. You are seeing an apocalyptic flood.

Host 1

Let's run the numbers for the listener to really visualize this. So at a 98% hit ratio, out of a thousand requests, 20 hit your origin.

Host 2

Yep.

Host 1

But at an 80% hit ratio out of a thousand requests, 200 are now hitting your origin.

Host 2

200. Right.

Host 1

That doesn't mean your traffic increased by a fraction. It means your origin requests per second, your RPS, just multiplied by 10 overnight.

Host 2

Exactly. 200 divided by 20. It is a 10x multiplier.

Host 1

Right. If your database connection pool was running at 40% utilization before, it cannot suddenly run at 400%.

Host 2

No, it can't.

Host 1

The connection pool exhausts a milliseconds. The CPU steal time goes through the roof, and that causes the worker nodes to start queuing requests.

Host 2

Which just cascades, right?

Host 1

Absolutely. It increases latency, which causes health checks to fail, which causes load balancers to pull nodes out of rotation, and that just further overloads the surviving nodes. The origin melts down spectacularly.

Host 2

It's a total death spiral.

Host 1

It is.

Host 2

And this specific catastrophic failure mode actually has a name in systems architecture. In those AWS builder center architecture notes you shared with us, they refer to this as cash addiction.

Host 1

Cash addiction. That phrasing implies the system literally cannot function without the substance it's hooked on.

Host 2

Well, yeah, because when a system is first built, a cache is often added as a pure performance optimization. Like it's a nice to have layer, you know.

Host 1

Right. It lowers latency for the user, saves some egress bandwidth.

Host 2

Exactly. But over time, as organic traffic grows, the downstream dependencies, the databases, the internal microservices, they look at their average load, they see it's manageable, and the engineering team scaled down their fleet sizes accordingly to save money.

Host 1

Oh, I see. They adjust their baseline to the cached reality, not the raw traffic reality.

Host 2

You got it.

Host 1

The system becomes structurally dependent on the modal behavior of the cache. The cache stops being an optimization, it just becomes a load bearing pillar.

Host 2

Because the cache introduces two entirely different modes of operation, right? The cache hit mode and the cache miss mode.

Host 1

Right.

Host 2

And any unanticipated shift in the distribution between those two modes leads to disaster. When the cache fails or the hit ratio drops even slightly, the origin instantly browns out because it is simply no longer provisioned to handle the raw, unmitigated internet.

Host 1

Okay, let's unpack this. Welcome to your custom deep dive. You asked us to look at the absolute bleeding edge of web architecture, specifically how caching actually works at a global scale.

Host 2

And we are skipping the 101 today.

Host 1

We absolutely are. Since you already know how DNS resolves, what an AnyCast edge point of presence is, and the basic anatomy of HTTP, we aren't going to waste time defining those terms. We are going straight into the machinery.

Host 2

Yeah, we have a really dense stack of sources to work through. We're pulling directly from RFC 911, which is the current HTTP caching standard, and RFC 5861, which governs stale content extensions.

Host 1

And we're also overlaying Mozilla Developer Network's advanced caching guides and those deeply practical AWS architecture notes you mentioned.

Host 2

Right. The mission today is to dissect exactly how CDN internals, request collapsing, origin shielding, and advanced cache headers dictate whether your system scales globally or suffers that cache addiction meltdown we just described.

Host 1

So I want to start with the absolute foundation, which looking through RFC 911, is incredibly rigorous. When a file sits on an edge server in Frankfurt or Tokyo, how does that physical server actually know if the file is still?

Host 2

That's a great question.

Host 1

Because I think the middle model most developers have is just a simple stopwatch. The file arrives, the stopwatch starts, and when it hits zero, it's stale. But looking at the RFC, it's a strict mathematical calculation evaluated on every single request.

Host 2

Yeah, the stopwatch analogy is way too simple. The edge server evaluates a very specific Boolean equation. It's simply is the freshness lifetime greater than the current age?

Host 1

Freshness lifetime greater than current age.

Host 2

Yeah.

Host 1

Okay.

Host 2

If it is, the object is fresh. It can be served to the client without ever opening a connection to the origin server. But unpacking those two variables, freshness lifetime and current age, reveals an astonishing amount of complexity and weird edge cases.

Host 1

Well let's break down freshness lifetime first. How does the cache determine what that value is? Because looking at the headers, it's not just a single field. There's a strict hierarchy of directives that the cache has to parse.

Host 2

Right. The cache looks up the HTTP response headers sent by the origin and it evaluates them in a specific order of precedence to lock in that freshness lifetime variable.

Host 1

And at the very top of that hierarchy is S-M A S M A G E X, right?

Host 2

Exactly.

Host 1

S Mag.

Host 2

Which stands for shared cache max age. If this directive is present in the cache control header, it completely overrides everything else for a shared cache, like a CDN or a corporate proxy.

Host 1

It dictates the exact number of seconds the response is considered fresh by the CDN.

Host 2

Yes. But crucially, private caches like the local browser cache on a user's laptop completely ignore SMAGE.

Host 1

Oh, that's interesting. So that allows for split brain caching policies. You could tell the CDN to hold on to a heavy payload for a month using S max age, but tell the browser not to hold on to it at all.

Host 2

Which is incredibly useful.

Host 1

Definitely.

Host 2

Think about assets you might need to purge globally from the edge, but you don't want them stuck in millions of local browser caches where you have absolutely no purge access.

Host 1

Right, because you can't reach into a user's laptop and clear their cache.

Host 2

Exactly. So moving down the hierarchy, if Smax Age isn't there, the cache looks for the standard max age directive. This applies universally to both shared and private caches, again, specifying the lifetime in seconds.

Host 1

And if neither of those are present?

Host 2

Then it falls back to the legacy HTTP 1.0 header, which is expires.

Host 1

Now, the MDN notes go out of their way to warn developers against using the expires header. It uses an explicit HTTP date timestamp, something like Thursday, 1st December, 2026, 1600 GMT.

Host 2

Yeah, it's very verbose.

Host 1

And to figure out the freshness lifetime from that stream, the cache has to take that expires value and subtract the value of the date header, which is when the origin generated the response. Why is this considered such an anti-pattern today?

Host 2

Relying on explicit timestamps and distributed systems introduces a massive vulnerability, which is clock synchronization.

Host 1

Right.

Host 2

The server generating the response and the cache evaluating it need to have perfectly synchronized clocks. If your origin server's clock drifts by even a few minutes, which honestly happens way more often than ops teams like to admit, the entire freshness calculation becomes corrupted.

Host 1

The stopwatch is broken before it even starts.

Host 2

Exactly. Furthermore, the HTTP date format is notoriously rigid. If a developer uses a slightly different time format or forgets a comma, the cache fails to parse it and defaults to a zero lifetime.

Host 1

So max age fix this by shifting from absolute timestamps to relative time elapsed in seconds.

Host 2

Yes, it completely sidesteps the clock drift problem. The RFC actually mandates that if a cache sees both a max age and an expires header, it must discard the expires header entirely.

Host 1

Okay, so that resolves one side of the Boolean equation. We have our freshness lifetime as a fixed integer of seconds.

Host 2

Right.

Host 1

But the other side of the equation is the current age. And reading the RFC, this is where my mental model of the stopwatch completely broke down.

Host 2

It usually does for most people.

Host 1

Because the age of a response is not just the time it has been sitting on the solid state drive of that specific edge node. The age is an additive estimate of the time since the origin originally generated or validated the response. It aggregates across the entire network path.

Host 2

Yeah. The protocol has to account for the fact that a response might travel through multiple independent cache hops before it reaches the end user.

Host 1

So imagine a topology where a request goes from the origin to a regional shield cache to an edge cache, then maybe through an enterprise network proxy and finally to the browser.

Host 2

Right. And each of those hops is caching the file.

Host 1

So how does the final edge node calculate the true age of the file across all those blind hops?

Host 2

The RFC outlines two parallel calculations for this. The first method is called a parent age.

Host 1

Okay. How does that work?

Host 2

The cache looks at the local time it received the response and subtracts the date header value that the origin stamped on the payload. Essentially it's asking what time is it on my local system right now? minus what time did the origin claim it created this?

Host 1

But wait, that again relies on that fragile clock synchronization between the edge node and the origin, right?

Host 2

It does, which is exactly why the RFC requires a second, completely independent calculation to compensate. It's called corrected age value.

Host 1

And this is where it calculates transit time?

Host 2

Yes, exactly. The corrected age value takes any age header that was already injected into the incoming response by previous upstream caches, and then it adds the network response delay. Network response delay meaning It calculates the time it took for the request to travel over the wire from the exact moment the edge asked for it to the moment the last byte arrived.

Host 1

That is fascinating. It's actually accounting for the speed of light and network congestion.

Host 2

Yes.

Host 1

So if a massive video file takes five seconds to physically stream from the origin in Virginia to the edge node in Tokyo, those five seconds of transit time count against its freshness lifetime. The file is already five seconds old before it's even written to the disk in Tokyo.

Host 2

Exactly right. The edge node executes both of these calculations. It takes the larger of the two, the apparent age or the corrected age value, and defines that as the corrected initial age.

Host 1

And then what?

Host 2

Then to arrive at the final current age variable for our equation, it adds the resident time. That is simply how long the file has been sitting in the current cache's local memory or disk since that initial calculation.

Host 1

Wow.

Host 2

This additive process ensures that the age is always a highly conservative worst-case estimate of how old the data actually is. It's completely independent of how many proxy layers it bounced through.

Host 1

The edge is running this rigorous timestamp algebra for billions of requests per second just to determine if freshness lifetime is greater than current age. If it is, boom, we get a cache hit.

Host 2

Yep, that's the core engine of the internet right there.

Host 1

But looking through the MDN documentation, there is a directive that seems to completely bypass this math. Let's talk about the no cache paradox.

Host 2

Oh no cache. It is arguably the most counterintuitive naming convention in the history of web standards.

Host 1

It really is. If you ask any junior developer or honestly, even many senior developers, what cache control.nocache does, they will confidently tell you it forces the browser and the CDN to never store the file.

Host 2

And according to the RFC, that is fundamentally incorrect.

Host 1

Right. No cache actually permits the cache to store the response on disk.

Host 2

It absolutely permits storage. The directive that completely forbids writing a response to nonvolatile storage is no store. No store. Right. If you send no store, you're telling the network do not write these bytes to disk under any circumstances.

Host 1

Okay, so then what is no cache doing?

Host 2

No cache means the cache will store the file, but it is strictly forbidden from reusing that stored response to satisfy a subsequent request without first revalidating it with the origin server.

Host 1

I want to make sure I have this crystal clear. If I think about this like a bouncer at a highly exclusive nightclub, the cache itself is the VIP holding area.

Host 2

Okay, I like this.

Host 1

If a file has no cache on it, the CDN is allowed to hold on to the file. You can stay in the VIP line. But when the user actually requests the page, the bouncer steps in. He says, Look, I see you standing there, but I cannot just let you in. I have to call the boss, the origin server, to check if your name is actually on the list tonight.

Host 2

That's perfect.

Host 1

You can't just walk in based on the cache's local authority. You must be revalidated.

Host 2

That analogy maps perfectly to the protocol. The cache possesses the content, but it is not authorized to trust its own copy. It has to make what HTTP calls a conditional request back to the origin.

Host 1

Walk me through the mechanics of that conditional request over the wire. How does the cache verify the file without just downloading the whole thing again? Because that would defeat the purpose.

Host 2

Right. So it hinges on validation headers. When the origin initially serves the file, it attaches a validator. Typically, this is an e-tag, which is essentially a unique cryptographic hash of the file's contents.

Host 1

Or it could be a last modified timestamp.

Host 2

Exactly. But let's use e tag for the example. Let's say the origin sends back a CSS file with an e-tag of hash XYZ123. The CDN stores the CSS file on disk and it stores that e tag string right alongside it in a lookup table.

Host 1

Then, you know, ten minutes later, a user requests that exact CSS file.

Host 2

The CDN intercepts the request.

Host 1

It pulls the file from disk, but it sees the no cache directive. It knows it cannot serve it immediately.

Host 2

The bouncer stops them. Right. So the CDN opens a connection back to your origin server, but it does not ask for the full CSS file. It sends a conditional request using the if none match header, injecting the stored E tag.

Host 1

Oh, so this CDN is effectively asking the origin. I have a cached version of this file with a signature hash XYZ123. Send me the new file, but only if your current signature does not match mine.

Host 2

If none match. The protocol wording is so painfully literal.

Host 1

It really is. So if the file hasn't been edited on the origin server, the current e tag is still hash XYZ123. The signatures do match.

Host 2

And because they match, the condition fails. The origin server does not transmit the file payload. Instead, it sends back a tiny, essentially empty response with the status code 304 not modified.

Host 1

304 not modified.

Host 2

Right. It has no body. It is just a set of headers saying your cached version is completely accurate. Go ahead and serve it to the client.

Host 1

The bandwidth savings here are just staggering. If you have a massive two-megabyte JavaScript bundle and you use no cache, the browser or the edge still has to make a network round trip to the origin. You still pay the latency cost of establishing the TCP connection and waiting for the origin to respond. True. But you avoid downloading the two megabytes. You download maybe a hundred bytes of headers instead. You enforce absolute real-time freshness, but you highly optimize the network payload size.

Host 2

It is a deliberate architectural trade-off. No cache trades latency for absolute freshness while minimizing bandwidth consumption.

Host 1

But in modern web architecture, trading away latency is often unacceptable. What if minimizing time to first byte is the absolute most critical metric for your application?

Host 2

Then no cache won't work for you.

Host 1

Right. So how do you guarantee millisecond response times without serving hopelessly outdated data? This is the exact problem RFC 5861 was authored to solve.

Host 2

Yes. That RFC is where caching transitions from a blunt instrument to an incredibly sharp precision tool. It introduces HTTP cache extensions for dealing with stale content.

Host 1

And the goal there is decoupling tail latency from the end user experience. I really want to spend some time on stale while revalidate. The mechanics of this are brilliant.

Host 2

Stale while revalidate, or SWR fundamentally rewrites the contract between the cache and the end user.

Host 1

How so?

Host 2

Well, think about a traditional caching model. When an object expires, when the current age we calculated earlier finally exceeds the freshness lifetime, the very next user who requests that file is going to suffer. Right. The cache intercepts the request, realizes the file is stale, and physically blocks the user's connection while it goes all the way back to the origin to fetch a fresh copy. That single unlucky user absorbs the full latency penalty of the cache miss.

Host 1

They just hit a brick wall. Everyone before them got a two-millisecond response and they get a 200 millisecond response just because they happen to ask of the file at the wrong second.

Host 2

Exactly. SWR changes the paradigm entirely. Let's say you configure your headers like this. Cache control, max age equals 600, stale while revalidate equals 30. You are issuing two distinct instructions to the edge node.

Host 1

Okay, break those down.

Host 2

First, the max age dictates the file is fresh for 600 seconds, serve it instantly for memory. That's normal.

Host 1

Right.

Host 2

Second, for a window of 30 seconds after it becomes stale, you authorize the cache to serve that stale file to the user instantly, while simultaneously triggering an asynchronous, non-blocking background fetch to the origin to update the cache.

Host 1

This reminds me of sitting down in a busy restaurant. You walk in, the host seats you, and a busboy immediately drops a basket of bread on the table.

Host 2

Okay.

Host 1

It is not fresh out of the oven. It is the day-old bread basket. But you have something in front of you to consume instantly the second you sit down. Meanwhile, the kitchen in the back is baking a fresh loaf of bread, but that fresh loaf isn't for you, it's for the next customer who walks through the door.

Host 2

That's a brilliant way to put it. You received the slightly stale bread instantly, and the baking latency was hidden from you entirely. Right. SWR completely abstracts origin latency and network transit time away from the end user during that predefined window. The user gets a millisecond response from the edge pop completely unaware that the data is technically expired.

Host 1

And the CDN does all the heavy lifting in the background.

Host 2

Exactly. It completely flattens out those tail latency spikes that just plague traditional expiration models.

Host 1

But RFC 5861 goes even further. It introduces a sister extension called stale if error reading through the AWS architecture notes, this directive seems like a literal cheat code for site reliability engineers.

Host 2

Oh stale if error is the ultimate failover mechanism. The RFC designed it specifically to handle scenarios where the origin server outright fails and returns an HTTP error code.

Host 1

Like a 500 internal server error of 502 bad gateway a 503.

Host 2

Or a 504 gateway timeout yeah. In a standard set of if the CDN asks the origin for a file and receives a 500 error it has no choice but to pass that catastrophic error page back to the end user.

Host 1

Right. The application appears entirely broken.

Host 2

Total downtime from the perspective of the customer.

Host 1

But if you have stale if error equals 1200 attached to your payload the CDN alters its behavior.

Host 2

What does it do?

Host 1

When it encounters one of those five XX errors from your origin, it checks its local storage. If it possesses a stale copy of the requested file that has been expired for less than 1200 seconds, it will gracefully ignore the origins 500 error.

Host 2

It just suppresses it.

Host 1

It suppresses it completely and serves the stale cached version to the user instead. It completely shields the user from the outage.

Host 2

Yes.

Host 1

If we connect this back to the AWS notes. This is the exact mechanism behind the architectural pattern of soft and hard TTLs. Time to live.

Host 2

Right.

Host 1

The max age is your soft TTL. That is when you politely want to refresh the data. But the stale if error window acts as a hard TTL. It is your ultimate safety net.

Host 2

Exactly.

Host 1

If your primary database cluster goes offline for 10 minutes, your CDN just keeps serving slightly stale HTML and JSON. Your users might see a product inventory count that is 10 minutes out of date, but they do not see a 502 bad gateway screen.

Host 2

It massively amplifies your perceived availability.

Host 1

Your origin could be literally on fire. Alarms going off in the data center. And the users browsing the site have absolutely no idea.

Host 2

It decisively severs the link between origin outages and the user experience. However, the RFC includes a strict caveat here to maintain protocol integrity.

Host 1

Oh, what's the caveat?

Host 2

When a cache serves a stale response under these error conditions, it is required to inject a specific warning header into the response, and it must ensure the age header accurately reflects the true non-zero age of the file.

Host 1

So the cache is not allowed to lie to the client and pretend the file is freshly generated.

Host 2

Right. It is simply authorized to serve it.

Host 1

So the client application can inspect the headers and realize, okay, I'm displaying data that is 30 minutes old because the back end is down. It allows the front end to handle the degraded state gracefully.

Host 2

Exactly.

Host 1

So we've discussed shielding the user from latency and shielding the user from origin errors. But let's look at this from the other direction. Let's go back to protecting the origin itself from the internet. What happens when a piece of dynamic content goes unexpectedly viral? Let's talk about the thundering herd problem and request collapsing.

Host 2

The thundering herd is the nightmare scenario that keeps infrastructure engineers awake at night. Imagine a completely uncached resource, perhaps a breaking news article, a highly anticipated product drop, or you know, a live election results endpoint. Right. It is suddenly requested by 10,000 users at the exact same millisecond.

Host 1

Cache is completely empty. The current age is null. So all 10,000 of those incoming connections register as an immediate cache miss.

Host 2

Right. And in a naive caching implementation, the edge node would simply act as a proxy. It would blindly open 10,000 concurrent TCP connections back to your origin server and just forward the requests. Ouch. Your origin database would receive 10,000 identical queries simultaneously, exhaust its memory, and crash instantly.

Host 1

Right.

Host 2

This is where advanced CDNs implement request collapsing, which is also known as request coalescing.

Host 1

Okay, how does that work?

Host 2

The edge node is intelligent enough to recognize that these 10,000 incoming sockets are requesting the exact same resource. They share the same cache key. Okay. So instead of opening 10,000 connections to your origin, the edge node holds 9,999 of those client sockets in a pending queue. It sends exactly one fetch request to your origin.

Host 1

And it sort of coalesces the massive spike into a single highly manageable request, right?

Host 2

Right. And the origin processes that single request, runs the database query once, generates the JSON payload, and streams it back to the edge node. And then when the edge receives those bytes, it multiplexes that single response stream back out to all 10,000 client sockets waiting in queue.

Host 1

Wow.

Host 2

You just successfully serve 10,000 concurrent users, but your origin infrastructure only had to process one request.

Host 1

That is an incredible piece of engineering. But wait, let's look at the global topology of this. CDNs like Cloudflare or Fastly, they operate highly distributed networks. They have hundreds of any cast points of presence around the world. Right. So a cluster of users in Tokyo hits the Tokyo pop, a cluster in London hits the London pop. The Tokyo pop collapses its local thundering herd and sends one fetch to the origin.

Host 2

Yep.

Host 1

The London pop collapses its local herd and sends one fetch. If I have viral traffic hitting all 300 pops globally, isn't my origin still going to receive 300 simultaneous requests?

Host 2

That is the exact limitation of flat edge networks, and it is why enterprise architectures mandate tiered caching, which is commonly referred to as an origin shield.

Host 1

An origin shield. Right. Okay.

Host 2

The edge pops, those hundreds of locations sitting close to the users, are configured so they do not actually possess a direct route to your origin server.

Host 1

Interesting.

Host 2

Instead, they collapse their local requests and then forward that single coalesced request to a massive centralized regional superpop. The superpop acts as the origin shield.

Host 1

I see. It's a hierarchical cache, a cache for the caches.

Host 2

Precisely. The origin shield sits directly in front of your origin infrastructure, usually in the same geographic region to minimize latency.

Host 1

So the Tokyo Edge node sends a request to the Shield. The London edge node sends a request to the shield.

Host 2

Right. And the shield performs a second layer of request collapsing on those incoming requests. Oh wow. The shield sends exactly one request to your origin, retrieves the payload, distributes it back to the edge POPs globally, which then multiplex it down to the end users.

Host 1

With a properly configured origin shield, your origin server only ever sees one single request globally for any given asset, regardless of whether a hundred users or ten million users request it simultaneously.

Host 2

It is the ultimate infrastructural defense mechanism.

Host 1

The edge collapses the users, the shield collapses the edge, and the origin remains perfectly insulated.

Host 2

Exactly.

Host 1

But you mentioned a phrase a few minutes ago that seems to be the absolute linchpin of this entire coalescing process. You said the requests have to share the same cache key.

Host 2

Right, the cache key.

Host 1

How does the edge node actually know two incoming requests are identical? It can't just be looking at the URL.

Host 2

Right. It's not just the URL. The cache key is the unique identifier, usually a cryptographic hash, that the cache uses to store and retrieve a response in its key value store. By default, most developers intuitively assume the cache key is just the URL path.

Host 1

Right. Like if the URL is API slash data, it's the same file.

Host 2

But the RFC dictates that the cache key is a complex composite. At a minimum, it is a combination of the HTTP request method, usually get or head, and the target URI.

Host 1

So a getRequest to API slash data and a post request to API slash data generate entirely different cache keys.

Host 2

Exactly.

Host 1

That makes sense. But it gets much more granular than that, doesn't it?

Host 2

It becomes significantly more complex when we introduce the very header.

Host 1

The very header.

Host 2

Right. The very header is how the origin server dictates the dimensions of the cache key to the edge network. If the origin includes a very header in its response, it is explicitly telling the cache do not just hash the URL and the method. You must also extract the values of these specific incoming request header and include them in your cache key hash.

Host 1

Let's trace a concrete example of that. What happens if the origin sends very except language?

Host 2

Okay, imagine you operate a localized website at example.com slash index.html. A user in Seattle requests it and their browser sends the header except language, EN-US.

Host 1

Right.

Host 2

The origin server processes this, generates the English version of the HTML, and returns it with the header very accept language. The CDN edge node receives this. Because of the very instruction, it generates a cache key that hashes the URL example.com slash index.html combined with the string EN-US.

Host 1

So it creates a specific bucket just for English.

Host 2

Yes. A moment later, a user in Kyoto requests the exact same URL, but their browser sends accept language JA-JP.

Host 1

Okay, so the edge node hashes the URL and JA-JP.

Host 2

Right. It compares this new key against its storage, realizes it does not possess the Japanese version of the page, and registers a cache miss. It forwards the request to the origin to fetch the Japanese HTML.

Host 1

The cache now maintains two separate isolated buckets for the English and Japanese responses, even though they share the exact same URI.

Host 2

Exactly.

Host 1

So if the cache key is the fundamental linchpin of request collapsing and the very header allows you to mutate that key, what happens when we manipulate it incorrectly? Because this seems like an area ripe for catastrophic configuration errors.

Host 2

Oh it absolutely is.

Host 1

I want to shift into the trade-offs and failure modes we found in the sources let's examine the very header cardinality explosion.

Host 2

This is a devastating failure mode that MDN specifically warns against in their advanced guides.

Host 1

Now does it happen?

Host 2

Let's say a well-intentioned developer decides to optimize their application by returning different image formats, maybe WebP for Chrome and JPEG for older browsers. Or maybe serving different HTML payloads based on the specific browser the user is running.

Host 1

Okay.

Host 2

They implement this by adding very user agent to the HTTP response.

Host 1

On paper, that sounds like a reasonable optimization. You want to serve Safari optimized code to Safari and Chrome optimized code to Chrome.

Host 2

It seems reasonable until you look at the raw data of how browsers identify themselves. The user agent string is not simply the word Chrome or Safari.

Host 1

But it's huge.

Host 2

It is a massive complex string that includes the browser rendering engine, the exact browser version, the underlying operating system, the OS version build, and often the specific mobile device hardware model. There are literally tens of thousands of highly distinct hyper specific user agent strings traversing the internet at any given moment.

Host 1

Oh, I see where this is going.

Host 2

The cache is a dumb, obedient system. It obeys the very header. Every time a request comes in, it hashes the exact literal string of the user agent.

Host 1

Oh no.

Host 2

Yeah. It ends up creating a unique cache bucket for a user on Chrome version 120 on Windows 11, a completely different bucket for Chrome 120 on Windows 10, and another bucket for Safari on an iPhone 14 running iOS 17.1. It fragments the cache into tens of thousands of microscopic buckets.

Host 1

Which means your cache hit ratio essentially plummets to zero.

Host 2

Precisely. Because the statistical probability of two independent users possessing the exact same hyper-specific user agent string hitting the exact same edge pop within the short TTL window of the cache is incredibly low.

Host 1

You have effectively bypassed the CDN entirely.

Host 2

You really have. The edge nodes are constantly writing data to disk, but they are almost never reading it back to serve a subsequent request. Your origin server is absorbing the full unmitigated brunt of the internet traffic.

Host 1

And your finance department is paying the CDN provider for massive amounts of storage and right bandwidth that is providing zero architectural benefit.

Host 2

Exactly.

Host 1

This is why MDN aggressively recommends using client-side feature detection using JavaScript in the browser to figure out what formats are supported instead of relying on fragile user agent sniffing on the back end.

Host 2

Right.

Host 1

And what about Very, the asterisk wildcard?

Host 2

Oh, very is the ultimate cache destroyer. The asterisk explicitly dictates this response varies based on some internal unnamable factor other than request headers. Wow. It instructs the cache that every single incoming request is fundamentally unique, guaranteeing zero cache hits. A stored response with very will always fail to match a new request. It is essentially a poorly implemented, highly inefficient version of no store.

Host 1

Manipulates with vary can clearly tank your performance. But let's pivot to something with much higher stakes. What happens if a developer accidentally caches data that belongs to a specific user? Let's analyze the security failures around misconfigured auth routes and PII leakage.

Host 2

This touches on a profoundly important distinction in the RFC between shared caches and private caches. The protocol was actually designed with built-in safety mechanisms.

Host 1

What kind of mechanisms?

Host 2

According to the specification, if an incoming request contains an authorization header, meaning the user is logged in, passing a JWT or session cookie. A shared cache like a CDN must absolutely refuse to cache the origins response by default.

Host 1

Oh, that's smart.

Host 2

The protocol naturally assumes that any authenticated data is highly personalized, sensitive, and not suitable for public distribution.

Host 1

So the system actively attempts to protect you from your own caching rules.

Host 2

It does. However, the RFC also provides escape hatches. A developer can explicitly override this default safety mechanism by appending certain directives to the cache control header.

Host 1

Like what?

Host 2

Specifically, if they add public must revalidate or smax age. They are officially signaling to the CDN, I understand the risks, but I authorize you to cache this authenticated response on the edge.

Host 1

And what are the consequences if they attach public to an API route that returns PII, personally identifiable information?

Host 2

It leads to an immediate catastrophic data breach.

Host 1

Walk me through that.

Host 2

Let's map out the scenario. Okay, you have an application dashboard at the URI slash API slash user slash profile. User A logs into the application, their browser attaches their secret auth token, and the origin server generates a JSON payload containing their profile data, their full name, their private email address, and their billing history.

Host 1

Okay.

Host 2

But an engineer accidentally hard-coded cache hyphen control, public max dash age equals 3600 on all slash API slash user endpoints.

Host 1

So the CDN reads the public directive.

Host 2

It sees public, which overrides the default auth protection. It takes user A's highly sensitive JSON payload and writes it directly to the EdgeNotes public storage. Crucially, unless instructed by a very header, the cache key is solely the URI/slash API slash user slash profile.

Host 1

Oh my God. The auth token isn't part of the key.

Host 2

Nope. A few seconds later, user B logs into their own account. Their browser makes a request to slash API slash user slash profile with user B's token. The CDN intercepts the request. It hashes the URI, looks up the cache key, and finds a fresh cache entry.

Host 1

It does not inspect the auth tokens because it wasn't instructed to include them in the hash.

Host 2

Exactly. It immediately serves the cache JSON response. User B is now staring at user A's full name, email, and billing history on their screen.

Host 1

That's a nightmare.

Host 2

It is a massive privacy violation, entirely caused by the misunderstanding of a single word, public, in an HTTP header.

Host 1

That is chilling. And it highlights exactly why cache control.private is so vital for authenticated routes. It instructs the shared cache to keep its hands off the payload entirely, but allows the user's local private browser cache to store the data to improve navigation speed. Definitely. Let's look at another highly nuanced architectural pattern discussed in the AWS Builder Center note. Negative caching.

Host 2

Negative caching is the practice of deliberately caching HTTP error responses. Normally we think of CDNs as mechanisms for caching 200 okay successful responses. Right. But we also need to consider caching 404 not found or 500 internal server error responses.

Host 1

Now, according to AWS best practices, you are highly encouraged to implement negative caching. Why is caching a missing page so critical to infrastructure survival?

Host 2

Imagine you operate a massive e-commerce platform. You delete a popular product page because the item is permanently discontinued. The URI now correctly returns a 404.

Host 1

Makes sense.

Host 2

If you do not instruct the CDN to cache that 404, every single time a user clicks an outdated link from a search engine or a malicious bot scrapes that dead URL, the request bypasses the edge and hits your origin.

Host 1

Uh-huh.

Host 2

Your database has to spin up, execute a query against the product tables, confirm the ID does not exist, and generate the 404 HTML template. This consumes valuable CPU and memory on your origin for a resource that isn't even there. If a botnet starts hammering a list of dead links, your origin could be brought down by traffic looking for ghosts.

Host 1

So by caching the 404, the CDN absorbs the blow. It instantly replies, nope, not here, without ever bothering the origin database. It's a shield against localized attacks or bad indexing.

Host 2

It protects your compute resources. However, AWS highlights a severe dilemma with this pattern. What if the error isn't a permanent 404, but a transient 500 internal server error?

Host 1

Okay. Let's say your primary database experience is a brief locking issue that lasts for exactly 10 seconds, causing your origin to throw 500 errors on the home page.

Host 2

If your CDN is configured to blindly negative cache any error response with the same long TTL you use for normal content, let's say an hour.

Host 1

You have just weaponized the cache against yourself.

Host 2

Exactly. The database lock resolves in 10 seconds. The origin is completely healthy again, but the CDN has already cached the 500 error page.

Host 1

Oh man.

Host 2

It will faithfully, efficiently serve that catastrophic error page to every single user who visits the homepage for the next hour, long after the origin has healed. You have effectively locked yourself into a prolonged self-inflicted outage.

Host 1

The cure becomes the disease.

Host 2

Right. But you should only cache a 500 internal server error for maybe five or ten seconds.

Host 1

Just long enough to absorb the thundering herd and protect the origin during the immediate hiccup.

Host 2

Exactly, but short enough that the CDN will quickly route a new request to the origin to test for recovery.

Host 1

That requires very fine-grained programmable control at the edge. You have to write logic that inspects the HTTP status code on the fly and applies dynamically different caching rules based on whether it's a 200, a 404, or a 500. It's not just a blanket rule.

Host 2

And implementing that level of programmatic control is exactly what separates basic CDN usage from resilient enterprise grade web architecture.

Host 1

Let's tackle one more failure mode before we move to application to design. Purge propagation and consistency lags. We spent a lot of time discussing time-based expiration. But what if a content editor publishes a massive typo on the home page and I need to delete that cache file right now, before the TTL expires?

Host 2

Right.

Host 1

The RFC specifies that caches should invalidate stored responses when they receive an unsafe method like a post put or delete request.

Host 2

The protocol does define that. If a client sends a delete request to example.com slash article one, the specific cache that physically processes that request is supposed to immediately invalidate its stored copy of Article One.

Host 1

Okay, so what's the issue?

Host 2

There is a massive, often misunderstood caveat here. The HTTP protocol only invalidates the specific localized cache node that the request passed through.

Host 1

I see the problem. If I'm an admin sitting in an office in New York and I update an article with a put request, my request routes through the New York Edge pop. The New York Pop updates its cache.

Host 2

Yes.

Host 1

But what about the edge node in London? It never saw that put request cross its wires. It has absolutely no idea the origin content was altered.

Host 2

The HTTP protocol's built-in invalidation is strictly localized. It does not possess a mechanism to magically sync state across a globally distributed network.

Host 1

So for global managed caches like CDNs, you cannot rely on standard HTTP invalidation methods.

Host 2

No, you have to bypass the protocol and rely on the CDN vendor's proprietary manual purge API. You have to execute an out-of-band API call to your provider sending a command that says purge the specific URI globally across all data centers.

Host 1

But even an API call isn't instantaneously global. We are still bound by the speed of light.

Host 2

Purge times have improved drastically.

Host 1

I mean, modern CDNs can often propagate a purge in under a second, but it still takes physical time for that command to replicate across hundreds of data centers worldwide.

Host 2

And during that propagation window, you experience consistency lags. A user requesting the page in New York might see the corrected article, while a user in Sydney is still being served the old version containing the typo.

Host 1

Right.

Host 2

If your application logic requires strict global transactional consistency, like a banking ledger or real-time inventory reservation system, you simply cannot safely cache that data at the edge. You have to architect the system around eventual consistency.

Host 1

Which brings us perfectly to the final section of our deep dive. Decision boundaries. Designing the architecture.

Host 2

It's a lot to consider.

Host 1

It is. How do we synthesize all of this? When you sit down to write an architecture document, how do you build a robust tiered caching policy based on these mechanics? Let's break it down by asset type.

Host 2

Okay, let's do it.

Host 1

First up, static assets. I'm talking about your compiled CSS files, your JavaScript bundles, your heavy image assets. What is the definitive policy?

Host 2

For static assets, the caching policy is absolute unyielding aggression. You want to instruct the edge network and the browser to hold on to these files for as long as theoretically possible. The standard policy is cache-control colon public max-age equals 31536000, comma, immutable.

Host 1

31,536,000 seconds.

Host 2

That works out to exactly one year. Is there a specific technical reason we use that exact number rather than just choosing a really high number like 99999999? It is actually a highly optimized value tied to modern protocol compression. In HTTP3, header compression is handled by a standard called QPAC. Q pack? Yeah. QPAC utilizes a static hard-coded table of predefined commonly used headers to minimize bytes over the wire. Max age equals 31536000 is literally entry number 41 in the QPAC static table.

Host 1

Wait, really?

Host 2

Yes. By utilizing that exact specific string, the network can compress the entire cache control header down to a single byte during transmission. It's an incredible micro optimization for network efficiency.

Host 1

That is wild. But if you instruct every browser on Earth to cache your core JavaScript application bundle for an entire year, how do you ever deploy a new feature? You can't issue a manual CDN purge to a million private laptops.

Host 2

You don't update the file. This is the architectural pattern of cache busting.

Host 1

Okay, walk me through that.

Host 2

When your CICD pipeline compiles the JavaScript, it generates a cryptographic hash of the file's contents and injects that hash directly into the output file name. So instead of deploying app.js, your server deploys app.a1b2c3d4.js.

Host 1

And if a developer changes a single line of code, the hash changes, and the resulting file name becomes app.f9e8d7c6.js.

Host 2

Exactly. And because we established earlier that the cache key is fundamentally based on the URI, a new file name generates a completely new, unique cache key.

Host 1

It is treated as an entirely new resource by the CDN and the browser.

Host 2

Right. The old file is simply ignored and slowly ages out of the cache storage. This pattern mathematically guarantees that the user always downloads the exact version of the JavaScript that matches the HTML they received, preventing version mismatch crashes.

Host 1

And what about that immutable directive we added at the end?

Host 2

That is the final optimization layer. We discussed conditional revalidation earlier. How a user hitting the refresh button forces the browser to check with a server, resulting in a 304 not modified. Right. The immutable directive instructs the browser this specific file will literally never ever change. If you possess it in your local cache, do not even bother asking the server if it has been updated.

Host 1

It completely suppresses conditional requests on manual page reloads, eliminating the network round trip entirely. It is the ultimate latency optimization for static bundles.

Host 2

It really is.

Host 1

Okay, that handles the static files flawlessly. But that brings us to the next category main resources. The actual HTML document, the ISR or SSR incremental static regeneration or server-side rendered HTML shells. You cannot cache bust your root index.html file.

Host 2

Right. The URL has to remain exactly example.com. You can't send users to example.com slash index dash hash one two three dot HTML.

Host 1

So what is the caching policy when the URL is fixed?

Host 2

Because the URI is fixed, you must pivot your strategy to prioritize absolute freshness over aggressive caching. You want to guarantee the user always receives the latest version of the application shell, so they get the updated links to the new hashed JavaScript bundles.

Host 1

Makes sense.

Host 2

The policy here relies heavily on our old friend, cache control.nocache, coupled closely with strong validators like e tag.

Host 1

We are back to the bouncer at the nightclub. The browser can hold the HTML file, but it must revalidate.

Host 2

Yes. NoCache forces the browser and the CDN to revalidate the HTML file on every single request. They must ping the origin and ask, has this HTML document changed?

Host 1

But because you configured the origin to include the E tag hash of the HTML document, if the marketing team hasn't updated the homepage, the origin simply responds with that tiny 304 not modified header.

Host 2

Exactly. So you guarantee the user never ever sees a stale application shell, but you don't waste massive amounts of bandwidth downloading the full HTML body over and over if they already possess the current version.

Host 1

It perfectly balances data integrity with payload size for the entry point of your application.

Host 2

You intentionally trade a tiny bit of latency the time it takes to execute the conditional request to ensure absolute consistency on your main document. You then instantly claw all of that latency back when the browser loads the static assets instantly from the local immutable cache.

Host 1

Okay, final category.

Host 2

High read JSON APIs.

Host 1

Let's say you have an endpoint serving a live comment section on a news article or a product inventory count during a sale.

Host 2

Right. Very common scenarios.

Host 1

It is dynamic, rapidly changing data, but it is heavily read by thousands of users. We can't cache it for a year, but if we use no cache, the thundering herd will overwhelm the origin database with conditional revalidations. How do we solve that?

Host 2

This is where we fully leverage the power of RFC 5861. For high read, rapidly changing data, strict transactional consistency is rarely required.

Host 1

Right. A comment section doesn't need to be millisecond accurate, it just needs to be fast.

Host 2

Exactly. The policy here balances brief freshness windows with asynchronous background updates. You implement a short max age, perhaps 10 seconds, combined with a much longer stale while revalidate window, perhaps five minutes.

Host 1

So the header reads cache control.max age equals 10, stale while revalidate equals 300.

Host 2

What this means in practice is that for any given 10 second window, every single user hitting that API endpoint receives the exact same instant JSON response directly from the edge cache.

Host 1

It completely collapses the thundering herd.

Host 2

It does. But as soon as that 10 seconds expires, the very next user who makes a request still gets served instantly. They receive the 10 second old stale JSON, but their request quietly triggers the CDN to open a background connection to the origin to fetch the new comments.

Host 1

It's the breadbasket analogy in action. The user gets the slightly stale data immediately, hiding the latency, while the CDN does the heavy lifting in the background. So 99% of your users receive a millisecond response from the edge. Your origin database only processes one query every 10 seconds instead of 10,000. And the data displayed is never more than a few seconds out of date.

Host 2

It is a masterful, highly resilient way to scale dynamic APIs without melting your database. When architected correctly, with a deep understanding of the RFCs, the cache stops being a liability, a source of cache addiction, and becomes a highly tuned, highly resilient shield that allows a small engineering team to serve traffic at a global scale.

Host 1

But before we wrap up this deep dive, I want to pivot to something entirely different. A concept we pulled from the very deep end of the AWS architecture notes, it is a security implication of caching that most developers never even consider.

Host 2

Oh, this is a fascinating topic.

Host 1

We spent this entire time talking about how caches speed things up. But that speed actually introduces a vulnerability. Let's discuss side channel timing attacks.

Host 2

Right. This is where the intersection of performance optimization and security becomes incredibly dangerous. Caches are, by their very definition, inherently vulnerable to side channel timing attacks.

Host 1

Why is that?

Host 2

To understand why, you have to look at the fundamental nature of what a cache does. It serves data significantly faster than the origin server. A response retrieved from an edge cache might be returned to the client in 20 milliseconds, while an uncache response requires a full database query and might take 200 milliseconds to fetch.

Host 1

But the discrepancy in response time is the feature. That's why we use it.

Host 2

Exactly. But that discrepancy in response time is also a massive leakage of information. A malicious actor does not need to bypass your authentication to see the raw data in the cache. They don't need to break your encryption. They only need to measure the response time.

Host 1

Walk me through a concrete scenario of how an attacker exploits this. How does measuring milliseconds lead to a data breach?

Host 2

Imagine an internal corporate HR application. It has a complex, dynamic search feature. If a manager searches for a specific employee's name, the system queries the database and returns their profile. Because this query is heavy, the application caches the result for a few minutes.

Host 1

Okay. Standard performance optimization.

Host 2

Now imagine an attacker, perhaps a curious employee without HR privileges, wants to know if executives are currently planning layoffs. The attacker simply writes a script that sends a search query for the term layoffs 2026 to the API.

Host 1

But they aren't authorized to see the results, so the API returns a 403 forbidden.

Host 2

Exactly. But the attacker doesn't care about the HTTP body. They only care about the stopwatch.

Host 1

I see.

Host 2

The first time they run the query, it takes 200 milliseconds before the server rejects it. But if a high-level HR executive actually runs that search a few minutes later, the complex database query executes and the system caches the fact that this query was run.

Host 1

Ooh, wow.

Host 2

If the attacker runs their script again and the API rejects them with a 403 forbidden, but this time the response comes back in 20 milliseconds, the attacker instantly knows with high mathematical probability that someone else with high privileges on the system recently searched for layoffs 2026.

Host 1

That is terrifying.

Host 2

And the data didn't leak, but the behavior of other users leaked through the timing variants. Exactly. Exactly. It's the ghost in the machine. In a multi-tenant SaaS application where thousands of different companies share the same infrastructure, an attacker could use timing attacks to map out exactly what sensitive reports their competitors are generating just by measuring how fast the API responds to arbitrary queries.

Host 1

So your cache doesn't just speed up your application. It actively broadcasts the digital footprint of your user's behavior to anybody observing the network timing.

Host 2

Exactly.

Host 1

Yeah. And it's fascinating and highly dangerous reality of optimized systems.

Host 2

It really is.

Host 1

So how does an architect defend against this? You can't just turn the cache off.

Host 2

The primary defense mechanism highlighted in advanced architecture is a technique called double keying the cache.

Host 1

Double keying.

Host 2

We discussed earlier how the cache key is a composite hash of the URL and the very headers. To prevent cross-tenant timing attacks, you must inject the tenant ID or the specific user ID directly into the cache key hash.

Host 1

So the cache key for the HR search isn't just hash URI plus layoffs. It becomes hash tenant ID plus user ID plus URI plus layoffs.

Host 2

Precisely. By injecting the user ID into the hash, you completely isolate the cache storage.

Host 1

So even if the HR executive runs the query and caches the result of their personal bucket, when the attacker runs the exact same query, the cache generates a completely different cache key based on the attacker's user ID.

Host 2

Exactly. The attacker suffers a cache miss, experiences the full 200 millisecond delay, and learns absolutely nothing about the executive's behavior.

Host 1

Double keying isolates the timing variant so you can only ever measure the speed of your own requests. It's an elegant solution to a terrifying problem.

Host 2

It is, but it heavily impacts your overall cache hit ratio because users can no longer share cache responses. It forces architects to make a hard decision between maximum performance and maximum operational security.

Host 1

That is a phenomenal place to leave it. We have covered a massive amount of architectural ground today. We started with the sheer terror of cache addiction and origin meltdowns, worked through the strict timestamp algebra of RFC 911, explored how stale extensions decouple latency, built origin shields to collapse the thundering herd, mapped out the destruction caused by very headers, and finally navigated the hidden dangers of side channeled timing attacks.

Host 2

It's been quite the journey.

Host 1

Thank you for bringing these sources to us and requesting this deep dive. Keep your origin shielded, audit your public headers, and always monitor your hit ratios.

Save to playlist

Manage playlists →

Share