Why DNS Failover Fails in Production
Transcript
Made automatically from the audio, so it may contain mistakes. Click a line to play from there.
You know, usually when we talk about a system failure, there's this expectation of like predictable mechanics.
Right, like clockwork.
Exactly. Like lockwork. You pull a lever, a gear stops, and the machine just halts. So you go into your control panel, you set your DNS time to live, your TTL, to exactly 60 seconds.
Because that's what the manual says to do.
Yeah, you just assume that if a physical region goes down, your traffic will gracefully fail over to the backup region in exactly one minute. I mean, that is the contract you believe you have with the architecture.
It's what we're all taught.
Right. But then a production server actually dies. Sixty seconds pass. 120 seconds pass. You look at the logs, and traffic is still stubbornly hitting that dead endpoint, generating, you know, thousands of errors a second. And you are just left staring at the screen, asking why.
Well, because the contract was an illusion. I mean, you gave the internet a strict instruction, and the infrastructure just fundamentally ignored it.
It is the absolute definition of a production nightmare. You think you have control via DNS, but the reality is the global network doesn't respect your TTLs at all.
Not even a little bit.
Like operating system stub resolvers cache things indefinitely. Java virtual machines have their own internal DNS caches that, well, they sometimes just default to infinite lifetimes.
Yeah, the JVM caching is notoriously brutal.
It really is. And then your local ISP's recursive resolvers routinely and intentionally ignore your authoritative TTLs just to save a little bandwidth on their end.
Which, you know, makes sense for them financially.
Sure. But it means relying on DNS failover as your primary real-time recovery mechanism is just fundamentally flawed. Because you are relying on thousands of autonomous third-party systems to respect a suggestion they have financial incentives to ignore.
Which is exactly the assumption so many architects build upon until they get burned in production. We treat DNS like a real-time remote control for traffic, when historically and structurally, it was designed as a loosely coherent, eventually consistent directory.
So today we are diving deep into our source material on web infrastructure to answer exactly why that is.
Getting right into the messy reality of it.
Exactly. Now, if you're listening to this, you already know the basic path of a web request. You know how a browser asks for an IP address, gets one, and fetches a page. We aren't going to re-explain the basics here.
Right, no basic recaps today.
Our mission today is to crack open the hood and examine the hardcore internal mechanics that govern why that path actually looks the way it does.
We're going to explore the severe physical limitations of DNS protocols and the wild economics driving BGP4 routing.
And the fragile illusion of any cast, plus the heavy math required to keep global server load balancing from just well, collapsing under its own weight.
Because our goal is to dissect why the underlying protocols, many of which were written decades ago in documents like RFC 1034, 1035, 4271, and 4786. Why they still dictate the modern economics of the Internet?
That's why those texts are still running the show.
It really is. Engineers are forced to make brutal trade-offs every single day based on these texts. So we are going to look at the spectacular ways these systems fail when those trade-offs collide in the real world.
So if DNS is fundamentally too slow and too restricted to save us during a data center fire, we have to look deeper at the physical constraints of the protocol itself.
Right. We have to look at the boundaries.
Because if you can't rely on the time to live expiring quickly, your next logical thought as an engineer might be brute force.
The classic brute force approach.
You know how it is. You might think, well, if I can't rely on the client asking me for a new IP when the primary dies, I'll just pack my initial DNS response with a hundred backup IP addresses up front.
Just give them the whole list.
Yeah. Hand the client an entire routing table so they can just try them all until one works. But we can't do that either.
No, that door is firmly, and it takes us right back to November 1987.
The dark ages of the internet.
Basically, yeah. RFC 1035. The foundational specification for DNS dictates a hard physical limit. Messages carried by UDP are strictly restricted to 512 bytes.
Wait, I have to pause there because 512 bytes is tiny.
It is remarkably small.
In a modern context, a single HTTP header can be larger than that. If I'm operating a massive global service, say I'm running dozens of data centers all over the world with thousands of edge nodes, how am I supposed to fit my entire list of fallback IPs into 512 bytes?
You don't. The design of the era prioritized reliability over capacity.
Meaning they just wanted to make sure the packet actually arrived.
Exactly. In the late 80s, the internet was highly fragmented. The maximum transmission unit, or MTU of various network links fluctuated wildly.
So you didn't know how big a packet the next router could actually handle.
Right. So by capping DNS UDP payloads at 512 bytes, the original architects guaranteed that a DNS packet would never fragment across any network link anywhere.
That's a defensive measure.
Completely defensive. It was to ensure the directory service always functioned.
Okay, but what happens today? If your response exceeds that 512-byte limit today, the authoritative name servers doesn't just send what it can and hope for the best, right?
No, it actively truncates the packet.
Okay.
It sets a specific flag in the DNS header called the TC bit, which stands for truncation.
Okay, so the client receives a DNS response, but the TC bit is flipped. The name server is essentially signaling, hey, I have more to tell you, but I couldn't fit it in the box.
Which creates a huge problem for the client.
Right. So what is the client's operating system forced to do next?
The client is forced into a TCP fallback. It realizes the UDP answer is incomplete, so it just discards it.
And it just throws it away.
Throws it away. The client must now initiate a full, stateful TCP connection directly to the name server just to retrieve that long list of IP addresses.
And this is where the performance penalty becomes lethal.
Absolutely lethal.
Before the actual HTTP request to load the website even begins, the client is forced to do a three-way TCP handshake with a DNS server.
Just to find out where to go.
Right. The client sends a SIN packet, waits for the server to send a SYNACK, and then sends an ACK back. Only then does it actually send the DNS query over TCP.
And depending on how far away that name server is, you are adding hundreds of milliseconds of latency.
You are making the user wait a noticeable amount of time before their browser even knows which web server to talk to.
Yeah, that performance penalty completely defeats the purpose of using a lightweight, stateless protocol like UDP in the first place.
It's like shooting yourself in the foot to buy a new shoe.
Exactly. Striving for extreme failover resilience by stuffing too many A or 4A records into your response guarantees a degraded experience for every single user on every single query.
Because they will all hit that 512-byte wall and suffer the TCP handshake tax.
Every single time.
So the brute force approach of sending every backup IP is dead on arrival.
We have to find another way.
Right. And the next logical workaround for an infrastructure engineer is to introduce indirection.
Using aliases.
Yeah. If I can't send a massive list of IPs, I'll use a C name, a canonical name. I'll point my root domain to an alias, like, you know, traffic dash manager-my company.com.
Which delegates the routing logic.
Exactly. And all that is sophisticated global load balancer dynamically change the IPs behind that alias based on regional health. That gives me the flexible failover I need without bloating the initial response.
Well, the logic is sound until it collides with the CNAME APEX constraint defined in RFC 1034.
Right. The apex constraint.
Yeah. The specification lays down an uncompromising rule regarding the zone apex, which is your root domain, like my company.com without the WWW.
Just the naked domain.
Right. The rule is if a CNAME resource record is present at a node, absolutely no other data can exist at that node.
I want to make sure the mechanics of this are totally clear. If I have my root domain, I am forbidden from putting a CNAME there if any other type of DNS record shares that same space.
Consider what fundamentally must live at the zone apex for a domain to function at all.
Well, you require a start of authority or SOA record.
Yes, and you require name server or NS records to delegate the zone to your providers.
And critically, you almost always have mail exchange or MX records, so your domain can actually receive email.
Exactly. So because RFC 1034 mandates that a CNAME must be the exclusive record at any given node, placing a CNAM at the apex creates an illegal state.
You cannot have an SOA and a CNAME side by side.
It breaks the protocol.
It's like a strict museum exhibit policy. Imagine a pedestal in a gallery.
Okay, I'm picturing it.
If you place a placard on that pedestal that says redirect to wing B, you are absolutely forbidden from placing an actual historical artifact on that same pedestal. It must be empty.
That's a great way to visualize it.
The placard is an alias. It's telling the observer my true identity and all my properties are located elsewhere.
Right. Go look over there.
If you were to put an artifact, say an MX record on that same pedestal next to the redirect sign, you create a logical paradox for the system trying to read it.
And the recursive resolvers logic breaks down completely in that scenario.
How so?
Well, when a resolver encounters a CNAM, its programming dictates that it must restart its query at the new target name to find the requested data. Okay. But if an MX record is sitting right next to the CNAM, the resolver is faced with an impossible choice. Does it use the local MX record sitting right there? Or does it follow the C name to look for MX records at the target destination?
Oh, I see. It doesn't know which source of truth to trust.
Right. RFC 1034 eliminates this ambiguity by making them mutually exclusive. To preserve the deterministic nature of DNS, you are barred from using CNAME at the zone Apex for your flexible failover architecture.
The friction here is incredible. We have TTLs being ignored by the rogue caching of ISP resolvers.
Yep.
We have the 512-byte UDP limit punishing us with TCP latency if we send too many fallback IPs.
Also true.
And we have RFC 1034 preventing us from using C names at the apex to abstract the problem away. DNS mechanics are absolutely fighting our real-world routing needs.
They really are. But the limitations get even more pronounced when we try to optimize for the user's geographic location.
Right, because we don't just want to send the user to an available server. We want to send them to the closest available server.
And DNS struggles to even identify where the user is in the first place.
Which is wild to think about.
The Google site reliability engineering documentation highlights this heavily as the DNS middleman problem.
The DNS middleman problem.
Right. The architecture of the internet dictates that end users rarely talk to an authoritative name server directly.
Their browser talks to a recursive resolver instead.
Exactly. Usually provided by their local ISP or a public DNS service like Google's 8.8.8.8, and that resolver goes out and talks to the authoritative server on the user's behalf.
So my authoritative name server, the one making the intelligent routing decisions, never actually sees the IP address of the user holding the phone.
Never.
It only sees the IP address of the massive ISP server asking the question.
The authoritative name server is entirely blind to the end user.
That seems like a massive flaw for geo routing.
It is. Imagine a large national ISP operating in the United States. To save on infrastructure costs, they might route all of their customers' DNS queries through one massive centralized data center located in Chicago.
Right. Because centralizing is cheaper for them.
But the ISP has customers everywhere. Miami, Seattle, Dallas. If a user sitting in a cafe in Miami types in your URL, their device queries the ISP, and the query pops out of the ISP's resolver in Chicago before hitting your authoritative server.
Okay. So my authoritative name server looks at the incoming query, sees a Chicago IP address, and executes its logic. Great. I have a data center in Chicago. Here is the IP for the local Chicago server.
Yes.
But the user is physically sitting in Miami.
Exactly. The name server returns an IP address heavily optimized for the central data center of the ISP, utterly failing the actual geographic location of the user.
There could be a pristine, highly available data center sitting idle in Miami, physically just miles away from the user.
But because of the DNS middleman problem, the routing logic sends their subsequent HTTP traffic all the way up to Chicago.
Adding dozens or hundreds of milliseconds of round trip latency to every single packet.
Which degrades the whole experience.
So if the design of the protocol masks the origin, how do infrastructure teams fix this? You can't just bypass the ISP resolvers without rewriting how every device on Earth connects to the network.
Well, the community developed an extension to the DNS protocol called EDNS0 client subnet.
EDNS Zero.
Right. It was engineered specifically to pierce this blind spot.
How does it work?
When a modern recursive resolver supports EDNS Zero, it intercepts the client's original IP address, masks off the last few bits to create a broader subnet, typically a /24 subnet in IPv4, and embeds that subnet information directly into the payload of the DNS query it forwards to the authoritative server.
Oh, that's clever. So the authoritative server received the query from the Chicago resolver, but now it can look inside the EDNS zero payload and realize the actual client subnet originates in Miami. I will ignore the Chicago origin of the query and hand back the IP address for the Miami data center.
It successfully restores geographic routing locality.
Your user gets routed to the edge node nearest to them. Okay, but utilizing EDNS zero introduces a significant, highly debated trade-off in the infrastructure community, doesn't it?
A massive trade-off.
The tension is immediately obvious. You are trading end user privacy for geographic routing performance.
Exactly.
You are taking network data that pinpoints the client's physical location, or at least their highly localized network neighborhood, and broadcasting it through the global DNS hierarchy to authoritative servers, they never directly contacted.
And the content provider suddenly gains visibility into the network origins of users pulling resources, even if those users are utilizing a centralized public resolver meant to anonymize their traffic. Which makes privacy advocates very uncomfortable.
I mean, rightly so. So if DNS is this fraud, if it's severely constrained by UDP size limits, plagued by TTL caching unreliability, legally prevented from using alias flexibility at the apex, and structurally blind to the user's location without leaking privacy data, then relying on DNS to point a user to a specific physical machine is a fragile strategy.
Very fragile.
It's clear that high availability infrastructure needs a way to decouple the concept of an IP address from a single piece of physical hardware.
Yes, we have to separate the identity from the metal.
Because if we can't change the IP address fast enough when a server dies, we need multiple servers to somehow share the exact same identity.
And the concept of decoupling the identity from the hardware leads directly to any cast routing.
Anycast.
To understand the magnitude of what anycast achieves, we have to contrast it with the foundational mental model most engineers start with Unicast.
Right, the standard way things work.
In a traditional Unicast architecture, an IP address is a globally unique identifier permanently bound to a single specific network interface on a single specific machine somewhere on Earth.
One IP, one machine.
Exactly. If a router sees a packet destined for that IP, there is only one valid physical destination.
But Anycast completely shatters that one-to-one mapping.
It redefines the IP address from a specific location to a generalized service identifier.
How so?
Well, AnyCast allows multiple geographically dispersed, entirely autonomous nodes to advertise the exact same IP address prefix to the global routing system simultaneously.
So you could provision a server rack in Tokyo, a server rack in London, and a server rack in New York, and have them all confidently announced to the internet I am IP address 203.0.113.5.
Yes. All three locations claim the exact same identity.
You know, when I first learned about this, it broke my brain.
It's counterintuitive.
It really is. If I am a core router sitting in the middle of the internet, peering at traffic, and I have a packet destined for this AnyCast IP, how does the network not completely collapse in confusion? You have three different continents claiming the exact same address.
The network doesn't collapse because of the underlying mechanics of BGP4, the border gateway protocol.
The protocol that holds the internet together.
Right. BGP is the routing protocol that glues the disparate networks of the global internet together. When BGP receives these identical advertisements from Tokyo, London, and New York, it doesn't view them as an IP conflict or an error.
What does it say them as?
BGP is a path vector protocol. It views these advertisements merely as multiple available paths to reach the same destination network.
So let's trace the mechanics of how BGP4 actually chooses the path based on RFC 4271.
Let's do it.
When a user in Paris sends a packet destined for this Anycast IP, how does BGP decide whether Tokyo, London, or New York gets it?
BGP evaluates paths based on a strict hierarchy of attributes. One of the most fundamental attributes is the AS path, the autonomous system path.
Okay, the AS path.
The internet is divided into thousands of autonomous systems, which are essentially large networks under a single administrative control, like Comcast, ATT, or a major cloud provider.
And each one has a number?
Exactly. Each AS is assigned a unique number. As a route advertisement propagates outward from our AnyCast node in London, every autonomous system that passes the advertisement along prepends its own AS number to the path.
It's leaving breadcrumb trail of networks it traversed.
That's a perfect way to put it. And the longer the breadcrumb trail, the less desirable the route.
Okay.
When a router in Paris needs to forward the user's packet, it looks at the routing table. It sees the AnyCast IP as reachable via Tokyo with an AS path of eight hops.
Kind of R.
Right. It sees New York with an AS path of five hops, and it sees London with an AS path of just two hops.
So BGP naturally selects the path with the shortest AS path length and hands the packet to the next neighbor on that route.
Exactly. BGP isn't looking for a specific server, it simply forwards the packet toward the topologically closest advertisement of that prefix.
That makes perfect sense for localizing traffic. The packet naturally gravitates toward the closest node. But here is where the reality of infrastructure diverges from the theory.
As it always does.
Right. Topological closeness in BGP, meaning the fewest number of network AS hops does not always translate to physical geographic closeness.
Not at all. Geographic distance is almost irrelevant to BGP.
Which is wild.
It is. The routing table is heavily distorted by the economics of internet transit.
Okay. To understand this distortion, we need to examine another crucial BGP attribute defined in RFC 4271.
The multi-exit disk or med.
The multi-exit discriminator.
Yes. Med is an optional non-transitive attribute utilized on external BGP links. Its primary function is to discriminate among multiple exit or entry points to the exact same neighboring autonomous system.
Okay. We need to ground this in a scenario. Give me a physical example of med overriding geography.
All right. Imagine your local ISP operates in Europe and they have two massive physical fiber connections to a global tier one transit provider.
Two big pipes.
Right. One interconnection point is located in London, and the second interconnection point is located in Frankfurt.
Okay.
Now the ISP has negotiated a complex business agreement where transit data flowing through the Frankfurt link is heavily discounted, say 50% cheaper than the bandwidth flowing through the premium London link.
So business logic dictates that the ISP wants to maximize its profit margins, so they want as much traffic as possible to use the cheaper Frankfurt pipe.
The ISP explicitly configures their BGP policies to reflect that financial preference. During the BGP decision process, if routes are learned from the same neighboring AS, the router evaluates the med value.
And the protocol dictates that the exit point with the lowest MED metric wins.
Exactly. The ISP assigns a low MED to the Frankfurt route and an artificially high MED to the London route.
I want to trace the real world implication of this configuration. I am a user sitting in a coffee shop in London. I request a web page hosted on an AnyCast IP. Okay. My packet goes to my local ISP's router in London. There's an AnyCast data center physically located a few miles away in London, reachable via the premium transit link.
Right down the street.
Right. But because of this MED configuration, my ISP's router evaluates the table, sees the Frankfurt link has a lower med, and routes my packet under the English channel all the way into Germany just to hand it off to the transit provider.
And then the transit provider routes it to their AnyCast node in Frankfurt.
The packet is routed geographically hundreds of miles away, simply to satisfy a peering agreement.
The industry terms this the BGP hairpinning effect.
Hairpinning.
Yeah. Traffic is hair pinned away from the geographically optimal node toward the economically optimal node.
So the internet routing table is fundamentally a map of business logic, peering truces and transit costs.
Exactly. Unless an architect deliberately engineers targeted peering arrangements at local internet exchange points, BGP remains entirely blind to fiber optic proximity.
But the capitalist reality of the internet takes precedence over the speed of light in fiber.
It really does.
So we rely on this AnyCast illusion where a single IP magically pulls you to the closest server, but we have to accept that closest actually means the cheapest administrative AS path according to upstream ISP configurations.
And while anycast is brilliant for solving the DNS constraints, because now one IP address can serve the entire globe without needing 500 twelve byte fallback lists, it introduces a terrifying new class of highly volatile problems.
As soon as we start talking about stateful connections, right?
The moment we stop talking about stateless traffic and move to state.
This is the crux of the architectural dilemma. DNS over UDP is generally a single stateless packet request, followed by a single packet response.
Right. Fire and forget.
If my UDP packet gets hairpinned to the Frankfurt Endicast node instead of the London one, I suffer a minor latency hit, but the transaction still works. The Frankfurt node answers the query, forgets about me, and we're done.
But loading a modern web application isn't stateless.
No. It relies on stateful TCP connections, or quick over UDP, both of which require establishing a connection, maintaining sequence numbers, and keeping a session in memory.
So what happens when the underlying BGP routing fluctuates in the middle of a complex stateful transaction.
This is exactly where anycast shows its sharpest edges.
To operate stateful services over anycast safely, RFC 4786 outlines strict operational practices regarding how upstream networks balance load.
And the danger lies in a common networking technique called equal cost multipath or ECMP, right?
Yes.
I understand the theory of ECMP. A router looks at its routing table and sees two different paths to the same destination prefix. Both paths have the exact same cost. The AS path length is identical, the med is identical.
So there's a tie.
Right. Instead of leaving one link idle, ECMP allows the router to distribute traffic across both links simultaneously to maximize bandwidth utilization.
Distributing load across equal cost links is highly efficient, provided the hashing algorithm utilized by the router respects the integrity of a flow.
Meaning all the packets from one session state together.
Exactly. A flow-based ECMP hash ensures that all packets sharing the same source IP, destination IP, source port, and destination port are mathematically bound to the exact same physical link.
Okay, but what's the failure mode?
The catastrophic failure mode occurs when network operators deploy per packet load balancing or PPLB.
Load balancing on a per packet basis instead of a per connection basis.
Per packet load balancing distributes packets across the equal cost links in a round robin fashion. Literally packet by packet.
Completely oblivious to the TCP flows those packets belong to.
Completely blind to it.
I want to walk through exactly how this destroys a system, because I've actually heard nightmare stories about this causing 3 a.m. pager duty alerts that take weeks to debug.
Oh, it's brutal to troubleshoot.
Let's trace it.
Let's trace it.
Okay, let's say I initiate a stateful TCP connection to an NDCast IP. I send my very first packet, the SYN packet, to begin the handshake.
Okay.
The upstream ISP router, utilizing PPLB, takes my SYN packet, picks path A, and routes it to the AnyCast data center in Chicago. The Chicago node is now waiting for my final acknowledgement to establish the session.
Right.
I receive the SYN-AC and immediately send my third packet, the ACK packet, back to the AnyCast IP. The packet hits that same upstream ISP router.
But because the router is doing per packet load balancing.
It says, I sent the last packet down path A, this next packet goes down path B. It routes my ACK packet away from Chicago and instead sends it to the AnyCast data center in Dallas.
And the Dallas AnyCast node receives an ACK packet for a TCP session it has absolutely no knowledge of.
Because the SYN went to Chicago.
Exactly. Its internal state table has no record of your initial SYN packet or the sequence numbers.
So the Dallas server assumes this is a rogue packet or a remnant of a connection that is already timed out.
It immediately drops the packet and fires back a TCP RST, a reset flag, killing the connection completely.
And because PPLB is happening packet by packet, the user might hit refresh, the round robin logic might happen to route the entire handshake to Chicago this time, and the page loads perfectly.
Right. It manifests as a chronic, intermittent, highly volatile instability that is incredibly difficult for the application owner to diagnose, because the problem is occurring deep within the routing logic of a third-party transit provider.
You are entirely at the mercy of how every single upstream router on the internet hashes its traffic.
If they use flow-based ECMP, your stateful AnyCast service works.
If a single provider in the chain switches to PPLB to optimize their link utilization, they will silently destroy your handshakes.
This is why operating stateful TCP over global Anycast is considered one of stateful connection resets doesn't stop at the most hostile engineering environments possible.
There's terrifying.
Even if the packet survives the BGP hairpinning, and even if it successfully traverses flow-based ECMP routers to arrive at the correct AnyCast data center, the journey is far from over.
We have to get to the actual server.
Right. As detailed in the Google Site Reliability Engineering Texts, you have to zoom into the architecture of the data center itself. The packet arrives at a local virtual IP or VIP residing on a load balancer.
That load balancer still has to forward the packet to one specific backend server out of a massive pool to actually process the request.
And inside the data center, you face a microscopic version of the exact same stateful problem.
The load balancer received the packet destined for the VIP. It looks at a pool of, let's say, a hundred back-end servers. It needs to pick one.
And critically, it needs to pick the exact same backend server for every subsequent packet in that specific TCP flow. Otherwise, the backend server fires a TCP RST, just like the Dallas Anycast node did.
The load balancer must maintain flow affinity.
Yes. The naive approach is to use a state table. The load balancer simply keeps a massive list in memory, tracking every single active connection and which server it belongs to.
But at hyperscale, maintaining a state table introduces a catastrophic single point of failure.
Completely. So to avoid that fragility, you have to use a stateless approach.
Hashing.
You use hashing. You take the four-tuple from the packet, the source IP, destination IP, and the ports, you run it through a fast hash function, and you use modulo arithmetic based on the total number of backend servers.
Okay. Let's unpack that. The standard formula is hash of the packet, modulo n, where n is the current number of active servers in the pool.
The math is highly efficient. If you have 100 servers, the hash generates a large integer and dividing it, modulo 100 yields a consistent server index between zero and ninety-nine.
The load balancer instantly knows where to send the packet without keeping a single byte of state in memory.
It seems perfectly elegant until the inevitable hardware failure occurs.
Right. What happens when server number 42's power supply catches fire and it dies?
The load balancer's active health checks detect the failure and pull server 42 out of the rotation to prevent traffic from black holeing.
The pool size shrinks.
Which means n becomes n minus one.
If n was 100 n is now 99, the denominator in the modulo math changes for every single packet arriving at the data center.
The mathematical foundation of the hash shifts entirely. When n becomes n minus 1, calculating the hash modulo 100 yields a completely different remainder than calculating the hash modulo 99.
That implies that almost every single active TCP packet suddenly maps to a completely different back-end server than it did a millisecond prior.
The mapping is totally scrambled.
The result is a massive cascading connection reset event across the entire cluster.
A single routine server death takes down every active user session in the building. You solve the global anycast problem only to nuke your own data center with a basic modulo operation.
So how do infrastructure giants actually prevent this cascade?
They rely on a mathematical algorithm introduced by David Carger and others in 1997 called consistent hashing.
Consistent hashing.
Right. Instead of utilizing a rigid modulo operation based on the total count of servers, consistent hashing maps both the backend servers and the packet hashes onto a conceptual ring or a circular key space.
Okay, I want to visualize this out loud for the listener. Instead of an Excel spreadsheet that constantly renumbers all the rows when one row gets deleted, imagine a large roulette wheel.
The roulette wheel is an excellent visualization.
Okay.
You take the IP address or identifier of each of your 100 back-end servers and you hash them. The resulting hash values dictate where each server sits around the perimeter of the roulette wheel.
They are fixed points on the circle.
Exactly. When a packet arrives at the load balancer, you hash the packet's flow details. That hash value acts like the ball dropping onto the roulette wheel, landing at a specific coordinate on the circle.
And to determine which server handles the packet, the load balancer's logic simply moves clockwise around the perimeter of the wheel until it encounters the very first server point.
The brilliance of this geometry becomes apparent during a failure.
Right. If server 42's power supply dies, its specific point is removed from the perimeter of the wheel.
And the only packets affected are the ones that were landing in the specific arc of the wheel that map to server 42.
Because the server is gone, those specific packets will now just travel a little further clockwise to find the next available server on the perimeter.
But all the other packets landing in all the other arcs mapping to the other 99 servers. Consistent hashing remains remarkably stable when the pool size fluctuates. It isolates the disruption geometrically.
So when n becomes n minus 1, only one over n of the active connections are rerouted and dropped.
The remaining 99% of connections stay perfectly stable.
It completely prevents the cascading TCP reset event.
Making it a fundamental non-negotiable requirement for operating stateful VIPs at hyperscale.
That is a beautiful application of mathematics solving a deeply physical networking problem.
It really is elegant.
So we've mapped the journey from the global BGP chaos down to the stability of the data center ring. But there is a mechanical step we haven't covered.
Getting the packet to the server.
Right. Once the load balancer uses consistent hashing to select the back-end server, how does it physically move the packet across the data center network to that server without breaking the IP protocols?
Because handling immense scale forces data centers to manipulate packets internally, introducing entirely new layers of overhead and risk. They do. In early architectures, to get the packet from the load balancer to the back end without acting as a stateful proxy, engineers utilize a technique called direct server response or DSR.
Let's trace the mechanics of DSR.
DSR operates by manipulating the data link layer, layer two of the OSI model.
Okay, the MAC address level.
Exactly. When the load balancer receives the packet destined for the virtual IP, it leaves the layer three IP addresses completely intact. Instead, it alters the destination MAC address of the Ethernet frame, replacing it with the specific MAC address of the chosen backend server.
So by leaving the layer three IP address intact, the back-end server receives the packet and believes it was sent directly from the client to the VIP.
Exactly, and that enables the critical optimization. When the back-end server processes the HTTP request and generates the payload, it sets the client's IP as the destination and the VIP as the source. The asymmetry there provides massive bandwidth savings.
Because in a typical web transaction, the incoming HTTP request is tiny, maybe a few hundred bytes of headers, but the response from the server could be a 10 megabyte video file.
Right. If the load balancer doesn't have to process the return traffic, you drastically reduce the CPU and network bottlenecks on your load balancing tier.
The efficiency gains are tremendous. But DSR imposes a severe structural constraint on your physical network, doesn't it?
It does. Because DSR relies exclusively on modifying layer two MAC addresses, the load balancer and all of its associated back-end servers must reside within the exact same layer two broadcast domain.
They essentially have to be plugged into the same contiguous local network segment.
And for a company operating at global scale, restricting a server pool to a single broadcast domain is a non-starter. Right. Google outgrew the broadcast domain limitation long ago. To scale server pools across giant data center floors, between different buildings, or even across continents, they transitioned to packet encapsulation.
Specifically utilizing GRE, generic routing encapsulation.
Yes.
We need a solid mental model for GRE.
Think of it like a Russian nesting doll made of shipping labels. Okay. Instead of manipulating the local MAC address, the load balancer takes the entire original IP packet arriving from the user and seals it completely intact inside a brand new larger IP packet.
This creates an IP and IP tunnel.
Exactly. The outermost shipping label has the load balancer as the source IP and the specific back-end server's internal routing IP as the destination.
So the internal data center network routes the packet based purely on the outer envelope. The routers don't even know there is another fully addressed packet hidden inside.
The network delivers the outer packet to the backend server. The back end server's networking stack receives it, strips off the outer GRE layer, and exposes the original user's packet hidden inside.
It processes the request in response.
This encapsulation severs the physical dependency of DSR. The load balancer and the back end can reside on separate subnets, separate buildings, or separate continents so long as they have layer three IP reachability. The tax is highly specific. Appending the outer IP header and the GRE protocol header adds exactly 24 bytes of overhead to the original packet.
24 bytes sounds trivial, but network links are governed by a maximum transmission unit, the MTU.
And on a standard Ethernet network, the MTU is hard-coded to 1500 bytes.
So if a user uploads a payload that maximizes their packet size to exactly 1500 bytes, and the load balancer slaps a 24-byte GRE header onto it, the resulting packet is now 1524 bytes. When a router encounters a packet that exceeds its MTU, it cannot forward it intact, right?
The router is forced to perform fragmentation.
It must halt, chop the 1524 byte packet into two separate fragments, append new headers to both fragments, and transmit them individually.
And fragmentation is computationally expensive.
It spikes the CPU utilization on the routers doing the chopping. Worse, the receiving backend server has to allocate memory buffers to hold the first fragment while it waits for the second fragment to arrive, often out of order, before it can reassemble the original packet.
If one fragment drops, the entire packet is corrupted.
Fragmentation causes catastrophic performance degradation and latency jitter at scale.
To successfully deploy VIP encapsulation with GRE, infrastructure teams are forced to re-engineer their entire internal data center network.
They have to configure jumbo frames.
Yes, increasing the MTU to 9,000 bytes across every single switch and router interface in the building just to absorb that 24-byte tax without triggering fragmentation.
The operational complexity required just to move a packet from the front door to a server rack is staggering.
It's an incredible chain of dependencies. Exhausting the RAM of every core router on Earth, Tier 1 ISPs enforce strict import filters based on prefix length.
If you attempt to advertise a single host route, slash 32 prefix in IPv4, which represents exactly one IP address, almost every major network provider will instantly drop the advertisement.
It's like you can't just submit a single standalone article to a global academic index. You are forced to submit the entire volume it belongs to, or they refuse to list your work at all.
To ensure your specific AnyCast IP is globally reachable, you were forced to advertise a covering prefix.
In modern IPv4 routing, this generally means advertising an entire slash 24 block, which encompasses 256 contiguous IP addresses, just to guarantee the reachability of your single AnyCast service IP.
Advertising an entire block of 256 IPs creates a terrifying operational dilemma if you were trying to be efficient.
Companies don't want to waste IPs.
What if I host multiple completely different AnyCast services within that same slash 24 block? Let's say I have my global DNS resolver service operating on IP.10 and a completely separate web application firewall service operating on IP.20.
Okay, two distinct services.
If the DNS service software crashes on my Tokyo node, but the firewall service is still running perfectly, what do I do with the BGP advertisement?
You are trapped by the covering prefix dilemma, forced to choose between two destructive options.
Choice one.
Choice one. You instruct your BGP router to withdraw the entire slash 24 covering prefix from the Tokyo node.
Which successfully routes all global traffic for the Crash DNS service away from Tokyo to the next nearest healthy Anycast node.
Which is the desired failover behavior.
But because you withdrew the entire slash 24 block, you also just intentionally took the perfectly healthy web application firewall on.20 completely offline for the entire region.
You deliberately sacrificed a healthy service to save a sick one. That's known as pessimistic withdrawal.
Okay, what is the alternative?
Choice two. You leave the slash 24 prefix advertised. The web application firewall remains online and continues serving Tokyo users locally.
However, because the prefix is still active, BGP will continue routing global traffic for the crash DNS service to the Tokyo node.
The DNS packets arrive at the data center, find the service dead, and fall into a black hole. Users experience a complete outage. And speaking of traffic falling into black holes, we haven't even touched on how security appliances react to anycast.
Let's define how RPF operates in a standard Unicast environment.
In Unicast, a specific IP address resides in one specific physical location. If a core ISP router has a port connected to a customer in New York, it expects traffic originating from that New York customer's IP block to enter through that specific port.
If the router suddenly sees packets claiming to be from the New York IP block, arriving on a port connected to appearing link in Los Angeles.
The router's logic concludes the source IP is being mathematically spoofed, likely as part of a distributed denial of service attack.
Under a strict RPF configuration, the router silently drops the spoof packets to protect the network.
But with any CAS, the IP address genuinely does live in New York and Los Angeles and London simultaneously.
Because anyCAST traffic legitimately originates from multiple diverse global locations at the exact same time.
So anycast nodes trigger strict RPF filters constantly.
Operating an AnyCast network requires relentless vigilance and constant out-of-band coordination with upstream transit providers to ensure they recognize your IP blocks as multi-homed any cast networks and configure exceptions to relax those strict RPF checks.
The friction at every layer is staggering.
Given the risks of stateful TCP handshakes being destroyed by per-packet load balancing, the absurdity of BGP hairpinning based on transit costs, the CPU tax of GRE fragmentation, the covering prefix dilemma, and the paranoia of RPF security filters. How do infrastructure architects actually synthesize this? When do you choose to deploy DNS versus anycast versus VIPs?
You deploy them as a layered nested defense strategy. Each tool solves a different tier of the routing problem.
First, DNS. Despite the reality of ignored TTLs and the 512 byte UDP truncation limits, DNS remains the most effective tool for coarse global traffic distribution. You use it when ultra low latency isn't the absolute critical factor, and when you can tolerate the eventual consistency of caching delays. DNS steers the user to a general geographic region.
It gets them to the right continent. And then you deploy anycast.
You deploy any cast at the edge of your network footprint.
You use it strictly for services that are natively stateless, like DNS resolution itself, or for extramination services that have been incredibly carefully tuned, working with ISPs to guarantee flow-based ECMP to handle stateful TCP safely.
What is the primary architectural benefit of placing Anycast at the edge rather than deep in the application tier?
Beyond the localization benefits, AnyCast nodes function as regional blast doors for DDoS attacks.
Because BGP routing inherently pulls traffic to the topologically closest advertised node, right?
Yes. A massive volumetric attack originating in eastern Europe gets absorbed entirely by the Eastern European Anycast nodes.
The malicious traffic never crosses the Atlantic to overwhelm the core US infrastructure. Anycast is fundamentally a structural defense mechanism against network layer saturation.
So DNS steers globally, anycast catches the traffic at the regional edge and filters the noise. And VIPs?
VIPLOVALANCE USI encapsulation and the consistent hashing rings we discussed operates exclusively inside the walls of the data center.
Once the validated traffic survives the BGP journey and enters the facility, VIPs provide the granular, stateful resource utilization required to distribute millions of concurrent TCP connections across thousands of backend application servers without dropping state when hardware inevitably fails.
It is a phenomenal, tightly woven architecture. Every layer compensates for the weaknesses of the layer above it.
It has to, or the whole thing falls apart.
But just when we think we have the rules of the internet neatly defined, a massive paradigm shift emerges that threatens to upend the entire structure. I'm talking about RFC 8484, DNS over HTTPS, universally known as DOT.
DOH is a fascinating protocol because it fundamentally subverts almost every historical constraint we discussed regarding DNS mechanics.
We started this deep dive exploring how you can't rely on OS stub resolvers and ISP caches because they arrogantly ignore your TTLs.
And we examined how the 512-byte UDP limit forces devastating TCP fallbacks.
DOH looks at that entire legacy system and essentially decrees we are going to bypass the operating system's network stack entirely.
It bypasses the legacy infrastructure by tunneling DNS queries inside a multiplexed, encrypted HTTP2 stream running over TLS.
DOH shifts the entire control plane of DNS resolution out of the operating system's deeply embedded network layer and elevates it directly into the web browser's application layer.
The implication for privacy is massive.
The local ISP routing the packets has absolutely no idea you're making a DNS query. Furthermore, because the queries are running over HTTP2, the archaic 512-byte UDP limit is completely irrelevant. You can transmit massive, complex DNS responses containing thousands of routing rules without ever triggering a truncation flag.
That sounds like DAW is the silver bullet. It solves the middleman privacy problem, it fixes the TTL caching nightmare, and it obliterates the UDP size limits.
But having spent an hour dissecting engineering trade-offs, I know there is a severe cost to this.
The cost is extreme statefulness. DAW takes DNS, which was intentionally designed in 1987 to be a lightweight, single packet, stateless UDP protocol and forces it into an incredibly heavy, stateful connection, requiring a full TCP handshake and a cryptographic TLS negotiation before a single query can be transmitted.
And we just spent half this deep dive exploring how stateful TCP is the mortal enemy of any cas stability due to ECMP hashing and packet load balancing.
The paradox is profound. DOH forces us to rely far more heavily on perfectly tuned stateful edge infrastructure to handle what used to be trivial stateless traffic.
It solves the privacy and caching problems, but it introduces massive computational and stateful overhead to the most foundational layer of the internet.
It's a huge trade-off.
This has been an unbelievable journey into the hidden mechanics of the web. To summarize how we got here, we started with the infuriating reality of a 60-second DNS TTL failing to execute a failover entirely because the legacy infrastructure of the internet refuses to obey the rules.
And the museum pedestal strictness of the CNAME APEX constraint that creates logical paradoxes if mixed with MX records.
We moved into the volatile economics of BGP 4 routing, where multi-exit desk configurations and hairpinning prove that packets often route based on transit discounts and capitalist truces rather than fiber optic geography.
We looked at how the illusion of AnyCast collides violently with stateful TCP if upstream ISPs utilize per packet load balancing, destroying handshakes in transit.
And how inside the data center, engineers rely on the mathematical elegance of consistent hashing, the roulette wheel of servers, to prevent cascading cluster failures when a server dies and N becomes N minus one.
We analyzed the 24-byte tax of GRE encapsulation and the resulting CPU death by MTU fragmentation, alongside the brutal covering prefix dilemma of BGP aggregation and the constant paranoia of strict RPF anti-spoofing checks dropping legitimate traffic.
And we wrapped up by seeing how DNS over HTTPS is currently rewriting the rules of the game, shifting the control plane up to the browser, solving caching issues, but introducing massive stateful overhead.
It's a lot to take in.
To you, the listener, who thought that loading a web page was just a simple matter of typing a URL and waiting for pixels to appear.
It really is a marvel of engineering that it works as reliably, as it does. But I want to leave you with a final, slightly provocative thought to mull over regarding DOH.
Go for it.
We spent decades building Anycast and distributed BGP routing specifically to avoid single points of failure. The Internet was structurally designed to be decentralized and resilient. But with the rapid adoption of DNS over HTTPS, because it requires complex browser integrations and heavy TLS infrastructure to operate at scale, DNS resolution is rapidly consolidating into the hands of just two or three massive centralized cloud providers.
So the question we have to ask ourselves is by trying to secure our DNS privacy and bypass those unreliable local ISP caches, are we accidentally recreating the exact centralized choke points that the original architects of the internet spent decades trying to design our way out of?
It's definitely something to think about.
Think about that the next time you set a 60-second TTL, assume it's gonna save your production environment, and you start helplessly at the logs when the internet decides it has other plans.
Yep, the internet always has other plans.
Thank you for joining us on this deep dive into our source material. We will catch you on the next one.