Web Arch & Infrastructure
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome back to the Explainer. Today we're completely bypassing the basics and diving straight into the protocol mechanics of global web infrastructure. We know you guys already understand the end-to-end web request topology from our previous deep dive, so yeah, we're absolutely skipping the beginner DNS is a phone book analogies today.
Instead, we're looking directly at the raw data from the Google SRE handbooks and core RFCs to figure out exactly how traffic gets steered across the globe. Literally from layer 7 DNS right the way down to layer 3 BGP Anycast. To get through all of that, here is our roadmap for today.
First, a deep dive into the DNS resolution pipeline, then record semantics, Unicast versus BGP Anycast, front-end VIP routing, and finally production failure modes. Right, let's kick things off with section one. DNS resolution pipeline.
Deep dive. What's really fascinating here is the explicit step-by-step iterative process. Rather than just making a direct magical connection from a client to a final answer, a query actually has to iteratively ascend.
It starts down at the stub resolver on the client, moves up to the recursive resolver, which then queries the root server. The route then refers it over to the top-level domain server or TLD, and finally it hits the authoritative name server.
Now, the absolutely crucial point here is the strict distinction between authority and caching. The recursive resolver just proxies those queries and caches the responses for a limited time to live, which, quite honestly, is the only thing stopping the entire internet from melting down under load.
On the other hand, the authoritative name server doesn't proxy a thing. It holds the true master zone data. It's your absolute final source of truth. Moving swiftly along to section two, record semantics and extensions.
Looking at standard record semantics pulled straight from RFC 1035, things are pretty cleanly categorized. A and AAA records map directly to IPv4 and IPv6 addresses respectively. C names handle your aliases, pointing one domain to another, but as you probably know, you literally cannot put a CNAM at the zone Apex, which is exactly why modern providers use alias or ANAM records for Apex flattening.
NS records delegate zone authority to specific name servers, and the SOA or start of authority manages all those core zone replication parameters like refresh and retry timers. But there is a fundamental routing flaw in this whole pipeline.
Think about it. How can an authoritative server optimise geographic routing if it only ever sees the IP address of the recursive middleman? So if a user in London uses a recursive resolver hosted in New York, the authoritative server just blindly assumes the user is in New York and routes them to a US data center.
That adds massive, completely avoidable latency. Well, the elegant answer to that routing flaw is EDNS client subnet, or ECS. This is a game-changing extension. It actually passes a portion of the client's real subnet data right along with the query through the recursive resolver.
Because of this, the authoritative server can calculate a geographically optimal IP response based on where the user actually is, rather than where their recursive resolver happens to be sitting. This is the exact mechanism CDNs rely heavily on to steer your users to the closest edge node.
Additionally, we really have to consider the security and integrity of these queries. RFC 8484 introduced DNS over HTTPS, or DOS. DOSH neatly maps DNS queries into HTTP exchanges to fully utilize TLS security.
This completely prevents unprivileged on path devices from interfering with DNS operations, actively mitigating things like passive surveillance and malicious traffic diversion. Let's drop down a layer for section three.
The divide between unicast vs anycast is a brilliant contrast. Under Unicast, every single node gets a completely unique IP address, meaning you have to rely heavily on layer 7 DNS to steer traffic to the right data center. But with BGP Anycast, it's a different story.
Multiple global edge points of presence advertise the exact same IP prefix, so the routing doesn't happen at layer 7 at all. It happens natively down at layer 3, strictly based on the physical network topology.
In an Anycast environment, the routers themselves are doing the heavy lifting, deciding exactly where the packet goes using the BGP4 decision process outlined in RFC 4271. When multiple routes to the exact same Anycast IP exist, the router evaluates the path sequentially.
First up, it prefers routes with the highest local preference. If there's a tie there, it looks for the shortest AS path length. And if it's still tied, it evaluates the lowest multi-exit discriminator, or MED. It is just a completely different paradigm to DNS-based steering.
Okay, section 4. VIP load balancing, front-end routing. So according to Google's SRE principles, a packet arrives and hits a shared virtual IP or VIP.
A network load balancer processes it, but rather than dealing with the severe scale limitations of layer 2 direct server return, the load balancer actually encapsulates the packet in an outer IP packet using generic routing encapsulation, or GRE. This is incredibly powerful because it allows the back end server to be anywhere, literally on a separate continent, stripping the GRE layer and processing the inner packet directly.
And, you know, keeping these massive data planes stable requires some seriously robust health checking. Take AWS Route 53's constant work architecture as a prime example. It completely decouples control plane API mutations from fixed size data plane status tables.
This is absolutely crucial because it means the system never accidentally overloads itself during a massive regional failover. But the real challenge here is handling stateful protocols like TCP. Think about it.
If a back end fails and you're just using simple modulo math to map connections, every single active connection across the entire data center gets reset instantly. Absolute chaos. To prevent this, Google SRE heavily mandates consistent hashing.
This ensures that when a single backend fails and the server pool resizes, only the connections to that specific failed machine are disrupted. All your other stateful TCP flows remain completely stable. Which brings us to our final part, section 5, failure modes in production.
Of course, the illusion of a flawless direct connection shatters when these protocols interact poorly at scale. The first major failure mode stems from rogue recursive resolvers. When engineers need to fail over traffic, they'll drop the DNS TTL to a very low value.
But if rogue recursive resolvers just flat out ignore those low TTL values, which let's be honest, they frequently do, they just continue to cache and serve the old IP. This completely breaks rapid DNS load balancing and actively prevents your traffic shifts from propagating.
Second, any cast introduces its own unique risks. According to RFC 4786, if you have BGP root flaps or if an equal cost multipath algorithm is performing per packet load balancing rather than per flow, packets from the exact same TCP flow can suddenly be routed to completely different Anycast nodes in entirely different cities.
The receiving node has no record of the handshake, instantly breaking that stateful TCP connection. Finally, we have the classic self-inflicted wound of extensive CNAM chains. When a domain is an alias to an alias to an alias, it incurs massive compounding latency penalties.
The poor recursive resolver is forced to restart the entire resolution loop from the top for every single new alias it encounters before it ever finds a usable A or AAA record. It's a completely avoidable architectural mistake that absolutely destroys your time to first byte.
So the crucial point we are left with today is this. As modern web infrastructure evolves, our layers of abstraction are only getting thicker. With BGP Anycast heavily hiding the true physical network topology and DNS over HTTPS completely obscuring queries from local network operators to prevent interference, the ultimate question remains in a fully distributed encrypted routing environment, who really controls the flow of your traffic?
It's a fascinating question, and one every infrastructure engineer needs to keep right at the forefront of their mind.