Web Architecture Deep Dive
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's jump straight into this explainer. Today we're unpacking the sophisticated mechanics of web architecture, specifically looking at the proxy boundary and beyond. Now, if you're an engineer working with modern distributed systems, I know you already understand the basics of edge layers, CDNs, and WAFs, so we're pushing right past that edge.
We are going to look at exactly what happens when your fleet is under immense pressure, and how the precise mechanics of reverse proxies, transport protocols, and load balancing algorithms actually keep your systems alive. So here is our agenda for this deep dive.
One, layer four versus layer 7 proxy architecture. Two, the evolution of HTTP transport, three, advanced load balancing algorithms, and four, health checks and the fail open rule. Let's kick things off with section one, layer four versus layer seven proxy architecture, defining the boundary.
There are some serious, raw mechanical differences between these two primary approaches. On one hand, you have the layer 4 network load balancer, or NLB, which is packet level routing at its absolute finest.
We're talking about systems like Google's Maglev operating here, effortlessly handling millions of requests per second with ultra low latency. And the key is they don't terminate the connection. They preserve the TCP stream state and use IP and generic routing encapsulation, or GRE, to seamlessly forward packets to your backends.
Now contrast that with a layer 7 application load balancer. ALBs like Envoy or Nginx are highly intelligent. They fully terminate TLS, they pass HTTP headers and paths, and they make complex, stateful routing decisions before pooling upstream connections to those backends.
But while L7 provides incredible routing flexibility, we have to acknowledge that it introduces some very specific computational taxes at the application layer. I mean, by terminating TLS, your proxy's taking on the heavy cryptographic lifting of encryption and decryption.
Passing HTTP headers and paths. That requires memory allocation and string manipulation, all of which adds processing latency. Plus, an ALB has to actively maintain connection pools to upstream servers.
So it's essentially a trade-off, right? You gain granular control and observability for sure, but you pay a premium in CPU cycles, latency, and ultimately your monthly cloud compute bill, especially when compared to a purely packet-forwarding L4 setup.
Moving right along to section two, the evolution of HTTP transport, breaking the bottlenecks. It's fascinating to trace the journey of HTTP transport across the proxy boundary. Way back in 1997, HTTP-1.1 required a separate TCP connection for almost every concurrent request, which severely limited throughput.
That was a bottleneck we just had to live with for nearly two decades. Then in 2015, HTTP-2 revolutionized this by introducing stream multiplexing over a single TCP socket, which was a massive leap forward for reverse proxies. But fast forward to 2022, and the industry had to push even further to HTTP-3, which abandons TCP entirely in favour of UDP-based QIC.
You might be wondering, why on earth did we need to abandon TCP if HTTP slash 2 was already multiplexed? Well, it's because HTTP2 solved one problem, but unfortunately exposed a massive vulnerability to OS-level packet loss, known as head-of-line blocking.
See, because HTTP2 multiplexes dozens of streams over one single TCP connection, TCP's strict requirement for in-order delivery actually becomes a major liability. If just a single packet is dropped on the network, the operating system's TCP stack halts the delivery of all the other perfectly received packets for all the other streams.
It just sits there, waiting for that one lost packet to be retransmitted. A tiny network blip essentially stalls your entire multiplexed pipeline. And that is exactly why HTTP3's UDP-based QIC protocol is such a game changer. It completely bypasses those TCP limits.
QUIC handles loss recovery independently on a per stream basis, meaning if packet three of Stream A is lost, streams B and C just continue processing without any interruption whatsoever. Furthermore, QIC brilliantly integrated directly into the transport layer.
So instead of requiring multiple round trips just to establish a TCP connection and then a TLS session, HTTP3 achieves a secure connection in just one round trip, or even zero round trips for returning clients. For reverse proxy, this translates to dramatically faster connection setups and vastly improve resilience, especially on unstable networks.
Okay, zooming out a bit for section three. Advanced load balancing algorithms, the routing math. We can actually quantify the intelligence of these routing strategies. Simple round robin, as we know, blindly distributes traffic, completely ignoring backend state.
Weighted least request definitely improves this by dynamically shifting traffic away from your burdened nodes. But the really interesting one is power of two random choices, or P2C. Instead of scanning the entire fleet, which, let's face it, is computationally expensive, P2C picks two nodes at random and sends the request to the less loaded of the two.
It's a highly efficient O1 algorithm. It's absolutely brilliant at preventing those thundering herd behaviors where proxies suddenly dogpile onto a newly recovered node. And for scenarios where you absolutely need stateful cache affinity, consistent hashing algorithms like Maglev ensure requests reliably hit the exact same backend, even as your fleet rapidly scales up or down.
To really grasp the real-world impact of these algorithms, we can look at data derived from Google's site reliability engineering practices. Failing to adapt to unforeseen performance limitations using something like simple round robin can result in a massive 2x spread in CPU consumption across a fleet.
That means your most loaded machine is literally working twice as hard as your least loaded machine, simply due to varying query costs and back-end machine diversity. You are essentially wasting precious CPU resources and artificially lowering the capacity of your entire fleet just because your load balancer isn't doing the advanced maths.
And finally, section four. Health checks and fail open. Fleet survival. How do we safely remove a node from rotation without dropping active user sessions?
Well, instead of an abrupt termination, Google engineers employ what they call a lame duck state. When a task needs to shut down, step one, it explicitly asks clients to stop sending new requests.
Step two, it broadcasts this status to all active clients. Crucially, for step three, it continues answering UDP health checks for inactive clients, informing them that it is in the process of draining. Finally, it cleanly completes its active requests and shuts down.
This seamless mapped-out draining prevents a wave of 500 errors from hitting your users during routine zero downtime deployments or hardware swaps. But what happens when a node fails unexpectedly? This is a terrifying failure mode.
If a task is seriously unhealthy, let's say a database connection suddenly drops, it might start rejecting requests with 100% errors instantly. Because it returns those errors so fast, a naive algorithm like least loaded round robin will actually think, wow, this node has zero active connections, and it processes requests in one millisecond.
I'll send all my traffic there. The algorithm mistakenly floods the unhealthy node, effectively sinkholing your traffic. To counter this, advanced load balancers absolutely must penalize nodes for returning errors, treating them as active, heavy load to force the algorithm to root traffic away.
The fail open rule. Imagine a scenario where a transient network partition occurs, and suddenly your load balancer cannot reach any of your backend nodes. If it trusts its health checks blindly, it will mark 100% of the fleet as unhealthy and start dropping all user traffic.
The fail open rule dictates that if a predetermined threshold, often 50% or 100%, of the downstream fleet is marked unhealthy, the proxy naturally assumes the health check mechanism itself is flawed. It deliberately ignores the failing health checks and distributes traffic across the fleet anyway.
It's a calculated risk, but a necessary one. It is far, far better to send traffic to potentially failing nodes than to guarantee a 100% outage by taking a perfectly capable feet offline just because of a transient telemetry blip.
As we wrap up this deep dive, I want to leave you with a provocative thought to consider. As you design your next generation proxies and edge architecture, how will you adapt these highly stateful, complex load balancing mechanisms for a UDP first world?
The underlying mechanics of the web are shifting fundamentally, and your proxy layer absolutely needs to be ready. Thank you so much for joining me for this explainer, and keep building those resilient systems.