Containers & K8s Internals
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's get straight into it. We are diving into the literal kernel level building blocks of containerization and orchestration. And right off the bat, I want you to completely forget all those tired analogies about metal shipping containers on cargo ships.
We are officially banning that analogy today. Instead, we're stripping away the fluff to look at exactly what a container is at the operating system level, using the actual Linux and Kubernetes specifications as our guide.
To do this, we're taking a highly structured bottom-up approach today. Our agenda covers one, VMs versus containers, two, namespaces, three, C Groups V2, four, OCI specs, five, Kubernetes workloads, and finally, six network flow. So, part one, VMs versus shared kernel containers.
Let's start our demystification right at the foundation by contrasting these two distinct deployment architectures. There's a stark contrast in isolation models here. Now you know the drill with the virtualized deployment era.
We relied on hardware virtualization. This gave each tenant a full guest operating system, which provided strict, secure multi-tenant isolation. But in the container deployment era, the paradigm shifts completely.
Containers share the host operating system kernel. They offer a much more relaxed isolation model, allowing for decoupled, incredibly lightweight execution. So the absolute crucial takeaway here is that a container isn't some miniature virtual machine.
Fundamentally, it's just a standard host Linux process. Namespaces and process isolation. If containers don't actually exist as a distinct object in the Linux kernel, how exactly does Linux create this convincing illusion of isolation?
The magic answer lies in namespaces. We essentially restrict what a standard Linux process can actually see using these eight namespace types. When a process is created using the clone system call, we pass in specific flags.
For example, if you pass clone new net, it creates a new network namespace, completely isolating network devices and stacks, whilst clone new pidd isolates process IDs. By applying these namespaces, we're effectively putting a blindfold on the process, hiding the rest of the host system from it.
The Linux manual pages hit the nail on the head here. They state that a namespace wraps a global system resource in an abstraction that makes it appear to the processes within the namespace that they have their own isolated instance of that global resource.
It's a brilliant bit of engineering. It prevents information leaks and allows for the complete secure confinement of these processes. Resource restrictions with C Groups V2.
Because now that we've restricted visibility with namespaces, we absolutely must restrict actual hardware consumption. So what restricts what a process can consume? Control groups version 2.
This is a unified Linux pseudo-file system hierarchy known as CgroupFS. It organizes processes into groups to strictly limit and monitor their usage of system resources. Now, unlike the rather complex and disjointed version 1 architecture, Cgroups V2 uses a single unified hierarchy.
It has a strict no-intern processes rule, meaning a process must reside in a leaf node. This completely prevents any ambiguity in how resources are distributed between parent and child processes.
By enabling specific controllers in the C group subtree control file, we enforce strict hardware consumption rules. Take the CPU controller for CFS quotas, or the memory controller for maximum memory limits.
This literally guarantees that our namespace isolated process won't go rogue, consume all the nodes' resources, and starve other vital applications on the cluster. Alright, section four, OCI specs and file bundles. With our kernel level isolation firmly established, we need to look at the standards that package these processes for distribution.
The Open Container Initiative, or OCI, defines the open industry standards for container formats and runtimes. The specifications broken down into three main areas the runtime specification, the image specification, and the distribution specification.
What's genuinely brilliant here is how the OCI completely decouples the container image format, those copy on write overlay FS, union image layers we all know and love, from the underlying runtime that actually executes the process on the kernel. Let's look at how this builds into an actual execution flow.
First, an implementation downloads an OCI image. Second, it unpacks that image into an OCI runtime file system bundle on the disk. And finally, the OCI runtime executes the bundle. The massive takeaway here is that a container runtime is essentially just a tool that takes a file system bundle and chucks it inside the namespaces and C groups we just explored.
Under the hood, it's elegantly simple. Let's step up a layer for part five Kubernetes workloads and reconciliation. We're zooming out to the control plane that observes and manages all these isolated processes across a massive fleet of nodes.
So the fundamental building block in Kubernetes isn't actually the container at all, it's the pod. A pod is the smallest deployable object, representing a set of one or more running containers.
Now, let's connect this back to what we just learned about kernel boundaries. Because the containers inside a pod share the exact same network and IPC namespaces, they can communicate locally with each other. They function as a single cohesive unit of execution, typically communicating over shared VETH pairs.
But you rarely manage pods directly, dear. Instead, you configure higher level workload resources. You might use a deployment for stateless, interchangeable replacements, a stateful set for persistent data, or a daemon set for node local facilities.
The Kubernetes control plane, your API server, scheduler, and controller manager, runs this constant obsessive reconciliation loop. It continuously compares your desired state with the actual state of the cluster, instructing the local kubelet on each node to spin up or tear down those underlying container processes as needed.
Finally, section six, tracing cluster network packet flow. Let's connect all these abstraction layers together by tracing the literal path of a network packet through this infrastructure. Let's follow the data.
When a pod is scheduled, the container runtime interface, or CRI, delegates the initial network namespace setup. From there, traffic exits the pod via VETH pairs to a CNI bridge or an eBPF data plane.
As traffic moves, KubeProxy manages virtual IPs for cluster IP services, actively rewriting rules using IPVS, NF tables, or eBPF. Finally, for external traffic entering the cluster, the gateway API or ingress controllers handle the complex layer 7 routing.
It's a beautifully orchestrated handover, literally from the kernel namespace level all the way out to global network routing. We've drawn a very clear technical line today between the strict hardware isolation of virtual machines and the shared kernel namespace isolation of containers.
But as engineering constantly evolves, we're seeing the rise of hardware virtualized micro VMs like Firecracker that attempt to offer the lightweight speed of a container with the strict security boundary of a VM. This leaves us with a fascinating engineering challenge regarding the future of secure lightweight workload orchestration.
As these technologies mature, will the boundary between a VM and a container disappear entirely? Definitely something to chew on. Thanks for tuning into this explainer, and I'll catch you next time.