Modern AI System Map
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Right, let's jump straight into this explainer and build a really crisp end-to-end visual systems map of exactly how modern AI pipelines operate. We're going to map the entire journey, literally from raw text bytes all the way to grounded tool calling inference.
If you're a software engineer or a systems architect building in this space, you already know that a static model weights file doesn't just magically become a complete product. Today we're stripping away all that marketing hype.
There's absolutely no magic here. It's just maths and distributed systems. So let's map exactly how these production pipelines are actually architected. To keep things structured, here is our exact roadmap for today.
We are going to cover the architectural system shift, offline training and alignment, the online inference pipeline, grounding and agency, macro engineering trade-offs, and finally the series roadmap and boundaries. The architectural system shift, legacy versus modern AI.
To properly understand why production AI is a multi-layer distributed system today, let's start with a rather hard-hitting contrast. Classical deterministic code and early recurrent neural networks, or RNNs, essentially hit a brick wall.
When you're dealing with unstructured reasoning and massive long-range context, brittle rule-based software simply couldn't scale. And RNNs? Well, they processed data sequentially, which created huge compute bottlenecks.
This limitation forced the industry's massive shift over to parallelisable transformer architectures that rely entirely on learned dense representations. It was a fundamental shift from sequential, hard coded logic over to parallel pattern matching matrix maths.
Offline training and alignment, building representations. So with that shift in mind, let's trace the offline phase. This is all about funneling massive amounts of raw data into a properly aligned checkpoint.
It starts with raw text corpora, passing through subword tokenization, right into self-supervised pre-training. This stage is completely driven by a next token cross-entropy loss objective. But here's the thing a pre-trained model is really just a glorified document completer.
To make it a useful instruction follower, it urgently needs post-training alignment, which we handle through supervised fine-tuning and preference optimization. Now, during that initial pre-training step, you can't just throw arbitrary amounts of compute at a model and hope for the best.
The world representations being built are strictly governed by compute and token scaling laws, most famously the chinchilla scaling laws. These mathematically define the absolute compute optimal ratio between your parameter count and your training tokens.
It ensures you aren't burning incredibly expensive GPU hours on a massive model that is completely starved of data. Once that compute heavy pre-training is done, what you have is a base completion engine.
But without crucial post-training alignment, it can't reliably format an output or act as an assistant at all. Supervised fine-tuning, or SFT, steps in here to teach the model the actual format of a human interaction.
Following that, reinforcement learning from human feedback, or alternatively, direct preference optimization, aligns the model's behavior with actual human preferences. This is what drastically reduces toxic outputs and makes the system genuinely helpful.
The online inference pipeline, tracing a live prompt. Now, let's take that aligned model and trace a live user prompt end to end through the online inference pipeline. First, the raw bytes of the user's text convert to subword token IDs.
These instantly transform into dense embeddings combined with positional encodings, which is absolutely vital because transformers process everything simultaneously, so they need to be mathematically told the order of the words. This massive tensor then flows through stacked multi-head self-attention and feed forward layers, exiting as sampled logits which finally map back into human readable text.
But physically, down on the silicon, this execution splits into two radically different phases. First up is the pre-fill phase. This initial phase basically maxes out the compute capacity of your GPU because it's processing the entire prompt context all at once.
It is heavily compute-bound. The hardware is crunching up gigantic matrix multiplication simultaneously just to understand the context of the input. The moment the prompt is processed, we flip into the deco phase.
This is an entirely different beast. Decoding is completely bottlenecked by memory bandwidth. It operates auto-regressively, meaning it generates tokens strictly one by one. For every single new token it creates, the system has to haul massive amounts of data from the GPU's high bandwidth memory directly into the compute cores.
So your bottleneck isn't compute power anymore, it's literally how fast data can travel across the chip. This physical reality creates a massive hardware tension that is the ultimate balancing act for production engineers.
If you are optimizing for latency and throughput, you have to carefully balance that compute heavy pre-fill phase against the memory-heavy decode phase. They demand completely contradictory optimization strategies when you're deploying at scale.
And of course, there is a catch to all of this parallel processing. Transformers beautifully solve sequential bottlenecks, but they create a nasty quadratic memory footprint. This necessitates the KV cache, where we store previous key and value states to avoid constantly recomputing them.
This setup exposes the severe limitations of relying purely on static parametric memory. Your frozen weights will inevitably hallucinate facts. Their world knowledge goes stale the very second training finishes, and that ever-growing KB cache creates massive memory pressure, essentially eating your VRAM for breakfast during long context generation.
Grounding, compression, and agency. Surrounding subsystems. To solve these parametric failures and memory bottlenecks, we have to build surrounding production subsystems.
First is grounding. Retrieval augmented generation, or RAG, acts as your non-parametric memory. Instead of blindly hoping the model memorized a specific fact during training, RAG swaps out that stale parametric recall for live, verifiable vector retrieval.
It queries an external database at runtime, grabs the exact factual context needed, and seamlessly injects it into the prompt right before the pre-fill phase. Next, we tackle the hardware constraints with compression and adaptation. We use post-training quantization and LoRa, low-rank adaptation, to aggressively shrink the VRAM footprint.
By crushing the precision of the model weights from 16-bit down to 8 or even 4-bit, or by bolting on tiny adapters rather than fine-tuning the entire model, you buy yourself massive deployability. This lets you run highly capable reasoning models on much cheaper hardware or even completely locally at the edge.
Finally, we have agency. Because let's face it, modern AI isn't just about passively answering questions anymore, it's about actually doing things. The Model Context Protocol, or MCP, provides the critical standardized architecture needed to turn a passive text generator into an active agent, capable of executing deterministic side effects.
It relies on a standard JSON RPC protocol, allowing models to securely handshake and interact with external tools and APIs. MCP works so well because it strictly separates the ecosystem. You have the MCP host, which is the overarching AI application.
You have the MCP client maintaining the connection, and you have the MCP server, which securely provides the actual tools. This separation ensures the language model interacts cleanly with external systems. It means you don't have to write messy, bespoke, hard-coded integrations for every single tool your agent needs to use.
Let's quickly trace how that actually works in practice, to see a text generation loop become a real-world action. Step one, the client dynamically discovers and lists all available tools from the server.
Step two, the language model decides it needs to use a specific tool and auto-regressively generates a structured call. Step three, the MCP server safely executes that deterministic side effect, like writing a local file or querying a live database, and then feeds the result right back into the model's context window.
Macro engineering trade-offs, the decision matrix. With all those primary components mapped out, solution architects have to navigate a very real decision matrix. Every single architectural addition has to be weighed carefully across axes like latency, cost, freshness, and predictability.
For instance, do you use RAG or fine-tuning? Well, you use RAG for dynamic, verifiable facts, but you fine-tune for formatting and stylistic behavior. What about deterministic workflows versus autonomous agents?
Workflows give you strict predictable latency, whereas agents give you open-ended, albeit slower, problem solving. And cloud APIs versus quantize local models. Cloud guarantees peak reasoning capability, but local SLMs hand you absolute latency and privacy control.
Series roadmap and boundaries. Component boundaries. As we lock down the boundaries for the upcoming deep dives in this explainer series, if you take away just one thing today, keep this fundamental rule of modern AI architecture in mind.
Weights store your capability and logic. Rag and retrieval supply your fresh, verifiable truth. Quantization and adapters by you speed and deployability. And tools, via protocols like MCP, execute your real world side effects.
If you keep these boundaries strict, you'll build robust, scalable distributed systems, rather than unpredictable, monolithic black boxes. We've successfully established the high level system interfaces today. But the real question is, are you ready to crack open these black boxes and look at the internal maths?
As we step into parts one through six of this explainer series, we are going to unpack exactly how QKV matrix projections calculate self attention, how HNSW graph traversals retrieve vectors in literal milliseconds, and how NF4 block quantization mathematically crushes these massive weights down to fit on edge devices.
It's going to get highly technical, so stay focused, and I'll see you in the next one.