Start here: AI Foundations for Engineers
Podcast · Start here: AI Foundations for Engineers

Systems Engineering for Modern AI

by ZumiLabs · 46m
0:00 46m

Transcript

Made automatically from the audio, so it may contain mistakes. Click a line to play from there.

Host 1

Imagine provisioning three terabytes of high speed GPU memory.

Host 2

Yeah, that's not cheap.

Host 1

Right. We are talking about top-tier silicon. It is costing your company thousands of dollars every single day.

Host 2

Absolutely.

Host 1

And then, you know, you realize it's not even holding the neural network's parameters.

Host 2

No, it's not.

Host 1

It's literally just holding temporary cash for a single batch of users asking questions.

Host 2

Which is wild.

Host 1

It is. If you're the systems architect signing off on that Cloudville, I mean, that is a terrifying realization.

Host 2

Oh, completely.

Host 1

And I think it perfectly illustrates how the modern AI stack has just utterly shattered the paradigms of classical software engineering.

Host 2

It really shatters them because you know classical engineering is built entirely on the comfort of determinism. Right.

Host 1

You know what to expect.

Host 2

Exactly. You write a function, you define the inputs, and you trace the exact logic path. Yeah. And then the compiler points to the exact memory address and basically says, there it is.

Host 1

It's visible.

Host 2

Right. It is perfectly categorized. It is binary.

Host 1

It's either broken or not broken.

Host 2

Yeah, exactly.

Host 1

You get a stack trace, you fix the bug, and you move on.

Host 2

Sure.

Host 1

But then, you know, as an industry, we tried to step into this world of unstructured reasoning.

Host 2

Which is a whole different ballgame.

Host 1

Totally. We try to write these rule-based deterministic systems to do things like summarize a dense legal contract.

Host 2

Or parse the natural language intent of like a really frustrated customer.

Host 1

Right. And suddenly that uncompromising precision just shatters.

Host 2

It falls apart.

Host 1

You hit a technical wall. And for a long time, classical software engineering and honestly, even early machine learning models just crashed headfirst into this wall.

Host 2

They did. I mean, we tried recurrent neural networks, we tried LSTMs.

Host 1

But they hit a severe architectural bottleneck, didn't they?

Host 2

They absolutely did. And it's crucial to understand why they failed mechanically.

Host 1

Because it sets up everything we do today, right?

Host 2

Exactly. So recurrent neural networks are RNNs, they process data sequentially. Okay. They take token one, they update a hidden state, then they take token two, combine it with the previous hidden state, and so on.

Host 1

One by one.

Host 2

Right. But from a systems perspective, that sequential dependency means you absolutely cannot parallelize the training process across a GPU cluster.

Host 1

Because you're awaiting.

Host 2

Yeah, you have to wait for step T to finish before you can compute step T plus one.

Host 1

I mean, that's the ultimate bottleneck.

Host 2

It really is.

Host 1

You have what? Thousands of compute cores just sitting idle waiting for a single sequential loop to finish.

Host 2

Exactly. It's incredibly inefficient. And on on top of the compute bottleneck, they suffered from the vanishing gradient problem.

Host 1

Oh, right.

Host 2

Yeah. By the time an RNN got to the end of a long paragraph, the mathematical influence of the very first word had been multiplied by fractions so many times.

Host 1

During the back propagation through time.

Host 2

Right. During back propagation. It was multiplied so many times that it effectively vanished to zero.

Host 1

Wow. So the model literally couldn't remember the beginning of the sentence it was reading.

Host 2

It was like trying to read a textbook through a straw.

Host 1

That is a great way to put it. And that architectural failure, that's exactly what brings us to the mission of this deep dive.

Host 2

It is.

Host 1

For those of you joining us, this is the foundational overview for Zoomy Lab's fundamental AI concept series.

Host 2

Getting right down to the basics.

Host 1

Exactly. Before we get into the mathematical weeds in our upcoming deep dives, our goal today is to provide you, the experienced software engineer, the systems architect, you know, the infrastructure lead, with a complete end-to-end mental model of the modern AI stack.

Host 2

Because the shift was massive.

Host 1

Right. We are moving away from that sequential processing bottleneck and diving into the transformer shift.

Host 2

Learned dense representations and parallelizable sequence modeling.

Host 1

Driven primarily by self-attention.

Host 2

So we need to map out exactly how all these layers fit together in a production environment.

Host 1

And just to set the tone right out of the gate for this series, we are going to be highly analytical today.

Host 2

Yes. Zero marketing fluff.

Host 1

Absolutely zero.

Host 2

And I am strictly banning any vague anthropomorphism.

Host 1

No, the AI is thinking stuff.

Host 2

Exactly. The AI does not think, it does not feel, it does not understand.

Host 1

It computes.

Host 2

It computes, it predicts, and it routes. We are dealing with massive matrix multiplications, mathematical optimizations, and severe memory bandwidth constraints.

Host 1

So it is not magic.

Host 2

No, it is systems engineering at an unprecedented scale.

Host 1

I love it. Ground rules established. So let's unpack this architecture.

Host 2

Let's do it.

Host 1

To understand how a modern inference system operates at runtime when a user actually hits your API, you first have to understand how the foundational weights are forged offline.

Host 2

Which is a massive process.

Host 1

Right. So let's map the creation funnel, starting right at the data pipeline. Because before we even get to neural networks, we have a massive data engineering problem.

Host 2

A petabyte scale data engineering problem, actually.

Host 1

Right.

Host 2

We begin with raw text bytes scraped from the internet, proprietary code bases, and you know, digitized books.

Host 1

But a language model doesn't see text.

Host 2

No, it doesn't. It doesn't read English syntax or Python logic.

Host 1

So what's the first step?

Host 2

The very first component in the pipeline is the tokenizer.

Host 1

Right, which is usually a byte pairing coding or BPE algorithm.

Host 2

Exactly. BPE is fundamentally a data compression technique.

Host 1

Okay. How does that work?

Host 2

We start by treating every individual byte as a token. Then the algorithm scans the entire training corpus and finds the most frequently adjacent pair of bytes.

Host 1

Give me an example.

Host 2

Let's say the letter T and the letter H appear together most often. It merges them into a new single token, T H.

Host 1

Oh, I see.

Host 2

Then it scans again, maybe finding that T H and E are commonly adjacent.

Host 1

So it merges those into T H E.

Host 2

Exactly. It repeats this process thousands of times until we reach a target vocabulary size.

Host 1

Which is usually what, around 100,000 tokens?

Host 2

Yeah, typically around 100,000 uni tokens.

Host 1

Which you know actually explains some of those weird edge cases we see in production.

Host 2

Like what?

Host 1

Like if you ask an LLM to spell a highly unusual word backward, it often fails miserably.

Host 2

Oh, right, yeah.

Host 1

Because it doesn't actually see the word as individual letters, right?

Host 2

No. It sees it as an opaque, indivisible chunk.

Host 1

A single token ID.

Host 2

Precisely. We take raw text and chunk it into these tokens and then map those tokens to integers.

Host 1

So that is all the model ever interacts with.

Host 2

Yes. Massive sequences of integer IDs.

Host 1

And those integers are then mapped to dense vectors, which we call embeddings. Right. And those live in a high-dimensional continuous space.

Host 2

This is a really important transition.

Host 1

Yeah, this is where the model transitions from discrete computer science to continuous geometry.

Host 2

Exactly. Words with similar contextual meanings are mathematically forced to cluster together in this vector space. Yeah, the self-supervised pre-training.

Host 1

So this is where we hit the compute optimal frontier.

Host 2

It is. And the core objective during this pre-training phase is beautifully simple. Almost deceptively so.

Host 1

It really is just next token prediction, right?

Host 2

Yes. Mathematically governed by cross entropy loss.

Host 1

Let's break that down for the engineers listening.

Host 2

Sure.

Host 1

We are feeding the model a sequence of tokens, say tokens one through ten.

Host 2

Okay.

Host 1

We pass those through the transformer layers. And at the very end, the model outputs a probability distribution over its entire 100,000 token vocabulary.

Host 2

Right, representing its prediction for token number 11.

Host 1

Correct. And because this is self-supervised, we already possess the actual token number 11 from our training corpus.

Host 2

Exactly. So we use the cross entropy loss function to calculate the mathematical distance, basically the error.

Host 1

The error between what?

Host 2

Between the model's predicted probability distribution and the true one hot encoded distribution of the actual next token.

Host 1

So we take that error gradient and backpropagate it.

Host 2

Right. Through the entire network.

Host 1

Adjusting billions of weights by these microscopic amounts, using an optimizer like Atom W.

Host 2

Yes. And the scale at which this happens is where the industry learns some very expensive, very public lessons about infrastructure.

Host 1

And resource allocation, yeah.

Host 2

If we look at the history of scaling laws, there was a massive paradigm shift.

Host 1

From the early Kaplan scaling laws in 2020 to the Hoffman or chinchilla scaling laws in 2022.

Host 2

This is a critical piece of architectural history for anyone in the field to understand.

Host 1

But to unpack the Kaplan laws first.

Host 2

So the Kaplan scaling laws published by OpenAI analyzed the relationship between compute, dataset size, and parameter count.

Host 1

And their empirical data led the industry down a very specific path.

Host 2

It did. The Kaplan power laws suggested that to optimally improve model performance, you should allocate the vast majority of any additional compute budget, primarily toward increasing the model size.

Host 1

The parameter count.

Host 2

Exactly. Just make the model bigger.

Host 1

Which triggered that absolute arms race of parameter counts. Oh, yeah. We saw the birth of massive models. GPD 3 launched with 175 billion parameters. Right. Other labs push models to 300 billion, even 500 billion parameters.

Host 2

But the underlying math of the Kaplan laws was actually flawed.

Host 1

Really? How so?

Host 2

It was due to the constrained compute budgets and fixed learning rate schedules they used in their original experiments.

Host 1

So the data was skewed.

Host 2

Yeah. GPT3 and its contemporaries were massively undertrained.

Host 1

Oh, I see.

Host 2

They were bloated parameter wise, but totally starved for data.

Host 1

Then DeepMind's chinchilla paper drops and completely flips the script.

Host 2

DeepMind ran an exhaustive set of isoflop experiments. They trained over 400 different language models, varying the size and the data to observe the true optimal allocation of compute.

Host 1

And the chinchilla rule of thumb emerged from that.

Host 2

Yes, and it completely reset the industry standard.

Host 1

Which was what?

Host 2

For compute optimal training, model size and training tokens must scale equally.

Host 1

So if you double the parameters in your architecture, you must also double the training tokens.

Host 2

Exactly. Specifically, they found an optimal ratio of roughly 20 to 1.

Host 1

Meaning?

Host 2

For every parameter in your model, you need about 20 tokens of high quality training data.

Host 1

Wow. Okay.

Host 2

They proved this by training chinchilla, which was a 70 billion parameter model on 1.4 trillion tokens.

Host 1

So chinchilla was a fraction of the size of models like DeepMind's own gopher or Microsoft's Megatron Touring.

Host 2

A tiny fraction. But because it adhered strictly to this compute optimal ratio, it uniformly outperformed those massive models on downstream tasks.

Host 1

Well being vastly cheaper to run in inference.

Host 2

Exactly.

Host 1

Which it's a huge lesson for any architect designing systems today.

Host 2

You can't just throw parameters at it.

Host 1

Right. Throwing parameters at a problem isn't the solution if you don't have the data to back it up.

Host 2

A bloated model just means you're going to pay exorbitant latency and memory costs during inference.

Host 1

For absolutely zero gain in reasoning capability.

Host 2

None at all.

Host 1

But let's keep moving down the pipeline. Pre-training a massive model and trillions of tokens only gets us a base model.

Host 2

Right.

Host 1

And a base model is essentially a superpowered, incredibly knowledgeable autocomplete.

Host 2

That's a good way to describe it.

Host 1

If you prompt a raw base model with what is the capital of France, it might output Paris.

Host 2

Or it might output what is the capital of Germany.

Host 1

Exactly. Because it statistically assumes you are writing a list of trivia questions.

Host 2

It has no concept of a user or a dialogue or an instruction.

Host 1

Precisely. The base model maps the statistical manifold of human language, but it lacks an interface.

Host 2

And that brings us to post-training alignment, which transitions the model from an autocomplete engine to an actual instruction following assistant.

Host 1

So what's the first step in that phase?

Host 2

The first step is supervised fine-tuning or SFT.

Host 1

Okay, this is where we teach the base model the required format of an interaction.

Host 2

Yes.

Host 1

We take tens of thousands of highly curated human-written examples of a prompt and a completion.

Host 2

Often wrapped in special formatting tokens like the user token and the assistant token.

Host 1

Right. And we train the model on this data set using the exact same cross-entropy loss as pre-training.

Host 2

We are strictly conditioning the model to recognize and conform to the dialogue structure.

Host 1

But SFT only teaches the format.

Host 2

Right. Preference alignment teaches the behavior, the tone, and the safety boundaries.

Host 1

And initially the industry relied heavily on RLHF for this, right? Reinforcement learning from human feedback?

Host 2

They did. And as any engineer who has actually tried to implement RLHF knows, it is incredibly complex to set up.

Host 1

It's a massive distributed systems headache.

Host 2

It really is. First, you have to train a completely separate neural network, a reward model.

Host 1

Okay.

Host 2

You feed it two different responses to the same prompt, and you train it to predict which response a human rater would prefer.

Host 1

So once you have that reward model, what happens?

Host 2

You use a reinforcement learning algorithm, most commonly proximal policy optimization or PPO, to update your main language model's weights.

Host 1

So the language model generates a response.

Host 2

The reward model scores it.

Host 1

And PPO calculates a gradient to push the language model to maximize that scalar reward in the future.

Host 2

Exactly. While simultaneously using a KL divergence penalty to ensure it doesn't drift too far from its original base knowledge.

Host 1

That sounds intense. You are essentially bouncing complex gradients and massive tensors between multiple separate models in memory during a single training loop.

Host 2

It's notoriously unstable.

Host 1

Highly sensitive to hyperparameters, I imagine.

Host 2

Very much so, and prone to reward hacking.

Host 1

Which is exactly why the recent architectural shift toward DPO, direct preference optimization, has been so significant.

Host 2

Especially for open weight models and enterprise fine-tuning.

Host 1

Yeah. DPO is a mathematical breakthrough that basically bypasses the need for a separate reward model entirely.

Host 2

It is brilliant.

Host 1

Let's explain the mechanism there, because it really is elegant. How does DPO actually bypass the reward model mathematically?

Host 2

So DPO leverages the Bradley Terry model of preferences.

Host 1

Okay.

Host 2

The researchers realize that you can mathematically reframe the reward function as an implicit function of the optimal policy itself.

Host 1

Meaning.

Host 2

Instead of training a separate model to output a reward score, you take your preference data, a prompt, a chosen response, and a rejected response. Right. You pass both responses through your language model to get their log probabilities.

Host 1

Okay, I follow.

Host 2

DPO uses a specialized loss function that directly increases the relative log probability of the chosen response compared to the rejected response.

Host 1

So it essentially makes the language model act as its own reward model.

Host 2

Exactly. It allows you to align the model using a standard supervised style gradient descent loop.

Host 1

It is mathematically elegant and vastly simplifies the alignment pipeline infrastructure.

Host 2

It really does.

Host 1

Okay, hold on. I need to push back here because this is the exact point where experienced software engineers usually raise an eyebrow.

Host 2

I know exactly what you're gonna ask.

Host 1

We just described a pipeline that is at its absolute core, just next token prediction governed by cross-entropy loss, right?

Host 2

Yes.

Host 1

And then it's nudged by some preference pairs.

Host 2

Right.

Host 1

So if we are freezing a model and we aren't explicitly programming any logic rules or state machines, where exactly does the reasoning capability emerge?

Host 2

Right.

Host 1

I mean i i i if it's just a statistical parrot minimizing loss, how is it writing complex multi-file React applications or accurately summarizing dense 100-page legal text?

Host 2

It is the most common skepticism engineers have, and honestly, addressing it requires us to step away from the classical idea of explicit programming.

Host 1

Okay, so where do we look?

Host 2

We have to look at information theory. Specifically Solomonov induction and the concept of representation learning.

Host 1

Alright, break that down for me.

Host 2

Reasoning in this context is not a programmed module. It emerges as a compressed representation of world models.

Host 1

It explain what you mean by a compressed world model.

Host 2

Think about the training objective.

Host 1

Okay.

Host 2

You have to minimize cross-entropy loss across a massive, highly diverse pre-training corpus. Right. The model has to accurately predict the next word in a quantum physics textbook just as accurately as it predicts the next word in a romance novel.

Host 1

Or just as accurately as it predicts the next token in a public GitHub repository.

Host 2

Exactly. And it can't just memorize all of that.

Host 1

Because memorizing a trillion tokens is physically impossible, even for a 70 billion parameter model.

Host 2

Right. The data is orders of magnitude larger than the parameter capacity.

Host 1

Because it cannot memorize, it is mathematically forced to compress.

Host 2

Exactly. And the most optimal way to compress data is to learn the underlying structural rules that generated that data in the first place.

Host 1

So if you want to accurately predict the output of a complex Python function found in a training data set, you cannot just guess.

Host 2

No, the only way to minimize the loss on that specific token is to implicitly learn how the Python interpreter evaluates logic.

Host 1

How he tracks variables and manages scope.

Host 2

Precisely. So by forcing it to predict the output, we are forcing it to build an internal representation of the machine that created the output.

Host 1

That is fascinating.

Host 2

And furthermore, this emergent structural logic is heavily sharpened during the alignment phase.

Host 1

When we apply DPO?

Host 2

Yes. When we apply DPO or RL on high-quality data sets containing complex reasoning traces like step-by-step mathematical proofs or chain of thought logic.

Host 1

We are actively penalizing the model for taking statistical shortcuts.

Host 2

We are mathematically rewarding the generation of intermediate logical steps.

Host 1

So the behavior we call reasoning is basically just a direct mathematical byproduct of relentless compression and loss minimization. That is a phenomenal way to frame it. The reasoning is the compression algorithm.

Host 2

Exactly.

Host 1

Okay. So we've forged the foundational weights. The multi-million dollar GPU training run is done.

Host 2

Finally.

Host 1

We have our aligned instruction following model checkpoint. But now you, the systems architect, have to actually serve this massive tensor file in production.

Host 2

Which brings its own set of massive challenges.

Host 1

A user hits your API and submits a prompt. Let's transition to the online serving pipeline. How does the inference engine physically process this at runtime?

Host 2

Let's trace the forward pass through the system. Okay. The user's text prompt hits the API and immediately undergoes tokenization, just like in training.

Host 1

And those token IDs flow into the abetting layer, mapping to dense vectors.

Host 2

Right. But remember the core architecture of the transformer, self-attention processes all tokens in a sequence simultaneously.

Host 1

Right. Unlike an RNN, it has no inherent concept of sequence order.

Host 2

It doesn't know if a word is at the beginning of a sentence, the end of a paragraph, or somewhere in the middle, unless we explicitly inject that metadata.

Host 1

So we inject positional encodings.

Host 2

We do. In modern models, this is typically rope, rotary position embeddings.

Host 1

How does rope work?

Host 2

Rope mathematically rotates the embedding vectors in a complex plane. The amount of rotation is proportional to the token's position in the sequence.

Host 1

So this preserves the relative distance between tokens.

Host 2

Exactly. Allowing the attention mechanism to understand that the word not immediately preceding the word good fundamentally changes the context compared to if they were 10 words apart.

Host 1

From there, the positionally aware vectors flow up through the stack transformer blocks.

Host 2

Passing through alternating layers of multi-head self-attention and feed-forward networks.

Host 1

And this right here is where the architecture of serving splits into two radically different computational profiles.

Host 2

The pre-fill phase and the autoregressive decode phase.

Host 1

This is the architectural dichotomy of inference serving.

Host 2

It is. If you are an infrastructure engineer managing latency, this is the single most important concept to grasp.

Host 1

Because your GPUs will behave entirely differently across these two phases.

Host 2

Let's break it down. The pre-fill phase happens first.

Host 1

This is when the model is processing the user's initial prompt.

Host 2

Correct. During the prefill phase, the entire prompt is already known. We have all the tokens.

Host 1

Therefore, the computation is highly parallelizable.

Host 2

We pass the entire sequence of prompt tokens through the network simultaneously.

Host 1

So from a hardware utilization perspective, this phase is strictly compute bound.

Host 2

We are performing massive matrix matrix multiplications or GEMM operations, general matrix multiply.

Host 1

The GPUs are basically operating at peak arithmetic intensity.

Host 2

Yes. The massive weight matrices are loaded from high bandwidth memory into the GPU's compute cores. And they are used to process hundreds or thousands of prompt tokens in parallel.

Host 1

It is highly efficient.

Host 2

But then the model finishes processing the prompt and generates the very first new token.

Host 1

And now we shift into the decode phase. This is where the model is generating the actual response.

Host 2

And this phase has an entirely different hardware profile.

Host 1

How so?

Host 2

The decode phase is auto-regressive. We can only generate one token at a time. The model needs the token it just generated at step one to act as the input for step two.

Host 1

So you cannot parallelize across time here.

Host 2

No, you can't.

Host 1

For our infrastructure engineers listening, I like to think about this like a single-threaded process that is constantly blocked by I.O.

Host 2

That is the perfect analogy. During the decode phase, we are no longer doing highly parallel matrix matrix multiplication.

Host 1

What are we doing?

Host 2

We are doing matrix vector multiplication or GEMV.

Host 1

Right.

Host 2

For every single new token we generate, we have to fetch the entire massive model weight matrix from the GPU's high bandwidth memory.

Host 1

Pull it into the registers, multiply it against a single token vector, and then flush it.

Host 2

Exactly. So the GPU compute cores, the actual flops are often just sitting idle.

Host 1

Because they are incredibly fast, but they are starved for data.

Host 2

The bottleneck isn't the processing speed.

Host 1

It's the physical memory bandwidth.

Host 2

Right. The speed at which we can shuttle terabytes of weights back and forth from the HBM to the compute cores for every single token step.

Host 1

And this severe memory bandwidth bottleneck is precisely what leads to the most critical optimization in AI infrastructure, right? Yes. Yes.

Host 2

The KV cache.

Host 1

Yes. The infamous key value cache. Let's explain the exact mechanics of why this exists.

Host 2

So during the autoregressive decode phase as we generate a new token, the self-attention mechanism needs to look back at all the previous tokens.

Host 1

Both the prompt and the previously generated tokens.

Host 2

Right. To establish contacts and calculate attention scores. If we didn't cache anything, we would have to take every single past token, multiply it by the key and evaluate matrices to project it into the attention space.

Host 1

And we calculate those tensors from scratch for every single new token we generate.

Host 2

Exactly. That would mean the compute cost would grow exponentially with every word generated.

Host 1

It would be computationally ruinous.

Host 2

So to avoid that redundant computation, we cache the results. Okay. As soon as a token is processed, we take its calculated key and value tensors and we store them in the GPU's VRAM.

Host 1

So in subsequent steps, the attention mechanism just pulls those pre-computed tensors from the cache.

Host 2

It solves the compute bottleneck, but it introduces a massive memory footprint nightmare.

Host 1

Let's look at the actual math, because this is where production architectures break. How big is this cache?

Host 2

The formula for the KV cache size is brutal.

Host 1

Walk us through it.

Host 2

It is the batch size multiplied by the sequence length, multiplied by the number of transformer layers, multiplied by the number of attention heads, multiplied by the head dimension, multiplied by two, because we store both keys and values, multiplied by the byte precision, which is usually two bytes for 16-bit floats.

Host 1

Let's give that concrete numbers. Yeah. Imagine you're running a production API server. Okay. You are serving a 70 billion parameter model. You want a decent batch size of, say, 512 concurrent user requests to maximize throughput.

Host 2

Sure.

Host 1

And let's say your context length is maxed out at 8,000 tokens for a rag application.

Host 2

If you run that math, just storing the KV cache for that single batch can require hundreds of gigabytes or even terabytes of VRAM.

Host 1

Depending on the exact architecture.

Host 2

Exactly. I've seen production deployments where a high throughput cluster requires up to three terabytes of VRAM just to hold the KV cache.

Host 1

That is often triple the size of the static model weights themselves.

Host 2

Because the model weights are fixed, but the KV cache grows dynamically with batch size and sequence length.

Host 1

And if you don't manage that memory properly, you get catastrophic fragmentation.

Host 2

Yes.

Host 1

Which is why innovations like page detention, pioneered by the VLLM project, are so critical.

Host 2

They recognize that the KV cache was suffering from the exact same memory fragmentation issues as early computer operating systems.

Host 1

It's a brilliant crossover of classical systems engineering into AI.

Host 2

It really is. Page detention treats the KV cache exactly like OS virtual memory paging.

Host 1

So instead of allocating contiguous blocks of memory for a sequence whose final length is unknown, which leads to massive wasted space.

Host 2

It breaks the KV cache into fixed size blocks or pages.

Host 1

The attention mechanism uses a block table to map logical tokens to physical memory pages non-contiguously.

Host 2

This nearly eliminates memory waste and allows you to drastically increase your batch size on the same hardware.

Host 1

It is incredible how the foundational principles of computer science keep reinventing themselves at the AI layer.

Host 2

Everything old is new again.

Host 1

Okay. So we've managed the memory, we've executed forward passes, and finally, at the very end of this pipeline, we have the output stream and logit sampling.

Host 2

Right.

Host 1

The model doesn't just output a string of text.

Host 2

No, the final layer outputs logits.

Host 1

Raw, unnormalized scores representing a probability distribution over the entire hundred thousand token vocabulary.

Host 2

And here we have architectural choices on how to decode those logits into actual text.

Host 1

Like greedy decoding.

Host 2

Exactly. With greedy decoding, we simply apply an argmax function and pick the token with the highest absolute probability.

Host 1

This is highly deterministic and great for tasks like code generation or data extraction.

Host 2

But it can lead to repetitive, sterile text in conversational applications.

Host 1

So we introduce sampling parameters. The most common is temperature.

Host 2

Temperature scaling modifies the logits before we apply the softmax function that turns them into probabilities.

Host 1

Mathematically, we divide the logits by the temperature value.

Host 2

If the temperature is exactly 1.0, the distribution is unchanged.

Host 1

But if we lower the temperature, say to 0.2.

Host 2

Dividing by a decimal less than one makes the differences between the logits much larger.

Host 1

It sharpens the probability distribution.

Host 2

The likely tokens become overwhelmingly likely, and the unlikely tokens are mathematically crushed to near zero.

Host 1

So it makes the model highly focused and deterministic.

Host 2

Conversely, raising the temperature above 1.0 flattens the distribution.

Host 1

It brings the probabilities closer together.

Host 2

Which introduces controlled randomness and variance, giving generative text a more creative, natural feel.

Host 1

We also combine this with top K or top P sampling, also known as nucleus sampling.

Host 2

Where we truncate the distribution entirely.

Host 1

Right. Only allowing the model to sample from the top K, most likely tokens, or the subset of tokens whose cumulative probability equals P.

Host 2

It prevents the model from ever sampling the absolute worst hallucinated tokens at the very long tail of the distribution.

Host 1

So we have a standalone inference loop. It works. The hardware is optimized.

Host 2

But here is where we hit the next wall.

Host 1

Because serving a static model isn't a product. It's an island.

Host 2

It is.

Host 1

As systems architects, we know that deploying a raw static model checkpoint directly to users is a recipe for disaster.

Host 2

It is severely limited.

Host 1

They hallucinate. Their VRAM footprints, as we just discussed, are massive. And most critically, they are completely isolated from our external enterprise systems, databases, and APIs.

Host 2

So how do we break these limitations?

Host 1

Let's move to component boundaries. Grounding, compression, and agency.

Host 2

Let's start with grounding and defining the exact boundary between parametric and non-parametric memory.

Host 1

Okay.

Host 2

The model's weights, those billions of parameters trained over trillions of tokens, represent its parametric memory.

Host 1

But this memory is frozen the moment the training run finishes.

Host 2

Exactly. It is inherently prone to staleness.

Host 1

A model trained in 2023 knows absolutely nothing about a zero-day vulnerability discovered this morning.

Host 2

And it is prone to hallucination because parametric memory relies on statistical likelihood rather than verifiable truth.

Host 1

It will confidently generate a highly probable, perfectly formatted API call that doesn't actually exist. What do we do?

Host 2

First, we send the query to a specialized embedding model. This model generates a dense vector representation of the query's semantic intent. Or a hybrid index combining vector search with classical BM25 keyword search.

Host 1

This database contains all of our proprietary documents, wiki pages, and code based documentation, all pre chunked and embedded.

Host 2

The vector database retrieves the top K most semantically relevant chunks of text.

Host 1

We take those raw text chunks and we dynamically inject them directly into the user's prompt at runtime.

Host 2

Along with strict instructions for the LLM to only answer based on the provided context.

Host 1

We are forcing the model to condition its autoregressive generation on verifiable external truth.

Host 2

Rather than relying on its statistical parametric memory.

Host 1

It perfectly bridges the gap between the static weights and dynamic, constantly updating enterprise data.

Host 2

But RAG only solves the truth and staleness problem. It doesn't solve the efficiency or behavioral adaptation problems.

Host 1

For that, we need to cross another boundary and look at quantization and LoRa.

Host 2

And let's tackle quantization first.

Host 1

Because this is how we actually shrink that massive VRAM footprint without spending millions on a full fine-tuning run.

Host 2

Quantization reduces the numerical precision of the numbers used to represent both the model's weights and the transient activations during the forward pass.

Host 1

Instead of using 16-bit floating point numbers, we compress them down to 8-bit integers or even 4-bit integers.

Host 2

We have post-training quantization, or PTQ, which happens offline after the model is trained, and quantization aware training, or QAT, which makes the precision reduction directly into the gradient descent process.

Host 1

But PTQ isn't free. You can't just blindly round numbers down.

Host 2

No, you can't.

Host 1

And this brings up a fascinating anomaly that systems engineers had to solve the problem of activation outliers.

Host 2

This is a critical mathematical hurdle. As language models scale up beyond roughly six billion parameters, researchers observed a strange phenomenon.

Host 1

Extreme outliers spontaneously emerge in the activation tensors.

Host 2

Certain feature channels during the forward pass will suddenly spike to values up to a hundred times larger than the rest of the activations in that layer.

Host 1

And if you blindly quantize that activation tensor to an 8-bit integer, you have to define a scale.

Host 2

Right. You have a finite range from negative 128 to 127.

Host 1

If you set the scale wide enough to capture that massive outlier, all the normal, subtle activation values get crushed to zero.

Host 2

You lose all the nuance and the model's reasoning performance completely collapses.

Host 1

Which is why techniques like SmoothQuant are so brilliant.

Host 2

And require a deep understanding of linear algebra.

Host 1

Yeah. SmoothQuant performs a mathematically equivalent transformation offline.

Host 2

It recognizes that while activations have massive outliers, the model weights themselves are typically very well behaved and normally distributed.

Host 1

So how do we fix the activations without changing the model's output?

Host 2

We migrate the variance. Mathematically, for a given linear layer where output y equals activation X multiplied by weight W, SmoothQuant introduces a per channel scaling factor. Let's call it S.

Host 1

Okay.

Host 2

It divides the activation channels by S, flattening the outliers.

Host 1

To ensure the mathematical output y remains completely identical, it simultaneously multiplies the corresponding channels in the weight matrix W by that exact same scale S. We've taken the difficulty of quantization and pushed it from the dynamic activations into the static weights.

Host 2

Because weights are much easier to quantize, the smoothing allows us to successfully deploy W8A8 inference.

Host 1

8-bit weights and 8-bit activations.

Host 2

Exactly. Without losing model capacity. It's a pure mathematical sleight of hand that saves gigabytes of VRAM.

Host 1

So that handles memory compression and efficiency. But what about behavioral adaptation? Let's talk about LoRa, low rank adaptation.

Host 2

If a company wants a model to output highly specific XML formats or adopt a very particular brand voice, doing a full parameter fine-tuning of a 70 billion parameter model is prohibitively expensive.

Host 1

It requires a massive GPU cluster just to hold the gradients and optimizer states.

Host 2

So LoRa is a parameter efficient fine-tuning technique.

Host 1

The mechanics of LoRa rely on rank decomposition.

Host 2

Instead of updating all the billions of weights and the dense matrices, we completely freeze the original pre-trained model. It becomes read-only.

Host 1

We then inject two small trainable matrices, let's call them A and B into the transformer layers.

Host 2

Typically targeting the query and value projection layers in the attention mechanism.

Host 1

And this is where the math is elegant. Matrix A takes the high dimensional input and projects it down to a very low dimensional space.

Host 2

A low rank, like eight or sixteen.

Host 1

Matrix B then projects it back up to the original high dimensional shape.

Host 2

Exactly. When you multiply matrix A and matrix B together, the resulting matrix has the exact same dimensions as the original frozen weight matrix.

Host 1

But it was constructed using orders of magnitude fewer trainable parameters.

Host 2

During the forward pass, the input is passed through the frozen weights and simultaneously passed through the LoRa adapter.

Host 1

And the outputs are simply added together.

Host 2

Yes.

Host 1

Wait, hold on. I need to challenge this from an engineering perspective.

Host 2

Let's hear it.

Host 1

If we are freezing a 70 billion parameter model and we are only training a tiny adapter with a rank of eight, meaning we are only updating maybe 0.1% of the total parameters.

Host 2

Right.

Host 1

Aren't we severely bottlenecking the model's expressivity? If I needed to learn an entirely new proprietary coding language that my company invented, a tiny matrix injection feels mathematically insufficient to overwrite the base training.

Host 2

It is a highly intuitive pushback, but it misunderstands how models store capability.

Host 1

Okay, correct me.

Host 2

This touches on the intrinsic dimension hypothesis.

Host 1

Which dictates what?

Host 2

The hypothesis dictates that while a massive model requires billions of parameters to initially learn the complex, high-dimensional manifold of human knowledge during pre-training, the delta, the update, required to shift the model's style, format, or domain focus actually exists in a very low-dimensional subspace.

Host 1

So the model already knows how to code. It already understands syntax and logic loops.

Host 2

We are just using the low-rank adapter to point it toward a specific syntax dialect.

Host 1

Oh, I see.

Host 2

Precisely. And in fact, freezing the base weights is advantageous.

Host 1

Why?

Host 2

If you do full fine-tuning on a small proprietary dataset, you risk catastrophic forgetting. The model might learn your XML format, but suddenly forget how to do basic arithmetic because you overwrote the foundational weights.

Host 1

So LoRa preserves the vast world model while injecting a highly specific behavioral steer.

Host 2

Exactly.

Host 1

Which brings up a practical engineering question we see all the time, the golden boundary rule.

Host 2

Yeah.

Host 1

If I am building a system, do I stuff documentation into a rag vector store, or do I train a LoRa adapter?

Host 2

Rag supplies fresh, verifiable truth and dynamic state.

Host 1

If the model needs to know the exact syntax of an internal API endpoint written yesterday, or a customer's current account balance, you use RAG.

Host 2

LoRa buys you stylistic adaptation formatting and domain behavior.

Host 1

So if you need the model to consistently output its answers in a rigid JSON schema, or adopt the diagnostic tone of a medical professional, you use LoRa.

Host 2

You do not use LoRa to teach the model new facts. You use it to teach it new behaviors.

Host 1

That distinction alone will save an engineering team six months of wasted sprint cycles.

Host 2

Absolutely.

Host 1

So we have memory, we have adaptation, but transitioning from a system that generates text to a production agent requires something more.

Host 2

We have to move beyond the statistical generation of text and cross the boundary into agency.

Host 1

How do we execute deterministic side effects? How does the AI actually do something in the real world?

Host 2

In the modern stack, that is standardized by the model context protocol or MCP.

Host 1

MCP is fascinating. It's the standardization of host client server boundaries for AI.

Host 2

The LLM waits store immense capability.

Host 1

They can reason about a problem.

Host 2

But they are mathematically isolated inside the GPU.

Host 1

Right.

Host 2

MCP provides a standardized JSON RPC interface that acts as the bridge. It allows the AI model, acting through an MCP client, to discover and request tool execution from an MCP server.

Host 1

So let's trace this. The user asks the model to check the production database for user 123's status.

Host 2

The model doesn't have a database connection.

Host 1

Instead, the model generates a highly structured JSON payload. It outputs a string that says effectively, I want to invoke the query database tool. And here are the arguments. The MCP harness, the actual classical deterministic code wrapping the L NLM, intercepts that JSON payload.

Host 2

It validates the schema and it executes the deterministic code. It runs the actual SQL query against the real database.

Host 1

It takes the result of that query, formats it back into text, injects it into the model's context window as a new message, and tells the model to resume generation.

Host 2

It perfectly separates the probabilistic generation of the LLM from the deterministic execution of your backend systems.

Host 1

The weights probabilistically decide what tool to use and what arguments to pass. But the MCP server executes the actual side effects in a sandboxed secure environment.

Host 2

This is how AI safely interacts with REST APIs, local file systems, and production databases without giving us statistical black box direct execute permissions.

Host 1

Okay. We've mapped the entire microservice architecture. But here's the reality check for section four. Macro trade-offs.

Host 2

We can design this beautiful quantized, RAG-enabled MCP-driven agent on a whiteboard.

Host 1

We can deploy it.

Host 2

And then we can watch our cloud bill bankrupt the company in a week.

Host 1

Or watch our latency spike to 15 seconds per request. Systems architects don't operate in a vacuum. We have to balance reasoning depth with the brutal reality of production constraints.

Host 2

Let's start with the big one.

Host 1

When do we use deterministic workflows versus autonomous agents?

Host 2

This is the spectrum of control and unpredictability. On the highly controlled end, we have workflows. We use prompt chaining, static routing, parallelization, and evaluator optimizer loops.

Host 1

Like a classic setup where model A generates a summary report, the output is deterministically passed to model B, which evaluates it against a strict rubric.

Host 2

And if it fails, it loops back to model A with the feedback.

Host 1

Workflows offer predictable latency and predictable costs. You know, exactly how many LLM API calls are going to be made and in what order.

Host 2

They are perfect for well-defined, repeatable enterprise tasks.

Host 1

But then on the other end of the spectrum, we have agents. Autonomous React loops, reasoning and acting.

Host 2

Agents are designed for open-ended problem solving.

Host 1

Think of Sweebench coding agents, where the AI is given a GitHub issue and told, fix this bug.

Host 2

The model dynamically decides its own control flow. It might decide to use an MCP tool to search the code base. The tests fail.

Host 1

It reads the error log, reasons about why it failed, and loops back to rewrite the code.

Host 2

It gives you incredible reasoning depth.

Host 1

It mimics human debugging.

Host 2

But at a steep, often unacceptable cost for real-time systems.

Host 1

You completely lose predictable latency.

Host 2

An agent might solve the problem in two steps, or it might take 50.

Host 1

You burn through tokens at a massive rate.

Host 2

And most critically, you introduce the risk of compounding errors.

Host 1

If an autonomous agent hallucinates a tool output in step two, by step 10, its internal context is completely corrupted and it is completely off the rails. Another massive architectural trade-off we are seeing right now.

Host 2

With models like Gemini 1.5 supporting million token context windows, when do you just stuff the entire 10,000 page PDF into the prompt versus building a complex RAG pipeline?

Host 1

This is a tension between the computational complexity of attention and the precision of retrieval.

Host 2

Remember the architecture of self-attention we discussed earlier? Every token has to attend to every other token.

Host 1

The compute cost scales quadratically with sequence length, big O of n squared.

Host 2

Even with innovations like flash attention, keeping the memory bounds in check, the compute cost is still massive.

Host 1

Stuffing a million tokens into a prompt for every single user query on a high traffic server is computationally exorbitant. It's brute force.

Host 2

It is. Long context models are incredible for analyzing a single massive document in isolation, like asking a model to find contradictions across an entire code base offline.

Host 1

But for high throughput real-time enterprise systems, targeted vector retrieval via RAG is vastly more efficient. Long context windows shouldn't replace RAG architectures. They should be viewed as expanding what a single chunk of retrieve context can contain, allowing for richer, more holistic retrieval.

Host 2

Finally, the ultimate deployment question that every infrastructure team debates. Frontier cloud APIs versus quantized local models.

Host 1

We are comparing massive closed-weight cloud models, which offer peak reasoning capabilities but come with high network latency, severe data privacy concerns, and strict vendor lock-in.

Host 2

Against small, specialized, quantized local models or SLMs running directly on your own infrastructure.

Host 1

If you're the architect on call and your latency spikes to 10 seconds per token on a Friday night because a cloud provider is throttling your API tier, that lack of control is terrifying.

Host 2

Local SLMs give you absolute control over tail latency.

Host 1

You aren't subject to network jetter.

Host 2

You have total data privacy because the PHI or PII data never leaves your VPC.

Host 1

And this is where knowledge distillation is fundamentally changing the deployment game.

Host 2

We use the massive, expensive frontier models as teachers.

Host 1

We prompt them to generate vast amounts of highly structured, complex synthetic data or reasoning traces.

Host 2

We then use that data to train highly efficient student SLMs for specific edge tasks.

Host 1

You do not need a trillion parameter model to accurately route a customer service ticket or extract JSON from a receipt. You need a highly distilled 8 billion parameter model.

Host 2

Running locally, heavily quantized via SmoothQuant, executing with subsecond latency on a single GPU. That perfectly synthesizes the core takeaway of this entire overview. The modern AI stack has moved far beyond a single magical checkpoint that you download from Hugging Face and just run.

Host 1

It is a multi-layered, highly complex distributed system.

Host 2

The weights store the foundational capability.

Host 1

Vector retrieval supplies the verifiable truth.

Host 2

Quantization and LoRa enabled deployability and efficiency.

Host 1

And protocols like MCP execute real world action.

Host 2

It is an ecosystem of deeply interconnected components, all bound by the laws of memory, compute, and latency.

Host 1

Exactly. And this was just the 10,000 foot architectural view. As part of our fundamental AI concept series, we are going to crack open the mathematical and algorithmic internals of these specific components in our upcoming deep dives.

Host 2

We'll be covering the complex mathematics of rope rotation, the actual backpropagation graph mechanics, and the underlying graph theory of HNSW vector indexing.

Host 1

But before we wrap up today, leave us with a final thought to mull over.

Host 2

Here's a concept to consider as you design your next production architecture. As we rely more and more heavily on RAG for our knowledge base, and MCP and external tool calling for our actions, are we rapidly approaching a future where the LLM itself is no longer the brain or the central repository of the application?

Host 1

That's a good point.

Host 2

Perhaps the LLM is simply becoming a lightweight, natural language routing protocol, sitting in the middle of a vast, highly deterministic microservice architecture.

Host 1

Which raises an important question for all of us.

Host 2

Is the future of AI actually just a return to a new paradigm of traditional systems engineering?

Host 1

So we're right back to where we started, trying to find deterministic precision in a probabilistic world.

Host 2

It certainly seems that way.

Host 1

The underlying mechanisms might still be complex, but we finally have the architectural tools to build reliably on top of them. Thanks for taking this deep dive with us.

Save to playlist

Manage playlists →

Share