The Mechanics of Embeddings and Backpropagation
Transcript
Made automatically from the audio, so it may contain mistakes. Click a line to play from there.
So imagine a catastrophic failure mode in production. You're you're building this massive system to solve a really deep high dimensional sequence problem. And you're dealing with a huge vocabulary. We're talking maybe tens of thousands of unique tokens. So you decide to use sparse, discrete representations like one hot vectors.
Oh boy. Yeah.
You feed that data into your architecture, you apply just naive gradient descent, hit compile, and run it. And instead of learning, the loss function just flatlines.
Completely stalls out.
Exactly. The server's chugging, the metric stall, and the entire system just completely collapses under its own weight. It just breaks. And you know, it feels like you followed the textbook perfectly, but under the hood, the math is just screaming. Why does this actually happen?
I mean, it is a brutal reality check for a lot of engineers, honestly. Because treating every single item in a massive vocabulary as a completely isolated, independent symbol, it mathematically guarantees that you're gonna stall out your optimization.
You're starving it of context.
Exactly. You are starving the system of context before it even begins to learn anything.
Well, welcome back to our series on fundamental AI concepts. You're already familiar with the high level pipeline. You know the difference between the training lifecycle and online inference from our previous deep dive. So our mission today is to open up the black box.
We're getting into the weeds today.
We really are. We're gonna explore the internal mechanics of neural networks, backpropagation, and representation geometry. And uncover exactly why these systems break down in the ways we just described.
Because to understand why that catastrophic failure happens with discrete representations, we have to look at how a system is actually forced to interpret that data geographically, you know?
Yeah. So let's talk about why one hot vectors fail so spectacularly. In a one hot encoding, every single word or token in your vocabulary gets its own unique dimension.
Right. A whole dimension to itself.
Right. So you have a vector full of zeros and a single one at the index of that specific word. The problem is geometrically, every single one of those vectors is perfectly orthogonal. They are all sitting at exact 90 degree angles to each other.
Which is, mathematically speaking, a disaster for semantic meaning. It means the mathematical distance between the vector for cat and the vector for dog is identically the same as the distance between cat and car.
Wow. So there's just zero semantic similarity embedded in the math whatsoever.
None at all. You've essentially created this massive empty coordinate system, which is what mathematicians call the curse of dimensionality. The system is just entirely blind to the relationships between concepts because every concept is its own isolated island.
I'd like to use an analogy for how we actually want this to work. So imagine trying to understand a person's personality. If you used one hot encoding, every single human would just be their own isolated category. There's no way to compare them.
Which isn't very helpful for predicting behavior.
Right. But what if we used the big five personality traits instead? You know, introversion versus extroversion, agreeableness, openness, and so on.
Oh, I see where you're going with this.
Yeah. So imagine scoring someone on each of those five traits on a continuous scale from say negative one to positive one. By mapping out those five numbers, you create a very specific coordinate in a five dimensional space. And people with similar personalities will be physically closer together in that space.
That's a great way to put it.
Dense embeddings do this exact same thing for data, but instead of five dimensions, they use hundreds, anywhere from like three hundred to four thousand ninety-six dimensions.
Yeah, it paints a perfect picture, really. Because learned dense embeddings project those discrete symbols into a continuous high-dimensional vector space. The geometry of that space physically encodes the meaning. Right. And the classic, most famous demonstration of this, you've probably seen it, is from the word to vec model. If you take the dense vector for the word king, geometrically subtract the vector for man, and add the vector for woman, the resulting coordinate you arrive at is shockingly close to the vector for queen.
See, that just blows my mind every time I think about it. Directional alignment actually encodes semantic and syntactic relationships.
It really does. Yeah. And we measure that alignment using things like cosine similarity or dot products. So if vectors point in the same direction in that high dimensional space, their dot product is high, which means they share semantic traits.
You've basically transformed abstract meaning into measurable geometry.
Exactly.
But okay, here is the engineering nightmare. We've got 4,000 dimensions and millions of words. How do you actually train a space like that? I mean, you can't possibly calculate the relationship between every single word and every other word in a massive data set all at once. The processing power would just be astronomical.
Oh, it's a computationally impossible. Doing that across a vocabulary of millions of words, you would be updating billions of probabilities every millisecond. The hardware would just melt.
Right. So what's the work around?
This is where a very specific technique called noise contrastive estimation comes in. And it's often implemented using a skip gram architecture.
Okay, skip grams. I've seen that mentioned in the documentation. That's where instead of looking at the whole vocabulary, you just light a small window across your text, right?
Exactly. You look at a target word in the center of that window and its actual neighbors in the text. Those are your positive examples. You want the math to pull those vectors closer together.
Makes sense. Birds of a feather.
Right. But if you only ever pull things together, the system will eventually collapse all words into a single point in space.
Oh, because everything is just gravitationally attracted to everything else.
Precisely. To prevent that collapse, you introduce negative sampling. You pick random noise words from the broader vocabulary that aren't in that specific window. And you tell the system, push the target word away from these.
So you are contrasting the actual positive signal with random noise.
Hence noise contrastive estimation.
So you're just sampling a tiny fraction of the dictionary at any given time to push and pull these coordinates around. And that allows you to build this rich geometry efficiently without having to compute probabilities for the entire dictionary every single step.
Yes. Which brings us to the next massive hurdle, honestly. We've successfully mapped our discrete tokens into this beautiful continuous 4,000 dimensional geometry. Great.
Mission accomplished.
Well, not quite, because finding the absolute perfect location for millions of vectors requires a massive update mechanism. The geometry is totally useless without a way to actually optimize it.
Right. Because we've got millions of coordinates. If we realize a word is in the wrong place, how on earth do we update all those parameters at the exact same time without the server catching fire? You need a totally different way to calculate the error.
And that is why we rely on the computational graph and reverse mode automatic differentiation. In the industry, we just call it backpropagation. Back prop. Yeah. So a computational graph breaks down really complex mathematical expressions into individual nodes and edges. Inputs flow forward through matrix transformations and nonlinearities until they produce a final output.
And at the very end of that forward pass, you calculate a loss function, something like cross-entropy, right? Which gives you a single scalar value representing basically how wrong the system's prediction was.
Exactly. Just one number summarizing all the error.
But the challenge is adjusting millions of weights based on that one single error value. Because if you do it naively, trying to sum up every possible path from the input to the output, you get a combinatorial explosion. I read it would take tens of thousands of years to compute.
Yeah, that combinatorial explosion is exactly why we use reverse mode auto diff. Instead of going forward and branching out exponentially, it factors the paths. It applies the chain rule of calculus backward, starting from that single scalar loss function and it distributes the error gradient back through the graph. Okay. And because it merges paths at every node going backward, it can update millions of parameters in what we call an O1 pass, meaning it touches each edge of the graph a single time.
Wait, wait. Let's unpack this for a second. If there are billions of pathways from the input to the output, how is it mathematically possible to only touch each edge once? That sounds kind of like a paradox. Like if reverse mode goes backward through the graph to calculate the gradients, doesn't the system need to remember exactly what happened going forward?
Yes. You've hit on a really crucial mechanical truth of these architectures. The forward activations absolutely must be cached in memory during training. Wow. Yeah. You strictly cannot calculate the backward gradient without knowing the exact state of all the nodes during the forward pass. Every single intermediate mathematical step has to be held in your hardware's VRAM.
Okay, we are definitely going to talk about the harsh reality of that memory tax in a moment, but even assuming you have perfect gradients and infinite memory, navigating that optimization space is just treacherous. We are dealing with a lost landscape that is wildly complex. It is not some smooth, simple bowl where gravity just pulls you neatly to the bottom.
Far from it. It's full of pathological curvature. You frequently encounter what mathematicians call ill-conditioned Hessian ravines.
That sounds intense.
It is. Picture a multidimensional canyon where the walls are incredibly steep. But the floor of the canyon slopes very, very gradually toward your destination. If you use vanilla stochastic gradient descent or SGD in that landscape, the results are chaotic.
It's like throwing a ping-pong ball down a hallway. It bounces violently wall to wall across the steep sides of the ravine, and it makes barely any forward progress along the actual galley floor. Just oscillates frantically.
That's a perfect analogy. And to stop that violent ping-ponging, you have to introduce physical inertia into the math, what we call momentum, usually denoted as beta.
Okay, momentum.
Right. Morantum dampens the oscillations by averaging the past steps. But modern systems take it a step further with optimizers like Adam and Adam W.
Yeah, Adam W is basically the industry standard right now. It uses adaptive step scaling, which sounds like a lot, but let's break it down.
So it maintains an exponential moving average of both the gradients and the squared gradients. We call these the first and second moments. The first moment acts like the momentum we just talked about, building up velocity in a consistent direction.
Like a snowball rolling down a hill.
Exactly. And the second moment tracks the variance. It's essentially measuring how erratic the gradients are. So Adam W uses that second moment to scale down the steps in those really steep, erratic directions while accelerating the steps in the shallow, consistent directions.
So it's actively reading the terrain and adjusting its footing as it goes.
Exactly.
And Atom W specifically decouples weight decay, which is, you know, a regularization technique to keep weights from growing too wildly large. It decouples that from the gradient updates. And that allows it to take larger, more confident steps through those pathological ravines without completely destabilizing the network.
Right. So we have our dense embeddings and we have Atom W navigating the lost landscape beautifully. And that works incredibly well for static data like images or standalone documents. But time series and sequential data introduce a structural nightmare that simple optimization tricks just cannot fix.
Right. Because if we connect this to the bigger picture, recurrent architectures like unrolled recurrent neural networks or RNNs, they force the system to do repeated matrix multiplications across time steps.
And that brings us back to basic arithmetic. Think about compounding. If you multiply a number by 0.9 a hundred times, it shrinks essentially to zero.
It just disappears.
Right. But if you multiply by 1.1 a hundred times, it explodes to astronomical numbers.
And that is the vanishing and exploding gradient problem in a nutshell. Over long sequences, the gradients required to learn long-term dependencies either disappear entirely, meaning the system completely forgets the past, or they explode, completely breaking the math and returning NAN errors.
It was a massive bottleneck for years.
LSTMs, right?
Yeah. LSTMs introduced this brilliant conveyor belt concept called the cell state, and they paired it with additive gates. This allows information to flow through time without being constantly multiplied. The gating logic just decides what to keep, what to forget, and what to output at each step.
Furthermore, modern deep architectures mandate two really critical stabilization primitives on top of that. First, layer normalization, which computes the mean and variance across a layer for a single case, keeping the values incredibly stable. And second, residual connections.
Yes. Residual connections are the identity shortcuts. So instead of just passing data through a heavy mathematical transformation, which we can call F of X, you add the original input back to the output at the very end. The equation literally becomes X plus F of X.
I always think of a residual connection like a bypass highway around a really congested city. The data can either go through the heavy traffic of the neural network layer, you know, the transformations, or it can take the bypass and arrive at the next layer completely untouched.
That's exactly how it functions.
It creates a pristine gradient highway that lets error signals bypass those heavy transformations and flow cleanly all the way back to the early layers of the network.
It basically guarantees the original signal doesn't get lost in the noise of deep processing.
Okay, but here is where I get confused, and I think a lot of people do too. If residual connections and layer norm create these beautiful gradient highways and they stabilize the hidden states perfectly, why did the RNN and LSTM gating mechanisms still completely bottleneck sequential parallelization? Like, why did the industry have to move past them entirely?
Well, that raises an incredibly important point about the difference between mathematical stability and computational efficiency.
Okay.
Because the fundamental flaw lies in the dimension of time itself. Despite all those stabilization primitives, the gating logic of an LSTM remains inherently strictly sequential.
The math requires the previous hidden state to calculate the current gates values.
Yes. You strictly cannot compute step 100 without first having computed step 99. And you can't compute step 99 without having finished step 98.
Oh wow.
So no matter how many thousands of GPU cores you have sitting totally idle on your massive server cluster, they have to wait in line. The architecture hardcaps your hardware parallelization.
That's I mean, that's brutal. Because these architectures force us to cache massive amounts of data for backprop and wait for sequential operations in recurrent layers, they dictate very harsh physical realities for our hardware.
They absolutely do.
This is where we see major production failure modes. Let's talk about the training memory overhead. It is a harsh engineering reality that training consumes roughly 4x to 8x more VRAM than inference.
And let's do the exact byte per parameter VRAM arithmetic here, because engineers really need to know this when they're provisioning servers. During inference, when you are just running the system to get a prediction, you only need VRAM for the weights themselves. Okay. So if you have a 7 billion parameter model in 16-bit precision, you need about 14 gigabytes of VRAM just to hold the model.
Right. And 14 gigs easily fits on a standard high-end commercial GPU.
Exactly. But during full atom training, your memory footprint just explodes. You need those 14 gigabytes for the weights, plus another 14 gigabytes for the gradients for every single weight.
Okay, so that's 28.
Plus 14 more gigabytes for the optimizer's first moment, the momentum we talked about, plus 14 more for the optimizer's second moment, the variance.
Wow. We are already at 56 gigabytes.
And we're not even done. On top of all of that, you need VRAM for all those cached forward activations we discussed earlier, which scale linearly with your batch size and your sequence length.
So a system that easily runs on a laptop GPU for inference might require a massive multicard enterprise server cluster just to execute a single training run.
Precisely. It's a totally different ballgame. And beyond just memory crashes, there are high-dimensional geometry pitfalls you have to watch out for in production.
Oh right.
One of the most insidious ones is anisotropy.
Anisotropy, right. This is when the entire embedding space basically collapses, right?
Yes. Instead of utilizing that beautiful spherical 4,000 dimensional space we envisioned earlier, all the vectors collapse into a very narrow cone. Everything gets clustered together incredibly tightly in one specific direction.
And why is that bad?
Because when that happens, the cosine similarity between completely unrelated tokens becomes artificially high. The system literally thinks everything is highly similar to everything else. It loses all ability to discriminate meaning.
It's like having a giant mansion, but stuffing all your furniture into one tiny closet. I mean, you have all this space, but you're not using any of it, and you can't tell the couch apart from the television because they're stacked on top of each other.
That is painfully accurate. You also have to carefully contrast overfitting with true generalization here. Because unregularized optimization often ends up chasing high frequency noise in adversarial artifacts instead of true semantic meaning. We see this all the time in feature visualization.
Oh, feature visualization is fascinating. That's when you optimize a random input image to see what a specific node in the network is actually looking for, right?
Exactly.
You would hope to see a clear image of, say, a dog ear or a car tire, but often instead of a clear concept, you get a nightmare of static and high frequency checkerboard patterns.
Which means what exactly?
It means the math found a structural cheat code. It figured out how to maximize the node's activation value without actually learning the underlying concept you wanted it to learn. It's exploiting the math. So given all of this, what does this all mean for someone actually building a system today? I mean, given the massive VRAM footprint of training and the latency bottlenecks of sequential generation, engineers have to be ruthless about what they deploy to production. When is deploying a massive full generative model actually just massive overkill?
It is a critical decision boundary for any engineering team right now. If your workload involves classification, clustering, or similarity searches, think of the recommendation engines used by Spotify to suggest songs or Airbnb to suggest properties.
Right.
Deploying a massive generative LLM decoder for that is often a terrible engineering choice.
Because you don't need the system to talk to you or write poetry. You just need it to find relationships between items.
Exactly. For those specific tasks, deploying lightweight, standalone dense embedding models vastly outperforms generative models. You just map your data into the vector space and you do math on the coordinates to find the nearest neighbors.
And the latency on that?
Fractions of a millisecond, the RAM requirements are minimal, and you completely avoid the sequential decoding overhead of generating tokens one by one. In those classification or search cases, embeddings are just the mathematically and financially superior choice.
It is entirely about using the right tool for the job. Well, let's recap the journey we've taken in this deep dive. We started by seeing exactly why discrete symbols like one-hot vectors fail. They suffer from orthogonality and the cursive dimensionality, starving the model of context.
Right. And then we explored how continuous geometry solves that semantic mapping using dense embeddings and noise contrastive estimation to place concepts in a measurable space by comparing positive neighbors against random noise.
And we looked at the mechanics of reverse mode auto diff on computational graphs. We discovered why calculating gradients in a single sweep creates such heavy memory burdens by forcing us to cache all those forward activations.
We navigated the pathological ravines of the lost landscape, seeing how Atom W adaptive step scaling uses momentum and variance to stabilize optimization without flying off the cliffs.
And finally, we saw how time series data introduces compounding vanishing gradients and why, despite the highway bypasses of residual connections and layer norm, the gating logic of LSTMs strictly bottlenecks parallelization because it is chained to the flow of time.
Because at the end of the day, the physical reality of the math ultimately defines the limits of the hardware.
Which leaves you with a final puzzle to mull over. If deep models are essentially navigating in activation space based on geometric relationships, and if LSTMs are fatally bottlenecked because their architecture forces us to process time sequentially, step by step, waiting for the previous step to finish. Right. What would happen if we could mathematically bypass time entirely? What if an architecture was designed to look at every part of a sequence all at once, paying attention to everything simultaneously?
Now that that is a fascinating concept.
Something to explore on your own. Thanks for joining us on this deep dive.