Transformer Bottlenecks
Transcript
Made automatically from the video's sound, so it may contain mistakes. Click a line to play from there.
Welcome to this explainer. Right, we're skipping the high level history today and diving straight into the engineering mechanics of modern transformers. Look, if you already have a handle on embeddings, computational graphs and residual blocks, you are in exactly the right place.
Today, we're looking right under the hood at the physical and computational constraints that actually dictate how these massive models operate, and crucially, the brilliant architectural breakthroughs designed to solve them. So here is our roadmap for today.
First up, tokenization and memory trade-offs. Second, scaled dot product self-attention. Third, rotary position embedding, or rope. Fourth, the decoding memory bottleneck, and finally, GQA and flash attention.
Let's kick things off with part one, tokenization and memory trade-offs. Now, subword algorithms like byte pair encoding, or BPE, essentially build a vocabulary from the ground up. Think about it.
Instead of assigning a unique token to every single whole word in existence, which is frankly physically impossible, or just using individual characters, these algorithms start with base bytes. They then iteratively merge the most frequently adjacent pairs into subwords.
The result? A highly efficient system that keeps the vocabulary compact, while still retaining the ability to represent completely unseen complex words by simply decomposing them into known chunks.
It's quite brilliant. You might be wondering, why not just have a massive vocabulary to capture everything perfectly from the get go? Well, it's a brutal engineering trade-off.
A huge vocabulary directly bloats the embedding table memory, literally eating up valuable GPU footprint before the processing even begins. But on the flip side, if you use a tiny vocabulary, like, say character level tokenization, you drastically increase the length of your input sequences.
And because the attention mechanism scales quadratically with sequence length, longer sequences mean your compute time absolutely skyrockets. So tokenization really is this delicate art of balancing these two extremes. Scale dot product self-attention.
Let's look at the core scaled dot product attention maths. First, we project our input embeddings into three distinct matrices. You can think of the query as what a token is looking for, the key as the label of what a token actually contains, and the value as the raw content it passes along.
We score the relevance of tokens by taking the dot product of the queries and keys. But notice that absolutely vital second step, scaling this by the square root of the head dimension. Without this scaling factor, large dimensions would push those dot products into extreme regions, leaving the softmax function with tiny, completely unworkable gradients.
Finally, we multiply those normalized scores by our value matrix. We then face a major architectural split in how we apply this attention. In auto-regressive generation, like a standard GPT model predicting the next word, we apply causal masking.
This is a decoder-only setup that explicitly prevents current positions from attending to future tokens, literally masking them out with negative infinity before the soft mask. Basically, looking into the future would be cheating.
Conversely, bidirectional attention, which you'll see in BERT-style models, allows an encoder to look at all tokens in a sequence simultaneously, to build a deep contextualized representation of the entire text at once. Alright, part three.
Rotary position embedding, or rope. Transformers actually process everything in parallel, meaning that without explicit help, they have absolutely no idea what order the tokens are in. They're completely blind to sequence.
Rotary position embedding, or rope, elegantly solves this spatial constraint. So instead of simply adding a static vector to our embeddings, rope encodes the absolute position of a token by applying a rotation matrix in a 2D complex space to the query and key vectors.
Because of the mathematical properties of rotation, taking the dot product of a rotated query and a rotated key naturally incorporates their relative distance. This is a real game changer. It gives us relative position dependency whilst entirely avoiding the need to materialize massive memory-heavy positional tables.
Let's dive into part four, the decoding memory bottleneck. With the foundational maths established, we really must pivot to the physical constraints of GPU hardware. There's a stark contrast between the two phases of running a language model.
When you send a prompt, the model processes it entirely at once in the pre-fill phase. This is highly parallel, blisteringly fast, and strictly compute bound. But once it starts actually generating the answer, it enters the decode phase.
This is auto-regressive, meaning it generates token by token. Because it must constantly fetch historical data from memory to calculate attention for just one single new token, this phase is incredibly starved for memory bandwidth.
To avoid expensively recomputing the states of all previous tokens every single time we generate a new one, we cache them. This is the famous KV cache. At stage N, we compute everything and store the keys and values.
Then at n plus one, we just compute the new tokens, QKV, append its key and value to the cache, and load the past KV to calculate attention. However, this KV cache grows linearly with every single token you generate. For long sequences, this cache becomes absolutely massive, literally clogging up the memory bandwidth as it's repeatedly hauled back and forth across the GPU for every single step.
GQA and flash attention. If the KV cache is the enemy of efficient inference, well we need to shrink it. Standard multi-head attention requires a unique key and value head for every single query head.
Grouped query attention, or GQA, cleverly changes the architecture by sharing a single key and value head across a group of multiple query heads. This drastically shrinks the footprint of the KV cache in memory, heavily reducing the bandwidth required to load it during decoding, whilst maintaining almost the exact same model quality.
It's a brilliant compromise. And our final breakthrough for today is flash attention, which is entirely about being I.O. aware, meaning being really smart about how data moves physically inside the GPU.
You see, GPUs have a large, slow high bandwidth memory, or HBM, and a tiny, blisteringly fast on chip SRAM. Standard attention writes the massive intermediate attention matrix to the slow HBM, which just takes forever.
Flash attention uses a technique called tiling. It loads blocks of data from HBM to the fast SRAM, computes the attention on chip without ever materializing that full quadratic matrix, and only writes the final results back to the slow memory.
It is an absolute masterclass in software, optimizing around hardware physics. So we've seen how tokenization trade-offs, rope, grouped query attention, and flash attention all work in concert to bypass the brutal computational and memory limits of standard transformers.
But as we wrap up this explainer, I want you to consider this question about the future horizon of context scaling. If we've successfully optimized our I.O. operations and drastically reduced our KV caching footprint, what becomes the next fundamental bottleneck as we push context windows past 1 million or even 10 million tokens?
It's a fascinating engineering puzzle to leave you with. Thanks for joining me and keep exploring.