The math and hardware behind AI
Transcript
Made automatically from the audio, so it may contain mistakes. Click a line to play from there.
Picture this. You you sit down with one of the most advanced large language models on the planet.
Right. Something with, you know, billions of parameters.
Exactly. And you ask it a totally simple trivia question, like, who is Tom Cruise's mother? In less than a second, it spits out the correct answer. Mary Lee Pfeiffer.
Which is pretty impressive, right?
Totally. But then you flip the question. You ask, who is Mary Lee Pfeiffer's son? And the AI just it completely fails. It either guesses wrong or you know says it doesn't know.
Yeah, and that right there that really shatters that illusion of a super genius machine we all kind of carry around in our heads.
It really does.
I mean, it reveals a massive disconnect in how we think these systems operate versus what they are actually doing under the hood.
Because usually when we look at a magic trick, there's this silent agreement.
Yeah.
You see the magician pull a rabbit out of an empty hat and you applaud the illusion. You accept that you don't know where the trapdoor is.
Right, you're just enjoying the show.
Yeah. But lately it feels like we're living inside this global magic show. And the magician is artificial intelligence. Every single day you unlock your phone with your face, or you get this perfectly tailored weavy recommendation, or your email automatically filters out a phishing scam.
And it feels like magic.
It does. But the stakes are way too high for us to just blindly applaud anymore. I mean, we really need to know where the trapdoor is.
Absolutely. Because when we don't understand the mechanics operating beneath the surface, we we basically surrender our agency to them. We stop being the audience and start being the props.
Yeah, and that is exactly our mission for this deep dive. We are stripping away the magic and looking at the actual mechanics of artificial intelligence.
Getting into the real nuts and bolts. Right.
We're getting into the difference between theoretical AI and the reality we actually live with. Plus, you know, the specific hardware fueling this entire revolution, the invisible math shaping your social media feeds, and and why the future of AI isn't just about building bigger models.
But making them smarter, yeah.
Exactly, making them smarter.
So to start, I think we can clear up a massive misconception. Hollywood has given us a very skewed idea of what AI actually is.
Oh, for sure. The whole Terminator or H A L 9000 thing.
Right. Most people watching sci-fi, they're thinking of artificial general intelligence or AGI. That's a system capable of abstract thoughts, solving novel problems across totally diverse fields without needing, you know, specific training for every new task.
The theoretical super brain. But I mean, that's not what's unlocking your phone. We all know the reality is artificial narrow intelligence or AI, weak AI.
Yeah, it's highly specialized. Like a chess bot can beat a grandmaster in 20 moves, but it absolutely cannot summarize a PDF or give you medical advice.
It operates entirely within its designated lane. Okay, let's unpack this. If we're dealing with narrow AI, the engine powering it is machine learning. And that shift to machine learning completely changed how computing works, didn't it?
It flipped the whole paradigm. Historically, if you wanted a computer to do something, you had to write explicit, rigid if-then rules for every single possible scenario.
Which is exhausting just to think about.
It is. But machine learning changed that. Instead of programming the rules, you give the system raw data and it learns the rules on its own.
So if you wanted to recognize a dog, you don't program the geometry of like floppy years and a tail.
Exactly. You feed it 10,000 pictures of dogs, and it identifies the underlying statistical patterns.
And this learning process essentially breaks down into four distinct stages. When I was going through the research, I kept picturing an athlete preparing for a massive game.
Okay, I like that analogy.
Yeah. So if stage one is data collection gathering, massive amounts of high quality, diverse training data, like images or text, that's basically our athlete sitting in a film room just watching hours and hours of game tape, right? Taking it all in.
That's spot on. And the quality of that game tape directly determines their performance on the field. I mean, if they only watch tape of left-handed pitchers, they will strike out the second they face a right hander.
Right. They wouldn't know what to do.
Exactly. So then stage two is pattern recognition. The algorithm processes all that data to find hidden structural relationships.
So that's the athlete watching the tape and suddenly noticing the opposing pitcher's tell. Like realizing the pitcher always dips his shoulder a fraction of an inch before throwing a curveball.
Yes. The system is identifying features that correlate with specific outcomes. In a retail data set, it's noticing which specific products are frequently bought together at, say, 100 p.m. on a Tuesday.
Okay, that makes sense.
And from there, we move to stage three, which is model optimization. This is the iterative cycle of testing and refinement. The model makes a prediction, compares it to the known outcome, realizes it made a mistake, and then adjusts its internal parameters.
Okay, so this is the athlete in the batting cage. They swing at that curveball, they miss, they realize their stance was too wide, they tweak their grip, and they try again over and over until they stop missing.
Precisely. That adjustment process is entirely mathematical. It's just minimizing errors until the predictions become reliable, which finally brings us to stage four deployment and prediction. The model is released to make decisions on new unseen data.
The athlete steps up to the plate in a real live game.
Exactly. And what's really fascinating here is that a well-designed model doesn't just, you know, stop learning once it's deployed.
Right, it keeps going.
Yeah, it continues to take in new inputs during its operational life cycle. It's constantly refining its swing based on the live pitches it sees every single day.
Here's where it gets really interesting, though. If those four stages are the process, what is the actual physical engine running it? Because doing millions of calculations in that virtual batting cage takes immense physical power.
Massive amounts of power.
And in the AI industry, that physical power comes down to this massive hardware war between GPUs and TPUs.
The silicon foundation of the entire AI boom. So let's look at GPUs or graphics processing units.
But the ones that were originally designed for video games.
Right. Rendering a complex 3D environment requires calculating the physics, the lighting, the movement of millions of individual pixels simultaneously.
And it turns out that exact type of parallel processing power is precisely what you need for the dense matrix multiplication required in deep learning.
Exactly. And GPUs, specifically the ones built by NVIDIA, they completely dominate. I mean, they hold roughly 80% of the AI market.
Which is huge.
It is. They use thousands of programmable CUDA cores. You can think of CDA cores as thousands of tiny independent calculators sitting on the chip, all crunching separate numbers at the exact same time.
So they're incredibly versatile. And they natively support massive open source software toolkits like PyCorch, which is what developers actually use to build the models.
Yeah. But Google saw this bottleneck and built an alternative, the TQU or tensor processing unit.
Okay, so how is that different?
Well, unlike a GPU, which is a versatile general purpose tool, a TPU is an ASIC and application-specific integrated circuit. It was custom built from the atomic level up for one single job. Neural network math.
So how does that actually change the physical processing compared to those, you know, independent CUDA calculators on a GPU?
Think of it this way. A GPU is like a massive room full of thousands of independent workers who can all do different tasks at the same time. A TPU uses a specialized architecture called a systolic array.
Systolic array, yeah.
It operates more like a highly synchronized factory assembly line. Data streams rhythmically across a rigid grid of processing elements. It sacrifices all flexibility for raw, unadulterated efficiency.
And the scale of that efficiency is staggering. I mean, Google's latest ironwood TPUs, the V7 generation, they can scale up into these massive supercomputing clusters called pods.
Right. Thousands of chips.
Yeah, we're talking about 9,216 chips wired together, delivering 42.5 exaflops of computational power. Which an exaflop is a quintillion calculations a second. That is a one followed by 18 zeros every single second.
It's almost hard to wrap your head around. And because of that specialized factory line design, TPUs offer incredible performance per watt. They're highly energy efficient. The ironwood architecture is roughly 30 times more efficient than Google's first generation TPU from a decade ago.
Wait, if TQUs are 30 times more efficient now and they can scale to 42 quintillion calculations a second with this synchronized assembly line, why does NVIDIA and the GPU still own 80% of the market? Why wouldn't every tech company just buy TPUs?
It basically comes down to the classic trade-off between specialization and versatility. TPUs are heavily locked into Google's walled garden. They're designed to run on Google specific software frameworks, and you primarily rent access to them through Google Cloud.
So you can't just buy a create a TPUs and put them in your garage.
Exactly. But GPUs are the Swiss army knife of computing. You can buy them, rack them in your own private servers, and run almost any open source framework you want. I see. Yeah. For researchers and companies who demand absolute control over their infrastructure, or who have, you know, mixed workloads running AI today, but maybe scientific simulations or video rendering tomorrow, the GPU remains the default choice.
Okay, so that physical hardware is what powers the batting cage. But how does the AI actually know it's missing the ball? How does a machine mathematically measure a mistake?
This is the core math of optimization, and it's built around a concept called a loss function. A loss function is a mathematical formula that measures how far the AI's prediction deviates from the ground truth.
With the ground truth being the correct answer provided in the training data.
Right. So if the model's guess is completely wrong, the loss function outputs a very high number. If it's perfectly accurate, the loss is zero. The entire purpose of a machine learning algorithm is just to minimize that loss function.
And there are different types of loss functions depending on the job you give the AI, right? Like if you're predicting a continuous number, say trying to estimate the price of a house based on its square footage, you would use a regression loss, like mean squared error.
Yeah, mean squared error, or MSE. It literally squares the difference between the model's prediction and the actual house price.
And because it squares the error, it heavily penalizes big mistakes. Being off by $10,000 creates a vastly larger penalty than being off by 1,000, which forces the model to aggressively correct major outliers.
Spot on. But if you're doing a categorization job like deciding if an incoming email is a phishing scam or not, you use a classification loss like cross-entropy.
Cross entropy. Okay, how does that work?
It measures the uncertainty in the prediction. It doesn't just want the model to guess scam correctly, it wants the model to be 99% confident that it is a scam. It actually paneletizes a model for being right but hesitant.
That's fascinating. And to actually fix these errors and get that loss number closer to zero, the model uses mechanisms called gradient descent and backpropagation.
Yeah, and the best way to visualize gradient descent is to imagine a hiker standing at the top of a rugged mountain trying to find the lowest point in the valley below.
Okay, I'm picturing it.
But the hiker is totally blindfolded, they can't see the valley. All they can do is feel the slope of the ground beneath their feet. They take a step in the direction that feels the steepest going down.
Just feeling their way down the mountain.
Right. They evaluate the new slope, take another step down, and repeat this until the ground flattens out. In AI, that valley floor is zero loss.
And back propagation is how the hiker figures out which specific muscles to move to take that downward step.
Yes, exactly.
After the model makes a guess, it looks at the loss and works backward through its neural network. It figures out exactly which simulated neuron contributed to the bad guess and tweaks the numerical weight of that specific neuron so it performs better on the next pitch.
Which is brilliant, but it raises a really important question. It's often referred to as the black box problem.
Right, because of the complexity.
Exactly. When you are adjusting billions of parameters simultaneously, the optimization process creates an architecture so complex that human beings can no longer read it. We know what the model decided, but we do not know why.
Because the math is too dense.
Yeah. It's distributed across too many hidden layers for a human engineer to trace the exact logical path from input to output.
Which is terrifying when you apply this technology to human lives. Yeah. I mean, it's one thing if a black box algorithm recommends a terrible sci-fi movie. It's an entirely different reality when black box denies your mortgage application or you know flags you as a high risk in a criminal justice system, and the developers themselves cannot explain the reasoning.
Accountability completely vanishes. Yes. And the European Union has actually tackled this directly in their ethics by design guidelines. They mandate that AI systems must respect human agency, ensuring fairness and accountability. If an AI makes a high stakes decision about a human life, there must be human oversight and a transparent way to explain the logic path.
And the danger compounds when you introduce algorithmic bias. It's the oldest rule in computer science. Garbage in, garbage out.
Yep, it always comes back to the data.
Exactly. If a corporate AI hiring tool is trained on 20 years of historical data from a heavily male dominated industry, the machine learning process will look at that historical ground truth and optimize its weights to penalize female candidates.
Because the math doesn't know what sexism is.
Right. It is simply minimizing its loss function by mathematically replicating the outdated societal patterns hidden in its training data.
It treats human prejudice as a statistical pattern to be optimized, which is why regulatory guidelines are heavily pushing for rigorous data governance and bias testing before these systems are ever deployed.
But this invisible math isn't just happening in corporate HR departments, though. It's happening in your pocket right now. Social media platforms completely run on this optimization logic.
Oh, absolutely.
Like years ago, Facebook used a manual formula for your feed called edge rank. Human engineers manually tuned the dials. A photo gets five points, a comment gets three points, a like gets one point.
But manual tuning just cannot scale to billions of users generating petabytes of content. The platforms entirely shifted to machine learning, specifically optimizing for a metric Facebook dubbed meaningful social interactions, or MSI.
And to figure out what a user finds meaningful, these platforms rely on techniques like matrix factorization and collaborative filtering. At its core, this is just the classic e-commerce engine. Customers who bought this also bought that.
But apply to human attention on a massive scale. Matrix factorization takes a giant, mostly empty grid of users and posts. Most people on a platform haven't interacted with most of the content. Sure. But the algorithm analyzes the sparse interactions that do exist and mathematically identifies hidden underlying attributes.
Think about it this way. Imagine you have videos that are broadly funny and videos that are informative. The algorithm doesn't speak English. It has no concept of humor or education.
None at all.
But by tracking engagement, it groups users and posts together. It realizes user A consistently engages with attribute X and attribute Y. Video 4 contains high concentrations of both attributes. Therefore, serve video 4 to user A.
And in practice, the system maps every user and every piece of content into a high-dimensional embedding space.
A high dimensional embedding space.
Yeah. Imagine a 3D map, but instead of three dimensions, it has thousands. It plots you as a specific coordinate based on your behavioral history. It plots every video as a coordinate. If your coordinate drifts close to a video's coordinate in that space, it injects that video right into your feed.
So what does this all mean? If you're scrolling on your phone right now, are you actually exercising your own unique autonomous taste? Or are you just fulfilling a mathematical probability score mapped out in a high-dimensional embedding space?
That is the core risk of what the tech industry calls frictionless design. The social media algorithm is minimizing its loss function by maximizing your time on screen.
It's just doing its job.
Right. It learns exactly which psychological buttons to press to keep you scrolling, seamlessly prioritizing your subconscious short-term impulses over your active human judgment or long-term satisfaction.
It is entirely bypassing our autonomy, which loops right back to the EU's demand that AI respect human agency. If our digital reality is increasingly governed by powerful, opaque black boxes that are susceptible to biased data and designed to manipulate engagement, how do we secure our future with this technology?
Well, we have to start by being realistic about their current limitations. We really need to stop treating large language models like GPT-4 as infallible superintelligences.
Which goes back to the Tom Cruise example from the beginning of the deep dive. That perfectly illustrates this. LLMs are incredible, but fundamentally they are highly advanced predictive text engines. Right. They predict the next most mathematically probable token based on their training data. They are not logic engines.
Researchers tested GPT-4's ability to verify prime numbers, a task requiring basic logical deduction.
And how did it do?
It was only successful in 2.4% of cases.
Oh, wow. And when it failed, the who is Mary Lee Pfeiffer's son test, it proved it isn't reasoning. It hasn't built a conceptual knowledge graph connecting a mother and son bidirectional.
No, it's just read the exact phrase Tom Cruise's mother is Mary Lee Cypher thousands of times, but rarely read the reverse. It is basically a statistical parrot.
And this reliance on probability over logic is what causes hallucinations, right? Because the model's primary objective is to confidently predict a plausible sounding sequence of words, it will seamlessly invent fake, peer-reviewed academic papers on nuclear physics if prompted.
Exactly. It possesses no internal mechanism to distinguish between empirical fact and generated fiction.
And even when a model is working perfectly, it faces model drift.
Oh, yeah. Model drift is a huge issue.
A model's predictive accuracy naturally decays over time because the real world changes. But the static training data does not. If an AI was trained to predict consumer buying habits using data from 2019, a massive global economic shift in 2026 makes those predictions actively harmful. The statistical relationship simply no longer exist.
If we connect this to the bigger picture, just throwing more data and more compute at these neural networks isn't going to fix their underlying inability to reason. Right. That is why the bleeding edge of the field is moving toward a counter movement called neurosymbolic AI.
Which is essentially combining two entirely different eras of computer science.
Precisely. Neurosymbolic AI aims to take the best of both worlds. It integrates the incredible messy pattern recognition capabilities of deep neural networks with the strict rules-based logic and factual frameworks of traditional symbolic AI.
So instead of just blindly guessing the next word probabilistically and hallucinating a fake legal precedent, the AI actually has a hard-coded symbolic framework of real laws and facts to check its own math against before it speaks.
It grounds the AI in reality. And this structural shift is crucial for high-stakes applications like medical diagnosis or autonomous driving, where you absolutely cannot afford a hallucination.
That makes total sense.
But even with the promise of neurosymbolic systems, the universal consensus among developers and ethicists is the absolute necessity of augmented intelligence.
Meaning we have to keep humans in the loop. We cannot just hand over the keys to the black box.
Exactly. AI is a tool, not a replacement for human cognition. Like any immensely powerful tool, it requires human domain knowledge, contextual common sense, and ethical judgment to function safely.
We need humans to audit the outputs.
We need humans to interpret the black box decisions and prevent fatal feedback loops.
A fatal feedback loop like model collapse. That really brings us full circle today. We set out to strip away the magic of AI. We broke down the reality of narrow AI and mapped out the four stages of machine learning.
From the film room to the real game.
Right. We looked inside the hardware war between the raw power of GPUs and the assembly line efficiency of TPUs. We unpacked the math of gradient descent, optimizing our social media feeds, and charted a path toward a neurosymbolic future.
And the overriding takeaway is that understanding the mechanics is the only defense you have. If you don't understand how the algorithm operates, you are simply data for it to optimize.
Which brings me to a final thought for you to take away today. We briefly mentioned model collapse. This is the mathematical degradation that happens when an AI starts training on synthetic data generated by other AI models rather than original human data.
It's like making a photocopy of a photocopy.
Exactly. Eventually the output loses all its sharp edges and turns into a blurry, unusable mess. As the internet becomes overwhelmingly flooded with AI-generated text, code, and images. Are we accelerating toward a future where these massive neural networks so you entirely lose touch with human reality, trapping themselves in an infinite automated echo chamber of their own making?
It definitely forces you to wonder who is really pulling the strings in the magic show.
The next time you unlock your phone or scroll your feed, ask yourself, are you looking at the rabbit or are you just staring at the trapdoor?