How a GPU Works: From Silicon to Superintelligence

What if I told you that the most important machine in the world right now isn't a rocket, a reactor, or a robot — it's a chip the size of your palm? The Graphics Processing Unit — the GPU — has become the engine of the artificial intelligence revolution. Today, we're going to take you on a journey from the bottom up: starting with the fundamental physics of how a GPU works, through the extraordinary process of designing and manufacturing one, all the way to the architectural innovations that power today's large language models. Let's dive in. To truly understand a GPU, we have to start at the very bottom. At its core, a GPU is built from billions of microscopic switches called transistors. By wiring these transistors together, engineers create logic gates — simple circuits that can perform basic operations like AND, OR, and NOT. These gates are the atoms of compute. When you ask an AI a question, the answer is ultimately calculated by electrical currents flowing through billions of these gates, opening and closing billions of times per second. If we zoom out, we see these logic gates combining to form functional blocks. The two most important blocks in a GPU are CUDA Cores and Tensor Cores. A CUDA core is an Arithmetic Logic Unit — a general worker that performs one calculation at a time. But AI fundamentally relies on matrix multiplication — multiplying huge grids of numbers together. This is where Tensor Cores come in. They are specialised blocks of logic gates designed to multiply entire matrices in a single clock cycle. This specific arrangement of gates is what allows GPUs to train massive language models in weeks rather than centuries. Zooming out further, these cores aren't just scattered randomly. They are grouped into organized neighbourhoods called Streaming Multiprocessors, or SMs. Each SM contains hundreds of CUDA cores, several Tensor Cores, and its own local memory called L1 Cache. This local memory is crucial. For the Tensor Cores to multiply matrices at lightning speed during AI training or inference, the data must be physically close to the compute gates. The SM is a self-contained engine of parallel processing. Now we see the full silicon die. A modern AI GPU, like NVIDIA's H100, packs 132 of these SMs onto a single chip. All these SM neighbourhoods are connected to a central, massive L2 Cache, and then out to towering stacks of High-Bandwidth Memory, or HBM. This is the core difference between a CPU and a GPU. A CPU has a few powerful cores for sequential tasks. A GPU is a superhighway of parallel processing, designed to distribute massive workloads — like the billions of calculations required for a neural network — across thousands of cores simultaneously. Now that we understand the hardware from gates to the full chip, let's look at the software running on it: the Transformer architecture. At its heart is the "Attention Mechanism." Every word in a prompt generates three matrices: a Query, a Key, and a Value. The model must multiply every word's Query against every other word's Key to understand the context. This requires massive, dense matrix multiplication — exactly the workload that the Tensor Cores we saw earlier were physically wired to execute. But there's a problem. A model like Llama 3 70B contains 70 billion parameters — the "weights" or knowledge it learned during training. This requires over 140 gigabytes of memory, which won't fit on a standard 80-gigabyte GPU. So, we zoom out beyond a single chip. Using a technique called Tensor Parallelism, the massive weight matrices are split up and stored across the High-Bandwidth Memory of multiple GPUs. During computation, the GPUs calculate their piece of the puzzle and use high-speed NVLink cables to instantly share the results. When you ask an LLM a question, it generates the answer one word at a time. To do this, it needs to look back at the entire conversation. Recalculating the Keys and Values for all previous words through the Tensor Cores every single time would be incredibly slow. The solution is the KV Cache. As the GPU processes each word, it saves its Key and Value data directly into the High-Bandwidth Memory. When generating the next word, it simply looks up the saved data. But this cache grows linearly with every word, quickly becoming the biggest memory bottleneck in AI. Because of this architecture, LLM inference happens in two distinct phases. First is the Prefill phase. When you send a prompt, the GPU processes all your words simultaneously. This maxes out the Tensor Cores — it is compute-bound. But once the prompt is processed, the model enters the Decode phase, generating the answer one word at a time. Now, the math is simple, but the GPU must load the entire model's weights from HBM for every single word. The Decode phase isn't limited by the logic gates; it's limited by how fast the memory can deliver data. It is memory-bandwidth-bound. Let's trace the journey of a single token through the hardware we just built. The GPU fetches the first layer of model weights from HBM, pulling them through the L2 cache and into the Streaming Multiprocessors. The Tensor Cores perform the matrix multiplications. The result is combined with the KV Cache data. This process repeats through dozens of neural network layers. Finally, the GPU outputs a probability distribution, selects the most likely next word, and the entire cycle begins again. Inference is just using the model. Training the model is far more complex. It requires a continuous loop. First, the Forward Pass: the Tensor Cores make a prediction. Next, Loss Calculation: it finds the error. Then comes the Backward Pass: the GPU calculates the gradients — the exact mathematical adjustments needed to fix the error. Finally, the Optimizer Step updates the model weights in the HBM. Because this happens across thousands of GPUs simultaneously, they must constantly pause to synchronize their gradients over the NVLink network before taking the next step. This bottom-up understanding explains the evolution of the hardware itself. Compare the older A100 to the new H200. The A100 was a powerhouse, with 80 gigabytes of memory delivering 2 terabytes per second of bandwidth. But remember, the decode phase is memory-bandwidth-bound. The H200 uses the exact same logic gates and compute cores as the H100, but upgrades the memory to 141 gigabytes of HBM3e, delivering a massive 4.8 terabytes per second. That 2.4x increase in bandwidth directly translates to faster token generation. And the extra capacity means a much larger KV Cache, allowing the model to remember vastly longer conversations. So how do you actually design one of these silicon supercomputers? It's a process called the IC design flow — Integrated Circuit design — and it takes thousands of engineers working for three to five years. The journey moves from abstract logic all the way down to the physical arrangement of billions of transistors. Let's walk through each stage. It all starts with code — but not the kind that runs on a computer. Engineers write in hardware description languages like Verilog or VHDL at what's called the Register Transfer Level, or RTL. They're describing how data should flow between storage elements on every tick of the chip's internal clock. For a GPU with 80 billion transistors, this means writing and verifying millions of lines of RTL code that define every Streaming Multiprocessor, every Tensor Core, every memory controller, and every interconnect. Once the RTL is verified, it goes through Logic Synthesis. Specialised EDA software — from companies like Synopsys and Cadence — translates the abstract code into a gate-level netlist. This is a massive blueprint of specific physical logic gates chosen from a standard cell library provided by the foundry. Think of it as translating a recipe into the exact molecular ingredients needed. The synthesis tool also optimises for speed, power consumption, and physical area — the three eternal trade-offs of chip design. Next comes the physical design phase — Place and Route. Imagine organising a city of 80 billion buildings, where every building must be connected by roads, and the total travel time on any road cannot exceed a few picoseconds. Algorithms determine the exact position of every logic gate on the silicon die, then route microscopic copper wiring to connect them. This routing happens across more than thirteen vertical layers of metal, creating a dense three-dimensional maze of interconnections. Before manufacturing, the design must pass rigorous verification. Design Rule Checks ensure no wires violate the foundry's manufacturing constraints — minimum spacing, width, and via sizes. Layout Versus Schematic checking confirms the physical layout exactly matches the intended circuit — catching shorts, opens, and missing connections. And Static Timing Analysis verifies that electrical signals can propagate across the chip fast enough to meet the target clock frequency. A single undetected error among billions of connections means a dead chip worth millions of dollars. When every verification check passes, the team reaches the most significant milestone in chip design: Tape-Out. The final physical layout is exported as a GDSII file — often terabytes in size — and transmitted to the semiconductor foundry. This is the point of no return. Any error discovered after tape-out means months of delay and tens of millions in re-spin costs. It's why verification consumes more engineering effort than the design itself. Manufacturing takes place in the most advanced factories on Earth — semiconductor foundries like TSMC in Taiwan. It begins with a 300-millimetre disc of ultra-pure silicon crystal, polished to atomic smoothness. Over the next three to four months, the foundry will print dozens of identical GPU dies onto this single wafer through hundreds of precisely orchestrated process steps. The magic happens through Extreme Ultraviolet lithography — EUV. These machines, built exclusively by the Dutch company ASML and costing over 300 million dollars each, use light with a wavelength of just 13.5 nanometres. That's light so energetic it's absorbed by air and glass, so it must be directed by ultra-smooth mirrors inside a vacuum. This invisible light passes through a stencil called a reticle, projecting the chip's circuit pattern onto the wafer with features smaller than a virus. Lithography is just one step in a cycle that repeats hundreds of times. First, thin films of material are deposited onto the wafer. Then, lithography patterns the next layer. Plasma etching carves away the unwanted material. Ion implantation dopes the silicon with specific electrical properties. Layer by layer, atom by atom, the foundry sculpts billions of three-dimensional transistor structures and connects them with over thirteen layers of copper wiring. The entire process takes months to complete. Here's the brutal economics of chip manufacturing: yield. Even in the world's cleanest rooms, microscopic defects are inevitable — a single particle of dust, an atomic irregularity. The larger the chip, the higher the probability that a defect lands on it. The H100's die is a massive 814 square millimetres — nearly the maximum size the lithography machine can print. To manage this, NVIDIA builds in redundancy: the full die has 144 Streaming Multiprocessors, but only 132 are enabled. If a defect hits one SM, it's simply disabled, and the chip still ships. Once the GPU die is cut from the wafer, it needs to be connected to its memory. This is where advanced packaging comes in — specifically TSMC's CoWoS technology: Chip-on-Wafer-on-Substrate. Instead of connecting chips through a traditional circuit board, the GPU die and its High-Bandwidth Memory stacks are placed side-by-side on a thin silicon interposer. This interposer contains thousands of microscopic wires, enabling communication at terabytes per second — bandwidth that would be physically impossible with conventional packaging. Those memory towers we keep mentioning are High-Bandwidth Memory, or HBM. As we've seen, LLMs require hundreds of gigabytes of weights to be loaded for every single word they generate. Standard memory chips are too slow. HBM solves this by stacking memory dies vertically, like a skyscraper, and punching microscopic copper wires — called Through-Silicon Vias — straight down through the stack. This delivers the massive data throughput that LLMs demand during the decode phase. To make LLMs run faster, GPU architects realized that AI doesn't always need perfect mathematical precision. Calculating with 8-bit or even 4-bit numbers is exponentially faster and uses less memory than traditional 32-bit numbers. Modern GPUs feature dedicated Transformer Engines. These hardware systems analyze the neural network layer by layer, dynamically dropping the precision down to 8-bit or 4-bit when speed is needed, and scaling it back up only when high accuracy is required. Another architectural trick is Structured Sparsity. During AI training, many connections in a neural network drop to zero — meaning they have no impact on the final answer. Multiplying by zero is a waste of time and energy. Modern GPUs have specialized hardware that detects specific patterns of zeros, instantly compresses the data, and skips the useless calculations entirely. This effectively doubles the processing speed for sparse AI models without losing accuracy. As we saw with Tensor Parallelism, the biggest AI models must be split across dozens, or even thousands, of chips. But standard network connections are too slow to keep the GPUs synchronized. The solution is dedicated interconnects, like NVIDIA's NVLink. This technology creates a massive, high-speed web directly between the GPUs, allowing them to share data at terabytes per second. It turns a rack of individual chips into one giant, unified supercomputer. Let's put it all together. When a large language model generates a response, here's what happens at the hardware level: HBM delivers the model's weight matrices at terabytes per second. The Tensor Cores multiply those matrices against the input data at petaflop speeds, with the Transformer Engine dynamically managing precision. Sparsity hardware skips unnecessary calculations. And NVLink synchronises the results across all GPUs in the system. This entire pipeline executes in milliseconds — thousands of times per second — to produce the fluent text you see from models like GPT-4. How did one company capture over 90 percent of the AI data center market? It didn't happen overnight. It started in 2006 with a massive gamble. While competitors focused entirely on making graphics chips for video games, NVIDIA CEO Jensen Huang launched CUDA — a software platform that allowed developers to use GPUs for general-purpose computing. Wall Street hated it. They questioned why NVIDIA was spending hundreds of millions of dollars on R&D with no immediate return. But Jensen persisted. When the deep learning boom finally arrived years later, NVIDIA was the only company ready for it. This brings us to the real reason competitors like AMD and Intel have struggled to catch up. They make excellent hardware — sometimes even matching NVIDIA's raw specifications. But NVIDIA's true moat isn't silicon; it's software. Over nearly two decades, NVIDIA built an ecosystem of over 5 million developers and dozens of deeply optimized libraries — cuDNN for deep learning, NCCL for multi-GPU communication, TensorRT for inference. Today, every major AI framework is deeply intertwined with CUDA. For an AI company to switch to a competitor's chip, they would have to rewrite years of optimized code. It's a switching cost most simply cannot afford. NVIDIA also realized that selling individual chips wasn't enough. To train massive LLMs, you need thousands of GPUs working in perfect harmony. So, NVIDIA vertically integrated. They built their own supercomputers, called DGX. They acquired Mellanox for nearly 7 billion dollars to own the InfiniBand networking layer. They developed NVLink to connect the chips. Today, NVIDIA doesn't just sell a component; they sell the entire AI factory — silicon, systems, networking, and software as one integrated product. This full-stack approach guarantees maximum performance and leaves competitors trying to piece together disparate parts from different vendors. Everything we've discussed was engineered to accelerate one architecture: the Transformer. But the Transformer has a fundamental limitation — its attention mechanism scales quadratically with sequence length. Double the context window, and you quadruple the compute. Researchers are now developing entirely new paradigms: Mixture of Experts models that activate only a fraction of their parameters, State Space Models like Mamba that scale linearly, and Agentic AI systems where multiple models collaborate autonomously. Each of these demands different things from the hardware. To maintain this dominance, NVIDIA has done something unprecedented in semiconductor manufacturing: they've accelerated their roadmap to a one-year release cadence. While traditional chipmakers operate on two-to-three-year