GPUs

Why AI needs GPUs

A neural network is, underneath, one operation repeated trillions of times: matrix multiplication. GPUs win because they do that one thing with thousands of cores at once — while a CPU, built for a few fast sequential tasks, does it a handful at a time.

01The answer, then the intuition

The same sum, a million times over

Strip a transformer down and almost all of its work is multiplying big grids of numbers — every attention step and every weight layer is a matrix multiply. And a matrix multiply is embarrassingly parallel: each output number is an independent dot product that doesn't depend on the others, so in principle you could compute all of them at the same time.

A CPU has a few very powerful cores tuned for fast, branchy, sequential work. A GPU has thousands of simpler cores built to run the exact same arithmetic on many numbers at once — plus huge memory bandwidth to feed them. For AI's one repeated operation, that's a perfect match. Race them on the same matrix multiply:

CPU vs GPU — the same matrix multiply

Illustrative. Each tile is a chunk of output; cores fill tiles in parallel. The animation shows the shape of the gap — the real number is in the readout.

CPU~8 cores · sequential

progress0%

GPUthousands of cores · parallel

progress0%

Same math. One finishes while the other is barely started.

02Mechanics

Built for throughput, not latency

Thousands of cores. A modern GPU has tens of thousands of arithmetic units running the same instruction across different data at once (SIMT). For one operation repeated over millions of numbers, that's ideal; for tangled, branchy logic, it's wasted.
Tensor cores. GPUs now include dedicated units that do a small matrix multiply-accumulate in a single step — hardware built specifically for the one operation neural networks need. This is most of where the headline FLOP/s comes from.
Memory bandwidth. All those cores are useless if you can't feed them. GPUs pair the compute with very fast high-bandwidth memory (HBM) — the subject of the next chapter — so weights and activations stream in fast enough to keep the cores busy.
The trade-off. A CPU minimizes the time for one task (latency); a GPU maximizes the total work done across many (throughput). AI is almost entirely the second kind of problem, which is why the GPU — originally built to shade millions of pixels in parallel — turned out to be the perfect AI chip by accident.

This is also why the field is a hardware story as much as a software one: progress is gated by how many parallel multiply-adds per second per dollar the silicon can deliver.

04The math

expand ▾

Why the gap is ~2,000×

Multiplying two $d \times d$ matrices produces $d^2$ outputs, each a dot product of length $d$ (a multiply and an add per term), so the cost is:

$$ \text{FLOPs} = 2\,d^3 $$

Wall-clock time is just that divided by the hardware's throughput. For $d = 8192$ that's about $1.1 \times 10^{12}$ FLOPs. A CPU delivering ~0.5 TFLOP/s of usable fp32 takes seconds; a GPU's tensor cores deliver on the order of ~1,000 TFLOP/s:

$$ t = \frac{2\,d^3}{\text{throughput}} \;\Rightarrow\; t_{\text{CPU}} \approx 2.2\ \text{s}, \quad t_{\text{GPU}} \approx 1.1\ \text{ms} $$

A roughly 2,000× difference on a single operation — and a model is billions of these. The gap isn't that the GPU's individual cores are faster (they're slower); it's that there are thousands of them doing independent work at once. Parallelism, not clock speed, is the whole game.

05The code

expand ▾

The race, in numbers

One large matrix multiply, timed against a CPU's and a GPU's throughput.

cpu_vs_gpu.py

def matmul_flops(d):
    return 2 * d**3            # d^2 outputs, each a length-d dot product

d = 8192
flops = matmul_flops(d)

cpu = 0.5e12     # CPU: ~0.5 TFLOP/s usable fp32
gpu = 990e12     # GPU tensor cores: ~990 TFLOP/s bf16 (H100-class)

print(f"{d}x{d} matmul: {flops:.2e} FLOPs")
print(f"CPU: {flops/cpu:.3f} s")
print(f"GPU: {flops/gpu*1000:.3f} ms")
print(f"speedup: {(flops/cpu)/(flops/gpu):.0f}x")
# 8192x8192 matmul: 1.10e+12 FLOPs
# CPU: 2.199 s
# GPU: 1.111 ms
# speedup: 1980x

06The economics

The build-out is a GPU build-out

Parallelism → money

Because AI's appetite is for parallel matrix-multiply throughput, the entire build-out resolves to one scarce thing: GPUs. The hundreds of billions in capital expenditure are, concretely, orders for accelerators and the power and buildings to run them. When people say a lab is "compute-constrained," they mean it cannot get enough of these chips.

That scarcity is why a single company, Nvidia, captures so much of the value in AI — it sells the one input everyone needs — and why its data-center revenue became a real-time gauge of the build-out itself. The chip is the bottleneck, the capital line, and the moat all at once.

For the Circuit, this is the supply side of the equation: the cost of intelligence is set by GPU throughput per dollar, and it falls only as fast as the silicon improves. Every efficiency trick in this section is ultimately about extracting more useful tokens from each very expensive chip.

07Going deeper

expand ▾

The primary sources

NVIDIA — GPU Performance Background · cores, throughput, and the roofline.
NVIDIA — Tensor Core architecture · the matrix-multiply-accumulate unit.
Dao et al. (2022) — FlashAttention · why memory movement, not FLOPs, often dominates.
SemiAnalysis · ongoing economic analysis of the AI hardware supply chain.

"First Principles — how AI actually works", chapter "Why AI needs GPUs", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Memory Wall

Memory, HBM & the memory wall

Chips can now do arithmetic far faster than they can fetch the numbers to work on. That widening gap is the memory wall — and it's why AI accelerators are defined by their high-bandwidth memory (HBM) as much as their compute.

01The answer, then the intuition

Compute outran memory — and never looked back

For decades, the arithmetic units on chips got faster far quicker than the memory feeding them. Compute has roughly tripled each hardware generation while memory bandwidth grew only about 1.6× — so with every generation, the processor spends more of its time waiting on data and less doing math. That is the memory wall, and modern AI slammed straight into it: decode is memory-bound precisely because it moves the whole model per token but does little arithmetic with it.

Drag through the generations and watch the two bars diverge — compute racing ahead, bandwidth trailing, and the gap you must somehow bridge growing wider each step:

The memory wall — compute vs bandwidth, per generation

Illustrative model of the documented trend (Gholami et al. 2024): compute ×3.0 / generation, bandwidth ×1.6 / generation. Bars are relative to generation 0.

×1

compute

×1

memory bandwidth

hardware generationgen 0 · +0 yr

2010s→ each step = ~2 years →now

The gap: ×1.0

At generation 0 they start even. Drag right to open the wall.

02Mechanics

What HBM is, and why it's never enough

The problem. A GPU's thousands of cores can only work as fast as memory delivers operands. When the arithmetic is cheap but the data movement is expensive, the cores sit idle — "memory-bound." Most LLM inference lives here.
HBM (high-bandwidth memory). The fix is to stack DRAM dies vertically, right next to the compute, connected by an extremely wide interface. That buys enormous bandwidth — an H100's HBM moves ~3.35 TB/s, versus tens of GB/s for a normal CPU's DRAM. It's the single feature that makes a GPU a viable AI chip.
The catches. HBM is expensive to manufacture, limited in capacity (tens of GB per chip), power-hungry, and made by only a handful of suppliers. So you're always short of it — short of capacity (the weights plus the KV cache must fit) and short of bandwidth (they must stream fast enough).
Why it keeps biting. Because compute keeps outpacing bandwidth, each new generation makes the imbalance worse, not better. Chip designers respond with bigger caches, lower precision, and clever data movement — but the wall recedes slowly at best.

So the mental model flips: for AI, a chip's memory system is often the real product, and the compute is the part that's easy to oversupply.

04The math

expand ▾

The roofline and the ridge point

The roofline model says achievable performance is capped by whichever runs out first — compute or bandwidth — given a workload's arithmetic intensity $I$ (FLOPs done per byte moved):

$$ \text{performance} = \min\!\big(\text{peak compute},\; I \times \text{bandwidth}\big) $$

The crossover — the ridge point — sits at:

$$ I^{*} = \frac{\text{peak compute}}{\text{bandwidth}} $$

A workload is memory-bound when $I < I^{*}$. As compute grows ×3 per generation and bandwidth only ×1.6, $I^{*}$ climbs by ×$(3/1.6) \approx 1.9$ each generation — the bar for staying compute-bound keeps rising. LLM decode at batch 1 has an intensity of roughly 1–2 FLOPs/byte, far below a modern GPU's ridge point of several hundred — which is exactly why it wastes most of the chip's compute, and why batching (raising $I$) is the escape.

05The code

expand ▾

The wall, generation by generation

Relative compute, bandwidth, and the widening gap, using the documented per-generation growth rates.

memory_wall.py

# Gholami et al. "AI and Memory Wall" (2024): per ~2-year generation,
# peak compute grows ~3x, memory bandwidth ~1.6x.
C, B = 3.0, 1.6

for k in range(6):
    comp, bw = C**k, B**k
    gap = comp / bw          # extra arithmetic-per-byte you must find to stay busy
    print(f"gen {k} (+{2*k}yr): compute x{comp:6.1f}  bandwidth x{bw:5.1f}  gap x{gap:5.2f}")
# gen 0 (+0yr): compute x   1.0  bandwidth x  1.0  gap x 1.00
# gen 3 (+6yr): compute x  27.0  bandwidth x  4.1  gap x 6.59
# gen 5 (+10yr): compute x 243.0  bandwidth x 10.5  gap x23.17  <- the wall

06The economics

You're buying memory, priced as compute

The wall → money

The memory wall rewrites what a GPU purchase really is. Because inference is memory-bound, buyers are often paying for capacity and bandwidth — and getting compute they can't fully use as a side effect. HBM is the scarce, expensive component inside the accelerator, made by just three suppliers (SK Hynix, Samsung, Micron), and its availability has become a genuine gate on how many AI chips can be built at all.

That reshapes the capex story. The bottleneck isn't only fabricating logic; it's stacking enough high-bandwidth memory around it. When a frontier chip is supply-constrained, HBM is frequently why — a detail that turns an obscure memory technology into a macro variable.

For the Circuit, the wall sets a hard floor under costs: no matter how cheap arithmetic gets, moving data stays expensive, so the price of serving a token can't fall faster than memory improves. Every efficiency lever in this section — fewer bits, batching, better caching — is ultimately a way to do more with each precious byte of bandwidth.

07Going deeper

expand ▾

The primary sources

Gholami et al. (2024) — AI and Memory Wall · the documented compute-vs-bandwidth divergence.
Williams, Waterman & Patterson (2009) — Roofline · the performance model.
Dao et al. (2022) — FlashAttention · beating the wall by minimizing memory traffic.
SemiAnalysis · HBM supply and its role in accelerator availability.

"First Principles — how AI actually works", chapter "Memory, HBM & the memory wall", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Quantization

Quantization & distillation

A frontier model is enormous. Two techniques shrink it to run cheaply: quantization stores each weight in fewer bits, and distillation trains a small "student" to copy a big "teacher" — keeping most of the quality at a fraction of the size.

01The answer, then the intuition

Same model, fewer bits — and a smaller copycat

From the parameters chapter you know the rule: memory = parameter count × bytes per parameter. The second factor is the lever. By default each weight is a 16-bit number, but most of that precision is wasted — the model works almost as well if each weight is squeezed into 8 bits, or even 4. That's quantization, and it cuts memory and cost in direct proportion.

Distillation attacks the first factor. Instead of compressing one model, you train a much smaller one to imitate the big model's outputs — learning from the teacher's full probability distribution, not just the right answer. The student ends up far smaller while inheriting much of the teacher's skill.

Quantization is the bigger everyday lever. Pick a model and drag the precision down — watch the hardware bill collapse:

Quantization calculator — drag the precision

Memory = parameters × (bits / 8). GPUs assume 80 GB each with ~20% runtime overhead.

—

weights in memory

—

80GB GPUs needed

—

vs 16-bit

precision16-bit

4-bit8-bit16-bit32-bit

02Mechanics

Compressing the weights, and the knowledge

Quantization. Map each high-precision weight to a low-bit integer by storing a shared scale per group of weights: w ≈ scale × round(w / scale). 16→8 bits roughly halves memory for almost no quality loss; 4-bit (with careful methods like GPTQ or AWQ) quarters it with a small, often acceptable hit. The weights are the same model — just stored coarsely.
Why it works. Neural networks are remarkably robust to noise in their weights. The signal lives in the overall pattern, not the 12th decimal place, so throwing away low-order bits costs surprisingly little.
Distillation. Run the big teacher on lots of inputs and record its full output distribution (the "soft labels"). Train a small student to match those distributions. The soft targets carry far more information than a single correct token — the teacher reveals how confident it is across all options — so the student learns faster and smaller.
Used together. Modern small models are often distilled from a big teacher and then quantized — the two compressions compound, which is how genuinely capable models end up running on a laptop or phone.

Both are lossy. The craft is spending the loss where it doesn't matter — and measuring honestly whether the smaller model still does your task, rather than trusting a benchmark.

04The math

expand ▾

Bits, scales, and soft targets

Memory for the weights is linear in the bit-width $b$:

$$ \text{memory} = N \times \frac{b}{8}\ \text{bytes} $$

Quantization to $b$ bits picks a scale $s$ for a group of weights and rounds each to the nearest representable level, then dequantizes on use:

$$ w_q = \mathrm{round}\!\left(\frac{w}{s}\right), \qquad \hat{w} = s\,w_q \approx w $$

Distillation trains the student distribution $p_S$ to match the teacher's softened distribution $p_T$ (temperature $T$ flattens both), minimizing the KL divergence:

$$ \mathcal{L}_{\text{distill}} = \mathrm{KL}\!\big(p_T \,\|\, p_S\big) = \sum_i p_T(i)\,\log\frac{p_T(i)}{p_S(i)} $$

The teacher's soft probabilities — "70% cat, 25% dog, 5% fox" — teach the student the structure a hard label ("cat") never could. That extra signal is why a small student can punch so far above its size.

05The code

expand ▾

The bill, bit by bit

The same 70B model at four precisions — memory and GPUs. This is the calculator above, in seven lines.

quantize.py

import math

def mem_gb(N, bits): return N * bits / 8 / 1e9
def gpus(m, cap=80, overhead=1.2): return math.ceil(m * overhead / cap)

N = 70e9                                  # a 70-billion-parameter model
for bits in [32, 16, 8, 4]:
    m = mem_gb(N, bits)
    print(f"{bits:>2}-bit: {m:6.1f} GB  ->  {gpus(m)} GPU(s)")
# 32-bit:  280.0 GB  ->  5 GPU(s)
# 16-bit:  140.0 GB  ->  3 GPU(s)
#  8-bit:   70.0 GB  ->  2 GPU(s)
#  4-bit:   35.0 GB  ->  1 GPU(s)   <- a 70B model on a single GPU

06The economics

The deflation underneath everything

Compression → money

Quantization and distillation are the deflationary force in AI. Every bit you drop is a proportional cut in memory, hardware, and energy per token served — and distillation moves a capability from an expensive frontier model into one a fraction of the size. Together they are why the price of a given level of intelligence keeps falling fast even as the frontier rises.

This is the optimistic half of the Circuit's ledger. The build-out spends ever more on the frontier, but compression relentlessly drives down the cost of serving everything below it — pushing models onto single GPUs, laptops, and phones, where the marginal cost approaches electricity. The open tier rides this hardest.

It also sharpens the central tension. If a distilled, quantized open model captures most of the value at a tenth of the cost, what exactly is the frontier premium being paid for — and for how long? That question, asked with real numbers, is the whole point of this desk.

Part II complete

You now know what a model is — and what it costs

From a base model learning language by guessing the next word, through alignment, the real differences between models, vision, and the compression that makes any of it affordable — you've followed the model from raw weights to a product with a price tag. Part I built the machine; Part II made it a model.

Part III opens the machinery that runs it — inference, GPUs, the memory wall, the KV cache, and the data-center cluster: what it actually takes to serve a model to millions at once, and where the real costs live. See the full curriculum →

07Going deeper

expand ▾

The primary sources

Hinton et al. (2015) — Distilling the Knowledge in a Neural Network · the soft-target idea.
Dettmers et al. (2022) — LLM.int8() · 8-bit inference for large models with no quality loss.
Frantar et al. (2022) — GPTQ · accurate 4-bit post-training quantization.
Dettmers et al. (2023) — QLoRA · fine-tuning a 4-bit model on a single GPU.

"First Principles — how AI actually works", chapter "Quantization & distillation", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Supply Chain

The compute supply chain

Behind every GPU is a chain that runs from a Dutch lithography machine to a Taiwanese fab to a power substation — and nearly every link is held by one to three companies. It's the most concentrated supply chain in modern industry, and the tightest link sets the pace of the whole build-out.

01The answer, then the intuition

A chain of near-monopolies

A finished AI accelerator passes through a startling gauntlet, and at almost every stage there is effectively one supplier the world depends on. The extreme-ultraviolet lithography machines that print the smallest features? One company, ASML. The fabrication of leading-edge chips? Overwhelmingly one, TSMC. The AI GPUs themselves? Overwhelmingly one, Nvidia. The high-bandwidth memory? Three makers. The advanced packaging that stitches it together? Capacity-constrained at a handful.

Because it's a chain, its output is set by the tightest link — not the average. Click through the stages; the highlighted one is the current bottleneck that caps how many chips the whole industry can actually ship:

Sand to a running model — the chokepoint chain

Illustrative relative capacity per stage; the red stage is the binding constraint. Click any node.

02Mechanics

The links, in order

EUV lithography → ASML. The machines that pattern the finest chip features use extreme-ultraviolet light. ASML is the sole maker on Earth; without it, the leading edge stops. A single point of failure at the very top of the stack.
Leading-edge fabrication → TSMC. Turning designs into wafers at the smallest nodes is dominated by TSMC, concentrated in Taiwan — the geopolitical fault line the whole industry watches.
GPU design → Nvidia. Nvidia designs the accelerators and, crucially, the CUDA software ecosystem that locks developers in. It captures enormous margin because it sells the one part everyone needs.
HBM & packaging → a few makers. High-bandwidth memory (SK Hynix, Samsung, Micron) and advanced packaging (CoWoS) are the frequent real bottlenecks — you can design the chip, but not get enough memory stacked onto it.
Assembly & power → hyperscalers and the grid. The chips become clusters inside data centers, and the final, growing constraint is electrical: gigawatts of power, gated by turbines, transmission, and permits.

Each link is both a pricing-power point — a near-monopoly that captures margin — and a fragility point, where one disruption ripples through everything downstream. Concentration is efficient and lucrative right up until it isn't.

04The math

expand ▾

The law of the minimum

A serial supply chain obeys Liebig's law of the minimum: total throughput is the capacity of its smallest stage, not the sum or the average:

$$ \text{output} = \min_{i}\; \text{capacity}_i $$

Adding capacity anywhere except the binding stage does nothing — a fact that makes bottleneck identification the entire game. And because each stage is a near-monopoly, its reliability $r_i$ is a single point of failure; the chain's reliability is the product:

$$ R_{\text{chain}} = \prod_{i} r_i $$

With many stages each held by one supplier, even high individual reliability multiplies down — six stages at 98% each give $0.98^{6} \approx 0.89$. Concentration raises margins at every link and lowers the reliability of the whole. The build-out can only proceed as fast as its scarcest input allows, which is why a shortage in one unglamorous component — memory, packaging, a transformer for the substation — can throttle the entire industry.

05The code

expand ▾

Finding the binding link

The chain's real output is its minimum stage — and which stage that is, is where all the leverage lives.

bottleneck.py

stages = {   # illustrative annual capacity, GPU-equivalent units
    "EUV litho (ASML)":        6.0, "Leading-edge fab (TSMC)": 5.0,
    "GPU design (Nvidia)":     5.5, "HBM (3 makers)":          3.5,
    "CoWoS packaging":         3.8, "Data-center power":       4.0,
}
cap = min(stages.values())
binding = next(n for n, v in stages.items() if v == cap)
print(f"chain output capped at {cap} by: {binding}")
# chain output capped at 3.5 by: HBM (3 makers)
# -> adding fab or litho capacity changes nothing; only more HBM raises output

06The economics

Where the margin — and the fragility — lives

The chain → money

This is the physical supply side of the Circuit, and it explains where the money pools. Each chokepoint is a near-monopoly, so each captures outsized margin — Nvidia's, TSMC's, and ASML's economics are what a single-supplier link looks like on an income statement. When a stage is the binding constraint, its owner has extraordinary pricing power; the shortages are the profits.

They're also the fragility. A serial chain of near-monopolies means one disrupted link — an HBM shortage, a packaging constraint, a grid it can't power, or a shock to Taiwan — throttles everything downstream. The build-out's pace isn't set by ambition or capital; it's set by the tightest link, whatever it happens to be this quarter.

For a research desk, tracking the chain is how you separate a genuine acceleration from a temporary squeeze. When one link loosens, output jumps; when another tightens, it stalls — regardless of demand. Reading the bottleneck is reading the real speed limit of AI, and it's exactly the kind of ground-truth signal the Circuit is built to watch.

07Going deeper

expand ▾

The primary sources

SemiAnalysis · deep coverage of HBM, CoWoS packaging, and accelerator supply.
ASML — EUV lithography · the sole supplier of leading-edge lithography.
IEA — Electricity & data-center demand · power as the emerging constraint.
CSIS — Mapping the Semiconductor Supply Chain · concentration and geographic risk.

"First Principles — how AI actually works", chapter "The compute supply chain", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.