Cluster

The data-center cluster

A frontier model is too big for any single GPU. It runs across thousands of them, wired together into racks, pods, and buildings — drawing tens of megawatts. At this scale the network, not the chip, becomes the bottleneck, and power becomes the real limit.

01The answer, then the intuition

One model, ten thousand chips, one machine

Everything so far assumed the model lived on a GPU or two. Frontier models break that assumption: their weights and caches overflow any single chip, and training them in a human lifetime needs thousands working in concert. So the real unit of AI isn't a GPU — it's a cluster: tens of thousands of GPUs stitched into one enormous parallel computer.

But you can't just add chips and get proportional speed. They have to talk — sharing weights, gradients, and activations over the network — and that communication is a tax that grows with scale. Drag a cluster from one GPU to hyperscale and watch three things move: compute soars, efficiency sags to the communication tax, and power climbs into the megawatts:

Scale a cluster — compute, efficiency, and power

~1 PFLOP/s and ~1.4 kW all-in per GPU; MFU (realized ÷ peak) declines with scale as communication grows. Illustrative, real-world magnitudes.

—

peak compute

—

efficiency (MFU)

—

realized compute

—

power draw

cluster scale1 GPU

1 GPU32,768 GPUs

02Mechanics

How thousands of GPUs become one computer

Splitting the model. When a model won't fit on one GPU, you split it. Tensor parallelism shards each layer's matrices across GPUs in a node; pipeline parallelism puts different layers on different nodes; data parallelism runs full replicas on different batches. Real clusters combine all three.
The interconnect. Split work means constant communication. Inside a node, GPUs talk over NVLink at terabytes per second; across nodes, over InfiniBand or specialized Ethernet — fast, but far slower than a GPU's own memory. Training's all-reduce (summing gradients across every GPU each step) makes the network a first-class bottleneck.
The physical hierarchy. 8 GPUs to a server, several servers to a rack, racks into pods, pods into a building. A modern AI data center is tens of thousands of GPUs, kilometers of fiber, and a power substation — a single machine the size of a warehouse.
The scaling tax. Because of communication, doubling the GPUs doesn't double the useful work. Model FLOPs utilization (MFU) — realized work over peak — drops as you scale, often from ~50–60% at modest size toward ~30–40% at the frontier. You pay for the chips; the network eats part of what you get.

So the frontier is an exercise in distributed systems as much as machine learning — and increasingly in electrical engineering, because the thing you eventually run out of is power.

04The math

expand ▾

Compute, the tax, and the wattage

Peak compute is just the count times per-GPU throughput; realized compute applies the efficiency $\eta$ (MFU):

$$ \text{realized} = G \cdot f_{\text{GPU}} \cdot \eta(G), \qquad \eta(G) \;\text{falls as}\; G \;\text{grows} $$

The efficiency drop is an Amdahl-style limit: if a fraction of each step is communication that grows with scale, the useful fraction shrinks. And power scales cleanly with the count, marked up by the facility's overhead (PUE):

$$ P = G \cdot P_{\text{GPU}} \cdot \text{PUE} $$

At $G = 32{,}768$, $f_{\text{GPU}} \approx 1$ PFLOP/s and $P_{\text{GPU}} \approx 1.4$ kW all-in, that's ~33 EFLOP/s of peak but only ~10 realized at ~32% MFU — drawing about 46 MW, the power of a small town. The compute grows linearly with money; the useful compute grows more slowly, and the power grows right along with the bill.

05The code

expand ▾

A cluster, by the numbers

Peak vs realized compute and power across five scales — watch MFU erode as the cluster grows.

cluster.py

import math
PER_GPU = 1e15          # ~1 PFLOP/s per GPU
W = 1400                # ~watts per GPU, all-in (chip + share of node & cooling)

def mfu(G):             # efficiency falls with scale (communication tax)
    return 0.62 - 0.02 * math.log2(G)

for G in [1, 8, 512, 8192, 32768]:
    peak = G * PER_GPU / 1e18                 # EFLOP/s
    realized = peak * mfu(G)
    power = G * W / 1e6                        # MW
    print(f"{G:6d} GPUs: peak {peak:5.2f} EF  MFU {mfu(G)*100:4.1f}%  "
          f"realized {realized:5.2f} EF  power {power:5.1f} MW")
# 32768 GPUs: peak 32.77 EF  MFU 32.0%  realized 10.49 EF  power 45.9 MW
#   -> compute grows linearly with spend; useful compute lags; power tracks the bill

06The economics

The build-out, made physical

Clusters → money

This is what the capital expenditure actually buys. When a hyperscaler reports tens of billions in spend, it resolves to this: buildings full of GPUs, the networking to bind them, and the substations to power them. The abstract "$500B build-out" is, concretely, a fleet of these clusters — and the reason the numbers are so large is that a frontier machine is a small industrial facility.

Two constraints now bind harder than chips. First, the scaling tax: because efficiency falls with size, the next doubling of a cluster costs 2× but delivers less than 2× the useful compute — diminishing returns the spending has to outrun. Second, power: at tens of megawatts per cluster and gigawatts in aggregate, AI has become an energy story, gated by grid capacity, turbines, and permits as much as by silicon.

For the Circuit, the cluster is the whole cost side made physical — GPUs, network, and power compounding into the denominator that revenue must eventually clear. It's the concrete thing the divergence between spending and payoff is measuring. Everything in this section — quantization, batching, beating the memory wall — exists to wring more value out of these very expensive, very hungry machines.

Part III complete

You now know what it takes to run a model

From a single forward pass, through the tokens-per-second speed limit, the GPU, the memory wall, the KV cache and batching, all the way to a warehouse of thirty thousand chips drawing the power of a town — you've followed inference from one operation to the physical build-out it demands. Part I built the machine; Part II made it a model; Part III is the cost of running it.

Part IV turns from mechanics to use: prompting, retrieval (RAG), agents and tools, evals, and vector search — how you actually build something with all this. See the full curriculum →

07Going deeper

expand ▾

The primary sources

Shoeybi et al. (2019) — Megatron-LM · tensor and pipeline model parallelism.
Narayanan et al. (2021) — Efficient Large-Scale Training on GPU Clusters · MFU and the scaling tax.
Llama 3 Herd of Models (2024) · a real 16K-GPU training cluster, described.
SemiAnalysis · data-center power, networking, and cluster economics.

"First Principles — how AI actually works", chapter "The data-center cluster", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Inference

What is inference?

Training builds the model once. Inference is running it — the forward passes that turn your prompt into an answer, every single time you hit send. It happens in two very different phases, and it is the cost you pay forever.

01The answer, then the intuition

Read the prompt fast; write the answer slow

Inference splits into two phases that feel nothing alike. Prefill reads your entire prompt in a single parallel pass — every token processed at once, the GPU running flat out. Then decode begins, and the model writes its answer one token at a time: each new token needs its own forward pass, and each pass depends on the one before, so they can't be parallelized.

That asymmetry is the whole story of inference cost. Reading is cheap and parallel; writing is sequential and slow. Click Run inference and watch it happen — the prompt lights up all at once, then the answer crawls out token by token:

Run inference — prefill, then decode

One parallel prefill pass over the prompt, then one forward pass per generated token.

Prompt & generated answer

prefill passes

decode passes

total forward passes

02Mechanics

Two phases, two bottlenecks

Prefill (compute-bound). The whole prompt is pushed through the model in one pass. Because all prompt tokens are available at once, the math is one big parallel matrix multiply — the GPU's compute is the limit, and it's used efficiently. This phase sets the "time to first token."
Decode (memory-bound). Now the model generates. Each token requires loading the model's weights from memory to do one forward pass, producing exactly one token, which is appended and fed back in. The arithmetic per step is tiny relative to the weights moved, so the bottleneck flips from compute to memory bandwidth — the GPU mostly waits on memory. This is why generation streams out at a steady, limited pace.
The KV cache. To avoid recomputing the whole prompt at every decode step, the model caches the per-token intermediate values (keys and values) — the KV cache. It's what makes decode merely slow instead of catastrophically slow, and it's a major consumer of GPU memory (Chapter 18).
No learning happens. Inference only runs the forward pass — no gradients, no weight updates. The model is frozen; it's a pure function from tokens to tokens.

So the model you spent $100M training does the same forward-pass arithmetic on every request for the rest of its life. Making that arithmetic fast and cheap is what all of Part III is about.

04The math

expand ▾

Two FLOPs per parameter, per token

One forward pass through a model with $N$ parameters costs about $2N$ floating-point operations per token (a multiply and an add per weight). For a request with a prompt of $P$ tokens and a generated answer of $G$ tokens:

$$ \text{prefill} \approx 2N\,P \quad(\text{1 parallel pass}), \qquad \text{decode} \approx 2N\,G \quad(G \text{ sequential passes}) $$

$$ \text{total} \approx 2N\,(P + G) $$

The FLOP counts can be equal, but the wall-clock isn't: prefill's $P$ tokens run together, while decode's $G$ tokens run strictly one after another. The decode phase is also memory-bound — its limiter isn't FLOPs but the bytes of weights streamed per token, roughly $2N \cdot (\text{bytes per parameter})$ moved from memory each step. That ratio, arithmetic intensity, is why decode leaves a GPU's compute mostly idle — and why fewer bits and batching matter so much.

05The code

expand ▾

The cost of one request

Prefill vs decode FLOPs for a 70B model answering with a 500-token prompt and 500-token reply.

inference.py

def inference_flops(N, prompt_len, gen_len):
    per_token = 2 * N                  # ~2 FLOPs per parameter, per token
    prefill = per_token * prompt_len   # all prompt tokens, ONE parallel pass
    decode  = per_token * gen_len      # gen tokens, that many SEQUENTIAL passes
    return prefill, decode

N = 70e9
pf, dc = inference_flops(N, 500, 500)
print(f"prefill: {pf:.2e} FLOPs  (1 parallel pass, 500 prompt tokens)")
print(f"decode:  {dc:.2e} FLOPs  (500 sequential passes)")
print(f"total:   {pf+dc:.2e} FLOPs per request")
# prefill: 7.00e+13 FLOPs  (1 parallel pass, 500 prompt tokens)
# decode:  7.00e+13 FLOPs  (500 sequential passes)
# total:   1.40e+14 FLOPs per request   <- and decode's are one-at-a-time

06The economics

The cost that never stops

Inference → money

Training is a one-time capital event. Inference is the bill that arrives forever — every request, from every user, runs the full forward pass again. Across a model's life, serving it almost always costs far more in total than training it did, which is why the data centers being built are sized for inference demand, not just training runs.

And the expensive half is decode. Because it's sequential and memory-bound, a single user's generation barely uses a GPU's compute — so providers pack many users' requests together (batching, Chapter 18) to fill the silicon. The entire discipline of inference engineering exists to claw back the efficiency that the one-token-at-a-time nature of decode throws away.

This is the meter the Circuit watches most closely: every token decoded is a real, recurring cost, and the question is whether the revenue per token clears it. Training built the asset; inference is where the money is actually spent — and, one hopes, made.

07Going deeper

expand ▾

The primary sources

Pope et al. (2022) — Efficiently Scaling Transformer Inference · prefill vs decode, the cost model.
Kwon et al. (2023) — PagedAttention / vLLM · the modern serving engine.
Kaplan et al. (2020) — Scaling Laws · the $2N$ FLOPs-per-token accounting.
How to Scale Your Model (Google DeepMind) · a deep, free reference on inference arithmetic.

"First Principles — how AI actually works", chapter "What is inference?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Latency

Latency, throughput, tokens/sec

How fast is an AI model? The honest answer is a formula. Because decode is memory-bound, a model's speed is set by how fast it can stream its weights out of memory — tokens per second ≈ bandwidth ÷ model bytes.

01The answer, then the intuition

Speed is bandwidth divided by size

You felt this in the last chapter: decode writes one token per forward pass, and each pass has to read the entire model out of memory. So the speed limit isn't how fast the GPU can compute — it's how fast it can move the weights. A bigger model means more bytes to stream per token, which means fewer tokens per second. It's almost that simple.

Two numbers matter to a user: time to first token (how long until the answer starts — set by prefill) and tokens per second (how fast it then types — set by decode). The calculator below computes the second from first principles. Pick a model size and drag the precision; watch a 70B model crawl, and watch quantization nearly quadruple it:

Tokens/sec calculator — the memory-bound speed limit

Single stream on one GPU with ~3.35 TB/s memory bandwidth (H100-class). tok/s ≈ bandwidth ÷ (params × bytes).

—

decode tokens / sec

—

time for a 1,000-token reply

—

bytes streamed / token

precision16-bit

16-bit8-bit4-bit

02Mechanics

Latency, throughput, and the tension between them

Time to first token (TTFT). Set by prefill — how long to process your whole prompt before the first word appears. Grows with prompt length; this is the "thinking…" pause.
Inter-token latency / tokens-per-second. Set by decode. Each token streams the model's weights from memory, so per-user speed ≈ memory bandwidth ÷ model bytes. This is the steady typing rate you watch.
End-to-end latency. Simply TTFT + (tokens × per-token time). A long answer from a big model is slow on both counts.
Throughput. The system's total tokens/sec across all users at once. Here's the twist: because one user's decode barely uses the GPU's compute, you can run many users' requests in the same forward pass — batching — and total throughput climbs almost for free, even though each individual user's speed is unchanged or slightly slower.
The tension. Latency is one user's experience; throughput is the system's efficiency. Bigger batches raise throughput (and lower cost per token) but can raise latency. Tuning that trade-off is the central job of an inference team.

So "tokens per second" is two metrics wearing one name: a per-user speed bounded by bandwidth, and an aggregate throughput bounded by compute once the batch is full. Quantization helps the first; batching helps the second.

04The math

expand ▾

The roofline of decode

Each decode step streams roughly the whole model — $N$ parameters at $b/8$ bytes each — from memory. With memory bandwidth $\text{BW}$ (bytes/sec), the time per token and the rate are:

$$ t_{\text{token}} \approx \frac{N \cdot (b/8)}{\text{BW}}, \qquad \text{tokens/sec} \approx \frac{\text{BW}}{N \cdot (b/8)} $$

End-to-end latency for a prompt of $P$ and an answer of $G$ tokens:

$$ \text{latency} \approx \underbrace{t_{\text{prefill}}(P)}_{\text{TTFT}} \;+\; G \cdot t_{\text{token}} $$

The roofline idea: decode lives on the memory-bandwidth slope, so halving the bytes (quantize) roughly doubles the rate. Batching $B$ requests reuses the same weight-load across all of them, so aggregate throughput rises toward $B \times$ — until the work becomes compute-bound and hits the GPU's FLOPs ceiling instead. Inference engineering is the art of climbing that roofline.

05The code

expand ▾

Speed, from the bandwidth up

Single-stream decode rate for three model sizes at three precisions, on an H100-class GPU.

tokens_per_sec.py

BW = 3.35e12   # H100 HBM3 memory bandwidth, ~3.35 TB/s

def toks_per_sec(N, bits):
    bytes_streamed = N * (bits / 8)     # ~whole model moved per token
    return BW / bytes_streamed

for N, nm in [(7e9,"7B"), (70e9,"70B"), (175e9,"175B")]:
    rates = "   ".join(f"{b}-bit {toks_per_sec(N,b):5.1f}" for b in [16,8,4])
    print(f"{nm:5s}: {rates}  tok/s")
# 7B   : 16-bit 239.3   8-bit 478.6   4-bit 957.1  tok/s
# 70B  : 16-bit  23.9   8-bit  47.9   4-bit  95.7  tok/s
# 175B : 16-bit   9.6   8-bit  19.1   4-bit  38.3  tok/s

print(f"70B 16-bit, 1000-token reply: {1000/toks_per_sec(70e9,16):.1f} s")  # 41.8 s

06The economics

Throughput is the unit of margin

tokens/sec → money

A GPU costs the same per hour whether it serves one user or a hundred. So the number that decides whether inference makes money is tokens per second per GPU — and therefore tokens per dollar. Everything in this chapter is a lever on that number: quantization raises per-user speed, batching raises aggregate throughput, and both cut the cost of every token served.

This is why providers obsess over batching and why they price output tokens the way they do: a memory-bound GPU running a single user is mostly idle silicon being paid for. Filling it is the difference between a gross margin and a loss. The same model can be a profitable product or a money pit depending entirely on how well its operator climbs this roofline.

For the Circuit, tokens/sec/dollar is the efficiency term in the whole equation: as it improves — through better hardware, lower precision, and smarter serving — the cost side of the ledger falls, buying the build-out more time for revenue to catch up. It is the most quietly important number in AI economics.

07Going deeper

expand ▾

The primary sources

Pope et al. (2022) — Efficiently Scaling Transformer Inference · latency vs throughput trade-offs.
Databricks — LLM Inference Performance Engineering · TTFT, tokens/sec, batching in practice.
NVIDIA — GPU Performance Background (roofline) · memory-bound vs compute-bound.
MLPerf Inference · the standard latency/throughput benchmark.

"First Principles — how AI actually works", chapter "Latency, throughput, tokens/sec", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

KV Cache

KV cache & batching

Two tricks make serving LLMs affordable. The KV cache stops the model recomputing the whole prompt every step; batching reuses one expensive weight-load across many users at once. Together they turn a wasteful, memory-bound GPU into a profit center — until the cache eats the memory.

01The answer, then the intuition

Cache the past; share the weights

Two facts from earlier collide here. From attention: each new token attends to every previous token — so naively, generating token 1,000 would recompute the first 999 from scratch, again and again. From inference: decode is memory-bound, so a single user leaves the GPU's compute mostly idle.

The KV cache fixes the first: store each token's attention keys and values once, and every later step just reads them — turning quadratic recompute into a cheap lookup, at the price of memory. Batching fixes the second: since one weight-load can serve many requests in the same pass, pack dozens of users together and aggregate throughput climbs almost for free.

But the KV cache and the batch compete for the same HBM. Drag the batch size: throughput soars — until the cache exhausts the memory and the whole thing falls over:

Batching — throughput vs the KV cache budget

A 70B model on an 8×80GB node (~500 GB free after weights), each request at 8K context. Illustrative.

—

aggregate throughput

—

per-user tok/s

—

KV cache used

KV cache vs 500 GB free HBM0%

batch size (concurrent requests)1

140

02Mechanics

Why each trick works, and where they fight

The KV cache. In attention, every token produces a key and a value vector. Since past tokens never change, you compute their K and V once and keep them. Each decode step then computes K/V for only the new token and attends against the cache — so per-step work is linear in context length, not quadratic. The cost is memory: the cache grows with every token generated.
Static batching. Group N requests and run their decode steps in one forward pass. The weights are loaded from HBM once and reused across all N, so you get ~N× the throughput for roughly the same memory traffic — the direct answer to decode being memory-bound.
Continuous batching. Real requests start and finish at different times. Continuous batching (the vLLM idea) adds and removes requests from the running batch every step, keeping the GPU packed instead of waiting for the slowest request. This is most of why modern serving is efficient.
The collision. Both the model weights and every request's KV cache must fit in the same HBM. A bigger batch means more caches, so memory — not compute — usually sets the ceiling on batch size. PagedAttention manages the cache like virtual memory (pages, not one big block) to pack far more requests into the same HBM.

So the serving problem is a memory allocation problem in disguise: fit the most revenue-generating requests you can into a fixed budget of very expensive bandwidth and capacity.

04The math

expand ▾

The size of the cache

For a model with $L$ layers and hidden size $d$, at $b$ bytes per number, the KV cache stores a key and a value per token per layer:

$$ \text{KV per token} = 2 \, L \, d \, b $$

For a batch of $B$ requests each holding $n$ tokens of context, the total cache is:

$$ \text{KV total} = 2 \, L \, d \, b \cdot n \cdot B $$

A large model with full multi-head attention ($L=80$, $d=8192$, 16-bit) caches $2\cdot80\cdot8192\cdot2 \approx 2.6$ MB per token — so 8,192 tokens is ~21 GB per request. (Real 70B-class models use grouped-query attention and cache roughly 8× less; the batch-vs-cache tension is identical, just at a smaller scale.) Aggregate throughput rises with the batch, throughput $\approx B \times (\text{single-stream rate})$, until either compute saturates or the cache hits the HBM limit:

$$ B_{\max} \approx \frac{\text{HBM free}}{2 \, L \, d \, b \cdot n} $$

Lower precision $b$ or shorter context $n$ shrinks the cache and lets you batch more — which is why serving stacks fight so hard for every byte.

05The code

expand ▾

How many users fit

The KV cache size and the batch it caps, for a 70B model on an 8-GPU node.

batching.py

layers, d, bytes_pp = 80, 8192, 2          # 70B-class, 16-bit
kv_per_token = 2 * layers * d * bytes_pp   # K and V per layer
ctx = 8192
kv_per_req = kv_per_token * ctx

print(f"KV per token: {kv_per_token/1e6:.2f} MB")
print(f"KV per request @ {ctx} ctx: {kv_per_req/1e9:.1f} GB")

free = 8*80e9 - 140e9                       # 8x80GB node minus 140GB weights
B_max = int(free // kv_per_req)
print(f"free HBM: {free/1e9:.0f} GB  ->  {B_max} concurrent requests")
print(f"throughput at batch {B_max}: {23.9*B_max:.0f} tok/s  (vs 23.9 for one)")
# KV per token: 2.62 MB
# KV per request @ 8192 ctx: 21.5 GB
# free HBM: 500 GB  ->  23 concurrent requests
# throughput at batch 23: 550 tok/s  (vs 23.9 for one)  <- 23x, for ~free

06The economics

Where inference stops losing money

Batching → money

This is the chapter where inference economics turns positive. A GPU serving one user is mostly idle silicon being paid for at full price. Batching spreads that fixed cost across many users, so the cost per token falls by nearly the batch factor — the difference between a business and a bonfire. When a provider quotes a low per-token price, efficient batching is how they can afford to.

And it explains the seams in the product. Long contexts are expensive not only in attention but because their KV caches crowd out other users, shrinking the batch and raising everyone's cost. That's why huge context windows carry premium pricing, and why the whole stack — quantization, PagedAttention, continuous batching — exists to squeeze more paying requests into a fixed HBM budget.

For the Circuit, batch efficiency is the hinge of the cost side: it's what lets falling prices coexist with expensive hardware. Improve it, and the break-even the whole build-out is chasing moves closer. It is unglamorous plumbing that quietly decides whether AI serving makes money.

07Going deeper

expand ▾

The primary sources

Kwon et al. (2023) — PagedAttention / vLLM · managing the KV cache like virtual memory.
Yu et al. (2022) — Orca · continuous (iteration-level) batching.
Pope et al. (2022) — Efficiently Scaling Transformer Inference · the batch/latency trade-off.
Dao et al. (2022) — FlashAttention · computing attention without materializing the big matrix.

"First Principles — how AI actually works", chapter "KV cache & batching", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.