cl100k_base · ~100k merges · ≈4 chars/token (EN) · 1 token ≈ ¾ word
A token is the basic unit of text an AI model reads and writes — not a word, not a letter, but a chunk. Models don't see language; they see tokens.
Tokenization isn't magic.
4 words become 6 tokens. The model reads the chunks on the bottom, never the sentence on top.
01The answer, then the intuition
Before a language model reads a single thing, your text is chopped into tokens by a piece of software called a tokenizer. A token can be a whole word (cat), a fragment of one ( token + ization), a single character, a space, or a punctuation mark. Common words usually become one token; rare words get split into pieces.
The model then works entirely in tokens — it predicts the next token, over and over, and the tokens are stitched back into text at the end. Three things you'll meet everywhere downstream are all measured in tokens: the context window (how much it can read at once), the price (APIs bill per token), and the speed (tokens per second).
A useful rule of thumb for English: 1 token ≈ 4 characters ≈ 0.75 words. So a 1,000-word essay is roughly 1,300 tokens.
Loading the real GPT-4 tokenizer (cl100k_base)…
02Mechanics
Almost every modern model tokenizes with a variant of Byte-Pair Encoding (BPE). The idea is borrowed from a 1994 data-compression trick and is delightfully simple. You build the vocabulary by merging:
cl100k at ~100,000, the newest at ~200,000.The result is a learned list of merges. To tokenize new text, the tokenizer greedily applies those merges: frequent strings like the survive as one token, while a rare word like antidisestablishmentarianism gets rebuilt from several pieces. Each final token maps to an integer ID, and that ID indexes a row in the model's embedding table — which is the next chapter. Text → tokens → IDs → vectors.
Two consequences worth holding onto: spaces are usually glued to the front of the next word ( token, with a leading space, is a different token from token), and the same word can tokenize differently depending on what's around it. That is why token counts feel slightly unpredictable — and why you should measure, not guess.
03The math
expand ▾Let the vocabulary be a finite set of size $|V|$. A tokenizer is a function that maps a string $s$ to a sequence of token IDs $\;t = (t_1, \dots, t_n),\; t_i \in \{0, \dots, |V|-1\}$, where $n$ is the token length of $s$.
BPE builds $V$ greedily. Beginning from the base alphabet $V_0$ (the 256 bytes), it repeatedly chooses the adjacent pair with the highest corpus frequency and adds its concatenation to the vocabulary:
stopping when $|V_k|$ reaches the target. There is no clean global optimum — BPE is a greedy compressor — but it reliably shortens sequences for a given vocabulary budget.
That budget is the whole game, because tokenization sets two of the model's biggest costs:
So a tokenizer is a dial between $n$ and $|V|$ — between sequence length and parameter count — and every model picks a point on it. Hold the $O(n^2)$; it returns in layer 06.
04The code
expand ▾A minimal BPE trainer. It learns merges from a toy corpus exactly as described above — runnable as-is in Python.
train_bpe.py
from collections import Counter
def get_pairs(tokens):
return Counter(zip(tokens, tokens[1:]))
def train_bpe(words, num_merges):
# words: list of strings; start from characters
corpus = [list(w) for w in words]
merges = []
for _ in range(num_merges):
pairs = Counter()
for tok in corpus:
pairs.update(get_pairs(tok))
if not pairs:
break
(a, b), _ = pairs.most_common(1)[0] # most frequent adjacent pair
merges.append((a, b))
# merge that pair everywhere
corpus = [merge(tok, a, b) for tok in corpus]
return merges
def merge(tok, a, b):
out, i = [], 0
while i < len(tok):
if i < len(tok) - 1 and tok[i] == a and tok[i+1] == b:
out.append(a + b); i += 2
else:
out.append(tok[i]); i += 1
return out
words = ["token"] * 6 + ["tokenizer"] * 3 + ["tokenization"] * 2
print(train_bpe(words, num_merges=5))
# -> [('t','o'), ('to','k'), ('tok','e'), ('toke','n'), ('token','i')] (pieces merge up into 'token')In production you don't train your own — you call the real one. OpenAI's tiktoken gives the exact tokenizer the models use:
count_tokens.py
import tiktoken
enc = tiktoken.get_encoding("cl100k_base") # GPT-4 / GPT-3.5
ids = enc.encode("Tokenization isn't magic.")
print(ids) # [3404, 2065, 4536, 956, 11204, 13]
print(len(ids)) # 6 tokens
print([enc.decode([i]) for i in ids])
# ['Token', 'ization', ' isn', "'t", ' magic', '.']The live widget at the top runs exactly this tokenizer in your browser — type into it and watch the chunks change.
05The economics — the economic dimension
Mechanics → money
Everything you just read has a price tag. The token isn't only the unit of text — it's the unit of cost. Models are billed per token; a frontier model runs roughly $2–$5 per million input tokens. One million tokens is roughly 750,000 words — several thousand pages. Multiply by hundreds of millions of users sending thousands of tokens each, and you arrive at the spend that built the data centers.
And remember the $O(n^2)$ from the math layer. Because attention compares every token to every other, a longer context costs compute that grows with the square of the token count — and the model must hold a KV cache whose memory grows linearly with every token in the conversation. That cache is a primary reason AI is starved for high-bandwidth memory. The token, in other words, is what runs straight into the memory wall.
So the chain is short and direct: more tokens → more compute and more memory → more HBM and more data centers → the build-out. The thing on this page is the atom that the whole Circuit is made of — see how it flows through the memory supply chain and the economics in pictures.
06Going deeper
expand ▾
Sennrich, Haddow & Birch (2016) — Neural Machine Translation of Rare Words with Subword Units · the paper that brought BPE to NLP.
Gage (1994) — A New Algorithm for Data Compression · the original byte-pair-encoding idea.
Radford et al. (2019) — GPT-2 · introduced byte-level BPE for language models.
OpenAI — tiktoken · the production tokenizer (cl100k_base, o200k_base).
Kudo & Richardson (2018) — SentencePiece · the language-agnostic alternative used by many open models.
"First Principles — how AI actually works", chapter "What is a token?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.
vector, d ≈ 1,536–12,288 · similarity = cosine · nearest-neighbor is the operation
An embedding turns a token into a point in space — a list of numbers — arranged so that meaning becomes distance. Similar things sit close together, and relationships become directions you can do arithmetic with.
01The answer, then the intuition
In the last chapter a token became an integer ID. But an ID like 3404 carries no meaning — 3405 isn't "one more" of anything. So the very first thing a model does is replace each ID with an embedding: a vector of a few hundred to a few thousand numbers.
The magic is in how those numbers are arranged. The model learns them so that tokens used in similar ways end up near each other in the space. "Dog" and "cat" land close; "dog" and "Tuesday" land far apart. Meaning is no longer a symbol — it's a location.
And because it's a real vector space, you can do arithmetic on meaning. The famous example: take the vector for king, subtract man, add woman — and you land almost exactly on queen. The "royalty" and "gender" relationships are directions in the space. Try it:
A 2-D schematic. Words that mean similar things cluster; relationships are parallel arrows.
Real embeddings live in hundreds of dimensions; this is flattened to two so we can see it.
02Mechanics
Every model holds an embedding table — a big matrix with one row per vocabulary token. If the vocabulary has $|V|$ tokens and the model width is $d$, that table is |V| × d numbers (for GPT-2: ~50,000 rows of 768). Tokenizing gives you IDs; the embedding step is just a lookup — row 3404 of the table is the vector for that token.
Those numbers aren't hand-set; they're learned. During training the model nudges the vectors so that tokens appearing in similar contexts drift together — the old word2vec insight that "you shall know a word by the company it keeps." Modern LLMs learn their embeddings jointly with everything else, and they're contextual: the vector for "bank" shifts depending on whether the sentence is about rivers or money.
The same trick works on anything you can show a model — sentences, images, audio, code. Turn it into a vector, and "similar" becomes "close." That single idea is the engine under semantic search, recommendations, and retrieval.
03The math
expand ▾An embedding is a vector $\mathbf{v} \in \mathbb{R}^d$. To ask "how similar are two tokens," we don't use straight-line distance — we use the angle between their vectors, via cosine similarity:
It runs from $-1$ (opposite) through $0$ (unrelated) to $1$ (identical direction). Angle, not magnitude, because what matters is which way a vector points in meaning-space, not how long it is.
Analogies fall out of the same geometry. If a relationship — "royal," say — is a consistent direction $\mathbf{r}$, then $\text{king} \approx \text{man} + \mathbf{r}$ and $\text{queen} \approx \text{woman} + \mathbf{r}$. Subtract to isolate the direction and re-add it elsewhere:
The model never "knew" that queens are royal women. The geometry encodes it, because that's how the words were used.
04The code
expand ▾Cosine similarity and the king/queen analogy, on toy vectors — runnable as-is.
embeddings.py
import numpy as np
# toy 4-d embeddings (illustrative; real ones are hundreds of dims)
emb = {
"king": np.array([0.9, 0.8, 0.1, 0.7]),
"queen": np.array([0.9, 0.1, 0.8, 0.7]),
"man": np.array([0.2, 0.8, 0.1, 0.1]),
"woman": np.array([0.2, 0.1, 0.8, 0.1]),
}
def cosine(a, b):
return a @ b / (np.linalg.norm(a) * np.linalg.norm(b))
# 1) similar words point the same way
print(round(cosine(emb["king"], emb["queen"]), 3)) # 0.749
# 2) analogy: king - man + woman ~ queen
v = emb["king"] - emb["man"] + emb["woman"]
best = max(emb, key=lambda w: cosine(v, emb[w]))
print(best, round(cosine(v, emb["queen"]), 3)) # queen 1.0The interactive space above is the same idea, with the vectors arranged by hand so you can watch the parallelogram close.
05The economics
Geometry → money
The moment meaning is a location, finding relevant information becomes finding nearby points. That single move powers semantic search, recommendations, and retrieval-augmented generation (RAG) — the dominant way companies put their own data into an LLM. An entire category of infrastructure, the vector database, exists just to store billions of embeddings and find the nearest ones in milliseconds.
It isn't free. Every document, product, and message gets embedded (compute), and the vectors are held in fast memory to be searched at scale (memory). The embedding table itself is |V| × d parameters — the same $|V|\cdot d$ that the token chapter traded against sequence length. At the scale of the modern web, "turn everything into a vector and keep it searchable" is its own slice of the data-center demand.
So embeddings are the second atom — after the token — of the AI economy: the layer that turns meaning into something you can store, search, and bill for. See where it lands in the Circuit.
06Going deeper
expand ▾
Mikolov et al. (2013) — Efficient Estimation of Word Representations (word2vec) · the paper that made king − man + woman famous.
Pennington, Socher & Manning (2014) — GloVe · global word vectors from co-occurrence.
Devlin et al. (2018) — BERT · contextual embeddings: the same word, different vector by context.
Reimers & Gurevych (2019) — Sentence-BERT · embedding whole sentences for search.
"First Principles — how AI actually works", chapter "What is an embedding?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.
weights × activations + bias → nonlinearity · stacked · learned by gradient descent
A neural network is a stack of weighted sums. Each unit multiplies its inputs by weights, adds them up, and passes the result through a simple bend. Learning is nothing more than adjusting those weights until the output is right.
01The answer, then the intuition
Strip away the mystique and a single artificial neuron does one humble thing: it takes some numbers in, multiplies each by a weight, adds a bias, and decides yes-or-no. Geometrically, that "decide" is a straight line (a plane, in higher dimensions) splitting the input space in two.
Below is exactly one neuron with two inputs. Its weights set the angle and position of its decision line. Drag the sliders until the line cleanly separates the two groups of points — and watch the accuracy climb. That's a neuron "learning," done by hand.
The line is where w₁x + w₂y + b = 0. Move the weights; separate the dots.
One neuron draws one straight line. Stack a layer of them with a nonlinear bend between, and the lines compose into curves — that's how a deep network carves any shape.
02Mechanics
Three pieces turn that one line into something powerful:
ReLU, which just zeroes out negatives, is the workhorse. Without it, stacking layers would collapse back to a single line; with it, the network can bend its boundaries into arbitrary shapes. This is the whole reason depth helps.That's the entire recipe. An LLM is this idea at staggering scale — the tokens and embeddings you've met flow through hundreds of such layers, and the "attention" of the next chapter is a particularly clever layer inside it.
04The math
expand ▾A single neuron with weights $\mathbf{w}$, bias $b$, and activation $\sigma$ computes:
A whole layer stacks many neurons, so the weights become a matrix $W$ and the operation is a matrix-vector product:
A deep network is just these composed — the output of one layer feeding the next:
Training adjusts every weight by walking downhill on the loss $L$, scaled by a learning rate $\eta$:
That gradient $\nabla_{\!W} L$ is computed by backpropagation — the chain rule, applied layer by layer from the output back to the input. Everything else in modern AI is detail on top of this.
05The code
expand ▾A two-layer network, front to back. Runnable as-is — this is the entire forward pass.
forward.py
import numpy as np
def relu(z): return np.maximum(0, z)
def sigmoid(z): return 1 / (1 + np.exp(-z))
# layer 1: 3 inputs -> 4 hidden units ; layer 2: 4 -> 1 output
W1 = np.array([[ 0.5,-0.3, 0.8],
[ 0.1, 0.9,-0.2],
[-0.4, 0.2, 0.7],
[ 0.3,-0.6, 0.1]]) # shape (4, 3)
b1 = np.array([0.0, 0.1, -0.2, 0.05])
W2 = np.array([[1.2, -0.7, 0.5, 0.9]]) # shape (1, 4)
b2 = np.array([0.05])
def forward(x):
h = relu(W1 @ x + b1) # hidden layer + the nonlinear bend
y = sigmoid(W2 @ h + b2) # output as a probability
return y
print(forward(np.array([1.0, 0.5, -1.0]))) # -> [0.36703]The interactive above is the single-neuron version of the first line; a real model is millions of these matrices, learned rather than set.
06The economics
Multiplies → money
A network's cost is set by one number: how many weighted multiplies it does. That's its parameter count, $N$. Running a model over one token takes roughly $2N$ floating-point operations; training it takes about $6ND$ — six times the parameters times the tokens it learns from. Those constants are not metaphors; they're the arithmetic that sizes the build-out.
So when a frontier model has $N \approx 10^{12}$ parameters and trains on $D \approx 10^{13}$ tokens, the training run is on the order of $10^{26}$ operations — months of a data center running flat out. The three sliders you moved are, at the frontier, a trillion of them, multiplied across trillions of tokens. That product is the demand for GPUs, power, and memory.
This is the hinge of the whole thesis: AI's compute bill is a direct function of network size, and size has only gone up. It's why "scaling laws" became an investment strategy — and why the Circuit exists. The humble weighted sum, multiplied enough times, built the data centers.
07Going deeper
expand ▾
Rosenblatt (1958) — The Perceptron · the original artificial neuron.
Rumelhart, Hinton & Williams (1986) — Learning representations by back-propagating errors · backprop, the engine of learning.
LeCun, Bengio & Hinton (2015) — Deep Learning (Nature review) · the modern synthesis.
Kaplan et al. (2020) — Scaling Laws for Neural Language Models · where the 6ND compute rule comes from.
"First Principles — how AI actually works", chapter "What is a neural network?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.
softmax(QKᵀ/√dₖ)V · O(n²) in sequence length · the n² is why context costs
score every token against every other, squash to weights, mix.
Attention is the move that lets every token look at every other token and decide which ones matter for it right now. It's how a model figures out that "it" refers to "the animal" — and it's the heart of the transformer.
01The answer, then the intuition
To understand a word, you need its context. "Bank" means something different by a river than in a sentence about money; "it" only makes sense once you know what it points to. Earlier models read left-to-right through a bottleneck and forgot. Attention fixed that with a brutally direct idea: let every token look at all the others at once, and learn how much to weight each one.
Click a word below. The arcs show what it pays attention to — thicker means more. Notice where "it" looks:
A single attention head over one sentence. Arcs show how much each word attends to the others.
Illustrative weights for one head. Real models run dozens of these heads in parallel, each learning a different kind of relationship.
02Mechanics
Each token produces three vectors from its embedding, via learned weight matrices:
The mechanism is a soft dictionary lookup. Compare one token's query against every token's key (a dot product) to get a relevance score; the better the match, the higher the score. Run those scores through a softmax so they become weights that sum to one, then take a weighted average of the values. That blend is the token's new, context-aware representation — "it" has now mixed in a large dose of "animal."
Stack this inside the network layers, run many heads in parallel (each catching a different relationship — syntax, coreference, topic), and repeat over dozens of layers. That stack is the transformer, and it's the next chapter.
04The math
expand ▾Pack the queries, keys and values for all $n$ tokens into matrices $Q, K, V$, each of shape $n \times d_k$. The entire operation is one line:
Read it inside-out. $QK^{\top}$ is an $n \times n$ matrix — every token's query dotted with every token's key, the full grid of relevance scores. Dividing by $\sqrt{d_k}$ keeps the numbers from blowing up as dimension grows. The $\text{softmax}$ turns each row into a probability distribution (the attention weights). Multiplying by $V$ takes the weighted average of the values.
Keep your eye on that $QK^{\top}$. It is $n \times n$ — its size grows with the square of the number of tokens. Hold that thought; it is the most expensive fact in all of AI, and it returns in layer 06.
05The code
expand ▾The whole mechanism, runnable. Note the attention weights — each row sums to one.
attention.py
import numpy as np
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True))
return e / e.sum(axis=-1, keepdims=True)
def attention(Q, K, V):
d_k = Q.shape[-1]
scores = Q @ K.T / np.sqrt(d_k) # (n, n): every query vs every key
weights = softmax(scores) # each row sums to 1
return weights @ V, weights # blended values, and the attention map
np.random.seed(0)
Q = np.random.randn(3, 4); K = np.random.randn(3, 4); V = np.random.randn(3, 4)
out, w = attention(Q, K, V)
print(w.round(2)) # [[0.68 0.3 0.02]
# [0.29 0.7 0.01]
# [0.5 0.19 0.31]]
print(w.sum(axis=1)) # [1. 1. 1.]06The economics
n² → money
Attention's superpower — letting every token see every other — is also its bill. That $QK^{\top}$ grid is $n \times n$, so the compute grows as $O(n^2)$ in the number of tokens. Double the context, and you quadruple the attention work. This single fact is the most consequential number in AI economics.
It's why early models capped context at a few thousand tokens, why "1 million token context" is a genuine engineering feat rather than a setting, and why running models is memory-bound: to avoid recomputing, models cache every token's keys and values — the KV cache — whose size grows with the sequence and devours high-bandwidth memory. The $n$ you met in the token chapter, attention squares.
So the line of demand runs straight through this layer: longer, smarter context → quadratic compute and linear-but-relentless memory → more GPUs and more HBM → the build-out. A whole research frontier (FlashAttention, sparse and linear attention) exists just to soften that exponent. When the Circuit talks about the memory wall, this $n^2$ is the wall.
07Going deeper
expand ▾
Vaswani et al. (2017) — Attention Is All You Need · the transformer, and this exact formula.
Bahdanau, Cho & Bengio (2014) — Neural Machine Translation by Jointly Learning to Align and Translate · attention, before transformers.
Dao et al. (2022) — FlashAttention · making the O(n²) memory-efficient on real GPUs.
Alammar — The Illustrated Transformer · the canonical visual walkthrough.
"First Principles — how AI actually works", chapter "What is attention?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.
attention + MLP, ×N layers · one token out per pass · every weight touched each token
A transformer is the assembly. Take embeddings, attention, and a small neural network, wire them into a repeatable block, and stack that block dozens of times. That stack is what an LLM is.
01The answer, then the intuition
You've met every piece. A transformer just connects them. Text becomes tokens, tokens become embeddings, and then the data flows through a stack of identical blocks. Inside each block, attention lets tokens share information, and a small neural network processes each one. Repeat, and out the top come probabilities for the next token.
Click through the architecture — each stage is one of the earlier chapters doing its job:
Data flows up. The shaded block is the unit that repeats.
Tap any stage in the diagram to see what it does — and which chapter it comes from.
02Mechanics
The whole architecture is one block applied over and over. Inside it, two ideas you've seen, plus two pieces of plumbing that make deep stacks trainable:
One detail matters: attention is order-blind — shuffle the tokens and it gives the same answer. So before the first block, each embedding gets a positional encoding added, a signal telling the model where each token sits. Stack ~12 blocks for a small model, ~100 for a frontier one, cap it with an output layer that turns the final vectors into next-token probabilities, and you have GPT.
04The math
expand ▾Let $\mathbf{x}$ be the matrix of token vectors entering a block. The block is exactly two residual-and-normalize steps — first attention, then the feed-forward network:
where the feed-forward network is a two-layer MLP applied to each token, $\text{FFN}(\mathbf{x}) = \max(0,\,\mathbf{x}W_1)\,W_2$. Crucially the block maps $\mathbb{R}^{n\times d} \to \mathbb{R}^{n\times d}$ — same shape in, same shape out — which is precisely why you can stack it $N$ times:
The final vectors are projected by an unembedding matrix to vocabulary-sized scores (logits) and softmaxed into the probability of the next token. That's the entire forward pass of a GPT.
05The code
expand ▾The block, composing the pieces from the earlier chapters. Note the output shape equals the input shape — that's the stacking property.
block.py
import numpy as np
np.random.seed(0)
def softmax(x):
e = np.exp(x - x.max(axis=-1, keepdims=True)); return e / e.sum(axis=-1, keepdims=True)
def attention(x, Wq, Wk, Wv):
Q, K, V = x@Wq, x@Wk, x@Wv
return softmax([email protected] / np.sqrt(Q.shape[-1])) @ V
def layernorm(x):
return (x - x.mean(-1, keepdims=True)) / (x.std(-1, keepdims=True) + 1e-5)
def ffn(x, W1, W2):
return np.maximum(0, x@W1) @ W2 # 2-layer ReLU MLP
def block(x, Wq, Wk, Wv, W1, W2):
x = layernorm(x + attention(x, Wq, Wk, Wv)) # attention + residual + norm
x = layernorm(x + ffn(x, W1, W2)) # feed-forward + residual + norm
return x
x = np.random.randn(3, 4) # 3 tokens, width 4
Wq, Wk, Wv = (np.random.randn(4, 4) for _ in range(3))
W1, W2 = np.random.randn(4, 8), np.random.randn(8, 4) # FFN hidden = 8
print(block(x, Wq, Wk, Wv, W1, W2).shape) # (3, 4) — stack it as many times as you like06The economics
Architecture → money
The transformer's bill is its shape. Each of the $N$ blocks runs an attention (the $O(n^2)$ from the last chapter) and a feed-forward network whose cost scales with the square of the width, $O(n\,d^2)$. Multiply by the number of layers and you have the whole forward pass. A frontier model is roughly $N \approx 100$ blocks, width $d$ in the tens of thousands, over $n$ in the thousands — and it runs that for every token, for hundreds of millions of users.
This is why the architecture is the capex. Every knob that makes models better — more layers, more width, longer context — multiplies straight into FLOPs, GPUs, power, and memory. The transformer turned "make it bigger" into a reliable recipe, and that recipe is what the entire build-out is paying for.
You now hold the whole chain: a token ($n$), its embedding ($d$), the attention ($n^2$), the network ($N$ layers of weights) — assembled here into the machine whose cost is the Circuit. The transformer is where the foundations become an economy.
07Going deeper
expand ▾
Vaswani et al. (2017) — Attention Is All You Need · the transformer architecture itself.
Radford et al. (2018) — GPT · the decoder-only transformer that became the LLM.
Phuong & Hutter (2022) — Formal Algorithms for Transformers · the architecture written out precisely.
Alammar — The Illustrated Transformer · the canonical visual walkthrough.
"First Principles — how AI actually works", chapter "What is a transformer?", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.