Prompting

A prompt is a program — written in plain English, for a machine whose only instinct is to continue text. You don't change the weights; you change the context you condition them on. Done well, that's enough to steer the model completely.

01The answer, then the intuition

Programming without changing the program

Because a model just predicts the next token given everything before it, the tokens you supply are the program. The frozen weights are the interpreter; your prompt is the code. This is why the same model can write poetry, extract JSON, or debug Python — you're not switching models, you're switching the context that conditions its predictions.

Three levers do most of the work. Zero-shot is just an instruction. Few-shot adds worked examples so the model infers the exact pattern you want — remarkably, with no training at all. Chain-of-thought asks it to reason step by step, spending more tokens to think before answering. Switch between them on the same task and watch the output sharpen:

Prompt lab — one task, three techniques

Task: classify a review's sentiment and return strict JSON. Illustrative outputs; token counts are representative.

The prompt sent to the model

Illustrative output

prompt tokens: — vs zero-shot: —

02Mechanics

Why context alone can steer a frozen model

Zero-shot. Just describe the task. The model relies entirely on what it learned in pretraining and alignment. Fast and cheap, but format and edge cases are hit-or-miss.
Few-shot (in-context learning). Put a few input→output examples in the prompt. The model infers the pattern and continues it — with no weight update. This was the headline surprise of GPT-3: examples in the context act like temporary training. It nails formats and conventions that are hard to describe in words.
Chain-of-thought. Ask it to "think step by step." By generating intermediate reasoning tokens before the answer, the model effectively does more computation — each token is another forward pass — which sharply improves multi-step and math problems.
System prompts & structure. A system message sets persistent role and rules; clear delimiters, explicit output schemas, and "return only JSON" instructions reduce ambiguity. You're shaping the probability distribution toward the tokens you want.

The craft is real but bounded: prompting can only elicit what's already in the weights. When the model simply lacks the knowledge or skill, no wording fixes it — that's when you reach for retrieval or fine-tuning (next chapters).

04The math

expand ▾

Conditioning, not learning

Everything a prompt does is condition the same distribution. The model samples the answer $y$ from:

$$ y \sim P(y \mid \text{prompt}) $$

Few-shot just makes the prompt longer — the examples $\{(x_i,y_i)\}$ are extra conditioning tokens, so "in-context learning" is Bayesian conditioning, not gradient descent. Nothing in the weights changes:

$$ P\big(y \mid x,\, (x_1,y_1),\dots,(x_k,y_k)\big) $$

Chain-of-thought factorizes the answer through an intermediate reasoning $r$, letting the model spend computation on the path before committing:

$$ P(y \mid x) = \sum_{r} P(y \mid r, x)\,P(r \mid x) $$

Generating $r$ token-by-token turns "think harder" into literal extra forward passes — more compute at inference time, which is why it helps on hard problems and costs more.

05The code

expand ▾

The price of a better prompt

Prompting is free to build, but every example and reasoning step is more tokens per call — the real tradeoff.

prompt_cost.py

instr, per_example, output = 18, 22, 12    # representative token counts

zero = instr + output                       # instruction only
few3 = instr + 3*per_example + output       # + three worked examples
cot  = instr + 45 + output                  # + a reasoning trace in the output

print(f"zero-shot: {zero} tokens/call")
print(f"3-shot:    {few3} tokens/call  ({few3/zero:.1f}x)")
print(f"CoT:       {cot} tokens/call  ({cot/zero:.1f}x)")
# zero-shot: 30 tokens/call
# 3-shot:    96 tokens/call  (3.2x)
# CoT:       75 tokens/call  (2.5x)   <- better answers, more tokens, every call

06The economics

The cheapest way to program — with a running meter

Prompting → money

Prompting is the cheapest possible way to customize a model: no training run, no data pipeline, just words — change it and redeploy in seconds. That's why most AI products start here. But the cost moves from upfront to per call: every example in a few-shot prompt and every step of chain-of-thought is more tokens, paid on every single request, forever.

At scale that arithmetic dominates. A prompt that's 3× longer is roughly 3× the inference bill across millions of calls — so serious teams trim prompts token by token, cache shared prefixes, and reserve chain-of-thought for the queries that truly need it. Prompt engineering is, underneath, cost engineering.

It also frames the build-vs-buy choice the next chapters unpack: prompting is a recurring per-token cost, while fine-tuning is an upfront cost that can shorten prompts later. For the Circuit, prompting is the demand side in miniature — the knob that turns a fixed model into useful work, one metered token at a time.

07Going deeper

expand ▾

The primary sources

Brown et al. (2020) — Language Models are Few-Shot Learners (GPT-3) · in-context learning.
Wei et al. (2022) — Chain-of-Thought Prompting · reasoning steps improve hard tasks.
Kojima et al. (2022) — "Let's think step by step" · zero-shot chain-of-thought.
Anthropic — Prompt Engineering Guide · practical, current techniques.

"First Principles — how AI actually works", chapter "Prompting", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Token Economics

The economics of a token

Strip away the narrative and all of AI's economics reduce to one unit: the token. What it costs to produce, what it sells for, and how many you must sell to repay the training bill. This is the Circuit's central question, made arithmetic.

01The answer, then the intuition

Two P&Ls hiding in one token

Every token has a cost and a price, and the gap between them is the entire business. But there are really two economics stacked inside it. The first is the gross one: does the revenue from a token exceed the inference cost of producing it? That depends almost entirely on utilization — a lightly-batched GPU makes every token a loss; a well-packed one makes it a profit.

The second is the fully-loaded one: even at a positive gross margin, you must sell enough tokens to repay the hundred-million-dollar training run and the hardware. Drag the batch size and watch a single token flip from a catastrophic loss to a thin profit — then see how many trillions it takes to reach the first real dollar:

Per-token unit economics — drag the batch

70B @ 4-bit, ~$30/hr node, $3 revenue per 1M tokens, $100M training capex. Illustrative but internally consistent.

—

cost / 1M tokens

—

gross margin / 1M

—

tokens to repay capex

revenue

inference cost

margin

batch size (utilization)1

1 (idle GPU)64 (packed)

02Mechanics

Where every number comes from

Cost per token. It's the hardware's hourly cost divided by the tokens it produces per hour. Throughput is set by everything in Part III — batching, quantization, and beating the memory wall. This is why utilization dominates: the same GPU, idle or packed, produces the same cost per hour but wildly different cost per token.
Revenue per token. What you charge, pressured downward by open-weight competition and the falling cost of "good enough" intelligence. Prices have fallen fast and keep falling.
Gross margin. Revenue minus inference cost per token. At low utilization it's deeply negative; batching is what carries it across zero. Most public arguments about "AI profitability" are really arguments about this one number.
The capex overhang. Above gross margin sits the fixed cost — training the model and buying the cluster. A positive per-token margin still has to be multiplied by an enormous volume to repay it, and the model may be obsolete before it does.

So "is AI profitable?" isn't one question. A token can be gross-margin positive and the company still deeply unprofitable, because the fixed costs are gigantic and the price per token keeps sliding. Both P&Ls have to work — and they're in tension.

04The math

expand ▾

Cost, margin, and the first dollar

Cost per token is hourly hardware cost over hourly throughput, where throughput scales with the batch:

$$ c_{\text{tok}} = \frac{\text{cost}_{\text{hr}}}{\text{throughput}_{\text{hr}}}, \qquad \text{throughput}_{\text{hr}} = B \cdot r \cdot 3600 $$

Gross margin per token is just price minus cost; the fully-loaded break-even is the training capex divided by that margin:

$$ m_{\text{tok}} = p_{\text{tok}} - c_{\text{tok}}, \qquad V^{*} = \frac{\text{capex}}{m_{\text{tok}}} \;\; (\text{requires } m_{\text{tok}} > 0) $$

The numbers are sobering. At batch 40 the margin is ~\$0.81 per million tokens — \$8.1\times10^{-7}$ each — so repaying \$100M in training takes $V^{*} \approx 1.24\times10^{14}$ tokens, 124 trillion. And $V^{*}$ moves the wrong way twice: as competition pushes $p_{\text{tok}}$ down, and as scaling pushes capex up. That widening gap between a shrinking margin and a growing fixed cost is the divergence the whole desk exists to measure.

05The code

expand ▾

A token's P&L, batch by batch

Utilization decides whether a token makes money — and repaying the training run is a different order of magnitude.

token_economics.py

node_cost_hr = 30.0     # $/hr, 8-GPU node
base_tps     = 95.0     # tokens/sec single stream (70B @ 4-bit)
price_1M     = 3.00     # $ revenue per 1M tokens
train_capex  = 100e6    # $ one-time training cost

def econ(batch):
    tok_per_hr = base_tps * batch * 3600
    cost_1M = node_cost_hr / (tok_per_hr / 1e6)
    return cost_1M, price_1M - cost_1M          # cost, margin per 1M

for b in (1, 8, 40):
    c, m = econ(b)
    print(f"batch {b:>2}: cost ${c:6.2f}/1M  margin ${m:6.2f}/1M  "
          f"-> {'profit' if m > 0 else 'LOSS'}")

_, m = econ(40)
print(f"tokens to clear ${train_capex:.0e} capex: {train_capex/(m/1e6):.2e}")
# batch  1: cost $ 87.72/1M  margin $-84.72/1M  -> LOSS
# batch  8: cost $ 10.96/1M  margin $ -7.96/1M  -> LOSS
# batch 40: cost $  2.19/1M  margin $  0.81/1M  -> profit
# tokens to clear $1e+08 capex: 1.24e+14   <- 124 trillion tokens

06The economics

The whole thesis, in one unit

The token → money

This chapter is the Circuit reduced to a single number you can hold. Everything upstream — the chips, the supply chain, the scaling laws, the clusters — exists to change the cost of a token. Everything downstream — the products, the agents, the enterprise deals — exists to raise the revenue from one. The business is the wedge between the two, multiplied by an almost unimaginable volume.

And the wedge is under attack from both sides. Revenue per token falls as competition and open weights commoditize intelligence; the capex per model rises as scaling demands more compute. A token that's gross-margin positive today can still leave a company far from repaying its fixed costs — and the finish line keeps moving away. That's not pessimism; it's the arithmetic.

So when someone claims AI is or isn't profitable, this is the calculation to demand. Which margin — gross or fully-loaded? At what utilization, what price, what capex? The honest answer is a spreadsheet, not a slogan — and building that spreadsheet, transparently, is precisely what an independent research desk is for.

07Going deeper

expand ▾

The primary sources

Sequoia — AI's $600B Question · the revenue-vs-capex gap, framed by an investor.
SemiAnalysis — inference cost economics · cost-per-token teardown from the hardware up.
Epoch AI — Training cost of frontier models · the capex side of the equation.
a16z — The Economics of Generative AI · unit economics and margin structure.

"First Principles — how AI actually works", chapter "The economics of a token", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.

Cost Optimization

Cost optimization

AI serving cost isn't one number to shave — it's a stack of independent levers that multiply. Each cuts cost by a factor; stacked, they compound. This is the operational discipline that turns a token that loses money into one that makes it.

01The answer, then the intuition

Independent factors, multiplied

The reason AI costs can fall so dramatically is that the savings multiply. Caching cuts cost in half; batching cuts what's left to a quarter; quantization halves that again; routing and prompt trimming take more still. Because each lever acts on a different part of the cost, they stack — five modest wins become one enormous one.

That's why "AI is too expensive" is usually a solvable engineering problem, not a fixed fact. Toggle the levers and watch a naive $100-per-million-tokens bill collapse toward a couple of dollars:

The cost stack — toggle the levers

Baseline $100 / 1M tokens (naive). Each lever's multiplier is illustrative but realistic.

$100/1M

baseline · nothing enabled

02Mechanics

The five levers, and what each one moves

Prompt caching. When many requests share a long prefix — a system prompt, a document, few-shot examples — cache its KV state once and reuse it, so you don't reprocess it every call. For repeated-context workloads this alone can halve cost or more.
Continuous batching. Pack many requests into each forward pass so one weight-load serves them all. The single biggest lever on a memory-bound GPU's economics.
Quantization. Serve at 4- or 8-bit instead of 16, cutting the memory moved per token and letting you batch more — usually for negligible quality loss.
Model routing. Send easy requests to a cheap model and escalate only the hard ones. Captures most of the frontier's quality at a fraction of the price.
Prompt & output trimming. Every token is billed, so shorter prompts, tighter instructions, and capped output lengths cut cost on every single call — the least glamorous lever, and often the easiest.

The discipline is to treat cost as a product of factors and attack each independently, always guarding quality with an eval so a saving doesn't quietly become a regression. Stacked carefully, order-of-magnitude reductions are routine.

04The math

expand ▾

Why savings compound

Because each lever scales a different part of the pipeline, total cost is the baseline times the product of the multipliers — not the sum:

$$ \text{cost} = \text{baseline} \times \prod_{i} f_i, \qquad 0 < f_i < 1 $$

Multiplication is what makes the effect so large. Five levers of $\{0.5, 0.25, 0.5, 0.4, 0.7\}$ give:

$$ 100 \times 0.5 \times 0.25 \times 0.5 \times 0.4 \times 0.7 = \$1.75 \;\;\Rightarrow\;\; 57\times \text{ cheaper} $$

No single lever did that — the biggest was only 4× on its own. The compounding is the point: a stack of merely-good optimizations produces a great one. It also explains why serving prices have fallen so steeply industry-wide — providers are stacking these same factors, and the product keeps shrinking. (The multipliers aren't truly independent — caching and batching interact — but the multiplicative model is the right first-order intuition.)

05The code

expand ▾

Stacking the levers

Apply each multiplier in turn and watch the running total fall.

cost_stack.py

baseline = 100.0    # $ per 1M tokens, naive setup
levers = {
    "Prompt caching":       0.50, "Continuous batching": 0.25,
    "Quantization (4-bit)": 0.50, "Model routing":       0.40,
    "Prompt trimming":      0.70,
}

total = baseline
for name, f in levers.items():
    total *= f
    print(f"+ {name:22s} x{f}  -> ${total:.2f}/1M")

print(f"final ${total:.2f} vs ${baseline:.0f} = {baseline/total:.0f}x cheaper")
# + Prompt caching         x0.5   -> $50.00/1M
# + Continuous batching    x0.25  -> $12.50/1M
# + Quantization (4-bit)   x0.5   -> $6.25/1M
# + Model routing          x0.4   -> $2.50/1M
# + Prompt trimming        x0.7   -> $1.75/1M
# final $1.75 vs $100 = 57x cheaper

06The economics

How a losing token becomes a winning one

Optimization → money

This chapter is the operational answer to Chapter 29's problem. There, a token was deeply unprofitable at low utilization; here, a stack of levers cuts its cost by ~50× — which is exactly what carries it across the line from loss to margin. Cost optimization isn't a nice-to-have; for most AI products it's the difference between a viable business and a subsidized demo.

The compounding is also why the industry's cost curve falls so fast, and why the same capability keeps getting cheaper to serve every year. Providers stack these factors continuously, so the price of a given quality of intelligence deflates — the optimistic half of the Circuit's ledger, pushing against the rising capex on the other side.

For the desk, this is a caution against static analysis. A token that looks unprofitable at today's naive cost may be comfortably profitable once optimized — and today's price may already assume optimizations a competitor hasn't made. Reading AI economics honestly means asking not just "what does it cost?" but "what could it cost, fully optimized?" — because that's the number the market is racing toward.

07Going deeper

expand ▾

The primary sources

Anthropic — Prompt Caching · reusing a cached prefix to cut cost and latency.
Kwon et al. (2023) — vLLM / PagedAttention · the batching engine behind the savings.
Chen et al. (2023) — FrugalGPT · routing and cascades as a cost lever.
SemiAnalysis · how inference cost is falling across the stack.

"First Principles — how AI actually works", chapter "Cost optimization", by The Catch. CC-BY 4.0. Hosted at TheCatch.AI / Tech Stack / AI Principles.