liberlume-content-v3
lang: en
title: A journey inside the machine that answers: how the bytes move in inference
summary: What happens, physically, when a language model answers, and what changes if the one answering is a single machine, at home? A path through the principles of how it works, from attention to the craft of inference, with its two phases, its organization of the data, the techniques that make it cost less, read along one thread: where the bytes are and how much it costs to move them. In the background, an idea: local inference, as a choice and as a possibility, already matters today and will matter more and more. Keeping it concrete, an engine written in C that brings a model of hundreds of billions of parameters to run locally: DwarfStar, with which antirez shows how far one can push, and that this road can be travelled.
btc-anchor: 958989,00000000000000000000e45855edc8644c6749248b727561a338a19637f2c7e9
prev: sha256:c2420f5a52461ddaad2238802eb311eb4339c2485008a7fe39d52b6cd2eb0a59
--- body ---
For some months now a part of my time has gone to a precise exercise: trying
to understand the operating principles of inference engines in general, and in
particular of those built to run locally, in principle on a single machine;
and to understand (as far as I can) why they are built the way they are. Any
resemblance to antirez's code is not purely coincidental. I am not an expert
programmer, and with respect to the AI world I am little more than a "curious
onlooker", but I have tried to understand all the same, I have tried to inform
myself. And I am well aware that the journey has, in any case, only just begun.
Starting from the inference engine is a strange vantage point but, I believe,
an instructive one: from in there the theory of transformers stops being a
list of acronyms and becomes a row of concrete problems, almost all of the
same kind: there is an enormous heap of numbers, and one has to decide where
to keep them and when and how to move them.
These pages are, first of all, my attempt to put back in order (with the help
of an LLM too, an almost ironic detail) what I have "accumulated" and tried to
understand. They were meant to be about inference, the local kind in
particular; but it felt almost natural to start from the beginning, from how a
model is trained and from the principles of how it works, because without
those foundations there is little one can say. The path therefore goes further
back than expected: first the basics, then the paper that started it all, in
2017, and from there the techniques with which a model is made to answer today
while optimizing the resources of whoever runs it.
Not a manual: a map with a few entry points, chosen because they open up the
rest. It grew more than expected, and I chose not to thin it out: too many
omissions would end up failing to convey what allows "the machine to answer
one word at a time". A line, however, is drawn. Some of the thorniest steps of
the algorithms involved are depicted but not explored in depth, taken as given
and rendered the way I understood them; for those details the right place
remains the specialist sources, and some are listed at the end.
The underlying thought, not too hidden, is that local inference, as a choice
and as a possibility, already matters today for companies, for communities
and, in some respects, for individual people, and will matter more and more:
perhaps the tipping point for the scenarios of adoption and development.
Understanding something of the mechanisms underneath is useful whatever one's
role.
From here on the voice returns to the impersonal; but the questions were born
there.
The map just announced begins with its stops; each title is an entry point:
- [1. What a machine learns](#1-what-a-machine-learns)
- [2. Training: the descent and the three phases](#2-training-the-descent-and-the-three-phases)
- [3. Interlude: the parrot and the library](#3-interlude-the-parrot-and-the-library)
- [4. 2017: attention, or managing the context](#4-2017-attention-or-managing-the-context)
- [5. The landscape of models](#5-the-landscape-of-models)
- [6. Inference: a craft made of two jobs](#6-inference-a-craft-made-of-two-jobs)
- [7. The notes on the table: the KV cache](#7-the-notes-on-the-table-the-kv-cache)
- [8. The optimizations: six answers to the same question](#8-the-optimizations-six-answers-to-the-same-question)
- [9. Inside the code: four close-ups](#9-inside-the-code-four-close-ups)
- [10. What you take away from the journey](#10-what-you-take-away-from-the-journey)
- [11. The questions that matter](#11-the-questions-that-matter)
- [References](#references)
## 1. What a machine learns
There is a useful way to arrive at LLMs starting from zero, and it passes
through three successive narrowings. Machine learning is the idea of not
writing the rules by hand: one chooses a function with many free parameters
and lets the examples fix them. A neural network is a particular family of
these functions, made of layers that pass numbers to one another: each layer
combines the input values with its own parameters and passes the result to the
next. A language model is a network trained on a single task, seemingly
modest: given a text, assign a probability to every possible immediate
continuation. The three narrowings, though, need to be filled in, because each
hides a choice, and choices explain more than definitions do.
**Where the correction comes from.** The classic taxonomy of machine learning
distinguishes three ways of learning, according to who tells the machine it is
making a mistake. In **supervised** learning every example carries the right
answer attached (this photo is a cat, this email is spam): one learns a
correspondence. In **unsupervised** learning the labels are absent, and one
looks for the structure the data already have on their own: groups,
regularities, ways of compressing. In **reinforcement** learning there is not
even the right answer: there is an environment, there are actions, and every
so often a reward arrives; one learns the strategy that maximizes it (it is
the family of programs that play Go). LLMs are born from an economically
decisive variant of the first family, **self-supervision** (self-supervised
learning): the text serves as its own label, because the continuation to
guess is, in the text, already there. No human labeller, and every existing
text becomes training material: it is the choice that unlocks scale.
The continuation to guess, moreover, is not a whole word: it is a **token**,
the unit the model actually sees. The text is broken by a tokenizer into
pieces taken from a vocabulary of tens of thousands of entries; a common word
is one token, a rare word or a proper name is broken into several tokens.
**Why not words.** It seems a gratuitous complication, given that words are
already there; and instead it is a tight choice among very concrete
constraints, worth seeing, because they all come back later.
The hard constraint is that the vocabulary must be **closed and finite**, and
decided before seeing any text: the model's output is a list with one cell per
entry, and a list is sized in advance. Words, however, are an open set: proper
names, neologisms, acronyms, typos, languages in which words compose at will.
Working with words one always ends up keeping in the list an "unknown entry"
cell, which in front of a real text is a surrender. With sub-word pieces it is
not needed: a few tens of thousands of entries are enough to write anything, by
composing. In the variant that breaks all the way down to bytes there is, by
construction, nothing inexpressible, not even a never-seen language or a piece
of binary file pasted in by mistake.
Once this is settled, it remains to decide how big the pieces should be. The
two extreme choices contradict each other, and each breaks what the other
fixes. Cutting fine, one character per token, gives a tiny vocabulary and no
out-of-list entry; in exchange it lengthens the sequences beyond measure, and
length is the most expensive quantity of this craft: we will see later that
the cost grows with the square of the length. Cutting coarse, one word per
token, keeps the sequences short but swells the vocabulary; and a large
vocabulary is paid for, because the model dedicates parameters to each entry,
to recognize it when it comes in and to be able to propose it when it comes
out. The bill thus grows together with the entries: a million entries would
mean billions of parameters spent on that alone. Between the two extremes lies
the whole space of sub-word pieces, and that is where usage has settled.
There is, finally, a statistical reason, and it concerns what one manages to
learn. An entry met three times in all of training remains badly estimated,
however one represents it; by breaking up, the frequent pieces receive very
many examples, and morphology is shared among different words (a prefix, a
suffix, a root), so that a never-seen word is not a wall but a composition of
known pieces. The vocabulary, after all, is designed by no one: it is derived
from the data, and the criterion is compression. The most widespread recipe
starts from the individual characters and repeatedly merges the most frequent
adjacent pair, until the desired size is reached; it is a compression
algorithm from the nineties, readapted to text. The pieces that come out do
not coincide with the units of meaning a linguist would recognize, do not
respect the grammar of any language and need not do so: they are the cuts that
make the text shorter, and nothing more.
The advantages have a price, and intuition pays it. Hence the first pair not
to be confused: the model does not reason in words, it reasons in tokens, and
many oddities (the counting of letters, the "per token" prices, the speed of
generation) are understood only from there.
**From numbers to points in space.** A token, just out of the tokenizer, is
only a vocabulary index, and an index has no meaning: entry 4,812 is no more
similar to 4,813 than to 40,000. The network's first move is therefore to turn
each token into an **embedding**: a list of thousands of numbers (a vector),
that is, a point in a many-dimensional space. It is easy to imagine a
dictionary compiled by someone, and it does not exist: the table that assigns
to each vocabulary entry its point is itself a block of the network's
parameters, random points at the start, arranged by training like everything
else. The idea has a history of its own. Before transformers, small networks
were trained only to predict nearby words, and the by-product turned out to be
more precious than the task: the vector representations that the network had
built for itself in order to succeed (word2vec, 2013). And it is here that
meaning begins to have a geometry: in the right representation, words used in
similar ways end up close, and the **directions** capture properties. The
classic example: the shift that leads from "king" to "queen" closely resembles
the one that leads from "man" to "woman"; in symbols, v(king) − v(man) +
v(woman) ≈ v(queen), and it is an arithmetic that on embeddings can really be
verified. It seems a miracle and is not: training rewards the representations
that compress, and if thousands of contexts treat "king" and "queen" with the
same scaffolding of sentences, except for gender, the most economical solution
is to give the two tokens almost the same point, separated by one and the same
direction. Meaning, as the network sees it, is use; the geometry is only the
most compact way to write it. It is the first appearance of an idea that will
return: in the right representation spaces meaning becomes geometry, and
geometry can be measured, compared, even pushed.
The model's input therefore lives in this space, one point per token; and so
does the output, as we will see shortly: the point the network arrives at at
the top is compared with the points of the vocabulary entries, and it is from
that comparison that the probabilities are born. From end to end, in short,
the model works on coordinates; only at the very last does a comparison turn
the geometry into a list of percentages.
**The shape of the function.** Each layer of the network, when it is
traversed, does just two things. The first is linear: to weight and sum, the
input values multiplied by the parameters (in bulk it is a multiplication of
matrices, the operation for which GPUs were born, and which will keep coming
back). It is worth seeing it written once, in miniature, because it is the
brick of all the rest:
```
[ y1 ] [ w11 w12 ] [ x1 ] y1 = w11·x1 + w12·x2
[ ] = [ ] · [ ] →
[ y2 ] [ w21 w22 ] [ x2 ] y2 = w21·x1 + w22·x2
```
two inputs, two outputs, four weights: each output is a weighted mix of all
the inputs, and the weights w are the parameters that training will fix. In
real layers x has thousands of components and W billions of cells, but the
operation is exactly this. The second is a **non-linearity**, in the jargon
the activation function: a fixed function, deliberately trivial, applied
number by number. The two things stand in a row, not side by side: first the
weighting, then the function on the result. In the miniature above the true
output of the layer is not y₁ but f(y₁) = f(w₁₁·x₁ + w₁₂·x₂), and so for each
component: the weighted sum is the argument of the non-linearity, and what
comes out of it goes to the next layer. Layer upon layer a stack forms, and it
is worth fixing right away the image with which the text will always look at
it: a building, one layer per floor. The tokens enter on the ground floor,
each floor works on the output of the one below, and traversing the network is
climbing to the top. The functions one meets in the names each deserve a
definition, because they are all simpler than their fame:
- **ReLU** (rectified linear unit): if the number is positive it passes
unchanged, if it is negative it becomes zero. That is all: max(0, x). A
switch, at the cost of one comparison.
- **GELU and SiLU**: the same idea with a rounded elbow. Instead of zeroing
the negatives all at once, they make them fade gently towards zero (SiLU,
for instance, is x·σ(x), with σ the sigmoid that slides from 0 to 1); the
smooth transition helps the derivatives not to die out.
- **softmax**: a tool apart, which inside the floors does not appear: it takes
a list of arbitrary scores and turns it into percentages that sum to 1,
amplifying the differences exponentially (the highest score
takes far more than its proportional share). In symbols: softmax(xᵢ) =
exp(xᵢ) / Σⱼ exp(xⱼ). It is needed wherever scores must be turned into
probabilities, and the places are two: at the top of the stack, once only,
to choose the next token; and, as we will see, inside attention.
It seems a department of details and is instead the point of everything:
composing linear functions still gives a linear function, so a hundred layers
of weightings alone would be equivalent to a single layer, and depth would be
an accounting illusion. It is the non-linearity that makes depth real, and
with it the capacity to represent what is not proportional: thresholds,
exceptions, interactions between distant words. Which non-linearity to use, on
the other hand, is an almost prosaic choice, with three criteria: to break the
linearity, to have derivatives that do not die out (why it matters is seen in
training, in a moment), to cost almost nothing, since it will be evaluated an
astronomical number of times. Inside the floors, then, the switch and the
elbow work, number by number, on every floor; the softmax is not the
non-linearity of any floor. It appears at the top of the stack, where it
closes the loop, and here it should be said where the scores it transforms
come from, because there is a jump that is usually glossed over: the last
floor does not hand over a score per entry, it still hands over a point in the
space of representations, thousands of coordinates like those of the
embedding. The scores are born from a comparison, the dot product between that
arrival point and the vector of each vocabulary entry: how much the arrival
point points in the direction of each entry. And in many models the table used
for the comparison is the same as the input one, reused in reverse (in the
jargon, weight tying): the map that on the ground floor brings the entries
into the space, at the top measures the closeness of the arrival point to each
of them. The rest is done by the softmax, from those scores, one per entry, to
the percentages that sum to 1; and going back from the point to the token
requires no more geometry: one chooses a cell of the list, and the position of
the cell is already the token, no coordinate to invert. It is there that the
network becomes the function promised at the start, text in, probability
distribution out. In the single pass there is nothing mysterious,
multiplications and sums plus a switch: the capacity lies in the accumulation,
dozens of layers and billions of parameters.
The "just two things" of a moment ago were two out of arithmetic honesty: it
is from those that the result comes out. Alongside them, however, works a third
ingredient, minor on its own and indispensable for the whole: a
**normalization**, which before the weighting brings the numbers of each token
back onto a common scale. It does not change what the layer computes, does not
put different tokens in communication and returns as many numbers as it
received; and the parameters it carries are a trifle compared with those of the
matrices. It serves to prevent the values, as one climbs from floor to floor,
from swelling or dying out, and why it is indispensable will be seen by looking
at training: it is the first of the two inventions that make tall stacks
governable.
And the numbers that one layer passes to the next have a name that will serve
later: the **activations**, the living part of the computation alongside the
fixed parameters.
**How much goes in and how much comes out.** Saying that the model takes a text
and returns probabilities is not enough: one needs to know in what shapes,
because it is from the shapes that one understands what can vary and what
cannot. The text, broken by the tokenizer, is a row of tokens, and on the
ground floor each is replaced by its own embedding, the point in the
many-dimensional space seen above: a list of some thousands of numbers. To
reason about shapes three names are needed, and it is worth taking them now:
**T** is how many tokens stand in a row, **d** how many numbers make up the
point of a token, **V** how many entries the vocabulary has. The input is
therefore not a vector, but a stack of vectors: T rows of d numbers each.
Of the two dimensions of the input, only one varies. The d is fixed by the
architecture and never changes: neither from one request to another, nor
climbing from floor to floor, because each floor receives T rows of d and
returns just as many. It is not a whim: we will see that each floor adds its
own result to what had entered it, and two things add up only if they have the
same shape. The T instead is the length of the text at that moment, and
changes continuously.
Which raises a legitimate question. A fixed-size network, if one changes the
length of its input, no longer works: the matrices do not match. How does this
one digest, with the same parameters, a prompt of ten tokens and one of ten
thousand? The answer lies entirely in one fact: **no parameter matrix has a
side of length T**.
```
input T × d T tokens, d numbers per token
each floor T × d → T × d the shape does not change on the way up
at the top T × d → T × V comparison with the vocabulary entries
(in inference only the last row is needed)
the parameters: embedding table V × d
inside each floor d × d, or d × (multiple of d)
final comparison V × d (often the same table)
```
The matrices, as one sees, are made of d and of V, never of T. And the reason
can be read in the two things a floor knows how to do. What works **one token
at a time** applies the same weights to each of the T rows, and how many there
are does not matter to it: they are T repetitions of the same computation. What
instead **mixes the tokens with one another** does produce a table as big as T
by T, but that table is computed on the spot, it is not made of parameters: it
is born from the numbers of that text and dies with it. The parameters, in
short, are indifferent to length; what grows with T is the work to be done and
the memory of the intermediate values, which is then the whole story of the
costs in these pages. In a classic stack a floor weighs roughly ten times d²
parameters, and the billions one speaks of are that number multiplied by the
floors: a matter of width and of height, never of the length of the text.
It is worth seeing the two pivotal multiplications with the rows and columns in
their place, because it is there that the flow of the computation becomes
concrete. The first is any floor; each row is a token, and one follows it on
its own:
```
input floor weights output
┌── d ──┐ ┌─── d ───┐ ┌── d ──┐
t1│ · · · │ │ \ │ t1│ o o o │
t2│ · · · │ · │ \ W │ = t2│ # # # │
t3│ · · · │ │ \ │ t3│ o o o │
└───────┘ └─────────┘ └───────┘
T × d d × d T × d
row t2 of the output arises ONLY from row t2 of the input;
each # is a weighted sum as in the miniature just now (the row
of t2 by a column of W). The rows do not talk to one another:
making them talk is the task of section 4.
```
The second is the move at the top of the stack, and uses a single row: the
last, the one that carries the prediction of the next token. It goes to meet
the vocabulary table and out of it comes a list as long as the vocabulary, one
score per entry:
```
last row vocabulary table scores
of the output (one column per entry)
┌── d ──┐ ┌────── V ──────┐
│▓ ▓ ▓ ▓│ · │ v1 v2 ... vV │ = │p1 p2 … pV│ → softmax
└───────┘ └───────────────┘ └──────────┘
1 × d d × V 1 × V
one score per entry: how much the arrival point "points" towards
each one. The table is the V × d of the input, reused in reverse
(the weight tying just now). In inference only this row is needed;
the other T−1 predict tokens that are already in the text, and are
not computed.
```
A single door could let length in among the parameters, and it concerns a
detail not yet spoken of: the model must somehow also be told in what order the
tokens stand, because a row of points, on its own, does not say which
comes first. If that information is put in a table with one row for each
possible position, then a side of length T among the parameters does exist, and
with it a rigid ceiling written into the weights. It is one of the reasons why
the more recent recipes prefer to compute the order, and in section 4 we will
see how. A ceiling, in any case, remains: it is the **context window**, the
maximum T that a model declares it can bear, and it is not a limit of shape but
of cost and of training.
There remains the output, and here the count must be done in full, because so
far the text has left it implicit. At the top there does not come out a list of
V percentages: there come out **T** of them, one for each position, each as
long as the vocabulary. The model, that is, does not only predict what comes
after the last token: it predicts, in one shot, what comes after each of the
tokens it has before it. In inference only one is needed, the last: the others
predict tokens already in the text, and the engines are careful not to compute
them. In training all are used, and it is what makes it sustainable to have a
machine read thousands of billions of tokens.
The first round, then, does not start from empty: the input tokens are those of
the prompt, and T is its length. Then the text lengthens by one row at a time,
and T with it. If there really were nothing to start from, one seeds with a
special start token: T equals 1, and never 0.
**What is made to descend.** "Learning" must become a number, otherwise there
is nothing to optimize: a **loss function** (the loss, as it is called almost
everywhere), which measures how far the prediction is from the truth. The scene
of the measurement is concrete: one takes a real text, runs through it position
by position, and at each one asks the model for its prediction. Here there is a
misunderstanding to clear up at once: the model does not produce a single token
to put next to the true one to say right or wrong. Its prediction, **for each
position**, is a vector, a list. The size of this list is that of the entire
vocabulary: one element for every possible entry (token), and in each element a
percentage, with the percentages summing to 1. Of lists, therefore, there come
out T, one per position, and it is better to look at one at a time. The true
token, which in the training text is already there, during this phase serves
only to establish **which cell to go and read** among those of the output
vector at that position:
```
text: "the cat sleeps on " true token, from the text: "cushions"
model output:
cushions 0.32 ← only this one is read: p(true token) = 0.32
rugs 0.25
beds 0.18
… (all the other vocabulary entries; the sum is 1) …
```
No direct comparison between two tokens, then: a reading. The number found in
the cell corresponding to the token's position in the vocabulary says how much
confidence (probability) the model attributed to the continuation of the text
materializing precisely with that token.
That number, however, as it is, is not yet of use: the loss must be **a single
one** for the whole text, and the probabilities of the individual positions do
not add up, they multiply. The probability that the model assigns to a whole
text is in fact the product of the probabilities it assigns to each of its
positions, one after another: thousands of numbers smaller than 1 multiplied
together, a result that plunges below any representable threshold. The
logarithm serves exactly this, and is not an ornament: it turns that product
into a sum. With its sign flipped, to have it positive, it gives the quantity
one really works on, the **surprise** of a single position:
```
surprise = −log p(true token)
```
It is zero when the prediction was certain and right (p = 1), and grows
without limit the more the model was wrong. The loss is the average of these
surprises over all the positions, and it is the canonical choice for language
models: in the jargon, the cross-entropy. It is worth seeing it at work on a
whole text, one row per position. They are different steps of the same reading,
not alternatives of the same step:
```
training text: "the cat sleeps on cushions"
pos. the model has before it true token p(true token) surprise
1 "the " cat 0.05 3.0
2 "the cat " sleeps 0.20 1.6
3 "the cat sleeps " on 0.45 0.8
4 "the cat sleeps on " cushions 0.32 1.1
────────
loss = average = 1.6
```
Each row is a scene like the one before: the whole vocabulary in output, a
single cell read. The loss is the average of the right-hand column, and nothing
else. One reads too, in that column, the character of the cross-entropy: it
punishes self-assured errors disproportionately. A position in which the model
gave the true token 0.1% is worth 6.9 on its own, as much as some seventy
positions guessed at 90%. It is a deliberate property, not a side effect: the loss
must reward **calibration** as well as getting it right, and a model that gives
60% to the right token while distributing the rest sensibly is worth more than
one that fires off certainties and every so often is badly wrong. The other
requirement is less visible but just as binding: the loss must be
differentiable, because the whole of training, as we will see, is made of
derivatives.
With the mechanics fixed, one can look at two things that were already there.
The first: this surprise is Shannon's, the same measure of information as
[Maxwell's demon](/en/maxwells-demon/): not a resemblance, the same quantity,
and a language model is trained by minimizing it. The second says what it
means, at bottom, to "learn the language": since the surprise is the logarithm
of the probability with its sign flipped, minimizing the average surprise is
the very same operation as maximizing the probability the model assigns to the
real text. The task, rewritten without jargon: to make the written world as
little surprising as possible.
**With what numbers it is measured.** The choices above define the system; to
describe it and compare it a few quantities suffice. Three define it: how many
**parameters** it has, how much **data** it has seen (measured in tokens), how
much **computation** it took to train it. They are not independent: an empirical
regularity by now robust (the scaling laws) says that quality improves
predictably when the three grow together, and it is the bet on which the last
decade of ever-larger models has been built. To judge, on the other hand, none
of the three suffices: one measures the loss on text **never seen** in
training, because a function with billions of parameters has every temptation
to learn by heart (overfitting), and generalizing is exactly what memory does
not give. The same loss on new text has a more readable translation, the
**perplexity**: among how many equiprobable alternatives the model is
hesitating, on average, at each token (perplexity 8, it hesitates among eight;
lower is better). The translation, after all, is literal: perplexity =
exp(loss), the exponential of the average loss, that is, the logarithm of a
moment ago undone. On the cat sentence, loss 1.6 means perplexity 5: as if at
each word the model were choosing, blindly, among five equivalent
possibilities. And since predicting the text well is not yet "doing things",
above all this stand the **benchmarks**, shared batteries of tests (questions,
problems, code) with their merit (they make comparable what would not be) and
their known ailments (ending up inside the training data, and models trained to
the test more than to the craft). The honest comparison declares the quantities
and the measures together: the parameters alone are an engine displacement, not
a speed.
Generating text, then, is calling this function in a loop: one gives it the
text so far, it chooses the next token, one appends it, one starts again. With
the shapes from a moment ago alongside, the round can be read in full:
```
round 1 text: "the cat sleeps on " T = 4 → row 4 → "cushions"
round 2 text: "the cat sleeps on cushions" T = 5 → row 5 → ","
round 3 text: "the cat sleeps on cushions," T = 6 → row 6 → " and"
```
It is worth looking closely at what passes from one round to the next, because
it is less than one imagines: **only the text** passes. The token just chosen
is appended and at the next round re-enters from the ground floor like all the
others, a vocabulary index that becomes an embedding, indistinguishable from
those of the prompt; no trace remains of the previous round, no intermediate
result, no hidden state. The model, every time, re-reads everything from
scratch. It is an evident waste, and it will be the first place where inference
will go to save.
It is worth pausing too on the word "chooses", which hides a step. The network
does not choose: the network ends with the distribution, and there its work is
finished. Taking a cell is a separate step, applied from outside, with a policy
of its own: one can always take the entry with the highest percentage, or draw
by lot respecting the percentages, letting a less obvious candidate through
every so often. It is the reason why the same question does not give the same
answer twice. The detail of how one draws has a craft of its own, which there
is no need to open here: it moves no bytes, occupies no memory and costs a
negligible fraction of the round. It is the only piece of the chain where the
cost does not dwell.
All the power and all the cost of LLMs lie instead inside this loop. It is here
that the word "generative" finds its precise meaning, stripped of marketing: a
classifier produces a label (this photo is a cat, this email is spam) and stops
there; a generative system learns the distribution of the data themselves, and
from a distribution one can draw something new, that was not written anywhere.
An LLM is the most literal case: its output that counts is a probability
distribution over the next token, and generating is drawing one, appending it,
repeating; it holds for text as for images or audio, the space changes, not the
idea. A language model is a function that, given a text, assigns a probability
to every possible next little piece of word: the rest of the text is the story
of how much it costs to evaluate it, billions of times a day.
The map of the whole piece. The function at the centre is always the same, with the parameters fixed once and for all; it is the text that lengthens, one token per round. All the power and all the cost of LLMs dwell inside this loop: the rest of the text tells how much a round costs, and how one makes it cost less.
## 2. Training: the descent and the three phases
It remains to say how they descend, the parameters, towards the minimum loss.
First, though, the link that holds everything together and risks going
unnoticed: the probability on the true token is not a judgement given from
outside, it is an output of the function itself, that cell of the distribution
which the softmax produces at the top of the stack. With the text fixed, the
cell depends only on the parameters; and it is worth, from here on, seeing each
parameter as a **knob**: the effect of turning one climbs the building, floor
after floor, up to the percentages at the top. The complete chain:
```
turn a knob → the activations of the floors above move
→ the scores at the top of the stack move
→ the softmax redistributes the percentages
→ p(true token) rises or falls
→ the surprise falls or rises
```
The true token, in all this, is not touched: it is in the text, and continues
to do its one and only job, to point out the cell to read. What moves are the
numbers in the cells; and since the percentages sum to 1, raising the cell of
the true token necessarily means taking probability away from the others.
Learning, seen from here, is decanting probability towards the right cells, one
hair of a knob at a time.
With the text fixed, therefore, the loss is a function of the parameters alone,
and the question "by how much does the surprise move if I turn this knob by a
hair?" is well posed for each of the billions of knobs. The answer has a name,
the **partial derivative**, and under the name there is a ratio between two
variations, which can be read in numbers. Take the miniature of section 1, with
the inputs fixed and, let us say, a loss that at the moment equals 2.000;
move only w₁₁ by one hundredth, leaving all the other knobs still, and redo the
computation:
```
w₁₁ = 0.500 → loss = 2.000
w₁₁ = 0.510 → loss = 2.003 variation: +0.003 for +0.010
partial derivative of the loss with respect to w₁₁ = 0.003 / 0.010 = 0.3
```
How much loss for how much turn: it is all there. The value says how much that
knob counts relative to the others, and the sign says the direction: positive
means "raising it, the loss rises", so to make it fall that knob must be
lowered. "Partial" adds no mystery: it means only that one moves one at a time,
holding all the others still. It should be said at once that no one computes
this way: shifting billions of knobs in turn and redoing the computation each
time would be the very definition of the impossible. But the quantity sought is
exactly this, and it is worth keeping it before one while looking at the real
procedure, which obtains the same numbers by an incomparably shorter road.
The miniature also says what they depend on: turning w₁₁ moves y₁ in proportion
to x₁, the input that weight multiplies, so a weight at work on a large input
shifts the output more than one at work on near zero, a detail that in a moment
will become the apportioning of blame. The mechanism of the descent, at this
point, has intimidating names and a simple idea, and it is worth seeing it as a
round of three steps, repeated billions of times:
1. **Forward: the prediction.** A block of text is shown to the network; for
each position the network produces its distribution over the next token, and
the loss function reads the cell of the token that is really in the text:
the average surprise over the whole block is the loss.
2. **Backward: backpropagation.** For each of the billions of parameters
exactly that ratio is needed, the partial derivative from a moment ago. One
at a time is out of the question: it would be billions of forwards for a
single step. Backpropagation obtains them **all together**, with a single
descent from the top to the ground floor, and its cost is the good news of
the whole procedure: those billions of numbers cost roughly as much as two
forwards. The set of all these derivatives, one number per knob and
therefore a list as long as the whole model, has a name that explains the
next step: the **gradient**, the direction, in the space of the parameters,
in which the loss rises most steeply.
3. **Step: gradient descent.** If the gradient indicates where the loss rises,
one moves the opposite way: each knob turns by a small step against its own
blame. The width of the small step (the learning rate) is the permanent
compromise: steps too long overshoot the valley, steps too short never reach
it, and every step tramples a little of what had been learned before. In
symbols, the whole step is: θ ← θ − η·∇L (θ the parameters, η the learning
rate, ∇L the gradient of the loss); the whole of training is this line,
repeated.
The building of the stack and the two directions of travel. Up on the forward: the tokens enter on the ground floor, each floor works on the output of the one below, and at the top the softmax delivers the percentages, where the surprise is read on a single cell. Down on the backward: the blame starts from the top and at each floor is rescaled by a factor; how much those factors count is told by the numbers just below.
How backpropagation manages to obtain all those derivatives in a single shot
deserves to be seen, because it is also the explanation of two or three things
that otherwise seem whims. A knob of the lower floors does not touch the loss
directly: it touches it through everything above it. It moves the activations
of its floor, which move those of the next floor, and so on up to the cell at
the top. Its overall effect is the **product** of the local effects met along
the way, and for a three-floor building it fits in one line:
```
∂L/∂w = ∂L/∂a₃ · ∂a₃/∂a₂ · ∂a₂/∂w
how much how much how much how much
the loss the loss floor 3 floor 2
feels feels feels feels
the knob floor 3 floor 2 the knob
```
It is the chain rule, and it should be read from right to left like a journey:
the knob moves its floor, its floor moves the one above, the one above moves
the loss. The point that makes everything work is that **every factor is
local**: it concerns one floor only, and to compute it it is enough to know
what that floor just did in the forward pass. That is why the count can be done
backwards once only. One starts from the top with the first factor; descending,
at each floor traversed one multiplies it by the factor of that floor. What
arrives at any floor is already the product of all those above: that floor need
only use it for its own weights and pass it, rescaled, to the one below. No
path is ever repeated, and it is from here that the cost of only two forwards
comes.
The historical jargon calls this accounting **credit assignment**, assigning to
each weight its share of merit and of blame, and "blame" is the right image,
because the apportioning is not fair on principle: whoever contributed more to
the error receives more of it. It is the detail of the miniature come to the
point: a weight that multiplied a large input moved the output more than one
that multiplied near zero, and the derivative records exactly this.
And it is the same line that explains why section 1 asked for derivatives that
"do not die out". On a real stack that product does not have three factors: it
has one per floor, dozens in a row. The blame does not pass whole from one
floor to the next, it is multiplied at every traversal, and chains of
multiplications are merciless with factors far from 1:
```
blame at the top = 1, stack of 30 floors
factor 1.0 per floor → 1.0³⁰ = 1 arrives at the bottom whole
factor 0.9 per floor → 0.9³⁰ ≈ 0.04 4% of it arrives
factor 0.5 per floor → 0.5³⁰ ≈ 10⁻⁹ dies out along the way
factor 1.1 per floor → 1.1³⁰ ≈ 17 the opposite ill: it explodes
```
Below 1 the blame dies before reaching the bottom and the lower floors are left
without guidance; above 1 it explodes and overwhelms everything: the stability
of a deep stack wants factors close to 1. The factor of each floor depends also
on the derivative of the activation function, and here are the "almost prosaic"
criteria of section 1 at work: the sharp switch of the ReLU passes the blame
whole on the positives but zeroes it in bulk on the negatives (clear road or
wall); the rounded elbow of GELU and SiLU keeps the factor small but never
zero, and a way to reach the bottom always stays open.
And it is here that the normalization named in section 1 earns its keep. By
bringing the numbers back onto a common scale at each floor, it keeps the
activations away from the extremes, and with the activations the factors of the
backward pass: the chain stays in a zone where it neither dies out nor
explodes. It adds no mathematics, teaches the model nothing, weighs almost
nothing in bytes; it is hygiene, and without hygiene tall stacks simply do not
train. A second invention will serve the same purpose, and will arrive with the
2017 architecture.
The whole round fits in a single image: a landscape. Each point is a complete
setting of the knobs, that is, one of the possible models; the elevation of
that point is the loss that model records on the text; taking a step is turning
all the knobs a little, moving to a nearby point. The one who walks is the
model itself, and the path is the training. The landscape has billions of
dimensions and thick fog: one does not see the valley, one only feels the slope
underfoot (the gradient), and one trusts that descending locally, one small
step at a time, leads somewhere. That it works was not at all obvious, and it is
largely an empirical discovery: there is no need to find the lowest point in
the world, a good valley is enough. No step of this procedure is intelligent;
learning is making a number descend, measuring the surprise and turning each
parameter by a small step in the direction that reduces it, thousands of
billions of times.
A practical detail completes the picture: the step is not computed on all the
available text, but on one block at a time (the batch), and the slope felt at
each step is therefore a noisy estimate of the true one; it is the reason for
the full name of the procedure, **stochastic** gradient descent. The noise is
not only a defect: it shakes the descent just enough not to stop in the first
hollow of the landscape.
The guiding thread, here, presents the steepest bill of the whole text: each
training step moves **all** the bytes of the model at least twice (forward to
predict, backward for the blame), plus the surrounding bytes that the
**optimizer** keeps for each parameter (the bookkeeping of the descent, which
for each knob preserves a few service numbers, like the average of the recent
steps). It is the physical reason why training occupies entire datacentres for
months, and it is the colossal expense paid **once**. What comes out of it is a
block of numbers: the shelves of the library, filled once and for all.
"Training", in the singular, is then a simplification: for a modern LLM the
phases are at least three, and they answer different questions. **Pretraining**
is the self-supervision of section 1 pushed to the maximum scale: it decides
what the model knows, and dominates the costs by orders of magnitude.
**Supervised fine-tuning** takes that model back up and corrects it on curated
examples of dialogue: it decides the form, that is, that a question is
answered, instead of continuing it with other similar questions as a pure text
completer would do. Learning from **preferences** (the RLHF family: answers put
side by side, human judgements or those of a judge model turned into a reward
with the tools of reinforcement) decides what the model chooses when the
admissible answers are many: tone, prudence, style. Same descent mechanics in
all three, incomparable budgets: knowledge costs like an infrastructure,
behaviour like a project.
From that moment the model no longer learns. Every answer, however brilliant,
leaves it identical to how it was: the conversation lives in the context, which
vanishes at the end of the session, not in the parameters. It has already been
told, as "empty memory", by [the energy cost of a language
model](/en/02-articolo/); here it is enough to fix its architectural
consequence: everything that follows, inference with its phases and its
optimizations, operates on **immutable** parameters. It is a constraint, but
also the advantage that makes inference optimizable: one can compress, copy,
distribute and cache what never changes. Training fixes the parameters once and
for all: from then on the model no longer learns, it just answers.
## 3. Interlude: the parrot and the library
"It just answers" opens a question that has divided the field for years: what
is it doing, exactly, while it answers? In 2021, when models were already
fluent but not yet in everyone's pocket, a much-discussed paper gave the
sceptical position its definitive name: **stochastic parrot**. The thesis is
clean: a system trained to minimize the surprise on text (section 1)
manipulates forms, not meanings; it stitches back together fragments seen in
training following the probabilities, without any anchoring to the world the
text speaks of. And the thesis seemed obvious for an honest reason: it is the
literal description of the recipe. In the loss function there is no term for
truth nor for understanding; there is how often the words follow one another.
Whoever coined the label was not making a caricature: they were reading the
definition. Moreover it warned of a real risk, mistaking fluency for
understanding, which remains sensible even if the thesis turned out to be
wrong.
Over time, though, the evidence on the ground has opened cracks hard to ignore.
The one closest to the thread of this text is a count of bytes: the training
text weighs tens of terabytes, the parameters a couple of orders of magnitude
less. A literal parrot would not fit: to predict well with so little space one
has to **compress**, and to compress at that level means finding regularities
(grammar, facts, relations, styles) that hold even on text never seen. To this
have been added behaviours that stitching explains badly: problems solved in
combinations never appeared in training, and learning from context (in-context
learning: with parameters strictly still, the model learns a new task from the
examples in the prompt, something none of the previous sections explicitly
foresaw obtaining).
The most interesting crack, however, does not come from behaviour: it comes
from looking **inside**. A by now classic experiment trained a small
transformer only on sequences of Othello moves, never a board: in its
activations one finds, readable and manipulable, a representation of the state
of the board, which the model built for itself because it compressed the moves
better. And in large language models the field called mechanistic
interpretability has learned to isolate the **features**: not single "neurons"
nor physical zones of the chip, but directions and sparse combinations in the
space of the activations that light up when generation touches a concept (a
place, a tone, an insecure coding practice), often independent of the language
and of the surface formulation. It is the precise version of the intuition that
"different zones activate depending on the subject": true, provided one adds
that the zones are geometries distributed in the reading table, not departments
of the library. And for the learning from context of a moment ago there even
exists a mechanical candidate: the induction heads, pairs of attention heads
that seem to implement "I have already seen A followed by B; now I see A again,
I propose B"; with the due caution, because saying that a head "does" something
is an after-the-fact reading of a pattern, not a programmed function.
If a feature can be isolated, it can also be pushed. Amplifying it or turning
it off by hand, during generation, shifts the behaviour predictably: it was
shown with a deliberately theatrical experiment (a model with the feature of a
famous bridge kept switched on, which slipped the bridge into every answer),
and in the laboratory directions are isolated for whole behaviours, sycophancy,
the refusal to answer. It is the concept at the base of **steering**:
intervening on the activations, not on the weights nor on the prompt. In the
lexicon of this text: acting on the reading table, leaving the shelves intact,
the parameters/activations pair seen at the start put to work. And it is no
longer only laboratory: there is already someone who exposes it as a
command-line option in a real inference engine, one direction per layer in a
half-megabyte file, to adjust verbosity or willingness on certain subjects
without touching the weights. It should be said with the same honesty: they are
young tools, the isolated features are a minimal fraction of those that exist,
and from being able to push a behaviour it does not follow that one can read
everything.
Why then do two parties remain? In part because half the quarrel dwells in the
word "knowledge": if to know requires experience of the world, intention,
anchoring to truth, the parrot remains a parrot by definition, and no
experiment on the activations scratches it; if internal structure that
generalizes is enough, lets itself be isolated and lets itself be manoeuvred,
the evidence accumulates on the other side. And in part because the stakes are
different: those who fear anthropomorphization and hype have serious prudential
reasons (fluency deceives, and marketing raises the bid); those who work inside
the models see structures that the parrot metaphor no longer covers. The
position of this text is minimal and declared: the library is too small to
contain what it has read, and this alone forces something more interesting than
repetition; whether one calls it knowledge is a choice of vocabulary, that
there is structure is by now an observation.
There is one last effect of looking inside, and it is worth declaring, because
it accompanies all the rest of the text. Opening the machine produces two
opposite sensations, and they do not cancel out: they coexist. The first is a
kind of aridity: in there is no one; there are sequences of numbers, weightings,
the same iteration repeated billions of times, and every brilliant word comes
out of that loop. The second is its opposite: precisely because no one wrote
those structures line by line, finding them (the board, the features, the
directions that let themselves be pushed) shows an organization that training
made emerge on its own, that no designer dictated and that one is still learning
to read. The boundary between what one can explain and what has simply emerged,
in the shelves, is not sharp. Whoever opens the machine takes away both things,
disenchantment and wonder; and it is probably the sign of having really opened
it.
## 4. 2017: attention, or managing the context
To predict the continuation of "the dog bit the" one needs to know what scene
is being spoken of, who the subject is, what has already happened: the
**context** is needed. The previous architectures (the recurrent networks) read
the tokens one at a time, carrying along a summary that faded: the distant past
was lost, and above all the reading was sequential by construction, impossible
to parallelize well on the hardware that meanwhile was becoming ever more
parallel.
The 2017 paper, "Attention is all you need", proposes to overturn the setup: no
reading in a row, no summary dragged along. All the tokens of the sequence are
processed **together**, and each one decides on its own, dynamically, which
others to pay attention to, whatever the distance. The mechanism is called
attention and the intuition sits comfortably in the library: each token
formulates a question (its query), displays a label (its key) and carries a
content (its value). Where these come from should be said at once, because they
are not new ingredients: question, label and content are three weightings of
the token's point, of the kind seen in section 1, with three parameter matrices
that training fixes like everything else. One compares each one's question with
everyone's labels, the scores become weights, and each token receives a
weighted combination of the others' contents: a tailor-made summary of the rest
of the sentence. Written as in the paper, the whole operation fits in one line:
Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V
and read in pieces it is exactly the library: Q·Kᵀ compares each question with
all the labels (a matrix of scores, one number for every pair of tokens); the
division by √dₖ puts them back in scale; the softmax turns them into
percentages; the final multiplication by V takes the weighted average of the
contents. Three matrix multiplications and a softmax: the formula's fame is all
in the scale at which it is executed.
It is worth following it too with the shapes in hand, because it is there that
one sees the birth of the count that will govern all the second half of the
text. In come T rows of d, as at every floor; the three weightings derive as
many, a question, a label and a content per token:
```
input T × d
normalization T × d row by row, mixes nothing
Q, K, V T × d each, from the three weightings (matrices d × d)
Q · Kᵀ T × T the table of scores: every token by every token
softmax T × T each row becomes percentages that sum to 1
(…) · V T × d back to the input shape
```
That **T × T** in the middle is the object to keep an eye on: it is the only
thing, in the whole transformer, that is not made of d, and doubling the length
of the text quadruples it. The parameter matrices, around it, stay d × d and do
not notice. Note too that at the output the shape is that of the input, T × d:
it is what allows the floors to be stacked at will, and the result to be added
to what had entered.
That √dₖ deserves a pause, because it seems an arbitrary adjustment and is not.
The comparison between question and label sums a contribution for every
dimension of the space: the more dimensions there are, the more terms are
summed, and the more the scores spread out. They grow, typically, like the
square root of the number of dimensions, which is precisely √dₖ. Dividing by
that root puts the scores back where the softmax works well: without it, in a
wide space the scores would arrive already so far apart as to collapse the
percentages onto a single winner, and the attention, instead of distributing,
would nail itself onto a single token.
The operation is not executed just once, but with several "heads" in parallel,
and here a frequent misconception must be cleared away: the heads do not
multiply the cost. The point of each token is divided into slices: with h
heads, each works on d/h numbers, and on that slice does its complete round of
questions, labels and contents, with its T × T table; at the end the h outputs,
d/h long each, come back together to form d again. It is the √dₖ of a moment
ago that becomes clear here, incidentally: that dₖ is the slice, d/h, not the
whole d. Eight heads on eighth-sized slices cost as much as a single head on
the whole: one buys variety of viewpoints, not computing power. What is multiplied
by h are the score tables, which are h instead of one, and indeed it is on
their side that one will go to save. What they look at, then, is a more open
question than is usually told: the heads stand in separate subspaces **so that
they can** specialize, and some let themselves really be read (one follows the
agreement between subject and verb, another brings the pronouns back to their
antecedent), but many show no clean role, and a substantial share can be
removed without the model noticing. Stacked for dozens of layers and alternated
with blocks of pure computation, this is the transformer. The floor of the
building, seen up close, thus has two rooms: the attention, the only place
where the tokens talk to one another, and the block of computation, the old
pair weighting and non-linearity at work on each token on its own. In front of
each of the two, in the modern recipes, stands the normalization of section 1:
it works on one row at a time, mixes nothing and hands back the shape it
received, T × d, with a handful of parameters, of the order of d against the d²
of the matrices around it. It is also the reason why, further on, one will see
that no one dreams of compressing it. And alongside the two rooms runs a direct
path: the result of each is added to what was entering, and the part that need
not be changed continues intact, without having to be rebuilt floor by floor.
It seems a plumbing detail and is instead the second of the two inventions
announced in section 1, the one that together with the normalization makes deep
stacks trainable: the sum offers the blame of the backward pass a shortcut in
which the factor is worth 1, and the chain of multiplications of section 2
stops being merciless.
One detail must be told in full, because left half-told it forces one to trust:
attention, on its own, does not know the order of the words. It compares each
token with all the others, not a sequence: "the dog bit the man" and "the man
bit the dog" would produce the same comparisons. The order must therefore be
reintroduced, and the solution is in the spirit of section 1: the position too
becomes geometry. The first recipe gives the position a point in space
(computed with a fixed formula in the 2017 paper, learned like the embeddings
in many models that came after) and adds it to the token's embedding: the point
that enters on the ground floor says together what it is and where it stands,
and the network learns to read the two pieces of information from the same
coordinates. The modern variants move the trick inside the attention (in the
jargon, the rotary embeddings, RoPE): instead of marking the point at the
input, they rotate question and label by an angle that grows with the position,
so that the comparison between two tokens feels their distance, not the
absolute place; and it is the distance, for language, that is the information
that counts. The craft of encoding it at best is still being refined, but the
principle remains one: the position is not a tag hung on the token, it is
written in the point.
In LLMs a single rule is added, the causal mask: each token can look only at
those that precede it, never at the future. The reason must be told in full,
because it is not a formal precaution but what makes training economical. In
section 1 the text was run through "position by position", and one might
imagine a pass for each: it is not so. A single forward on a block of text
produces **simultaneously** the prediction of all its positions, and it is for
this that training on thousands of billions of tokens is thinkable. But if
every position sees the whole sentence, the one in turn also sees the token it
should guess, and the task becomes copying. The mask is the price that makes
the saving legitimate: all the positions together, each walled in its own past. Here
too a miniature is worth a thousand words: here are the attention weights of a
sentence of four tokens, after softmax and mask,
```
t1 t2 t3 t4
t1 [ 1.00 0 0 0 ] each row: how that token
t2 [ 0.33 0.67 0 0 ] distributes its attention
t3 [ 0.25 0.25 0.50 0 ] (and sums to 1)
t4 [ 0.10 0.20 0.30 0.40 ]
```
where the zeros above the diagonal are the mask at work (the scores towards the
future are set to −∞ before the softmax, which turns them into exact zeros).
This table, one number for every pair of tokens, is the central object of the
attention: it is worth fixing it well, because it is the one that in a moment
will grow with the square.
A floor of the stack, drawn from top to bottom for ease of reading (in the building of section 1, this route is climbing one floor). The attention is the only point where the tokens talk to one another: each query interrogates the keys of all, the scores weight the values, and the causal mask (the triangle) forbids looking ahead. The block of computation instead works on each token on its own. The dashed paths are the sums with the direct path: what need not be changed passes undisturbed, and it is what keeps the training of deep stacks standing. Dozens of these floors, stacked, make the transformer.
The reason the attention won, however, is not the elegance: it is that it is
computed in parallel. The comparison of all the questions with all the labels
is still it, the multiplication of matrices. The price is the table of a moment
ago: T × T comparisons, a cost that grows with the **square** of the length. A
thousand tokens are a million comparisons and no one notices them; a million
tokens are 10^12 comparisons, and there is no hardware that can bear them done
naively. It is worth being precise about that "no one notices them", because
the other room of the floor, the block of computation, grows only in proportion
to the length: on short contexts it is the one that dominates the count, and
the square remains a minor term. The overtaking happens further on, and the
more the context lengthens the sharper it becomes: it is for this that the
square is not an always-problem, but the problem of the long context. This
square is the character that will return in almost all the optimizations of the
second half of the text. The attention is the mechanism by which each token
looks at all the others and decides on its own which count; it won because it
is computed in parallel, and its price, the square of the length, is the debt
that inference has carried with it ever since.
## 5. The landscape of models
Before entering the craft of inference it is worth a photograph of what is
around: "model" is a single word for very different objects, and the
differences that count can be ordered along a few axes. The axes age slowly;
the answers, quickly (more on this at the end of the section). Six questions
are enough to frame almost every model one meets:
| The question | The answers, today | Why it counts |
|---|---|---|
| Who can have it? | closed (used only via a service); open-weights (downloaded, with a licence to read); truly open (weights, code and data) | without open weights, no machine of one's own |
| How much does it weigh? | from a few billion to thousands of billions of parameters | the memory to hold it: the first item of every bill |
| How does it work per token? | dense (all the parameters, always) or with experts (few at work at a time) | computation and bandwidth per token; more on this among the optimizations |
| What notes does it take? | full attention; compressed; selective | the cost of the long context (sections 7 and 8) |
| What craft can it do? | base; assistant; "reasoner" (spends tokens to think before answering); multimodal (images and audio too) | it is the legacy of the three phases of section 2 |
| In what form does it travel? | full precision; already compressed at birth; community quantizations | the bytes to download, hold and move |
On the first axis a clarification is needed, because the current lexicon
confuses: most of the models called "open" are **open-weights**, not open
source. Since 2024 there exists a formal definition of open-source AI (weights,
code and sufficient information on the data, with full freedom of use: it is in
the references), and almost no flagship model satisfies it: the weights are
downloaded, the recipe stays at home. It is not a quibble: with the weights
alone one can run, inspect, adapt; one cannot remake. And even among open
weights the licences are not all equal: some are full free licences, others
carry conditions of use to read before building on top. On the other axes a
regularity of the moment is worth noting: almost all the large open-weights
models of today are with experts, because it is the architecture that makes the
size sustainable, and compressed or selective attentions are spreading from the
leaders to the rest of the field.
With these axes in hand, the choice of the tailor-made engine that one will
meet shortly stops seeming a coincidence: DeepSeek V4 Flash is open-weights with
a free licence, with experts (few parameters at work per token), with the most
compressed notes of its category, distributed already compressed at birth, and
it bears aggressive compression well. Each of these items is a requirement for
the single machine: a closed model cannot even be downloaded; a dense one of
the same size would not fit the bandwidth budget; one with cumbersome notes
would burn the memory on the context alone. Not all models can be run at home,
and it is not a matter of effort: the choice of the model is half the choice.
A caution closes the photograph, and is itself a lesson: this taxonomy is a
snapshot of mid-2026, and the frontier moves by weeks, not by years; the names
and numbers cited in these pages will age soon, some perhaps have already aged.
The axes, though, move much more slowly than the answers, and it is the reason
why it is worth learning them. Understanding the basics and staying up to date
are not alternatives: the first thing serves precisely to make the second
possible, and this text tries to give the first knowing it cannot give the
second.
## 6. Inference: a craft made of two jobs
Here the text arrives at its centre, and it is worth entering it with the
reader's question: what happens between the sending of the prompt and the first
word that appears? Two different jobs happen, and almost all the engineering of
inference is born from the fact that they are really different.
The first job is the **prefill**: the prompt, which can be long (instructions,
documents, the previous conversation), is digested all together. The tokens are
all there already, so the attention can work in parallel on the whole sequence:
it is the regime for which the transformer was born, the matrices are large,
the GPUs full, the limit is the computing capacity. It is the waiting time
before the first word.
The second job is the **generation**: from there on the model produces one
token at a time, and each new token depends on all the previous ones. No more
parallelism on the sequence: a queue, one step per token. And at each step, to
produce a little piece of word, the model must consult **all** its parameters:
every shelf of the library, reread in full, to choose a syllable. The
computations to do on those numbers are few; the real work is carrying them
from the memory chips to the computing circuits. The energy account already
recalled gave the physical measure (inside a chip, going to fetch a number from
memory costs from a hundred to seven hundred times more than doing a
computation with it); here one sees the architectural consequence: the prefill
is limited by computation, the generation by the **memory bandwidth**. Two
different bottlenecks in the same answer.
At this point the on-the-ground proof announced at the start comes onto the
scene, and it deserves a proper introduction. **DwarfStar** (ds4) is an
open-source inference engine, MIT licence, that Salvatore Sanfilippo (antirez)
wrote in C with a radical choice: not a general-purpose engine, but a
**tailor-made** engine, for a specific model and for the hardware it has in
front of it. It is born with DeepSeek V4 Flash on unified-memory Macs, with the
declared aim of making it run well on a single machine; the perimeter, however,
has already widened, and in the direction that counts: not only to new hardware
(CUDA systems, consumer APUs), but to a model of another family, GLM 5.2, set
alongside DeepSeek. Specialization is the method, not the fence; that the fence
has moved without falling is the best proof that the bet holds, and how the
transplant happened can be read in the code, in section 9. Optimizing for one
model-hardware pair at a time is what allows one to pull out the maximum; what
use it is to know how much the maximum is, the final questions will say. A few
files of C with no dependencies, the kernels rewritten for each piece of
hardware, and inside, readable one by one, almost all the moves of this text:
asymmetric quantization, the compressed cache with its indexer, the cache on
disk, streaming from SSD, distributed inference, even the steering of the
interlude (it is ds4, the "real inference engine" cited there). It is the
reason it makes an ideal on-the-ground proof: in a general-purpose engine the
choices hide behind the abstraction, in a specialized one they stand naked in
the code; and a suite of test vectors, compared byte by byte with the answers
of the official API, keeps all the rest honest. The project's README is itself
a small lesson in inference, and it is in the references at the end. And a
transparency of the project deserves an echo here, because it pairs with the
opening of these pages: the authors openly declare that the code is written
with strong assistance from an LLM, with the humans guiding the ideas, the
tests and the debugging.
It is worth looking closely at what "tailor-made" means, because it is not a
slogan: it is written in the code, starting from the opening comment of the
main file. There the engine declares its fixed shapes: it accepts the known
layouts of its model and refuses to start in front of any other. A
general-purpose engine does the opposite by trade: it reads the shape of the
network from the file's metadata and must know how to run whatever shape it
finds there, and the flexibility is paid for in layers of abstraction, edge
cases, missed opportunities. Knowing the shape in advance means that every data
structure and every kernel can assume it. And the specialization is visible to
the naked eye, in the list of the kernel files:
```
softmax.metal norm.metal flash_attn.metal the operations of any engine
dsv4_kv.metal dsv4_hc.metal dsv4_rope.metal the compressed cache, the heads,
the position: of that model
```
alongside the universal tools, kernels that carry the model in their name: its
compressed cache, its compression of the heads, its encoding of the position
(the rotation of section 4), each with the kernel written on purpose.
The tailoring does not stop at the kernels: it is a supply chain. The weights
are not the "community" files: they are files co-designed with the engine,
produced by a dedicated quantizer, a couple of thousand lines of C that declare
in their own opening comment that they keep only the pieces the recipes of that
model need (it is there that the asymmetric quantization just named lives). And
there is the safety net that allows one to push this far: the test vectors of a
moment ago, captured from the API with the most prudent decoding and reproduced
locally to the letter, so that a regression of the tokenizer or of the
attention emerges at once, before becoming wrong answers; and a suite of about
ninety problems that runs the same inference path as the users', to be
re-launched after every change to a kernel or a quantization. The declared
ambition is to make a local model seem "finished", not only executable; the
lesson of method is that every optimization arrives together with the tool that
measures its damage. There is, finally, a debt, declared with the same
frankness: without llama.cpp and GGML this engine would not exist (the
quantization layouts, the tables, even some adapted kernels), to the point that
the copyright of those authors remains in the licence file. The tailor-made
engine stands on the shoulders of the general-purpose one; it surpasses it in a
single point because it gives up all the rest.
Its numbers fix the signature of the two phases: on the same machine and with
the same model, the benchmark shows on the order of 250 tokens per second in
prefill and ~20 in generation (numbers of mid-2026, orders of magnitude and not
fixed measures: the project itself warns that optimization runs faster than the
published benchmarks). It is not a defect of the software: one finds it again,
with different numbers, on any system. Even sharper the counter-proof of
distributed inference: dividing the model over two machines, the prefill
**speeds up** (the pieces of prompt are worked in a pipeline), the generation
**slows down** (each token pays one more network trip, and the trips do not
shorten by pairing up).
How much the trips count is told by the scale of the connections, and it is a
proof anyone can redo with two computers. Same two machines, same model (two M5
Max, 91 GB Flash quant, a prompt of a few thousand tokens): on a direct cable,
with half a millisecond of latency, the generation does 25 tokens per second;
on WiFi, with the latency rising towards a hundred milliseconds, it drops to
10; on a VPN across the Internet, to less than 4. The prefill travels the same
scale in a different way: it travels in large blocks in an assembly line, so
what counts is how much stuff passes per second; the generation ships tiny
packets but one per token, so what counts is how long the single packet takes
to arrive. It is the question of the guiding thread with one more currency:
"how much it costs to move them" is paid in bytes and in milliseconds, and the
two phases pay in different currencies. The cleanest confirmation is a negative
experiment of the project: halving the bytes of the activations in transit
shifted almost nothing, so much that the option risks disappearing. At
generation the count is not the volume of the parcel: it is the number of
deliveries.
The same signature is read, reversed, in the hardware. With the same model and
engine, the generation speed chases the memory bandwidth almost in proportion:
a workstation with traditional RAM (80-90 GB/s) generates 2-5 tokens per
second, a laptop with 400 GB/s unified memory does about twenty, an 800 GB/s
machine about thirty. And the counter-example closes the count better than any
argument: a graphics card with the highest bandwidth of all (1000 GB/s) but 24
GB on board **does not even start**, because the model does not fit. For
generation, bandwidth and capacity are needed together: it is the reason why
unified-memory machines, which have less bandwidth but share dozens or hundreds
of GB between CPU and GPU, have won precisely this use case. The symmetric
paradox also holds: a small system born specifically for AI, with capacity and
computation in abundance but modest bandwidth, goes very fast in prefill and
generates at half the speed of a desktop computer. No spec sheet reports the
item "bandwidth to generate": that is the one to go and look for. Answering is
made of two different jobs, digesting the prompt all together and producing one
token at a time, and whoever optimizes, whoever buys, whoever compares must
know which of the two they are paying.
The two phases of the same answer. The prefill works all the tokens of the prompt together: the matrices are large, the limit is computation, and its duration is the wait before the first word. The generation is a comb of identical steps, one token per step: the dominant work is rereading the parameters at each step, and the limit is the memory bandwidth. The teeth of the comb are drawn at the same distance, but their cost grows slowly with the context: it is the theme of the next section.
## 7. The notes on the table: the KV cache
There is an absurdity hidden in the generation, as it has been described: if
every new token has to look at all the previous ones, at step one thousand the
model would have to recompute from scratch the labels and the contents (the
keys and the values) of a thousand tokens, at step one thousand and one of a
thousand and one, and so on. The already quadratic cost would become
intolerable at every single step.
The solution is the most important and least told object of inference: the keys
and the values of the tokens already seen **never change** (the parameters are
fixed, the past too), so they are computed once and kept aside. They are the
notes on the reading table: the **KV cache**. At each step the model computes
the key and value only of the new token and reuses all the rest. The generation
becomes linear again; in exchange, the notes occupy memory, and grow with the
length of the conversation, layer by layer, head by head. And "memory", here,
is not a metaphor: they are precise bytes, with an address, to the point that
they can be saved and reloaded, as we will see among the optimizations. When
one discusses whether a model "remembers", this is the only working memory that
really exists.
This trade (recomputation avoided against memory occupied) makes the KV cache
the hinge-character of the whole text. It is the reason why the long context
costs even when the model "is doing nothing"; it is the memory item that
competes with the parameters for the same space; and it is the object that
almost every optimization of the next section touches, compresses or moves. And
its weight, for once, can be estimated by hand: bytes of the cache ≈ 2 (for K
and V) × layers × heads × dimension per head × bytes per number × tokens in the
context. A number to fix the scale: in a standard transformer, the cache of a
context of a million tokens would occupy hundreds of gigabytes, more than the
model itself. DeepSeek V4 Flash, with an attention designed to compress the
notes (more on this shortly), brings it to ~26 GB: enormous in absolute terms,
small for the category, and it is the single choice that makes a context of a
million tokens thinkable on a single machine. The generation does not recompute
the past because it keeps it in the notes: the KV cache is the true memory of
the conversation, and its weight in bytes governs more design decisions than
any other number.
The KV cache trade, in its two sides. On the left the gain: the keys and the values of the tokens already seen never change, so they stay written on the table and at each step a single row is added, that of the new token. On the right the price: the notes grow in a straight line with the length of the context, layer by layer and head by head, and it is the memory item that competes with the shelves for the same space.
## 8. The optimizations: six answers to the same question
Seen from outside, the engineering of inference seems a catalogue of acronyms.
Seen from the guiding thread, it is a family of answers to the same question:
where the bytes are and how much it costs to move them. Here six are chosen, not
for completeness but because they cover the fundamental moves; each is declared
for what it is, a way to have fewer bytes, to move them less, or to move them
while doing something else.
**Fewer bytes per parameter: quantization.** The parameters are born as 16- or
32-bit numbers; writing them with fewer bits (8, 4, even 2) makes them less
precise but much smaller, and since the dominant cost is transporting them, a
halved model is (almost) a model twice as fast at generating. The how is less
brutal than it seems: the weights are grouped into small blocks (about thirty
values) and for each block a scale factor is saved; the few bits of each weight
say only the relative position inside the block, and the scale repositions the
whole block. And the bits are not spent blindly: some calibration text is
passed through the model, one measures which weights are really solicited (the
profile is called imatrix), and the precision is concentrated there. It should
not be confused with distillation, which is a **different** and smaller model
trained to imitate the large one: quantization is the same model, written with
shorter numbers.
The on-the-ground proof of ds4 shows how fine the craft is. DeepSeek V4 Flash
in full precision would weigh ~570 GB; the 2-bit version of ds4 weighs ~80 GB
with quality almost intact, but only because the quantization is
**asymmetric**: the parts of the model active in fits and starts are compressed
to the minimum (the "experts", about 90% of the parameters: the following move
presents them), while what works on every token, the normalizations of sections
1 and 4, the router that chooses the experts, the projections of the attention,
stays in high precision. The result is not "everything at 2 bits", it is a
**hierarchy of precisions**, from the 2 bits of the experts up to the 16 of the
most delicate parts, decided component by component. Compressing is not a
uniform operation: one compresses where the error can be afforded.
The result travels in a container file, the GGUF of the llama.cpp ecosystem:
inside are the already-quantized tensors, the metadata of the architecture and
the tokenizer, so the file is self-sufficient. And the engine does not copy it
into memory: it **maps** it (mmap) and reads the pieces on demand; where the
memory is unified, even the GPU works directly inside the mapped file, without
making a copy. Even opening a model, seen from the guiding thread, is a choice
about where the bytes are. Two honest notes close the move. The first: the
model "in the clear" often never existed (DeepSeek distributes weights already
compressed at birth, in 8- and 4-bit formats: the ds4 quantizer translates
between two already-compressed forms, it does not degrade a perfect original).
The second: that most of the bits can be thrown away without collapse says
something profound about the shelves: knowledge does not dwell in a few numbers
of surgical precision, it is written in a distributed and redundant way, robust
to noise; and the fact that the imatrix improves things confirms that not all
weights are equal. It is the same lesson as the interlude, seen from the side
of the bytes: in the shelves there is structure, not dictation.
**Fewer bytes at work per token: the MoE.** The Mixture of Experts changes the
architecture: instead of a single block of computation per layer, many smaller
"experts", and a router that for each token activates a few. DeepSeek V4 Flash
has 284 billion total parameters but activates ~13 per token. Beware the pair:
the **total** parameters must all be kept in memory anyway (the router could
call anyone at the next token), it is the **active** parameters that determine
computation and bandwidth per token. The MoE is therefore an explicit pact with
the guiding thread: abundant memory in exchange for reduced transport. It is the
ideal architecture where memory is wide and computation scarce, and indeed it
thrives on unified-memory machines.
**Fewer bytes of notes: compressing the KV cache.** The attention can be
redesigned so that the notes are born already compressed: small latent
representations to "reopen" on the fly (DeepSeek's MLA), cache rows shared among
groups of positions, local windows for the first layers. It is the road of the
~26 GB for a million tokens of section 7. Here the mathematics **changes**, and
the quality must be revalidated: it is the difference, to hold on to tightly,
from the pure implementation optimization. The canonical example of the latter
is Flash Attention: the very same mathematics, but computed in little blocks
inside the very fast memory of the chip instead of materializing the matrix of
comparisons in the slow one. No approximation, 2-4 times faster: the square of
section 4 does not disappear, but stops travelling. There is then a twin move,
more radical, and it is the signature of DeepSeek V4: **do not reread all the
notes**. A small indexer gives a score to the past tokens, and the real
attention is computed only on the most relevant (the order of magnitude: a few
hundred, chosen in a context of a million). Compressing answers "how much the
cache weighs"; selecting answers "how many comparisons do I make", and it is
this, more than compression, the move that really tames the square on the long
context. Here too the mathematics changes, and the quality must be revalidated.
**The same bytes for more users: batching.** If the generation is limited by
the transport of the parameters, and the parameters are the same for everyone,
then serving several requests together is almost free: the shelves are reread
once and the computations are done for ten reading tables. It is the reason why
the throughput of a service grows well before the latency of the single user
worsens, and it is the true economy of scale of the providers: the cost per
token collapses with the load. And it is no longer only a thing for big
providers: the same local engine can keep several independent sessions resident
and serve them together, so a machine at home or in a company answers several
conversations by rereading the shelves once only. The latency/throughput pair
must be kept distinct precisely here: optimizing one is not optimizing the
other, and benchmarks that do not declare which they are measuring are measuring
nothing.
**Not redoing the prefill: the cache that persists.** The notes, once taken, can
also be saved. If many requests share the same prefix (the system prompt, an
agent's documents, the conversation so far), the KV cache of that prefix is
computed once and reloaded, skipping the prefill. ds4 pushes it all the way:
cache on disk, indexed by the hash of the prompt, with the declared bet that
modern SSDs are fast enough to make it a first-class citizen. It is the same
idea as the providers' "prompt caching", seen from the engine's side.
**Moving the bytes while computing: hierarchical memory.** When the model does
not fit in the fast memory, the alternative to surrender is the hierarchy:
keeping close what is always needed, and making the rest travel (from RAM to
VRAM, or from SSD to RAM) while the circuits work on something else, hiding the
latency behind the computation. ds4 does it in both directions (a managed cache
between RAM and VRAM for the enormous contexts, streaming of the experts from
SSD for the models larger than the RAM), with two refinements that say
everything about the craft: the experts' cache refuses to exist if it cannot be
locked in RAM (better small and predictable than large and subject to paging),
and the preloading starts from a hotlist compiled into the program, the ranking
of the most requested experts measured on real loads: even among the experts,
some bytes are hotter than others. The overall effect of the streaming is a
change of category: the RAM stops being a threshold (the model fits, or it does
not fit) and becomes a scale of speed. Speculative decoding is a relative of the
same family: someone proposes a few tokens, the large model verifies them in a
single shot, turning sequential steps into a parallel step. In the engine two
ways of getting the proposals coexist, and it is instructive. In the first the
suggester is not a second model at all: DeepSeek is also trained to propose the
token after the next (multi-token prediction, MTP), so the model acts as its own
suggester and the engine is left only to verify. In the second (DSpark, an
auxiliary draft of a little over five gigabytes released on purpose) a small
separate model reads the internal state of the large one and proposes up to five
tokens ahead. In both cases the large one remains the only judge: a rejected or
unsure suffix falls back on the ordinary decoding. And in both cases the project
admits without mincing words that for now the gain is marginal, with the reason
coherent with all this text: speculation pays when the single step wastes
bandwidth, but on an architecture that has already reduced the bytes per token
(experts, compressed cache, indexer) little margin is left for the trick;
predictable continuations like code gain more, prompts poor in structure can
even slow down. Optimizations too have a budget, and it is always the same: the
bytes the single step leaves on the table.
The moral of the section is not the catalogue: it is that the six moves compose,
and that each declares its price (memory, quality to revalidate, complexity).
Almost every inference optimization is a different answer to the same question,
where the bytes are and how much it costs to move them; when one meets a new
one, asking which of the six families it belongs to is more useful than learning
its acronym.
The map of the section: six moves around the same question. On the left one reduces the bytes to move (writing them shorter, activating fewer, making the notes born small); on the right one moves better those that remain (one reading of the shelves for many readers, the work already done that is reused, the transport hidden under the computation). No move is free: the declared price is part of the move.
## 9. Inside the code: four close-ups
The moves can also be looked at from much closer, at the level where they really
live: the code. This section goes down there, four times: on three of the six
moves (the notes that persist, the shelves in streaming, the quantization) and
on the distributed craft met in section 6, chosen because each teaches
something that from outside is not seen. Whoever prefers not to go down can skip
to the following section without losing the thread. The on-the-ground proof is
always the same, DwarfStar, for the reason given at its introduction: in a
tailor-made engine the choices stand naked in the code.
**The notes on disk, up close.** The persistent cache seems a simple idea, the
notes are saved and reloaded; the code shows how much care it takes for it to
really work. The key of each file is the fingerprint of the prompt taken as a
sequence of bytes, not as a sequence of tokens, and the choice has a subtle
reason: the model may have generated a token whose text, sent back by the
client at the next round, splits into two different tokens; the bytes instead
coincide, so the comparison on the bytes finds the reusable prefix anyway, and
only the new continuation is re-tokenized. Inside the file, before the actual
notes, there is a 48-byte header documented field by field:
```
"KVC" version quantization bits of the experts
reason for saving (new / continuation / eviction / shutdown)
how many tokens it covers how many reuses it has had (the "hits")
when it was born when it was last used
```
and it is not pedantry: it is what the eviction needs to be fair. When the
space runs out, the policy is not a simple "out with the oldest": the hits cool
down with a half-life of six hours, so a prefix used very often yesterday does
not forever jump ahead of one used a moment ago. There is then a refinement
that is the guiding thread in its pure state: these files are read with ordinary
reading, not with the memory mapping used for the model, and the reason is
declared: not to add other virtual-memory maps to a process that already has an
enormous one. Even the choice of the system call is a decision about where the
bytes are. On this basis the project builds the next step, which here it is
enough to name: its native agent (a terminal assistant built directly on top of
the engine) treats every conversation as its file of notes, and moving from one
session to another is opening a different file, without redoing any prefill. If
the memory of the conversation is a file, the conversation becomes portable like
a file.
**The shelves in streaming, up close.** The streaming from SSD of section 8 has
an interface that is already half a lesson: one does not list files nor experts,
one declares a budget, how much memory to dedicate to the experts' cache, and
the plan is made by the engine. First it puts under cover what must stay in RAM
anyway (the parts of the model that work on every token), then it translates the
declared gigabytes into a number of resident experts, choosing them starting
from the hotlist seen among the optimizations; all the rest dwells on the SSD
and is read when the router calls it, while the circuits work on something else.
The discipline remains the one already stated, either locked in RAM or nothing;
what the zoom adds is the contract: whoever uses the machine does not manage the
shelves, declares how much space they have, and the engine derives the best
possible plan from that constraint. The result is the spectrum of speed of
section 8, made operational: more budget, more resident experts, more tokens per
second; less budget, one drops a gear, but it runs. Put side by side, the zoom
on the notes and this one say the same thing from two sides: in this project the
disk is not a fallback, it is first-class memory; the notes are saved onto it,
the shelves dwell on it, and the RAM stops being an entry requirement to become
a choice of speed.
**The quantization, pushed to the single floor.** The hierarchy of precisions
seen among the optimizations can be pushed further, and it has already happened,
with an instructive experiment. The question: do all the floors of the building
suffer compression in the same way? The tool: an instrument that dequantizes
nothing; it reads two already-compressed versions of the same model, one at 2
bits and one at 4, uses the first as a base and copies into it, byte by byte,
only the experts of the last six floors taken from the 4-bit version. The
result: a file of ~91 GB instead of the ~153 of the full 4-bit version that, in
the agreement checks of the answers, behaves statistically closer to the full 4
than to the pure 2. And the measure is declared for what it is, in the project's
words: it is not a benchmark, it is a useful signal. Hypothesis, surgical
intervention byte by byte, measure against a reference, honesty about the value
of the measure: the method, in four steps.
**Cutting the building, in two ways.** The distributed inference of section 6
has a protocol, and the protocol tells the architecture better than a diagram.
The model is sliced by contiguous floors: one machine owns the ground floor and
the first thirty floors, the other the remaining ones up to the output; the
coordinator keeps for itself tokenization, extraction of the token and prompt.
Every generation step is thus a vertical journey that crosses the machines: the
state of the token climbs the floors of the first, crosses the network, climbs
the floors of the second, and the scores return to the coordinator. It is,
physically, the extra network trip per token of section 6. The workers introduce
themselves declaring the model's identity, quantization profile and slice of
floors; every work parcel carries with it the fingerprints of the history of the
tokens before and after its own span, so each machine verifies that it is
working on the same conversation. And the error handling distinguishes two
different ills: if the fingerprints do not match (the bytes arrived, but the
history is wrong) the token history is replayed on the same route; if the
connection dies (the bytes did not arrive at all) the route is abandoned and
another machine is awaited. And the notes? They stay where they are born: each
machine keeps the KV cache of its own floors, because shipping it at every step
would cost more than the recomputation it avoids; what travels is only the state
of the token, at the border between the slices. When a session is saved to disk,
the coordinator asks each machine for the piece of its floors and recomposes a
single file, the same as the notes seen in the first zoom: the building is
sliced, the memory of the conversation is not. The rest, the parcel formats and
the tricks that keep the prefill assembly line full, is in the code, readable;
and the documentation closes with the customary honesty: no encryption nor
authentication, trusted machines, same version of the program on both sides. A
young protocol that declares its limits. At the network level, in the code, it
all fits in a few packet types: the worker introduces itself with a `HELLO`
(model identity, family, quantization profile, slice of floors), the coordinator
sends `WORK` and receives `RESULT`, and with every parcel travels the 64-bit
fingerprint of the token history before and after the span, the one that unmasks
the machine working on the wrong conversation.
There is a second way of cutting, and it cuts in the other direction. Instead of
giving whole floors to different machines, the work is split in two *inside each
floor*, and the two machines work the same token at the same moment, exchanging
partial sums of a handful of kilobytes at the "gates" internal to the
computation. The two cuts serve different things:
```
BY FLOORS (pipeline) BY WIDTH (tensor parallel)
┌─────┐ A: the first floors ┌──┬──┐ A and B on the SAME floor,
│ ▓▓▓ │A ↑ a token climbs │AA│BB│ on the SAME token, together
│ ▓▓▓ │ crosses the network │AA│BB│ each floor split in half
══════════ 1 hop / token │AA│BB│ exchange: partial sums,
│ ░░░ │B then B's floors └──┴──┘ a few KB at each "gate"
│ ░░░ │
└─────┘ serves to: FIT IT IN serves to: LOWER LATENCY
(sum the RAM) + whole model RESIDENT
```
The logic is all in the guiding thread. With a single machine and a model too
large, the experts shuttle from the SSD, and it goes slowly; pairing up in width
means that each machine keeps **half** the experts resident and never touches
the other's half, so the summed RAM holds the whole model and only those few
kilobytes per floor travel. It is the difference from the cut by floors, where
the two machines do sum the RAM but pay one network trip per token: here the
trips are many but tiny, and instead of fitting in a larger model they lower the
latency. The measure says it without appeal, on the same model that on its own
does not fit:
```
GLM 5.2, 188 GB two Macs (tensor) one Mac (streaming from SSD)
generation ~17 tokens/s ~5 tokens/s
prefill (4096 tokens) ~94 tokens/s ~3–5 tokens/s
memory all resident experts read from the SSD
```
From here one also understands what is transferable. The project's bet is one
model at a time, not one model forever: the model may change, the constraint
remains (credible local inference on high-end personal machines). And the
transplant has already happened twice, inside the project itself: when the
perimeter widened to consumer APUs the engine was not generalized, the kernels
were rewritten for the new hardware reusing the weight format, the cache, the
test chain; and when GLM 5.2 arrived, a model of another family, its
tailor-made pieces were added (its kernels, its weight layout, its quality
fixtures) without dismantling the rest. New hardware and new model: the same
move, in the two directions in which the fence could shift. In the code one even
sees the different form of the three backends (a folder of kernels for the Macs,
a single file for the NVIDIA GPUs, a collection of headers for the AMD APUs):
the mathematics is the same, the form of the code follows the hardware. It is
the practical sense of specialization as method: whoever has a different piece
of hardware in front of them, or tomorrow a different model, can redo the
tailor-made part while keeping the findings, because the findings live above the
kernels, not inside.
The four zooms share a spirit, and it is the true content of this section. At
every point of the system, even the smallest (the system call with which a
cache file is read, the precision of six floors at the top of the building),
there is margin to do better; and the method is always the same: one does the
detail as well as possible, measures the impact, if the impact is there one
keeps it and moves to the next detail; if it is not, one says so and goes back,
as with the negative experiment on the activations in transit or the "for now
marginal" gain of speculation. There is even the inverse move of quantization,
in the NVIDIA GPU backend: where the fast memory is to spare, some compressed
weights are re-expanded into a form more convenient for the computing circuits,
spending bytes to buy time, with a controlled budget and a declared fallback
(if the space is not there, one returns to the compressed kernels, and the
program says why). No move is decisive on its own: it is the disciplined
accumulation of measured details that brings a model of hundreds of billions of
parameters onto a single machine. In an inference engine every detail has margin
to be made more efficient; the method is to do it as well as possible, measure
the impact, keep what pays and move to the next detail. And the code is written
to be read: whoever wants to see the craft at work need not trust this account,
they can open the repository and find the same ideas, with the numbers
alongside.
At a glance, then, the surface of the project and the moves of this text that
are read inside it:
```
DWARFSTAR (ds4): snapshot of 21 July 2026
models DeepSeek V4 (Flash and PRO) · GLM 5.2 (another family)
backends Metal (Mac) · CUDA (NVIDIA) · ROCm (AMD APU)
tailor-made kernels per model+hardware · dedicated quantizer ·
byte-by-byte test vectors · suite of ~90 problems
reduce the bytes to move move the same bytes better
· asymmetric quantization · batching (locally too)
· Mixture of Experts · cache that persists (on disk)
· compressed KV cache + indexer · hierarchical memory (SSD streaming)
in two machines (or more)
· by floors (pipeline) sums the RAM, one network hop per token
· by width (tensor) same token together: less latency,
whole model resident
reduce the steps
· speculation MTP (itself) or DSpark (external draft),
gain for now marginal
```
It is not the map of a product to buy: it is the inventory of the ideas this
text has crossed, each readable in the point of the code where it lives. The
same thing said at the start, now with the names in their place.
## 10. What you take away from the journey
The synthesis offered here takes the form of a list of questions and answers:
a map of the topics treated so far. This map
precedes the other questions (those of the closing section) that, it is hoped,
the reading of this text helps to face a little more knowingly.
**Why does the answer arrive in bursts, one word at a time?** Because the machine
is made that way: it produces a little piece of word, appends it to what is
already there, and starts over from the beginning. There is no ready-made answer
transmitted in instalments: every little piece is born from a complete
consultation of the shelves, and the stutter one sees is the true rhythm of those
consultations.
**Why do price lists distinguish the words sent from those received?** Because
they are two different jobs. The words sent are digested all together, in
parallel: a fast job per piece. The words received are born one at a time, and
each costs a complete rereading of the shelves: a slow job per piece. Whoever
sells charges more for the job that costs more.
**Why does a long conversation cost and slow down, even in the moments when no
one is writing?** Because the conversation lives in the notes on the table: for
every piece of text the machine keeps its row of notes, and the table grows with
the conversation. Those notes occupy precious memory even at rest, and at every
new word they must be consulted: a fuller table is a more expensive memory and a
slower consultation.
**Does the machine learn anything from what I write to it?** No, not while it
answers. The shelves are filled once only, before; the conversation lives on the
reading table, and at the end of the session the table is cleared. For the
machine to really learn something new one would have to redo, at least in part,
the filling of the shelves: an enormous job, which no conversation sets off on
its own.
**Is a compressed model a worse model?** The right question is not how much it
has been compressed, but where. The knowledge on the shelves does not dwell in a
few digits of surgical precision: it is written in a distributed and redundant
way, and it bears rounding well, provided one compresses more where the error
can be afforded and almost not at all where it cannot. And a well-compressed
model is not presumed: it is measured, on the same tests as the original.
**Why does the same question not give the same answer twice?** Because the
machine's output is not a word: it is a ranking of probabilities over all the
possible words, and from that ranking one draws. The drawing has a knob (in the
jargon, the temperature): all prudence, always the candidate at the top,
repeatable but flat answers; or a bit of gamble, every so often a less obvious
candidate, more varied answers. Usually the knob sits a little above zero, and
it is for this that two answers to the same question resemble each other without
coinciding.
**Can the machine also run at home?** Yes, and it is less strange than it seems:
almost all the craft told so far works in that direction. Of some machines the
shelves are public and can be copied; they can be written denser, squeezing more
where the error can be afforded; in many modern machines, for every word only a
small part of it works; and the notes can be born compact and, when needed, live
on disk. The result is a whole library inside a computer one can buy, on a
person's desk as in a company's cabinet: slower than a datacentre's, often a few
steps behind the most advanced, but complete, with the data that does not leave
the door and with a few tools in hand that the service from outside does not
lend. How much it really delivers on one's own case, however, is written on no
spec sheet: it is a proof one can do, with the questions of the closing section
in hand; and it concerns anyone who has opportunity and need, the curiosity of a
person as much as the hardware and the skills of an organization.
## 11. The questions that matter
There remains the point declared at the start: what use is all this to someone
who does not write kernels, but has to choose, evaluate, or even just read the
news without being swept along. It is useful for knowing which questions to
ask, and the list is short.
**How much memory is needed, and for what?** (Total parameters, not active;
plus the KV cache at the context one really intends to use: it is the question
that decides whether a system fits in a machine, in a GPU, in a phone.) **Is
the load prefill or generation?** (Long documents and short answers are one
craft; long chats and long answers another; the bottlenecks are opposite.)
**Which quantization, and validated how?** (The number of bits is not enough:
the question is where it was compressed and with which quality checks.)
**Latency or throughput?** (For a single user the first counts; for a service
the second; the marketing numbers always choose the more photogenic one.) **And
the long context, how much does it really cost?** (Not "how much context it
supports", but how many bytes of notes it carries with it and who pays for
them.)
They are questions about memory, almost all of them. It is not by chance, and it
is the way this text would like to be remembered: to evaluate a system one need
not know everything, one needs a few right questions, and they are almost all
questions about the bytes, about where they are and how much it costs to move
them.
There is a concrete comparison in which these questions all make themselves felt
together: the model served in the cloud against the model run at home. The gap
in quality must be told without sugarcoating: the frontier models live in the
datacentres of those who trained them, and what runs on one's own machine
usually chases, with months or years of delay. But the six moves of section 8
are shifting the window: asymmetric quantization, MoE on unified-memory
machines, cache on disk are exactly the techniques that bring a model of
hundreds of billions of parameters onto a single machine, and it is the reason
why an engine like ds4 can exist. The questions of this section are the
technical way to decide the comparison: how much memory, which load, which
quantization validated how, which latency is really needed. And for whoever runs
at home, the cost also has domestic units, heat, noise, battery: not by chance a
local engine exposes a knob that trades speed for silence.
The rest of the decision is not technical, and it is well to declare it: where
the data goes (locally it does not leave), who can change price, conditions or
model from one day to the next (the lock-in), what it means for a company or a
country to depend on someone else's inference (sovereignty is also this), and
whether inference will become a commodity or remain a service of the few. They
are stakes that deserve a discourse of their own, not an aside; here the point
of contact is enough: whoever faces them without the questions of this section
is choosing in the dark, because cloud and local are not two prices, they are
two profiles of bytes.
There is, finally, a reading of DwarfStar worth making explicit, because it
holds together the two halves of the decision, and the project suggests it none
too subtly: a tailor-made engine, pushed all the way on a single model,
is first of all a **measuring instrument**. It establishes where local inference
really reaches, today, on machines one can buy. One arrives at local inference
by very different roads: because it is the only possible solution, because it is
an acceptable compromise or a way-station, for an opportunity to build, for one
of the stakes just mentioned (sovereignty, privacy), to have in hand tools the
cloud does not expose (the steering of the interlude); the list is long on
purpose. But whatever the road, the sensible choice has the same prerequisite:
knowing what one can really obtain, measured and not imagined. To discard the
local route only because today more convenient or apparently more obvious
solutions exist means giving up knowing its potential, and in certain scenarios
that route might turn out to be the only viable one. A simple rule holds, then,
which is the sense of all this text applied to a single decision: an option
never measured is not an option, it is a hope.
The library, to close the circle. The parameters are shelves filled once and
never touched again; answering is consulting them, a little piece of word at a
time, with the notes on the table so as not to reread everything at each step;
and almost all the engineering seen here is the craft of making an immense
library served by narrow corridors work: writing denser, bringing the shelves
closer, serving more readers with the same round. When the next acronym arrives
(it will), the questions to ask it are already on the table.
There remains a measure this text borrows from the energy account it started
from, and which it deliberately leaves open: there it concluded that the
adequate metric is not the energy per token, it is **the energy per right
result**. The techniques seen here all lower the cost of the token; whether that
token is really needed, none of the quantities of this text can say. And it is
the honest boundary to stop at: knowing where the bytes are and how much it
costs to move them says how much an answer costs; how much it is worth is
another question, and remains with whoever reads it.
## References
- A. Vaswani et al., *Attention Is All You Need*, 2017 —
[arxiv.org/abs/1706.03762](https://arxiv.org/abs/1706.03762)
- J. Su et al., *RoFormer: Enhanced Transformer with Rotary Position
Embedding*, 2021 — [arxiv.org/abs/2104.09864](https://arxiv.org/abs/2104.09864) (the rotary
embeddings/RoPE cited in §4: position as the rotation of question and label)
- R. Sennrich, B. Haddow, A. Birch, *Neural Machine Translation of Rare Words
with Subword Units*, ACL 2016 — [arxiv.org/abs/1508.07909](https://arxiv.org/abs/1508.07909) (the
sub-word tokenization of §1: the closed vocabulary that can still write
everything, derived from the data with a compression criterion; the BPE
method adapted from P. Gage, *A New Algorithm for Data Compression*, 1994)
- P. Michel, O. Levy, G. Neubig, *Are Sixteen Heads Really Better than One?*,
NeurIPS 2019 — [arxiv.org/abs/1905.10650](https://arxiv.org/abs/1905.10650) (the attention heads of §4:
many can be removed without harm; with E. Voita et al., *Analyzing
Multi-Head Self-Attention*, ACL 2019 — [arxiv.org/abs/1905.09418](https://arxiv.org/abs/1905.09418),
for the few heads with a readable role and the many without)
- D. Rumelhart, G. Hinton, R. Williams, *Learning representations by
back-propagating errors*, Nature 1986 —
[doi.org/10.1038/323533a0](https://doi.org/10.1038/323533a0) (the backpropagation of §2: the derivatives
of all the parameters in a single descent)
- C. E. Shannon, *A Mathematical Theory of Communication*, 1948 —
[doi.org/10.1002/j.1538-7305.1948.tb01338.x](https://doi.org/10.1002/j.1538-7305.1948.tb01338.x) (the surprise −log p of
§1, the same quantity as the demon post)
- T. Mikolov et al., *Efficient Estimation of Word Representations in Vector
Space*, 2013 — [arxiv.org/abs/1301.3781](https://arxiv.org/abs/1301.3781) (word2vec, now cited in §1:
the linear regularities of the embeddings and the king/queen example)
- J. Kaplan et al., *Scaling Laws for Neural Language Models*, 2020 —
[arxiv.org/abs/2001.08361](https://arxiv.org/abs/2001.08361) (for the mention of the scaling laws in §1)
- T. Dao et al., *FlashAttention: Fast and Memory-Efficient Exact Attention
with IO-Awareness*, 2022 — [arxiv.org/abs/2205.14135](https://arxiv.org/abs/2205.14135)
- DeepSeek-AI, *DeepSeek-V2* (introduction of Multi-head Latent Attention),
2024 — [arxiv.org/abs/2405.04434](https://arxiv.org/abs/2405.04434) (the origin of the idea of the
compressed notes cited in §8 as MLA)
- DeepSeek-AI, model card of *DeepSeek V4 Flash* —
[huggingface.co/deepseek-ai/DeepSeek-V4-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash) (the source of the
model numbers cited in the text; terminological note: the model card calls
the mechanism CSA+HCA, Compressed Sparse Attention + Heavily Compressed
Attention, where the literature says MLA; the indexer of §8 is the "Sparse"
part of CSA)
- C. Olsson et al., *In-context Learning and Induction Heads*, Anthropic
2022 —
[transformer-circuits.pub/2022/in-context-learning-and-induction-heads/](https://transformer-circuits.pub/2022/in-context-learning-and-induction-heads/)
(the mechanical candidate cited in the interlude)
- Specification of the GGUF format (ggml/llama.cpp ecosystem) —
[github.com/ggml-org/ggml/blob/master/docs/gguf.md](https://github.com/ggml-org/ggml/blob/master/docs/gguf.md)
- Open Source Initiative, *The Open Source AI Definition* (OSAID), v1.0,
2024 — [opensource.org/ai/open-source-ai-definition](https://opensource.org/ai/open-source-ai-definition) (the formal
definition of open-source AI cited in the landscape of models: weights, code
and information on the data, with full freedom of use)
- Y. Leviathan et al., *Fast Inference from Transformers via Speculative
Decoding*, 2022 — [arxiv.org/abs/2211.17192](https://arxiv.org/abs/2211.17192)
- E. M. Bender, T. Gebru et al., *On the Dangers of Stochastic Parrots: Can
Language Models Be Too Big?*, FAccT 2021 —
[dl.acm.org/doi/10.1145/3442188.3445922](https://dl.acm.org/doi/10.1145/3442188.3445922) (the origin of the label, for
the interlude)
- T. Brown et al., *Language Models are Few-Shot Learners*, 2020 —
[arxiv.org/abs/2005.14165](https://arxiv.org/abs/2005.14165) (the in-context learning cited in the
interlude)
- K. Li et al., *Emergent World Representations: Exploring a Sequence Model
Trained on a Synthetic Task*, ICLR 2023 —
[arxiv.org/abs/2210.13382](https://arxiv.org/abs/2210.13382) (the Othello experiment of the interlude)
- A. Templeton et al., *Scaling Monosemanticity: Extracting Interpretable
Features from Claude 3 Sonnet*, Anthropic 2024 —
[transformer-circuits.pub/2024/scaling-monosemanticity/](https://transformer-circuits.pub/2024/scaling-monosemanticity/) (the features,
and the experiment of the bridge kept switched on)
- A. Arditi et al., *Refusal in Language Models Is Mediated by a Single
Direction*, 2024 — [arxiv.org/abs/2406.11717](https://arxiv.org/abs/2406.11717) (one direction for a
whole behaviour: the base of steering)
- S. Sanfilippo (antirez), *DwarfStar (ds4)*, inference engine in C for
DeepSeek V4 Flash and GLM 5.2, MIT licence — [github.com/antirez/ds4](https://github.com/antirez/ds4)
(the state of the project and all its numbers cited in the text are a
snapshot verified on 21 July 2026: the project moves fast, the repository is
the up-to-date source)
- ["The energy cost of a language model"](/en/02-articolo/), on this site (the
cost of inference, the "empty memory", the computation/transport ratio)
- M. Horowitz, *Computing's energy problem (and what we can do about it)*,
ISSCC 2014 — [doi.org/10.1109/ISSCC.2014.6757323](https://doi.org/10.1109/ISSCC.2014.6757323) (the 100-700x ratio
between memory access and computation recalled in §6: the same source as the
energy account)