For some months now a part of my time has gone to a precise exercise: trying to understand the operating principles of inference engines in general, and in particular of those built to run locally, in principle on a single machine; and to understand (as far as I can) why they are built the way they are. Any resemblance to antirez's code is not purely coincidental. I am not an expert programmer, and with respect to the AI world I am little more than a "curious onlooker", but I have tried to understand all the same, I have tried to inform myself. And I am well aware that the journey has, in any case, only just begun.

Starting from the inference engine is a strange vantage point but, I believe, an instructive one: from in there the theory of transformers stops being a list of acronyms and becomes a row of concrete problems, almost all of the same kind: there is an enormous heap of numbers, and one has to decide where to keep them and when and how to move them.

These pages are, first of all, my attempt to put back in order (with the help of an LLM too, an almost ironic detail) what I have "accumulated" and tried to understand. They were meant to be about inference, the local kind in particular; but it felt almost natural to start from the beginning, from how a model is trained and from the principles of how it works, because without those foundations there is little one can say. The path therefore goes further back than expected: first the basics, then the paper that started it all, in 2017, and from there the techniques with which a model is made to answer today while optimizing the resources of whoever runs it.

Not a manual: a map with a few entry points, chosen because they open up the rest. It grew more than expected, and I chose not to thin it out: too many omissions would end up failing to convey what allows "the machine to answer one word at a time". A line, however, is drawn. Some of the thorniest steps of the algorithms involved are depicted but not explored in depth, taken as given and rendered the way I understood them; for those details the right place remains the specialist sources, and some are listed at the end.

The underlying thought, not too hidden, is that local inference, as a choice and as a possibility, already matters today for companies, for communities and, in some respects, for individual people, and will matter more and more: perhaps the tipping point for the scenarios of adoption and development. Understanding something of the mechanisms underneath is useful whatever one's role.

From here on the voice returns to the impersonal; but the questions were born there.

The map just announced begins with its stops; each title is an entry point:

1. What a machine learns

There is a useful way to arrive at LLMs starting from zero, and it passes through three successive narrowings. Machine learning is the idea of not writing the rules by hand: one chooses a function with many free parameters and lets the examples fix them. A neural network is a particular family of these functions, made of layers that pass numbers to one another: each layer combines the input values with its own parameters and passes the result to the next. A language model is a network trained on a single task, seemingly modest: given a text, assign a probability to every possible immediate continuation. The three narrowings, though, need to be filled in, because each hides a choice, and choices explain more than definitions do.

Where the correction comes from. The classic taxonomy of machine learning distinguishes three ways of learning, according to who tells the machine it is making a mistake. In supervised learning every example carries the right answer attached (this photo is a cat, this email is spam): one learns a correspondence. In unsupervised learning the labels are absent, and one looks for the structure the data already have on their own: groups, regularities, ways of compressing. In reinforcement learning there is not even the right answer: there is an environment, there are actions, and every so often a reward arrives; one learns the strategy that maximizes it (it is the family of programs that play Go). LLMs are born from an economically decisive variant of the first family, self-supervision (self-supervised learning): the text serves as its own label, because the continuation to guess is, in the text, already there. No human labeller, and every existing text becomes training material: it is the choice that unlocks scale.

The continuation to guess, moreover, is not a whole word: it is a token, the unit the model actually sees. The text is broken by a tokenizer into pieces taken from a vocabulary of tens of thousands of entries; a common word is one token, a rare word or a proper name is broken into several tokens.

Why not words. It seems a gratuitous complication, given that words are already there; and instead it is a tight choice among very concrete constraints, worth seeing, because they all come back later.

The hard constraint is that the vocabulary must be closed and finite, and decided before seeing any text: the model's output is a list with one cell per entry, and a list is sized in advance. Words, however, are an open set: proper names, neologisms, acronyms, typos, languages in which words compose at will. Working with words one always ends up keeping in the list an "unknown entry" cell, which in front of a real text is a surrender. With sub-word pieces it is not needed: a few tens of thousands of entries are enough to write anything, by composing. In the variant that breaks all the way down to bytes there is, by construction, nothing inexpressible, not even a never-seen language or a piece of binary file pasted in by mistake.

Once this is settled, it remains to decide how big the pieces should be. The two extreme choices contradict each other, and each breaks what the other fixes. Cutting fine, one character per token, gives a tiny vocabulary and no out-of-list entry; in exchange it lengthens the sequences beyond measure, and length is the most expensive quantity of this craft: we will see later that the cost grows with the square of the length. Cutting coarse, one word per token, keeps the sequences short but swells the vocabulary; and a large vocabulary is paid for, because the model dedicates parameters to each entry, to recognize it when it comes in and to be able to propose it when it comes out. The bill thus grows together with the entries: a million entries would mean billions of parameters spent on that alone. Between the two extremes lies the whole space of sub-word pieces, and that is where usage has settled.

There is, finally, a statistical reason, and it concerns what one manages to learn. An entry met three times in all of training remains badly estimated, however one represents it; by breaking up, the frequent pieces receive very many examples, and morphology is shared among different words (a prefix, a suffix, a root), so that a never-seen word is not a wall but a composition of known pieces. The vocabulary, after all, is designed by no one: it is derived from the data, and the criterion is compression. The most widespread recipe starts from the individual characters and repeatedly merges the most frequent adjacent pair, until the desired size is reached; it is a compression algorithm from the nineties, readapted to text. The pieces that come out do not coincide with the units of meaning a linguist would recognize, do not respect the grammar of any language and need not do so: they are the cuts that make the text shorter, and nothing more.

The advantages have a price, and intuition pays it. Hence the first pair not to be confused: the model does not reason in words, it reasons in tokens, and many oddities (the counting of letters, the "per token" prices, the speed of generation) are understood only from there.

From numbers to points in space. A token, just out of the tokenizer, is only a vocabulary index, and an index has no meaning: entry 4,812 is no more similar to 4,813 than to 40,000. The network's first move is therefore to turn each token into an embedding: a list of thousands of numbers (a vector), that is, a point in a many-dimensional space. It is easy to imagine a dictionary compiled by someone, and it does not exist: the table that assigns to each vocabulary entry its point is itself a block of the network's parameters, random points at the start, arranged by training like everything else. The idea has a history of its own. Before transformers, small networks were trained only to predict nearby words, and the by-product turned out to be more precious than the task: the vector representations that the network had built for itself in order to succeed (word2vec, 2013). And it is here that meaning begins to have a geometry: in the right representation, words used in similar ways end up close, and the directions capture properties. The classic example: the shift that leads from "king" to "queen" closely resembles the one that leads from "man" to "woman"; in symbols, v(king) − v(man) + v(woman) ≈ v(queen), and it is an arithmetic that on embeddings can really be verified. It seems a miracle and is not: training rewards the representations that compress, and if thousands of contexts treat "king" and "queen" with the same scaffolding of sentences, except for gender, the most economical solution is to give the two tokens almost the same point, separated by one and the same direction. Meaning, as the network sees it, is use; the geometry is only the most compact way to write it. It is the first appearance of an idea that will return: in the right representation spaces meaning becomes geometry, and geometry can be measured, compared, even pushed. The model's input therefore lives in this space, one point per token; and so does the output, as we will see shortly: the point the network arrives at at the top is compared with the points of the vocabulary entries, and it is from that comparison that the probabilities are born. From end to end, in short, the model works on coordinates; only at the very last does a comparison turn the geometry into a list of percentages.

The shape of the function. Each layer of the network, when it is traversed, does just two things. The first is linear: to weight and sum, the input values multiplied by the parameters (in bulk it is a multiplication of matrices, the operation for which GPUs were born, and which will keep coming back). It is worth seeing it written once, in miniature, because it is the brick of all the rest:

[ y1 ]   [ w11  w12 ]   [ x1 ]        y1 = w11·x1 + w12·x2
[    ] = [          ] · [    ]   →
[ y2 ]   [ w21  w22 ]   [ x2 ]        y2 = w21·x1 + w22·x2

two inputs, two outputs, four weights: each output is a weighted mix of all the inputs, and the weights w are the parameters that training will fix. In real layers x has thousands of components and W billions of cells, but the operation is exactly this. The second is a non-linearity, in the jargon the activation function: a fixed function, deliberately trivial, applied number by number. The two things stand in a row, not side by side: first the weighting, then the function on the result. In the miniature above the true output of the layer is not y₁ but f(y₁) = f(w₁₁·x₁ + w₁₂·x₂), and so for each component: the weighted sum is the argument of the non-linearity, and what comes out of it goes to the next layer. Layer upon layer a stack forms, and it is worth fixing right away the image with which the text will always look at it: a building, one layer per floor. The tokens enter on the ground floor, each floor works on the output of the one below, and traversing the network is climbing to the top. The functions one meets in the names each deserve a definition, because they are all simpler than their fame:

  • ReLU (rectified linear unit): if the number is positive it passes unchanged, if it is negative it becomes zero. That is all: max(0, x). A switch, at the cost of one comparison.
  • GELU and SiLU: the same idea with a rounded elbow. Instead of zeroing the negatives all at once, they make them fade gently towards zero (SiLU, for instance, is x·σ(x), with σ the sigmoid that slides from 0 to 1); the smooth transition helps the derivatives not to die out.
  • softmax: a tool apart, which inside the floors does not appear: it takes a list of arbitrary scores and turns it into percentages that sum to 1, amplifying the differences exponentially (the highest score takes far more than its proportional share). In symbols: softmax(xᵢ) = exp(xᵢ) / Σⱼ exp(xⱼ). It is needed wherever scores must be turned into probabilities, and the places are two: at the top of the stack, once only, to choose the next token; and, as we will see, inside attention.

It seems a department of details and is instead the point of everything: composing linear functions still gives a linear function, so a hundred layers of weightings alone would be equivalent to a single layer, and depth would be an accounting illusion. It is the non-linearity that makes depth real, and with it the capacity to represent what is not proportional: thresholds, exceptions, interactions between distant words. Which non-linearity to use, on the other hand, is an almost prosaic choice, with three criteria: to break the linearity, to have derivatives that do not die out (why it matters is seen in training, in a moment), to cost almost nothing, since it will be evaluated an astronomical number of times. Inside the floors, then, the switch and the elbow work, number by number, on every floor; the softmax is not the non-linearity of any floor. It appears at the top of the stack, where it closes the loop, and here it should be said where the scores it transforms come from, because there is a jump that is usually glossed over: the last floor does not hand over a score per entry, it still hands over a point in the space of representations, thousands of coordinates like those of the embedding. The scores are born from a comparison, the dot product between that arrival point and the vector of each vocabulary entry: how much the arrival point points in the direction of each entry. And in many models the table used for the comparison is the same as the input one, reused in reverse (in the jargon, weight tying): the map that on the ground floor brings the entries into the space, at the top measures the closeness of the arrival point to each of them. The rest is done by the softmax, from those scores, one per entry, to the percentages that sum to 1; and going back from the point to the token requires no more geometry: one chooses a cell of the list, and the position of the cell is already the token, no coordinate to invert. It is there that the network becomes the function promised at the start, text in, probability distribution out. In the single pass there is nothing mysterious, multiplications and sums plus a switch: the capacity lies in the accumulation, dozens of layers and billions of parameters.

The "just two things" of a moment ago were two out of arithmetic honesty: it is from those that the result comes out. Alongside them, however, works a third ingredient, minor on its own and indispensable for the whole: a normalization, which before the weighting brings the numbers of each token back onto a common scale. It does not change what the layer computes, does not put different tokens in communication and returns as many numbers as it received; and the parameters it carries are a trifle compared with those of the matrices. It serves to prevent the values, as one climbs from floor to floor, from swelling or dying out, and why it is indispensable will be seen by looking at training: it is the first of the two inventions that make tall stacks governable.

And the numbers that one layer passes to the next have a name that will serve later: the activations, the living part of the computation alongside the fixed parameters.

How much goes in and how much comes out. Saying that the model takes a text and returns probabilities is not enough: one needs to know in what shapes, because it is from the shapes that one understands what can vary and what cannot. The text, broken by the tokenizer, is a row of tokens, and on the ground floor each is replaced by its own embedding, the point in the many-dimensional space seen above: a list of some thousands of numbers. To reason about shapes three names are needed, and it is worth taking them now: T is how many tokens stand in a row, d how many numbers make up the point of a token, V how many entries the vocabulary has. The input is therefore not a vector, but a stack of vectors: T rows of d numbers each.

Of the two dimensions of the input, only one varies. The d is fixed by the architecture and never changes: neither from one request to another, nor climbing from floor to floor, because each floor receives T rows of d and returns just as many. It is not a whim: we will see that each floor adds its own result to what had entered it, and two things add up only if they have the same shape. The T instead is the length of the text at that moment, and changes continuously.

Which raises a legitimate question. A fixed-size network, if one changes the length of its input, no longer works: the matrices do not match. How does this one digest, with the same parameters, a prompt of ten tokens and one of ten thousand? The answer lies entirely in one fact: no parameter matrix has a side of length T.

input               T × d        T tokens, d numbers per token
each floor      T × d  →  T × d   the shape does not change on the way up
at the top      T × d  →  T × V   comparison with the vocabulary entries
                                  (in inference only the last row is needed)

the parameters:  embedding table         V × d
                 inside each floor        d × d, or d × (multiple of d)
                 final comparison         V × d  (often the same table)

The matrices, as one sees, are made of d and of V, never of T. And the reason can be read in the two things a floor knows how to do. What works one token at a time applies the same weights to each of the T rows, and how many there are does not matter to it: they are T repetitions of the same computation. What instead mixes the tokens with one another does produce a table as big as T by T, but that table is computed on the spot, it is not made of parameters: it is born from the numbers of that text and dies with it. The parameters, in short, are indifferent to length; what grows with T is the work to be done and the memory of the intermediate values, which is then the whole story of the costs in these pages. In a classic stack a floor weighs roughly ten times d² parameters, and the billions one speaks of are that number multiplied by the floors: a matter of width and of height, never of the length of the text.

It is worth seeing the two pivotal multiplications with the rows and columns in their place, because it is there that the flow of the computation becomes concrete. The first is any floor; each row is a token, and one follows it on its own:

   input                 floor weights           output
   ┌── d ──┐             ┌─── d ───┐             ┌── d ──┐
 t1│ · · · │             │ \       │           t1│ o o o │
 t2│ · · · │      ·      │   \  W  │     =     t2│ # # # │
 t3│ · · · │             │      \  │           t3│ o o o │
   └───────┘             └─────────┘             └───────┘
     T × d                  d × d                  T × d

   row t2 of the output arises ONLY from row t2 of the input;
   each # is a weighted sum as in the miniature just now (the row
   of t2 by a column of W). The rows do not talk to one another:
   making them talk is the task of section 4.

The second is the move at the top of the stack, and uses a single row: the last, the one that carries the prediction of the next token. It goes to meet the vocabulary table and out of it comes a list as long as the vocabulary, one score per entry:

   last row            vocabulary table                scores
   of the output       (one column per entry)
   ┌── d ──┐           ┌────── V ──────┐
   │▓ ▓ ▓ ▓│     ·     │ v1  v2 ... vV │      =    │p1 p2 … pV│ → softmax
   └───────┘           └───────────────┘           └──────────┘
     1 × d                   d × V                     1 × V

   one score per entry: how much the arrival point "points" towards
   each one. The table is the V × d of the input, reused in reverse
   (the weight tying just now). In inference only this row is needed;
   the other T−1 predict tokens that are already in the text, and are
   not computed.

A single door could let length in among the parameters, and it concerns a detail not yet spoken of: the model must somehow also be told in what order the tokens stand, because a row of points, on its own, does not say which comes first. If that information is put in a table with one row for each possible position, then a side of length T among the parameters does exist, and with it a rigid ceiling written into the weights. It is one of the reasons why the more recent recipes prefer to compute the order, and in section 4 we will see how. A ceiling, in any case, remains: it is the context window, the maximum T that a model declares it can bear, and it is not a limit of shape but of cost and of training.

There remains the output, and here the count must be done in full, because so far the text has left it implicit. At the top there does not come out a list of V percentages: there come out T of them, one for each position, each as long as the vocabulary. The model, that is, does not only predict what comes after the last token: it predicts, in one shot, what comes after each of the tokens it has before it. In inference only one is needed, the last: the others predict tokens already in the text, and the engines are careful not to compute them. In training all are used, and it is what makes it sustainable to have a machine read thousands of billions of tokens.

The first round, then, does not start from empty: the input tokens are those of the prompt, and T is its length. Then the text lengthens by one row at a time, and T with it. If there really were nothing to start from, one seeds with a special start token: T equals 1, and never 0.

What is made to descend. "Learning" must become a number, otherwise there is nothing to optimize: a loss function (the loss, as it is called almost everywhere), which measures how far the prediction is from the truth. The scene of the measurement is concrete: one takes a real text, runs through it position by position, and at each one asks the model for its prediction. Here there is a misunderstanding to clear up at once: the model does not produce a single token to put next to the true one to say right or wrong. Its prediction, for each position, is a vector, a list. The size of this list is that of the entire vocabulary: one element for every possible entry (token), and in each element a percentage, with the percentages summing to 1. Of lists, therefore, there come out T, one per position, and it is better to look at one at a time. The true token, which in the training text is already there, during this phase serves only to establish which cell to go and read among those of the output vector at that position:

text: "the cat sleeps on "      true token, from the text: "cushions"
model output:
   cushions 0.32   ←  only this one is read: p(true token) = 0.32
   rugs     0.25
   beds     0.18
   … (all the other vocabulary entries; the sum is 1) …

No direct comparison between two tokens, then: a reading. The number found in the cell corresponding to the token's position in the vocabulary says how much confidence (probability) the model attributed to the continuation of the text materializing precisely with that token.

That number, however, as it is, is not yet of use: the loss must be a single one for the whole text, and the probabilities of the individual positions do not add up, they multiply. The probability that the model assigns to a whole text is in fact the product of the probabilities it assigns to each of its positions, one after another: thousands of numbers smaller than 1 multiplied together, a result that plunges below any representable threshold. The logarithm serves exactly this, and is not an ornament: it turns that product into a sum. With its sign flipped, to have it positive, it gives the quantity one really works on, the surprise of a single position:

surprise = −log p(true token)

It is zero when the prediction was certain and right (p = 1), and grows without limit the more the model was wrong. The loss is the average of these surprises over all the positions, and it is the canonical choice for language models: in the jargon, the cross-entropy. It is worth seeing it at work on a whole text, one row per position. They are different steps of the same reading, not alternatives of the same step:

training text: "the cat sleeps on cushions"

pos.  the model has before it     true token   p(true token)   surprise
 1    "the "                        cat           0.05            3.0
 2    "the cat "                    sleeps        0.20            1.6
 3    "the cat sleeps "             on            0.45            0.8
 4    "the cat sleeps on "          cushions      0.32            1.1
                                                             ────────
                                               loss = average = 1.6

Each row is a scene like the one before: the whole vocabulary in output, a single cell read. The loss is the average of the right-hand column, and nothing else. One reads too, in that column, the character of the cross-entropy: it punishes self-assured errors disproportionately. A position in which the model gave the true token 0.1% is worth 6.9 on its own, as much as some seventy positions guessed at 90%. It is a deliberate property, not a side effect: the loss must reward calibration as well as getting it right, and a model that gives 60% to the right token while distributing the rest sensibly is worth more than one that fires off certainties and every so often is badly wrong. The other requirement is less visible but just as binding: the loss must be differentiable, because the whole of training, as we will see, is made of derivatives.

With the mechanics fixed, one can look at two things that were already there. The first: this surprise is Shannon's, the same measure of information as Maxwell's demon: not a resemblance, the same quantity, and a language model is trained by minimizing it. The second says what it means, at bottom, to "learn the language": since the surprise is the logarithm of the probability with its sign flipped, minimizing the average surprise is the very same operation as maximizing the probability the model assigns to the real text. The task, rewritten without jargon: to make the written world as little surprising as possible.

With what numbers it is measured. The choices above define the system; to describe it and compare it a few quantities suffice. Three define it: how many parameters it has, how much data it has seen (measured in tokens), how much computation it took to train it. They are not independent: an empirical regularity by now robust (the scaling laws) says that quality improves predictably when the three grow together, and it is the bet on which the last decade of ever-larger models has been built. To judge, on the other hand, none of the three suffices: one measures the loss on text never seen in training, because a function with billions of parameters has every temptation to learn by heart (overfitting), and generalizing is exactly what memory does not give. The same loss on new text has a more readable translation, the perplexity: among how many equiprobable alternatives the model is hesitating, on average, at each token (perplexity 8, it hesitates among eight; lower is better). The translation, after all, is literal: perplexity = exp(loss), the exponential of the average loss, that is, the logarithm of a moment ago undone. On the cat sentence, loss 1.6 means perplexity 5: as if at each word the model were choosing, blindly, among five equivalent possibilities. And since predicting the text well is not yet "doing things", above all this stand the benchmarks, shared batteries of tests (questions, problems, code) with their merit (they make comparable what would not be) and their known ailments (ending up inside the training data, and models trained to the test more than to the craft). The honest comparison declares the quantities and the measures together: the parameters alone are an engine displacement, not a speed.

Generating text, then, is calling this function in a loop: one gives it the text so far, it chooses the next token, one appends it, one starts again. With the shapes from a moment ago alongside, the round can be read in full:

round 1   text: "the cat sleeps on "            T = 4  →  row 4  →  "cushions"
round 2   text: "the cat sleeps on cushions"    T = 5  →  row 5  →  ","
round 3   text: "the cat sleeps on cushions,"   T = 6  →  row 6  →  " and"

It is worth looking closely at what passes from one round to the next, because it is less than one imagines: only the text passes. The token just chosen is appended and at the next round re-enters from the ground floor like all the others, a vocabulary index that becomes an embedding, indistinguishable from those of the prompt; no trace remains of the previous round, no intermediate result, no hidden state. The model, every time, re-reads everything from scratch. It is an evident waste, and it will be the first place where inference will go to save.

It is worth pausing too on the word "chooses", which hides a step. The network does not choose: the network ends with the distribution, and there its work is finished. Taking a cell is a separate step, applied from outside, with a policy of its own: one can always take the entry with the highest percentage, or draw by lot respecting the percentages, letting a less obvious candidate through every so often. It is the reason why the same question does not give the same answer twice. The detail of how one draws has a craft of its own, which there is no need to open here: it moves no bytes, occupies no memory and costs a negligible fraction of the round. It is the only piece of the chain where the cost does not dwell.

All the power and all the cost of LLMs lie instead inside this loop. It is here that the word "generative" finds its precise meaning, stripped of marketing: a classifier produces a label (this photo is a cat, this email is spam) and stops there; a generative system learns the distribution of the data themselves, and from a distribution one can draw something new, that was not written anywhere. An LLM is the most literal case: its output that counts is a probability distribution over the next token, and generating is drawing one, appending it, repeating; it holds for text as for images or audio, the space changes, not the idea. A language model is a function that, given a text, assigns a probability to every possible next little piece of word: the rest of the text is the story of how much it costs to evaluate it, billions of times a day.

text so far the prompt and what follows tokens numbers, via tokenizer the function the parameters, fixed the cost dwells here probabilities over the vocabulary chosen token appended start again, one token per round The generation loop
The map of the whole piece. The function at the centre is always the same, with the parameters fixed once and for all; it is the text that lengthens, one token per round. All the power and all the cost of LLMs dwell inside this loop: the rest of the text tells how much a round costs, and how one makes it cost less.

2. Training: the descent and the three phases

It remains to say how they descend, the parameters, towards the minimum loss. First, though, the link that holds everything together and risks going unnoticed: the probability on the true token is not a judgement given from outside, it is an output of the function itself, that cell of the distribution which the softmax produces at the top of the stack. With the text fixed, the cell depends only on the parameters; and it is worth, from here on, seeing each parameter as a knob: the effect of turning one climbs the building, floor after floor, up to the percentages at the top. The complete chain:

turn a knob        →  the activations of the floors above move
                   →  the scores at the top of the stack move
                   →  the softmax redistributes the percentages
                   →  p(true token) rises or falls
                   →  the surprise falls or rises

The true token, in all this, is not touched: it is in the text, and continues to do its one and only job, to point out the cell to read. What moves are the numbers in the cells; and since the percentages sum to 1, raising the cell of the true token necessarily means taking probability away from the others. Learning, seen from here, is decanting probability towards the right cells, one hair of a knob at a time.

With the text fixed, therefore, the loss is a function of the parameters alone, and the question "by how much does the surprise move if I turn this knob by a hair?" is well posed for each of the billions of knobs. The answer has a name, the partial derivative, and under the name there is a ratio between two variations, which can be read in numbers. Take the miniature of section 1, with the inputs fixed and, let us say, a loss that at the moment equals 2.000; move only w₁₁ by one hundredth, leaving all the other knobs still, and redo the computation:

w₁₁ = 0.500   →   loss = 2.000
w₁₁ = 0.510   →   loss = 2.003        variation: +0.003 for +0.010

partial derivative of the loss with respect to w₁₁  =  0.003 / 0.010  =  0.3

How much loss for how much turn: it is all there. The value says how much that knob counts relative to the others, and the sign says the direction: positive means "raising it, the loss rises", so to make it fall that knob must be lowered. "Partial" adds no mystery: it means only that one moves one at a time, holding all the others still. It should be said at once that no one computes this way: shifting billions of knobs in turn and redoing the computation each time would be the very definition of the impossible. But the quantity sought is exactly this, and it is worth keeping it before one while looking at the real procedure, which obtains the same numbers by an incomparably shorter road.

The miniature also says what they depend on: turning w₁₁ moves y₁ in proportion to x₁, the input that weight multiplies, so a weight at work on a large input shifts the output more than one at work on near zero, a detail that in a moment will become the apportioning of blame. The mechanism of the descent, at this point, has intimidating names and a simple idea, and it is worth seeing it as a round of three steps, repeated billions of times:

  1. Forward: the prediction. A block of text is shown to the network; for each position the network produces its distribution over the next token, and the loss function reads the cell of the token that is really in the text: the average surprise over the whole block is the loss.
  2. Backward: backpropagation. For each of the billions of parameters exactly that ratio is needed, the partial derivative from a moment ago. One at a time is out of the question: it would be billions of forwards for a single step. Backpropagation obtains them all together, with a single descent from the top to the ground floor, and its cost is the good news of the whole procedure: those billions of numbers cost roughly as much as two forwards. The set of all these derivatives, one number per knob and therefore a list as long as the whole model, has a name that explains the next step: the gradient, the direction, in the space of the parameters, in which the loss rises most steeply.
  3. Step: gradient descent. If the gradient indicates where the loss rises, one moves the opposite way: each knob turns by a small step against its own blame. The width of the small step (the learning rate) is the permanent compromise: steps too long overshoot the valley, steps too short never reach it, and every step tramples a little of what had been learned before. In symbols, the whole step is: θ ← θ − η·∇L (θ the parameters, η the learning rate, ∇L the gradient of the loss); the whole of training is this line, repeated.
The building of the stack: up on the forward, down on the backward at the top a single cell is read: p(true token) → surprise at the top: the softmax scores → percentages top floor weighting + non-linearity … dozens of floors … first floor weighting + non-linearity ground floor the tokens enter (the embeddings) forward the effect rises, floor after floor blame at the top = 1 backward the blame descends, rescaled at each floor ×f ×f ×f factors far from 1: the blame dies out or explodes
The building of the stack and the two directions of travel. Up on the forward: the tokens enter on the ground floor, each floor works on the output of the one below, and at the top the softmax delivers the percentages, where the surprise is read on a single cell. Down on the backward: the blame starts from the top and at each floor is rescaled by a factor; how much those factors count is told by the numbers just below.

How backpropagation manages to obtain all those derivatives in a single shot deserves to be seen, because it is also the explanation of two or three things that otherwise seem whims. A knob of the lower floors does not touch the loss directly: it touches it through everything above it. It moves the activations of its floor, which move those of the next floor, and so on up to the cell at the top. Its overall effect is the product of the local effects met along the way, and for a three-floor building it fits in one line:

 ∂L/∂w      =      ∂L/∂a₃      ·      ∂a₃/∂a₂      ·      ∂a₂/∂w

 how much          how much           how much          how much
 the loss          the loss           floor 3           floor 2
 feels             feels              feels             feels
 the knob          floor 3            floor 2           the knob

It is the chain rule, and it should be read from right to left like a journey: the knob moves its floor, its floor moves the one above, the one above moves the loss. The point that makes everything work is that every factor is local: it concerns one floor only, and to compute it it is enough to know what that floor just did in the forward pass. That is why the count can be done backwards once only. One starts from the top with the first factor; descending, at each floor traversed one multiplies it by the factor of that floor. What arrives at any floor is already the product of all those above: that floor need only use it for its own weights and pass it, rescaled, to the one below. No path is ever repeated, and it is from here that the cost of only two forwards comes.

The historical jargon calls this accounting credit assignment, assigning to each weight its share of merit and of blame, and "blame" is the right image, because the apportioning is not fair on principle: whoever contributed more to the error receives more of it. It is the detail of the miniature come to the point: a weight that multiplied a large input moved the output more than one that multiplied near zero, and the derivative records exactly this.

And it is the same line that explains why section 1 asked for derivatives that "do not die out". On a real stack that product does not have three factors: it has one per floor, dozens in a row. The blame does not pass whole from one floor to the next, it is multiplied at every traversal, and chains of multiplications are merciless with factors far from 1:

blame at the top = 1, stack of 30 floors
factor 1.0 per floor  →  1.0³⁰ = 1        arrives at the bottom whole
factor 0.9 per floor  →  0.9³⁰ ≈ 0.04     4% of it arrives
factor 0.5 per floor  →  0.5³⁰ ≈ 10⁻⁹     dies out along the way
factor 1.1 per floor  →  1.1³⁰ ≈ 17       the opposite ill: it explodes

Below 1 the blame dies before reaching the bottom and the lower floors are left without guidance; above 1 it explodes and overwhelms everything: the stability of a deep stack wants factors close to 1. The factor of each floor depends also on the derivative of the activation function, and here are the "almost prosaic" criteria of section 1 at work: the sharp switch of the ReLU passes the blame whole on the positives but zeroes it in bulk on the negatives (clear road or wall); the rounded elbow of GELU and SiLU keeps the factor small but never zero, and a way to reach the bottom always stays open.

And it is here that the normalization named in section 1 earns its keep. By bringing the numbers back onto a common scale at each floor, it keeps the activations away from the extremes, and with the activations the factors of the backward pass: the chain stays in a zone where it neither dies out nor explodes. It adds no mathematics, teaches the model nothing, weighs almost nothing in bytes; it is hygiene, and without hygiene tall stacks simply do not train. A second invention will serve the same purpose, and will arrive with the 2017 architecture.

The whole round fits in a single image: a landscape. Each point is a complete setting of the knobs, that is, one of the possible models; the elevation of that point is the loss that model records on the text; taking a step is turning all the knobs a little, moving to a nearby point. The one who walks is the model itself, and the path is the training. The landscape has billions of dimensions and thick fog: one does not see the valley, one only feels the slope underfoot (the gradient), and one trusts that descending locally, one small step at a time, leads somewhere. That it works was not at all obvious, and it is largely an empirical discovery: there is no need to find the lowest point in the world, a good valley is enough. No step of this procedure is intelligent; learning is making a number descend, measuring the surprise and turning each parameter by a small step in the direction that reduces it, thousands of billions of times.

A practical detail completes the picture: the step is not computed on all the available text, but on one block at a time (the batch), and the slope felt at each step is therefore a noisy estimate of the true one; it is the reason for the full name of the procedure, stochastic gradient descent. The noise is not only a defect: it shakes the descent just enough not to stop in the first hollow of the landscape.

The guiding thread, here, presents the steepest bill of the whole text: each training step moves all the bytes of the model at least twice (forward to predict, backward for the blame), plus the surrounding bytes that the optimizer keeps for each parameter (the bookkeeping of the descent, which for each knob preserves a few service numbers, like the average of the recent steps). It is the physical reason why training occupies entire datacentres for months, and it is the colossal expense paid once. What comes out of it is a block of numbers: the shelves of the library, filled once and for all.

"Training", in the singular, is then a simplification: for a modern LLM the phases are at least three, and they answer different questions. Pretraining is the self-supervision of section 1 pushed to the maximum scale: it decides what the model knows, and dominates the costs by orders of magnitude. Supervised fine-tuning takes that model back up and corrects it on curated examples of dialogue: it decides the form, that is, that a question is answered, instead of continuing it with other similar questions as a pure text completer would do. Learning from preferences (the RLHF family: answers put side by side, human judgements or those of a judge model turned into a reward with the tools of reinforcement) decides what the model chooses when the admissible answers are many: tone, prudence, style. Same descent mechanics in all three, incomparable budgets: knowledge costs like an infrastructure, behaviour like a project.

From that moment the model no longer learns. Every answer, however brilliant, leaves it identical to how it was: the conversation lives in the context, which vanishes at the end of the session, not in the parameters. It has already been told, as "empty memory", by the energy cost of a language model; here it is enough to fix its architectural consequence: everything that follows, inference with its phases and its optimizations, operates on immutable parameters. It is a constraint, but also the advantage that makes inference optimizable: one can compress, copy, distribute and cache what never changes. Training fixes the parameters once and for all: from then on the model no longer learns, it just answers.

3. Interlude: the parrot and the library

"It just answers" opens a question that has divided the field for years: what is it doing, exactly, while it answers? In 2021, when models were already fluent but not yet in everyone's pocket, a much-discussed paper gave the sceptical position its definitive name: stochastic parrot. The thesis is clean: a system trained to minimize the surprise on text (section 1) manipulates forms, not meanings; it stitches back together fragments seen in training following the probabilities, without any anchoring to the world the text speaks of. And the thesis seemed obvious for an honest reason: it is the literal description of the recipe. In the loss function there is no term for truth nor for understanding; there is how often the words follow one another. Whoever coined the label was not making a caricature: they were reading the definition. Moreover it warned of a real risk, mistaking fluency for understanding, which remains sensible even if the thesis turned out to be wrong.

Over time, though, the evidence on the ground has opened cracks hard to ignore. The one closest to the thread of this text is a count of bytes: the training text weighs tens of terabytes, the parameters a couple of orders of magnitude less. A literal parrot would not fit: to predict well with so little space one has to compress, and to compress at that level means finding regularities (grammar, facts, relations, styles) that hold even on text never seen. To this have been added behaviours that stitching explains badly: problems solved in combinations never appeared in training, and learning from context (in-context learning: with parameters strictly still, the model learns a new task from the examples in the prompt, something none of the previous sections explicitly foresaw obtaining).

The most interesting crack, however, does not come from behaviour: it comes from looking inside. A by now classic experiment trained a small transformer only on sequences of Othello moves, never a board: in its activations one finds, readable and manipulable, a representation of the state of the board, which the model built for itself because it compressed the moves better. And in large language models the field called mechanistic interpretability has learned to isolate the features: not single "neurons" nor physical zones of the chip, but directions and sparse combinations in the space of the activations that light up when generation touches a concept (a place, a tone, an insecure coding practice), often independent of the language and of the surface formulation. It is the precise version of the intuition that "different zones activate depending on the subject": true, provided one adds that the zones are geometries distributed in the reading table, not departments of the library. And for the learning from context of a moment ago there even exists a mechanical candidate: the induction heads, pairs of attention heads that seem to implement "I have already seen A followed by B; now I see A again, I propose B"; with the due caution, because saying that a head "does" something is an after-the-fact reading of a pattern, not a programmed function.

If a feature can be isolated, it can also be pushed. Amplifying it or turning it off by hand, during generation, shifts the behaviour predictably: it was shown with a deliberately theatrical experiment (a model with the feature of a famous bridge kept switched on, which slipped the bridge into every answer), and in the laboratory directions are isolated for whole behaviours, sycophancy, the refusal to answer. It is the concept at the base of steering: intervening on the activations, not on the weights nor on the prompt. In the lexicon of this text: acting on the reading table, leaving the shelves intact, the parameters/activations pair seen at the start put to work. And it is no longer only laboratory: there is already someone who exposes it as a command-line option in a real inference engine, one direction per layer in a half-megabyte file, to adjust verbosity or willingness on certain subjects without touching the weights. It should be said with the same honesty: they are young tools, the isolated features are a minimal fraction of those that exist, and from being able to push a behaviour it does not follow that one can read everything.

Why then do two parties remain? In part because half the quarrel dwells in the word "knowledge": if to know requires experience of the world, intention, anchoring to truth, the parrot remains a parrot by definition, and no experiment on the activations scratches it; if internal structure that generalizes is enough, lets itself be isolated and lets itself be manoeuvred, the evidence accumulates on the other side. And in part because the stakes are different: those who fear anthropomorphization and hype have serious prudential reasons (fluency deceives, and marketing raises the bid); those who work inside the models see structures that the parrot metaphor no longer covers. The position of this text is minimal and declared: the library is too small to contain what it has read, and this alone forces something more interesting than repetition; whether one calls it knowledge is a choice of vocabulary, that there is structure is by now an observation.

There is one last effect of looking inside, and it is worth declaring, because it accompanies all the rest of the text. Opening the machine produces two opposite sensations, and they do not cancel out: they coexist. The first is a kind of aridity: in there is no one; there are sequences of numbers, weightings, the same iteration repeated billions of times, and every brilliant word comes out of that loop. The second is its opposite: precisely because no one wrote those structures line by line, finding them (the board, the features, the directions that let themselves be pushed) shows an organization that training made emerge on its own, that no designer dictated and that one is still learning to read. The boundary between what one can explain and what has simply emerged, in the shelves, is not sharp. Whoever opens the machine takes away both things, disenchantment and wonder; and it is probably the sign of having really opened it.

4. 2017: attention, or managing the context

To predict the continuation of "the dog bit the" one needs to know what scene is being spoken of, who the subject is, what has already happened: the context is needed. The previous architectures (the recurrent networks) read the tokens one at a time, carrying along a summary that faded: the distant past was lost, and above all the reading was sequential by construction, impossible to parallelize well on the hardware that meanwhile was becoming ever more parallel.

The 2017 paper, "Attention is all you need", proposes to overturn the setup: no reading in a row, no summary dragged along. All the tokens of the sequence are processed together, and each one decides on its own, dynamically, which others to pay attention to, whatever the distance. The mechanism is called attention and the intuition sits comfortably in the library: each token formulates a question (its query), displays a label (its key) and carries a content (its value). Where these come from should be said at once, because they are not new ingredients: question, label and content are three weightings of the token's point, of the kind seen in section 1, with three parameter matrices that training fixes like everything else. One compares each one's question with everyone's labels, the scores become weights, and each token receives a weighted combination of the others' contents: a tailor-made summary of the rest of the sentence. Written as in the paper, the whole operation fits in one line:

Attention(Q, K, V) = softmax(Q·Kᵀ / √dₖ) · V

and read in pieces it is exactly the library: Q·Kᵀ compares each question with all the labels (a matrix of scores, one number for every pair of tokens); the division by √dₖ puts them back in scale; the softmax turns them into percentages; the final multiplication by V takes the weighted average of the contents. Three matrix multiplications and a softmax: the formula's fame is all in the scale at which it is executed.

It is worth following it too with the shapes in hand, because it is there that one sees the birth of the count that will govern all the second half of the text. In come T rows of d, as at every floor; the three weightings derive as many, a question, a label and a content per token:

input           T × d
normalization   T × d      row by row, mixes nothing
Q, K, V         T × d      each, from the three weightings (matrices d × d)
Q · Kᵀ          T × T      the table of scores: every token by every token
softmax         T × T      each row becomes percentages that sum to 1
(…) · V         T × d      back to the input shape

That T × T in the middle is the object to keep an eye on: it is the only thing, in the whole transformer, that is not made of d, and doubling the length of the text quadruples it. The parameter matrices, around it, stay d × d and do not notice. Note too that at the output the shape is that of the input, T × d: it is what allows the floors to be stacked at will, and the result to be added to what had entered.

That √dₖ deserves a pause, because it seems an arbitrary adjustment and is not. The comparison between question and label sums a contribution for every dimension of the space: the more dimensions there are, the more terms are summed, and the more the scores spread out. They grow, typically, like the square root of the number of dimensions, which is precisely √dₖ. Dividing by that root puts the scores back where the softmax works well: without it, in a wide space the scores would arrive already so far apart as to collapse the percentages onto a single winner, and the attention, instead of distributing, would nail itself onto a single token.

The operation is not executed just once, but with several "heads" in parallel, and here a frequent misconception must be cleared away: the heads do not multiply the cost. The point of each token is divided into slices: with h heads, each works on d/h numbers, and on that slice does its complete round of questions, labels and contents, with its T × T table; at the end the h outputs, d/h long each, come back together to form d again. It is the √dₖ of a moment ago that becomes clear here, incidentally: that dₖ is the slice, d/h, not the whole d. Eight heads on eighth-sized slices cost as much as a single head on the whole: one buys variety of viewpoints, not computing power. What is multiplied by h are the score tables, which are h instead of one, and indeed it is on their side that one will go to save. What they look at, then, is a more open question than is usually told: the heads stand in separate subspaces so that they can specialize, and some let themselves really be read (one follows the agreement between subject and verb, another brings the pronouns back to their antecedent), but many show no clean role, and a substantial share can be removed without the model noticing. Stacked for dozens of layers and alternated with blocks of pure computation, this is the transformer. The floor of the building, seen up close, thus has two rooms: the attention, the only place where the tokens talk to one another, and the block of computation, the old pair weighting and non-linearity at work on each token on its own. In front of each of the two, in the modern recipes, stands the normalization of section 1: it works on one row at a time, mixes nothing and hands back the shape it received, T × d, with a handful of parameters, of the order of d against the d² of the matrices around it. It is also the reason why, further on, one will see that no one dreams of compressing it. And alongside the two rooms runs a direct path: the result of each is added to what was entering, and the part that need not be changed continues intact, without having to be rebuilt floor by floor. It seems a plumbing detail and is instead the second of the two inventions announced in section 1, the one that together with the normalization makes deep stacks trainable: the sum offers the blame of the backward pass a shortcut in which the factor is worth 1, and the chain of multiplications of section 2 stops being merciless.

One detail must be told in full, because left half-told it forces one to trust: attention, on its own, does not know the order of the words. It compares each token with all the others, not a sequence: "the dog bit the man" and "the man bit the dog" would produce the same comparisons. The order must therefore be reintroduced, and the solution is in the spirit of section 1: the position too becomes geometry. The first recipe gives the position a point in space (computed with a fixed formula in the 2017 paper, learned like the embeddings in many models that came after) and adds it to the token's embedding: the point that enters on the ground floor says together what it is and where it stands, and the network learns to read the two pieces of information from the same coordinates. The modern variants move the trick inside the attention (in the jargon, the rotary embeddings, RoPE): instead of marking the point at the input, they rotate question and label by an angle that grows with the position, so that the comparison between two tokens feels their distance, not the absolute place; and it is the distance, for language, that is the information that counts. The craft of encoding it at best is still being refined, but the principle remains one: the position is not a tag hung on the token, it is written in the point.

In LLMs a single rule is added, the causal mask: each token can look only at those that precede it, never at the future. The reason must be told in full, because it is not a formal precaution but what makes training economical. In section 1 the text was run through "position by position", and one might imagine a pass for each: it is not so. A single forward on a block of text produces simultaneously the prediction of all its positions, and it is for this that training on thousands of billions of tokens is thinkable. But if every position sees the whole sentence, the one in turn also sees the token it should guess, and the task becomes copying. The mask is the price that makes the saving legitimate: all the positions together, each walled in its own past. Here too a miniature is worth a thousand words: here are the attention weights of a sentence of four tokens, after softmax and mask,

            t1     t2     t3     t4
t1  [ 1.00    0      0      0  ]     each row: how that token
t2  [ 0.33  0.67     0      0  ]     distributes its attention
t3  [ 0.25  0.25  0.50      0  ]     (and sums to 1)
t4  [ 0.10  0.20  0.30   0.40  ]

where the zeros above the diagonal are the mask at work (the scores towards the future are set to −∞ before the softmax, which turns them into exact zeros). This table, one number for every pair of tokens, is the central object of the attention: it is worth fixing it well, because it is the one that in a moment will grow with the square.

from the tokens' embeddings (or from the previous floor) Attention Q·K → weights → mixture of the V the tokens mix with one another sum with the direct path Block of computation each token on its own sum with the direct path to the next floor (for dozens of floors) the causal mask past future: forbidden each token sees only those before it
A floor of the stack, drawn from top to bottom for ease of reading (in the building of section 1, this route is climbing one floor). The attention is the only point where the tokens talk to one another: each query interrogates the keys of all, the scores weight the values, and the causal mask (the triangle) forbids looking ahead. The block of computation instead works on each token on its own. The dashed paths are the sums with the direct path: what need not be changed passes undisturbed, and it is what keeps the training of deep stacks standing. Dozens of these floors, stacked, make the transformer.

The reason the attention won, however, is not the elegance: it is that it is computed in parallel. The comparison of all the questions with all the labels is still it, the multiplication of matrices. The price is the table of a moment ago: T × T comparisons, a cost that grows with the square of the length. A thousand tokens are a million comparisons and no one notices them; a million tokens are 10^12 comparisons, and there is no hardware that can bear them done naively. It is worth being precise about that "no one notices them", because the other room of the floor, the block of computation, grows only in proportion to the length: on short contexts it is the one that dominates the count, and the square remains a minor term. The overtaking happens further on, and the more the context lengthens the sharper it becomes: it is for this that the square is not an always-problem, but the problem of the long context. This square is the character that will return in almost all the optimizations of the second half of the text. The attention is the mechanism by which each token looks at all the others and decides on its own which count; it won because it is computed in parallel, and its price, the square of the length, is the debt that inference has carried with it ever since.

5. The landscape of models

Before entering the craft of inference it is worth a photograph of what is around: "model" is a single word for very different objects, and the differences that count can be ordered along a few axes. The axes age slowly; the answers, quickly (more on this at the end of the section). Six questions are enough to frame almost every model one meets:

The question The answers, today Why it counts
Who can have it? closed (used only via a service); open-weights (downloaded, with a licence to read); truly open (weights, code and data) without open weights, no machine of one's own
How much does it weigh? from a few billion to thousands of billions of parameters the memory to hold it: the first item of every bill
How does it work per token? dense (all the parameters, always) or with experts (few at work at a time) computation and bandwidth per token; more on this among the optimizations
What notes does it take? full attention; compressed; selective the cost of the long context (sections 7 and 8)
What craft can it do? base; assistant; "reasoner" (spends tokens to think before answering); multimodal (images and audio too) it is the legacy of the three phases of section 2
In what form does it travel? full precision; already compressed at birth; community quantizations the bytes to download, hold and move

On the first axis a clarification is needed, because the current lexicon confuses: most of the models called "open" are open-weights, not open source. Since 2024 there exists a formal definition of open-source AI (weights, code and sufficient information on the data, with full freedom of use: it is in the references), and almost no flagship model satisfies it: the weights are downloaded, the recipe stays at home. It is not a quibble: with the weights alone one can run, inspect, adapt; one cannot remake. And even among open weights the licences are not all equal: some are full free licences, others carry conditions of use to read before building on top. On the other axes a regularity of the moment is worth noting: almost all the large open-weights models of today are with experts, because it is the architecture that makes the size sustainable, and compressed or selective attentions are spreading from the leaders to the rest of the field.

With these axes in hand, the choice of the tailor-made engine that one will meet shortly stops seeming a coincidence: DeepSeek V4 Flash is open-weights with a free licence, with experts (few parameters at work per token), with the most compressed notes of its category, distributed already compressed at birth, and it bears aggressive compression well. Each of these items is a requirement for the single machine: a closed model cannot even be downloaded; a dense one of the same size would not fit the bandwidth budget; one with cumbersome notes would burn the memory on the context alone. Not all models can be run at home, and it is not a matter of effort: the choice of the model is half the choice.

A caution closes the photograph, and is itself a lesson: this taxonomy is a snapshot of mid-2026, and the frontier moves by weeks, not by years; the names and numbers cited in these pages will age soon, some perhaps have already aged. The axes, though, move much more slowly than the answers, and it is the reason why it is worth learning them. Understanding the basics and staying up to date are not alternatives: the first thing serves precisely to make the second possible, and this text tries to give the first knowing it cannot give the second.

6. Inference: a craft made of two jobs

Here the text arrives at its centre, and it is worth entering it with the reader's question: what happens between the sending of the prompt and the first word that appears? Two different jobs happen, and almost all the engineering of inference is born from the fact that they are really different.

The first job is the prefill: the prompt, which can be long (instructions, documents, the previous conversation), is digested all together. The tokens are all there already, so the attention can work in parallel on the whole sequence: it is the regime for which the transformer was born, the matrices are large, the GPUs full, the limit is the computing capacity. It is the waiting time before the first word.

The second job is the generation: from there on the model produces one token at a time, and each new token depends on all the previous ones. No more parallelism on the sequence: a queue, one step per token. And at each step, to produce a little piece of word, the model must consult all its parameters: every shelf of the library, reread in full, to choose a syllable. The computations to do on those numbers are few; the real work is carrying them from the memory chips to the computing circuits. The energy account already recalled gave the physical measure (inside a chip, going to fetch a number from memory costs from a hundred to seven hundred times more than doing a computation with it); here one sees the architectural consequence: the prefill is limited by computation, the generation by the memory bandwidth. Two different bottlenecks in the same answer.

At this point the on-the-ground proof announced at the start comes onto the scene, and it deserves a proper introduction. DwarfStar (ds4) is an open-source inference engine, MIT licence, that Salvatore Sanfilippo (antirez) wrote in C with a radical choice: not a general-purpose engine, but a tailor-made engine, for a specific model and for the hardware it has in front of it. It is born with DeepSeek V4 Flash on unified-memory Macs, with the declared aim of making it run well on a single machine; the perimeter, however, has already widened, and in the direction that counts: not only to new hardware (CUDA systems, consumer APUs), but to a model of another family, GLM 5.2, set alongside DeepSeek. Specialization is the method, not the fence; that the fence has moved without falling is the best proof that the bet holds, and how the transplant happened can be read in the code, in section 9. Optimizing for one model-hardware pair at a time is what allows one to pull out the maximum; what use it is to know how much the maximum is, the final questions will say. A few files of C with no dependencies, the kernels rewritten for each piece of hardware, and inside, readable one by one, almost all the moves of this text: asymmetric quantization, the compressed cache with its indexer, the cache on disk, streaming from SSD, distributed inference, even the steering of the interlude (it is ds4, the "real inference engine" cited there). It is the reason it makes an ideal on-the-ground proof: in a general-purpose engine the choices hide behind the abstraction, in a specialized one they stand naked in the code; and a suite of test vectors, compared byte by byte with the answers of the official API, keeps all the rest honest. The project's README is itself a small lesson in inference, and it is in the references at the end. And a transparency of the project deserves an echo here, because it pairs with the opening of these pages: the authors openly declare that the code is written with strong assistance from an LLM, with the humans guiding the ideas, the tests and the debugging.

It is worth looking closely at what "tailor-made" means, because it is not a slogan: it is written in the code, starting from the opening comment of the main file. There the engine declares its fixed shapes: it accepts the known layouts of its model and refuses to start in front of any other. A general-purpose engine does the opposite by trade: it reads the shape of the network from the file's metadata and must know how to run whatever shape it finds there, and the flexibility is paid for in layers of abstraction, edge cases, missed opportunities. Knowing the shape in advance means that every data structure and every kernel can assume it. And the specialization is visible to the naked eye, in the list of the kernel files:

softmax.metal   norm.metal    flash_attn.metal    the operations of any engine
dsv4_kv.metal   dsv4_hc.metal dsv4_rope.metal     the compressed cache, the heads,
                                                  the position: of that model

alongside the universal tools, kernels that carry the model in their name: its compressed cache, its compression of the heads, its encoding of the position (the rotation of section 4), each with the kernel written on purpose.

The tailoring does not stop at the kernels: it is a supply chain. The weights are not the "community" files: they are files co-designed with the engine, produced by a dedicated quantizer, a couple of thousand lines of C that declare in their own opening comment that they keep only the pieces the recipes of that model need (it is there that the asymmetric quantization just named lives). And there is the safety net that allows one to push this far: the test vectors of a moment ago, captured from the API with the most prudent decoding and reproduced locally to the letter, so that a regression of the tokenizer or of the attention emerges at once, before becoming wrong answers; and a suite of about ninety problems that runs the same inference path as the users', to be re-launched after every change to a kernel or a quantization. The declared ambition is to make a local model seem "finished", not only executable; the lesson of method is that every optimization arrives together with the tool that measures its damage. There is, finally, a debt, declared with the same frankness: without llama.cpp and GGML this engine would not exist (the quantization layouts, the tables, even some adapted kernels), to the point that the copyright of those authors remains in the licence file. The tailor-made engine stands on the shoulders of the general-purpose one; it surpasses it in a single point because it gives up all the rest.

Its numbers fix the signature of the two phases: on the same machine and with the same model, the benchmark shows on the order of 250 tokens per second in prefill and ~20 in generation (numbers of mid-2026, orders of magnitude and not fixed measures: the project itself warns that optimization runs faster than the published benchmarks). It is not a defect of the software: one finds it again, with different numbers, on any system. Even sharper the counter-proof of distributed inference: dividing the model over two machines, the prefill speeds up (the pieces of prompt are worked in a pipeline), the generation slows down (each token pays one more network trip, and the trips do not shorten by pairing up).

How much the trips count is told by the scale of the connections, and it is a proof anyone can redo with two computers. Same two machines, same model (two M5 Max, 91 GB Flash quant, a prompt of a few thousand tokens): on a direct cable, with half a millisecond of latency, the generation does 25 tokens per second; on WiFi, with the latency rising towards a hundred milliseconds, it drops to 10; on a VPN across the Internet, to less than 4. The prefill travels the same scale in a different way: it travels in large blocks in an assembly line, so what counts is how much stuff passes per second; the generation ships tiny packets but one per token, so what counts is how long the single packet takes to arrive. It is the question of the guiding thread with one more currency: "how much it costs to move them" is paid in bytes and in milliseconds, and the two phases pay in different currencies. The cleanest confirmation is a negative experiment of the project: halving the bytes of the activations in transit shifted almost nothing, so much that the option risks disappearing. At generation the count is not the volume of the parcel: it is the number of deliveries.

The same signature is read, reversed, in the hardware. With the same model and engine, the generation speed chases the memory bandwidth almost in proportion: a workstation with traditional RAM (80-90 GB/s) generates 2-5 tokens per second, a laptop with 400 GB/s unified memory does about twenty, an 800 GB/s machine about thirty. And the counter-example closes the count better than any argument: a graphics card with the highest bandwidth of all (1000 GB/s) but 24 GB on board does not even start, because the model does not fit. For generation, bandwidth and capacity are needed together: it is the reason why unified-memory machines, which have less bandwidth but share dozens or hundreds of GB between CPU and GPU, have won precisely this use case. The symmetric paradox also holds: a small system born specifically for AI, with capacity and computation in abundance but modest bandwidth, goes very fast in prefill and generates at half the speed of a desktop computer. No spec sheet reports the item "bandwidth to generate": that is the one to go and look for. Answering is made of two different jobs, digesting the prompt all together and producing one token at a time, and whoever optimizes, whoever buys, whoever compares must know which of the two they are paying.

Prefill the whole prompt together, in parallel limit: computation first word Generation one token at a time, in a queue limit: the memory bandwidth at each step, all the parameters reread time
The two phases of the same answer. The prefill works all the tokens of the prompt together: the matrices are large, the limit is computation, and its duration is the wait before the first word. The generation is a comb of identical steps, one token per step: the dominant work is rereading the parameters at each step, and the limit is the memory bandwidth. The teeth of the comb are drawn at the same distance, but their cost grows slowly with the context: it is the theme of the next section.

7. The notes on the table: the KV cache

There is an absurdity hidden in the generation, as it has been described: if every new token has to look at all the previous ones, at step one thousand the model would have to recompute from scratch the labels and the contents (the keys and the values) of a thousand tokens, at step one thousand and one of a thousand and one, and so on. The already quadratic cost would become intolerable at every single step.

The solution is the most important and least told object of inference: the keys and the values of the tokens already seen never change (the parameters are fixed, the past too), so they are computed once and kept aside. They are the notes on the reading table: the KV cache. At each step the model computes the key and value only of the new token and reuses all the rest. The generation becomes linear again; in exchange, the notes occupy memory, and grow with the length of the conversation, layer by layer, head by head. And "memory", here, is not a metaphor: they are precise bytes, with an address, to the point that they can be saved and reloaded, as we will see among the optimizations. When one discusses whether a model "remembers", this is the only working memory that really exists.

This trade (recomputation avoided against memory occupied) makes the KV cache the hinge-character of the whole text. It is the reason why the long context costs even when the model "is doing nothing"; it is the memory item that competes with the parameters for the same space; and it is the object that almost every optimization of the next section touches, compresses or moves. And its weight, for once, can be estimated by hand: bytes of the cache ≈ 2 (for K and V) × layers × heads × dimension per head × bytes per number × tokens in the context. A number to fix the scale: in a standard transformer, the cache of a context of a million tokens would occupy hundreds of gigabytes, more than the model itself. DeepSeek V4 Flash, with an attention designed to compress the notes (more on this shortly), brings it to ~26 GB: enormous in absolute terms, small for the category, and it is the single choice that makes a context of a million tokens thinkable on a single machine. The generation does not recompute the past because it keeps it in the notes: the KV cache is the true memory of the conversation, and its weight in bytes governs more design decisions than any other number.

The table at step N K and V of token 1 K and V of token 2 ⋮ K and V of token N−1 already written: reused K and V of token N only the new row is computed no rereads, no recomputations How much the notes weigh bytes of notes tokens in the context linear with the context
The KV cache trade, in its two sides. On the left the gain: the keys and the values of the tokens already seen never change, so they stay written on the table and at each step a single row is added, that of the new token. On the right the price: the notes grow in a straight line with the length of the context, layer by layer and head by head, and it is the memory item that competes with the shelves for the same space.

8. The optimizations: six answers to the same question

Seen from outside, the engineering of inference seems a catalogue of acronyms. Seen from the guiding thread, it is a family of answers to the same question: where the bytes are and how much it costs to move them. Here six are chosen, not for completeness but because they cover the fundamental moves; each is declared for what it is, a way to have fewer bytes, to move them less, or to move them while doing something else.

Fewer bytes per parameter: quantization. The parameters are born as 16- or 32-bit numbers; writing them with fewer bits (8, 4, even 2) makes them less precise but much smaller, and since the dominant cost is transporting them, a halved model is (almost) a model twice as fast at generating. The how is less brutal than it seems: the weights are grouped into small blocks (about thirty values) and for each block a scale factor is saved; the few bits of each weight say only the relative position inside the block, and the scale repositions the whole block. And the bits are not spent blindly: some calibration text is passed through the model, one measures which weights are really solicited (the profile is called imatrix), and the precision is concentrated there. It should not be confused with distillation, which is a different and smaller model trained to imitate the large one: quantization is the same model, written with shorter numbers.

The on-the-ground proof of ds4 shows how fine the craft is. DeepSeek V4 Flash in full precision would weigh ~570 GB; the 2-bit version of ds4 weighs ~80 GB with quality almost intact, but only because the quantization is asymmetric: the parts of the model active in fits and starts are compressed to the minimum (the "experts", about 90% of the parameters: the following move presents them), while what works on every token, the normalizations of sections 1 and 4, the router that chooses the experts, the projections of the attention, stays in high precision. The result is not "everything at 2 bits", it is a hierarchy of precisions, from the 2 bits of the experts up to the 16 of the most delicate parts, decided component by component. Compressing is not a uniform operation: one compresses where the error can be afforded.

The result travels in a container file, the GGUF of the llama.cpp ecosystem: inside are the already-quantized tensors, the metadata of the architecture and the tokenizer, so the file is self-sufficient. And the engine does not copy it into memory: it maps it (mmap) and reads the pieces on demand; where the memory is unified, even the GPU works directly inside the mapped file, without making a copy. Even opening a model, seen from the guiding thread, is a choice about where the bytes are. Two honest notes close the move. The first: the model "in the clear" often never existed (DeepSeek distributes weights already compressed at birth, in 8- and 4-bit formats: the ds4 quantizer translates between two already-compressed forms, it does not degrade a perfect original). The second: that most of the bits can be thrown away without collapse says something profound about the shelves: knowledge does not dwell in a few numbers of surgical precision, it is written in a distributed and redundant way, robust to noise; and the fact that the imatrix improves things confirms that not all weights are equal. It is the same lesson as the interlude, seen from the side of the bytes: in the shelves there is structure, not dictation.

Fewer bytes at work per token: the MoE. The Mixture of Experts changes the architecture: instead of a single block of computation per layer, many smaller "experts", and a router that for each token activates a few. DeepSeek V4 Flash has 284 billion total parameters but activates ~13 per token. Beware the pair: the total parameters must all be kept in memory anyway (the router could call anyone at the next token), it is the active parameters that determine computation and bandwidth per token. The MoE is therefore an explicit pact with the guiding thread: abundant memory in exchange for reduced transport. It is the ideal architecture where memory is wide and computation scarce, and indeed it thrives on unified-memory machines.

Fewer bytes of notes: compressing the KV cache. The attention can be redesigned so that the notes are born already compressed: small latent representations to "reopen" on the fly (DeepSeek's MLA), cache rows shared among groups of positions, local windows for the first layers. It is the road of the ~26 GB for a million tokens of section 7. Here the mathematics changes, and the quality must be revalidated: it is the difference, to hold on to tightly, from the pure implementation optimization. The canonical example of the latter is Flash Attention: the very same mathematics, but computed in little blocks inside the very fast memory of the chip instead of materializing the matrix of comparisons in the slow one. No approximation, 2-4 times faster: the square of section 4 does not disappear, but stops travelling. There is then a twin move, more radical, and it is the signature of DeepSeek V4: do not reread all the notes. A small indexer gives a score to the past tokens, and the real attention is computed only on the most relevant (the order of magnitude: a few hundred, chosen in a context of a million). Compressing answers "how much the cache weighs"; selecting answers "how many comparisons do I make", and it is this, more than compression, the move that really tames the square on the long context. Here too the mathematics changes, and the quality must be revalidated.

The same bytes for more users: batching. If the generation is limited by the transport of the parameters, and the parameters are the same for everyone, then serving several requests together is almost free: the shelves are reread once and the computations are done for ten reading tables. It is the reason why the throughput of a service grows well before the latency of the single user worsens, and it is the true economy of scale of the providers: the cost per token collapses with the load. And it is no longer only a thing for big providers: the same local engine can keep several independent sessions resident and serve them together, so a machine at home or in a company answers several conversations by rereading the shelves once only. The latency/throughput pair must be kept distinct precisely here: optimizing one is not optimizing the other, and benchmarks that do not declare which they are measuring are measuring nothing.

Not redoing the prefill: the cache that persists. The notes, once taken, can also be saved. If many requests share the same prefix (the system prompt, an agent's documents, the conversation so far), the KV cache of that prefix is computed once and reloaded, skipping the prefill. ds4 pushes it all the way: cache on disk, indexed by the hash of the prompt, with the declared bet that modern SSDs are fast enough to make it a first-class citizen. It is the same idea as the providers' "prompt caching", seen from the engine's side.

Moving the bytes while computing: hierarchical memory. When the model does not fit in the fast memory, the alternative to surrender is the hierarchy: keeping close what is always needed, and making the rest travel (from RAM to VRAM, or from SSD to RAM) while the circuits work on something else, hiding the latency behind the computation. ds4 does it in both directions (a managed cache between RAM and VRAM for the enormous contexts, streaming of the experts from SSD for the models larger than the RAM), with two refinements that say everything about the craft: the experts' cache refuses to exist if it cannot be locked in RAM (better small and predictable than large and subject to paging), and the preloading starts from a hotlist compiled into the program, the ranking of the most requested experts measured on real loads: even among the experts, some bytes are hotter than others. The overall effect of the streaming is a change of category: the RAM stops being a threshold (the model fits, or it does not fit) and becomes a scale of speed. Speculative decoding is a relative of the same family: someone proposes a few tokens, the large model verifies them in a single shot, turning sequential steps into a parallel step. In the engine two ways of getting the proposals coexist, and it is instructive. In the first the suggester is not a second model at all: DeepSeek is also trained to propose the token after the next (multi-token prediction, MTP), so the model acts as its own suggester and the engine is left only to verify. In the second (DSpark, an auxiliary draft of a little over five gigabytes released on purpose) a small separate model reads the internal state of the large one and proposes up to five tokens ahead. In both cases the large one remains the only judge: a rejected or unsure suffix falls back on the ordinary decoding. And in both cases the project admits without mincing words that for now the gain is marginal, with the reason coherent with all this text: speculation pays when the single step wastes bandwidth, but on an architecture that has already reduced the bytes per token (experts, compressed cache, indexer) little margin is left for the trick; predictable continuations like code gain more, prompts poor in structure can even slow down. Optimizations too have a budget, and it is always the same: the bytes the single step leaves on the table.

The moral of the section is not the catalogue: it is that the six moves compose, and that each declares its price (memory, quality to revalidate, complexity). Almost every inference optimization is a different answer to the same question, where the bytes are and how much it costs to move them; when one meets a new one, asking which of the six families it belongs to is more useful than learning its acronym.

FEWER BYTES TO MOVE SAME BYTES, MOVED BETTER the question of every move Where the bytes are, how much to move them? Quantization same weights, shorter numbers price: precision, compressed only where the error is allowed MoE (experts) few parameters per token price: the total parameters all stay in memory Compressed KV cache notes born small price: new mathematics, quality to revalidate Batching same shelves, more readers price: the single one's latency beyond a certain load Persistent cache the prefill already done is reused price: disk space and prefix management Hierarchical memory in transit while computing price: complexity; latency only hidden, not gone
The map of the section: six moves around the same question. On the left one reduces the bytes to move (writing them shorter, activating fewer, making the notes born small); on the right one moves better those that remain (one reading of the shelves for many readers, the work already done that is reused, the transport hidden under the computation). No move is free: the declared price is part of the move.

9. Inside the code: four close-ups

The moves can also be looked at from much closer, at the level where they really live: the code. This section goes down there, four times: on three of the six moves (the notes that persist, the shelves in streaming, the quantization) and on the distributed craft met in section 6, chosen because each teaches something that from outside is not seen. Whoever prefers not to go down can skip to the following section without losing the thread. The on-the-ground proof is always the same, DwarfStar, for the reason given at its introduction: in a tailor-made engine the choices stand naked in the code.

The notes on disk, up close. The persistent cache seems a simple idea, the notes are saved and reloaded; the code shows how much care it takes for it to really work. The key of each file is the fingerprint of the prompt taken as a sequence of bytes, not as a sequence of tokens, and the choice has a subtle reason: the model may have generated a token whose text, sent back by the client at the next round, splits into two different tokens; the bytes instead coincide, so the comparison on the bytes finds the reusable prefix anyway, and only the new continuation is re-tokenized. Inside the file, before the actual notes, there is a 48-byte header documented field by field:

"KVC"   version   quantization bits of the experts
reason for saving (new / continuation / eviction / shutdown)
how many tokens it covers      how many reuses it has had (the "hits")
when it was born               when it was last used

and it is not pedantry: it is what the eviction needs to be fair. When the space runs out, the policy is not a simple "out with the oldest": the hits cool down with a half-life of six hours, so a prefix used very often yesterday does not forever jump ahead of one used a moment ago. There is then a refinement that is the guiding thread in its pure state: these files are read with ordinary reading, not with the memory mapping used for the model, and the reason is declared: not to add other virtual-memory maps to a process that already has an enormous one. Even the choice of the system call is a decision about where the bytes are. On this basis the project builds the next step, which here it is enough to name: its native agent (a terminal assistant built directly on top of the engine) treats every conversation as its file of notes, and moving from one session to another is opening a different file, without redoing any prefill. If the memory of the conversation is a file, the conversation becomes portable like a file.

The shelves in streaming, up close. The streaming from SSD of section 8 has an interface that is already half a lesson: one does not list files nor experts, one declares a budget, how much memory to dedicate to the experts' cache, and the plan is made by the engine. First it puts under cover what must stay in RAM anyway (the parts of the model that work on every token), then it translates the declared gigabytes into a number of resident experts, choosing them starting from the hotlist seen among the optimizations; all the rest dwells on the SSD and is read when the router calls it, while the circuits work on something else. The discipline remains the one already stated, either locked in RAM or nothing; what the zoom adds is the contract: whoever uses the machine does not manage the shelves, declares how much space they have, and the engine derives the best possible plan from that constraint. The result is the spectrum of speed of section 8, made operational: more budget, more resident experts, more tokens per second; less budget, one drops a gear, but it runs. Put side by side, the zoom on the notes and this one say the same thing from two sides: in this project the disk is not a fallback, it is first-class memory; the notes are saved onto it, the shelves dwell on it, and the RAM stops being an entry requirement to become a choice of speed.

The quantization, pushed to the single floor. The hierarchy of precisions seen among the optimizations can be pushed further, and it has already happened, with an instructive experiment. The question: do all the floors of the building suffer compression in the same way? The tool: an instrument that dequantizes nothing; it reads two already-compressed versions of the same model, one at 2 bits and one at 4, uses the first as a base and copies into it, byte by byte, only the experts of the last six floors taken from the 4-bit version. The result: a file of ~91 GB instead of the ~153 of the full 4-bit version that, in the agreement checks of the answers, behaves statistically closer to the full 4 than to the pure 2. And the measure is declared for what it is, in the project's words: it is not a benchmark, it is a useful signal. Hypothesis, surgical intervention byte by byte, measure against a reference, honesty about the value of the measure: the method, in four steps.

Cutting the building, in two ways. The distributed inference of section 6 has a protocol, and the protocol tells the architecture better than a diagram. The model is sliced by contiguous floors: one machine owns the ground floor and the first thirty floors, the other the remaining ones up to the output; the coordinator keeps for itself tokenization, extraction of the token and prompt. Every generation step is thus a vertical journey that crosses the machines: the state of the token climbs the floors of the first, crosses the network, climbs the floors of the second, and the scores return to the coordinator. It is, physically, the extra network trip per token of section 6. The workers introduce themselves declaring the model's identity, quantization profile and slice of floors; every work parcel carries with it the fingerprints of the history of the tokens before and after its own span, so each machine verifies that it is working on the same conversation. And the error handling distinguishes two different ills: if the fingerprints do not match (the bytes arrived, but the history is wrong) the token history is replayed on the same route; if the connection dies (the bytes did not arrive at all) the route is abandoned and another machine is awaited. And the notes? They stay where they are born: each machine keeps the KV cache of its own floors, because shipping it at every step would cost more than the recomputation it avoids; what travels is only the state of the token, at the border between the slices. When a session is saved to disk, the coordinator asks each machine for the piece of its floors and recomposes a single file, the same as the notes seen in the first zoom: the building is sliced, the memory of the conversation is not. The rest, the parcel formats and the tricks that keep the prefill assembly line full, is in the code, readable; and the documentation closes with the customary honesty: no encryption nor authentication, trusted machines, same version of the program on both sides. A young protocol that declares its limits. At the network level, in the code, it all fits in a few packet types: the worker introduces itself with a HELLO (model identity, family, quantization profile, slice of floors), the coordinator sends WORK and receives RESULT, and with every parcel travels the 64-bit fingerprint of the token history before and after the span, the one that unmasks the machine working on the wrong conversation.

There is a second way of cutting, and it cuts in the other direction. Instead of giving whole floors to different machines, the work is split in two inside each floor, and the two machines work the same token at the same moment, exchanging partial sums of a handful of kilobytes at the "gates" internal to the computation. The two cuts serve different things:

   BY FLOORS (pipeline)                  BY WIDTH (tensor parallel)
   ┌─────┐  A: the first floors       ┌──┬──┐  A and B on the SAME floor,
   │ ▓▓▓ │A  ↑ a token climbs         │AA│BB│  on the SAME token, together
   │ ▓▓▓ │   crosses the network      │AA│BB│  each floor split in half
   ══════════  1 hop / token          │AA│BB│  exchange: partial sums,
   │ ░░░ │B  then B's floors          └──┴──┘  a few KB at each "gate"
   │ ░░░ │
   └─────┘  serves to: FIT IT IN         serves to: LOWER LATENCY
            (sum the RAM)                + whole model RESIDENT

The logic is all in the guiding thread. With a single machine and a model too large, the experts shuttle from the SSD, and it goes slowly; pairing up in width means that each machine keeps half the experts resident and never touches the other's half, so the summed RAM holds the whole model and only those few kilobytes per floor travel. It is the difference from the cut by floors, where the two machines do sum the RAM but pay one network trip per token: here the trips are many but tiny, and instead of fitting in a larger model they lower the latency. The measure says it without appeal, on the same model that on its own does not fit:

   GLM 5.2, 188 GB          two Macs (tensor)     one Mac (streaming from SSD)
   generation               ~17 tokens/s          ~5 tokens/s
   prefill (4096 tokens)    ~94 tokens/s          ~3–5 tokens/s
   memory                   all resident          experts read from the SSD

From here one also understands what is transferable. The project's bet is one model at a time, not one model forever: the model may change, the constraint remains (credible local inference on high-end personal machines). And the transplant has already happened twice, inside the project itself: when the perimeter widened to consumer APUs the engine was not generalized, the kernels were rewritten for the new hardware reusing the weight format, the cache, the test chain; and when GLM 5.2 arrived, a model of another family, its tailor-made pieces were added (its kernels, its weight layout, its quality fixtures) without dismantling the rest. New hardware and new model: the same move, in the two directions in which the fence could shift. In the code one even sees the different form of the three backends (a folder of kernels for the Macs, a single file for the NVIDIA GPUs, a collection of headers for the AMD APUs): the mathematics is the same, the form of the code follows the hardware. It is the practical sense of specialization as method: whoever has a different piece of hardware in front of them, or tomorrow a different model, can redo the tailor-made part while keeping the findings, because the findings live above the kernels, not inside.

The four zooms share a spirit, and it is the true content of this section. At every point of the system, even the smallest (the system call with which a cache file is read, the precision of six floors at the top of the building), there is margin to do better; and the method is always the same: one does the detail as well as possible, measures the impact, if the impact is there one keeps it and moves to the next detail; if it is not, one says so and goes back, as with the negative experiment on the activations in transit or the "for now marginal" gain of speculation. There is even the inverse move of quantization, in the NVIDIA GPU backend: where the fast memory is to spare, some compressed weights are re-expanded into a form more convenient for the computing circuits, spending bytes to buy time, with a controlled budget and a declared fallback (if the space is not there, one returns to the compressed kernels, and the program says why). No move is decisive on its own: it is the disciplined accumulation of measured details that brings a model of hundreds of billions of parameters onto a single machine. In an inference engine every detail has margin to be made more efficient; the method is to do it as well as possible, measure the impact, keep what pays and move to the next detail. And the code is written to be read: whoever wants to see the craft at work need not trust this account, they can open the repository and find the same ideas, with the numbers alongside.

At a glance, then, the surface of the project and the moves of this text that are read inside it:

   DWARFSTAR (ds4): snapshot of 21 July 2026

   models       DeepSeek V4 (Flash and PRO) · GLM 5.2 (another family)
   backends     Metal (Mac) · CUDA (NVIDIA) · ROCm (AMD APU)
   tailor-made  kernels per model+hardware · dedicated quantizer ·
                byte-by-byte test vectors · suite of ~90 problems

   reduce the bytes to move           move the same bytes better
   · asymmetric quantization          · batching (locally too)
   · Mixture of Experts               · cache that persists (on disk)
   · compressed KV cache + indexer    · hierarchical memory (SSD streaming)

   in two machines (or more)
   · by floors (pipeline)     sums the RAM, one network hop per token
   · by width (tensor)        same token together: less latency,
                              whole model resident

   reduce the steps
   · speculation              MTP (itself) or DSpark (external draft),
                              gain for now marginal

It is not the map of a product to buy: it is the inventory of the ideas this text has crossed, each readable in the point of the code where it lives. The same thing said at the start, now with the names in their place.

10. What you take away from the journey

The synthesis offered here takes the form of a list of questions and answers: a map of the topics treated so far. This map precedes the other questions (those of the closing section) that, it is hoped, the reading of this text helps to face a little more knowingly.

Why does the answer arrive in bursts, one word at a time? Because the machine is made that way: it produces a little piece of word, appends it to what is already there, and starts over from the beginning. There is no ready-made answer transmitted in instalments: every little piece is born from a complete consultation of the shelves, and the stutter one sees is the true rhythm of those consultations.

Why do price lists distinguish the words sent from those received? Because they are two different jobs. The words sent are digested all together, in parallel: a fast job per piece. The words received are born one at a time, and each costs a complete rereading of the shelves: a slow job per piece. Whoever sells charges more for the job that costs more.

Why does a long conversation cost and slow down, even in the moments when no one is writing? Because the conversation lives in the notes on the table: for every piece of text the machine keeps its row of notes, and the table grows with the conversation. Those notes occupy precious memory even at rest, and at every new word they must be consulted: a fuller table is a more expensive memory and a slower consultation.

Does the machine learn anything from what I write to it? No, not while it answers. The shelves are filled once only, before; the conversation lives on the reading table, and at the end of the session the table is cleared. For the machine to really learn something new one would have to redo, at least in part, the filling of the shelves: an enormous job, which no conversation sets off on its own.

Is a compressed model a worse model? The right question is not how much it has been compressed, but where. The knowledge on the shelves does not dwell in a few digits of surgical precision: it is written in a distributed and redundant way, and it bears rounding well, provided one compresses more where the error can be afforded and almost not at all where it cannot. And a well-compressed model is not presumed: it is measured, on the same tests as the original.

Why does the same question not give the same answer twice? Because the machine's output is not a word: it is a ranking of probabilities over all the possible words, and from that ranking one draws. The drawing has a knob (in the jargon, the temperature): all prudence, always the candidate at the top, repeatable but flat answers; or a bit of gamble, every so often a less obvious candidate, more varied answers. Usually the knob sits a little above zero, and it is for this that two answers to the same question resemble each other without coinciding.

Can the machine also run at home? Yes, and it is less strange than it seems: almost all the craft told so far works in that direction. Of some machines the shelves are public and can be copied; they can be written denser, squeezing more where the error can be afforded; in many modern machines, for every word only a small part of it works; and the notes can be born compact and, when needed, live on disk. The result is a whole library inside a computer one can buy, on a person's desk as in a company's cabinet: slower than a datacentre's, often a few steps behind the most advanced, but complete, with the data that does not leave the door and with a few tools in hand that the service from outside does not lend. How much it really delivers on one's own case, however, is written on no spec sheet: it is a proof one can do, with the questions of the closing section in hand; and it concerns anyone who has opportunity and need, the curiosity of a person as much as the hardware and the skills of an organization.

11. The questions that matter

There remains the point declared at the start: what use is all this to someone who does not write kernels, but has to choose, evaluate, or even just read the news without being swept along. It is useful for knowing which questions to ask, and the list is short.

How much memory is needed, and for what? (Total parameters, not active; plus the KV cache at the context one really intends to use: it is the question that decides whether a system fits in a machine, in a GPU, in a phone.) Is the load prefill or generation? (Long documents and short answers are one craft; long chats and long answers another; the bottlenecks are opposite.) Which quantization, and validated how? (The number of bits is not enough: the question is where it was compressed and with which quality checks.) Latency or throughput? (For a single user the first counts; for a service the second; the marketing numbers always choose the more photogenic one.) And the long context, how much does it really cost? (Not "how much context it supports", but how many bytes of notes it carries with it and who pays for them.)

They are questions about memory, almost all of them. It is not by chance, and it is the way this text would like to be remembered: to evaluate a system one need not know everything, one needs a few right questions, and they are almost all questions about the bytes, about where they are and how much it costs to move them.

There is a concrete comparison in which these questions all make themselves felt together: the model served in the cloud against the model run at home. The gap in quality must be told without sugarcoating: the frontier models live in the datacentres of those who trained them, and what runs on one's own machine usually chases, with months or years of delay. But the six moves of section 8 are shifting the window: asymmetric quantization, MoE on unified-memory machines, cache on disk are exactly the techniques that bring a model of hundreds of billions of parameters onto a single machine, and it is the reason why an engine like ds4 can exist. The questions of this section are the technical way to decide the comparison: how much memory, which load, which quantization validated how, which latency is really needed. And for whoever runs at home, the cost also has domestic units, heat, noise, battery: not by chance a local engine exposes a knob that trades speed for silence.

The rest of the decision is not technical, and it is well to declare it: where the data goes (locally it does not leave), who can change price, conditions or model from one day to the next (the lock-in), what it means for a company or a country to depend on someone else's inference (sovereignty is also this), and whether inference will become a commodity or remain a service of the few. They are stakes that deserve a discourse of their own, not an aside; here the point of contact is enough: whoever faces them without the questions of this section is choosing in the dark, because cloud and local are not two prices, they are two profiles of bytes.

There is, finally, a reading of DwarfStar worth making explicit, because it holds together the two halves of the decision, and the project suggests it none too subtly: a tailor-made engine, pushed all the way on a single model, is first of all a measuring instrument. It establishes where local inference really reaches, today, on machines one can buy. One arrives at local inference by very different roads: because it is the only possible solution, because it is an acceptable compromise or a way-station, for an opportunity to build, for one of the stakes just mentioned (sovereignty, privacy), to have in hand tools the cloud does not expose (the steering of the interlude); the list is long on purpose. But whatever the road, the sensible choice has the same prerequisite: knowing what one can really obtain, measured and not imagined. To discard the local route only because today more convenient or apparently more obvious solutions exist means giving up knowing its potential, and in certain scenarios that route might turn out to be the only viable one. A simple rule holds, then, which is the sense of all this text applied to a single decision: an option never measured is not an option, it is a hope.

The library, to close the circle. The parameters are shelves filled once and never touched again; answering is consulting them, a little piece of word at a time, with the notes on the table so as not to reread everything at each step; and almost all the engineering seen here is the craft of making an immense library served by narrow corridors work: writing denser, bringing the shelves closer, serving more readers with the same round. When the next acronym arrives (it will), the questions to ask it are already on the table.

There remains a measure this text borrows from the energy account it started from, and which it deliberately leaves open: there it concluded that the adequate metric is not the energy per token, it is the energy per right result. The techniques seen here all lower the cost of the token; whether that token is really needed, none of the quantities of this text can say. And it is the honest boundary to stop at: knowing where the bytes are and how much it costs to move them says how much an answer costs; how much it is worth is another question, and remains with whoever reads it.

References

  • A. Vaswani et al., Attention Is All You Need, 2017 — arxiv.org/abs/1706.03762
  • J. Su et al., RoFormer: Enhanced Transformer with Rotary Position Embedding, 2021 — arxiv.org/abs/2104.09864 (the rotary embeddings/RoPE cited in §4: position as the rotation of question and label)
  • R. Sennrich, B. Haddow, A. Birch, Neural Machine Translation of Rare Words with Subword Units, ACL 2016 — arxiv.org/abs/1508.07909 (the sub-word tokenization of §1: the closed vocabulary that can still write everything, derived from the data with a compression criterion; the BPE method adapted from P. Gage, A New Algorithm for Data Compression, 1994)
  • P. Michel, O. Levy, G. Neubig, Are Sixteen Heads Really Better than One?, NeurIPS 2019 — arxiv.org/abs/1905.10650 (the attention heads of §4: many can be removed without harm; with E. Voita et al., Analyzing Multi-Head Self-Attention, ACL 2019 — arxiv.org/abs/1905.09418, for the few heads with a readable role and the many without)
  • D. Rumelhart, G. Hinton, R. Williams, Learning representations by back-propagating errors, Nature 1986 — doi.org/10.1038/323533a0 (the backpropagation of §2: the derivatives of all the parameters in a single descent)
  • C. E. Shannon, A Mathematical Theory of Communication, 1948 — doi.org/10.1002/j.1538-7305.1948.tb01338.x (the surprise −log p of §1, the same quantity as the demon post)
  • T. Mikolov et al., Efficient Estimation of Word Representations in Vector Space, 2013 — arxiv.org/abs/1301.3781 (word2vec, now cited in §1: the linear regularities of the embeddings and the king/queen example)
  • J. Kaplan et al., Scaling Laws for Neural Language Models, 2020 — arxiv.org/abs/2001.08361 (for the mention of the scaling laws in §1)
  • T. Dao et al., FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness, 2022 — arxiv.org/abs/2205.14135
  • DeepSeek-AI, DeepSeek-V2 (introduction of Multi-head Latent Attention), 2024 — arxiv.org/abs/2405.04434 (the origin of the idea of the compressed notes cited in §8 as MLA)
  • DeepSeek-AI, model card of DeepSeek V4 Flash — huggingface.co/deepseek-ai/DeepSeek-V4-Flash (the source of the model numbers cited in the text; terminological note: the model card calls the mechanism CSA+HCA, Compressed Sparse Attention + Heavily Compressed Attention, where the literature says MLA; the indexer of §8 is the "Sparse" part of CSA)
  • C. Olsson et al., In-context Learning and Induction Heads, Anthropic 2022 — transformer-circuits.pub/2022/in-context-learning-and-induction-heads/ (the mechanical candidate cited in the interlude)
  • Specification of the GGUF format (ggml/llama.cpp ecosystem) — github.com/ggml-org/ggml/blob/master/docs/gguf.md
  • Open Source Initiative, The Open Source AI Definition (OSAID), v1.0, 2024 — opensource.org/ai/open-source-ai-definition (the formal definition of open-source AI cited in the landscape of models: weights, code and information on the data, with full freedom of use)
  • Y. Leviathan et al., Fast Inference from Transformers via Speculative Decoding, 2022 — arxiv.org/abs/2211.17192
  • E. M. Bender, T. Gebru et al., On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, FAccT 2021 — dl.acm.org/doi/10.1145/3442188.3445922 (the origin of the label, for the interlude)
  • T. Brown et al., Language Models are Few-Shot Learners, 2020 — arxiv.org/abs/2005.14165 (the in-context learning cited in the interlude)
  • K. Li et al., Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Task, ICLR 2023 — arxiv.org/abs/2210.13382 (the Othello experiment of the interlude)
  • A. Templeton et al., Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet, Anthropic 2024 — transformer-circuits.pub/2024/scaling-monosemanticity/ (the features, and the experiment of the bridge kept switched on)
  • A. Arditi et al., Refusal in Language Models Is Mediated by a Single Direction, 2024 — arxiv.org/abs/2406.11717 (one direction for a whole behaviour: the base of steering)
  • S. Sanfilippo (antirez), DwarfStar (ds4), inference engine in C for DeepSeek V4 Flash and GLM 5.2, MIT licence — github.com/antirez/ds4 (the state of the project and all its numbers cited in the text are a snapshot verified on 21 July 2026: the project moves fast, the repository is the up-to-date source)
  • "The energy cost of a language model", on this site (the cost of inference, the "empty memory", the computation/transport ratio)
  • M. Horowitz, Computing's energy problem (and what we can do about it), ISSCC 2014 — doi.org/10.1109/ISSCC.2014.6757323 (the 100-700x ratio between memory access and computation recalled in §6: the same source as the energy account)