Posts tagged #KV cache
-
A journey inside the machine that answers: how the bytes move in inference
What happens, physically, when a language model answers, and what changes if the one answering is a single machine, at home? A path through the principles of how it works, from attention to the craft of inference, with its two phases, its organization of the data, the techniques that make it cost less, read along one thread: where the bytes are and how much it costs to move them. In the background, an idea: local inference, as a choice and as a possibility, already matters today and will matter more and more. Keeping it concrete, an engine written in C that brings a model of hundreds of billions of parameters to run locally: DwarfStar, with which antirez shows how far one can push, and that this road can be travelled.