Posts tagged #Information
-
The Chinese room opens: intelligence, understanding and consciousness, between language and anthropocentrism
Intelligence, understanding and consciousness are three different questions, with three different standards of proof: the first is answered by measuring behaviour, the second by looking inside the system, the third has no answer today, and one can say precisely why. The path runs from Turing to Searle to Chalmers and reaches the new fact: the Chinese room has actually been built, it is a language model (LLM), and the tools of interpretability are beginning to open it, finding representations of the world built from text alone and written in categories that are not ours. Language runs through the whole picture: it describes reality as humans see it, translating our senses and the representation evolution granted us; a system trained on text therefore starts from an anthropocentric description of the world (perhaps an intermediate stage, until senses different from ours exist) and inside that description builds categories that may not be anthropocentric. In the middle, a test anyone can replicate on an ordinary computer with a half-billion-parameter model: the jagged profile of its abilities, its mechanical explanation, the distance between the system's scale of difficulty and ours.
-
A journey inside the machine that answers: how the bytes move in inference
What happens, physically, when a language model answers, and what changes if the one answering is a single machine, at home? A path through the principles of how it works, from attention to the craft of inference, with its two phases, its organization of the data, the techniques that make it cost less, read along one thread: where the bytes are and how much it costs to move them. In the background, an idea: local inference, as a choice and as a possibility, already matters today and will matter more and more. Keeping it concrete, an engine written in C that brings a model of hundreds of billions of parameters to run locally: DwarfStar, with which antirez shows how far one can push, and that this road can be travelled.
-
Maxwell's demon, taken seriously
An imaginary being watching a single molecule seems to extract work out of nothing, in defiance of the second law of thermodynamics. Dissolving the paradox took a century and a new idea: information is physical. Shannon's entropy of messages and Boltzmann and Gibbs's entropy of molecules answer the same question, Landauer's principle fixes the exchange rate between the two, and Szilárd's engine puts it to the test: how much a bit weighs, where it ends up when it is erased, and why there is no free lunch either way.