Maxwell's demon, taken seriously

"Entropy" shows up in contexts that could hardly be further apart, with definitions that seem unrelated; in fact they all measure the same thing: how much information is missing to know the exact state of a system. Maxwell's demon is the point where the two worlds touch

I have always been drawn to the subjects that touch Understanding, the kind with a capital U: the ones that, while you work through them, give you the feeling that, if you manage to master them and hold them all in your head at once, even if only for a few minutes, they bring you a little closer to grasping how the world works from the physical point of view. (And, come to think of it, are there really any other points of view that are in some way "concrete"?)

In some of these subjects, on reflection, one concept keeps appearing, in various roles, in forms that seem to have nothing in common: entropy. And what has always fascinated me is that, in those rare moments when something truly falls into place in your head, you sense that they are all the same thing. The reflections that follow are, first of all, something I need for myself, to try to put things in order.

From here on the voice returns to impersonal; but the thread of the argument is born of that intuition.

1. The hook: the demon and the "free lunch"

A single molecule in a box. A "demon" watching it. And, out of this nothing, useful work, apparently for free, apparently in defiance of the second law of thermodynamics.

This is the version stripped to the bone, and we will get there step by step. The original scene, the one everything starts from, was imagined by James Clerk Maxwell, and it is more crowded: it is worth setting it up with care, because the paradox lives in the details.

A closed box, full of gas, at the same temperature throughout. Seen up close, a gas is a swarm of molecules flying in every direction, bouncing off one another and off the walls. They do not all travel at the same speed: even in a lukewarm, uniform gas, faster and slower molecules coexist, and what we call temperature is the measure of their average agitation. Hot means high average agitation, cold means low. It is a point that will come back often; better to keep it in mind from the start.

Now divide the box in half with a wall, and open in the wall a tiny door, ideal, with no friction and no weight: opening and closing it costs nothing. At the door Maxwell places a little being with senses sharp enough to see individual molecules and anticipate their arrival. Its task is disarmingly simple: when a fast molecule comes from the left, it opens and lets it through to the right; when a slow one comes from the right, it opens and lets it through to the left; in every other case it keeps the door shut. It pushes no molecule, it adds no energy: it only chooses when to open.

And yet, choice after choice, the fast ones pile up on the right and the slow ones on the left: the right half grows hot, the left half cold. Out of a lukewarm, uniform box, a temperature difference has appeared, with no work spent.

To see why this is a scandal one needs the second law of thermodynamics, which is worth stating here in two of its forms. First form: heat, left to itself, flows from hot bodies to cold ones and never the other way; coffee cools down on its own, and you never find it hotter than before at the room's expense. Second form, equivalent: you cannot take heat from surroundings at a single temperature and turn it entirely into work. This is why every heat engine lives off a difference, a hot side it draws heat from and a cold side it discharges part of it into: work is extracted from the difference, not from heat itself. If a single temperature were enough, a ship could drive itself forward simply by cooling the ocean it crosses: there is as much energy around as one could want, but without a difference it cannot be spent. (The bookkeeping of this prohibition physicists entrust to a quantity called entropy; for now the prohibition is enough, entropy will have its space shortly.)

Here is the demon's "scandal": it manufactures the difference itself. Out of a uniform box it pulls a hot side and a cold side; at that point it would suffice to place an engine between them to obtain work, let the temperatures mix back together, and start again. Work out of nothing, forever: what physicists call a perpetual motion machine of the second kind.

Before the verdict, though, there is an honest objection to clear away, because it is the first that comes to mind: the door. A real door has weight, friction, a physical presence; is dismissing it as "ideal" not the real magic of the whole scene? The answer has two parts. The mechanical part: the door can be pictured as a small plate sliding inside the thickness of the wall. The gas presses on it flat, perpendicular to the sliding, and a force perpendicular to the displacement does no work: opening does not mean overcoming the gas's pressure. The push that moves it, on the other hand, acts along the motion and does perform work; but that work ends up as speed of the plate, not as smoke, and when you brake it to a stop you get it all back. Only friction remains, which would eat a share at every pass; but friction can be reduced as much as one likes, and no law of mechanics imposes a minimum on it. In the ideal limit the balance of the open-and-close cycle is exactly zero, and it is the same idealization as the frictionless piston that thermodynamics has reasoned with all along. The deeper part: the second law does not hide behind imperfections. If it needed friction to hold, it would be a law about badly made gears, not about the nature of heat; the prohibition claims to hold for the perfect machine too. Granting the demon ideal equipment is not cheating: it is putting it to the test in its most severe form. (And we will see shortly that physicists really did go looking for the trick in the door as well.)

The paradox, then, really bites: no obvious trick is left in the scene. The demon does not push, does not lift, does not heat: it watches and chooses. The whole mechanism is made of knowledge and choice. One of two things, then: either the second law has a hole in it, or watching and choosing have a physical cost that must show up somewhere in the account. Indeed, the whole story that follows can be read this way: every cost in this scene can be whittled down toward zero, except one.

Sixty years later Leó Szilárd stripped the scene to the bone and made it quantitative: away with the gas, away with the sorting; a single molecule, one bit of knowledge, one piston. It is the bare version of the opening, the one on which, further on, the account will be settled in full.

For a century this paradox held physicists in check. Its solution is one of the deepest ideas of the twentieth century: information is physical, and it has an exact price. But to get there one must first answer two questions that seem to have nothing to do with each other: what is information? and what is the entropy of thermodynamics? The demon seems to be the point where the two answers meet.

A century-long paradox

Born to illustrate, matured into a paradox.

It is worth knowing where it comes from, because its history is physics itself told in chronological order: at every stage, the "cost" that saves the second law has moved.

Maxwell conceived it in 1867 (in a letter to Tait, then in Theory of Heat, 1871) not as a real violation, but to show that the second law is statistical: it holds because we do not know how to manipulate individual molecules, not because a mechanical law enforces it. It was Kelvin, in 1874, who named it a "demon".

The real problem arose with the question: if the demon is itself a physical system, where does its entropy go? Smoluchowski (1912) showed that an automatic demon, a trapdoor on a spring, is defeated by its own thermal fluctuations, but left open the case of the intelligent demon. Szilárd (1929) reduced everything to the one-molecule engine and was the first to tie information to entropy, placing the cost in the measurement; Brillouin, around 1951, strengthened that reading: to see the molecule you need photons, and photons cost entropy. This solution held for some thirty years.

Then the centre of gravity shifted one last time. Rolf Landauer (1961) and Charles Bennett (1982) found the definitive location of the cost, and it was not where it had been sought for thirty years: not in the measurement, but at another point of the cycle. Exactly where, and why the bill cannot be dodged, will become clear further on, with the engine in plain view. It is the resolution that still stands today, confirmed in the laboratory in 2012 by measuring, on a single bit, the heat Landauer had predicted.

So, was it born a paradox? Yes and no. For Maxwell it was an illustration, and the solution (the statistical character of the law) was already implicit in the formulation. It became a genuine paradox only when an accounting was demanded that included the demon itself, and that took a century and an idea Maxwell did not have: information is physical. The words that follow try to rebuild that idea from the ground up.

2. What information is: Shannon's entropy

In information theory, entropy measures the amount of information or, which is the same thing, the uncertainty.

The starting point: given a set of messages (or events) with their probabilities, the information gained by reading a given message is greater the more improbable that message was. A foregone outcome tells you nothing; an unexpected one does. At the limit, a source that can emit only one message carries no uncertainty at all: reading it adds nothing.

Shifting the gaze from the single message to the source as a whole: the more messages are possible, and the more improbable they are on average, the larger the entropy of the source. The maximum is reached when the N messages are all equally likely, the situation of greatest uncertainty and therefore of greatest average information.

The entropy of a source, measured in bits, is then the weighted average of the information carried by each message.

At bottom, Shannon's entropy answers a single question: how much information is missing to know the exact state of the system? Here the "exact state" is which message was emitted, and entropy measures the average ignorance of it. That question is worth fixing in mind: it will return, identical, asked of a physical system.

In mathematical form: for a source X in which the probability of a message occurring does not depend on the messages already produced, and which can emit N different messages {X₁, X₂, …, X_N}, each with probability p(Xᵢ), the entropy H associated with it is:

H(X) = − Σ(i=1..N) p(Xᵢ) · log₂ p(Xᵢ)

The limiting cases close the circle with the opening observations. With N equally likely messages (p = 1/N) the entropy reaches its maximum, H = log₂ N: a fair coin is worth exactly 1 bit. Not by accident: the bit is the uncertainty of a clean choice between two alternatives (log₂ 2 = 1), and this is what fixes the base 2 of the logarithm; the ln2 that will appear later is just the price of changing base. A six-sided die is worth log₂ 6 ≈ 2.58 bits. At the opposite extreme, a single certain outcome (p = 1) gives H = 0: no uncertainty, no information.

Operational meaning. The entropy H is not just a formula: by Shannon's source coding theorem it is the limit of lossless compression. On average, those messages cannot be represented with fewer than H bits. This gives the "bit" a concrete meaning, and it is precisely the unit the demon will write and erase.

3. What entropy is in physics: Boltzmann and Gibbs

The other entropy is born in a world that seems to have nothing to do with messages: heat, gases, the bustle of molecules. It enters the scene with Clausius, in the mid-nineteenth century, as pure bookkeeping of heat: the heat exchanged divided by the temperature at which the exchange happens (hence its units, Joules per Kelvin). What that ratio was really counting was said later by Boltzmann and Gibbs:

• Boltzmann: S = k · ln W, where W is the number of microstates (the microscopic configurations of molecules, positions, velocities) compatible with the observed macroscopic state. The more microstates are possible, the less we know about the exact state.
• Gibbs (general form): S = − k · Σ pᵢ · ln pᵢ.

Those who wrote them were not thinking of messages: thermodynamics was counting the heat of gases decades before Shannon counted the bits of a message. And yet it is the same equation, up to the constant k and the base of the logarithm. And an exact resemblance, not a vague one, is already a clue that they are measuring the same thing. With equally likely microstates (pᵢ = 1/W) Gibbs reduces to Boltzmann's S = k ln W, exactly as Shannon gives log₂ N with equally likely messages. Both answer the same question, the same as before, now asked of molecules: how much information is missing to know the exact state of the system?

And they coincide even in the limiting case. A perfect crystal at absolute zero has, ideally, a single possible microstate: W = 1, hence S = k·ln 1 = 0. That is the content of the third law of thermodynamics, and it is the same zero as the source with a single certain message: when nothing about the exact state is missing, entropy is zero, in both worlds and for the same reason.

4. They are the same quantity: Landauer, the exchange rate

Two identical formulas in different units (bits on one side, Joules per Kelvin on the other) suggest that they are the same quantity. Landauer's principle (1961) makes it quantitative:

1 bit ↔ k·ln2 of thermodynamic entropy
at room temperature:  k·T·ln2 ≈ 3×10⁻²¹ J of energy

Erasing one bit from memory makes at least k·ln2 of thermodynamic entropy reappear in the environment, as heat: the second law does not let total entropy decrease. Equality holds in the ideal case; a real memory dissipates more.

Up to this point, though, the equivalence lives on paper: it is a factor that converts bits into joules, and nothing forbids the suspicion that it is a mere coincidence of units. But an exchange rate is proven only by spending it. That is exactly what the demon does.

5. The demon as the junction: Szilárd's engine

Here the two worlds touch: Shannon's bit becomes physical work, and its erasure pays the bill in heat. The demon is the experiment that makes Landauer's exchange rate literal, not analogical.

5a. The cycle

A starting condition, not to be taken for granted: the box is at constant temperature T, in thermal contact with the environment. It is a single heat reservoir: why this is decisive becomes clear a little further on.

1. A box (in thermal contact with the environment, temperature T) with ONE molecule.
2. A partition is inserted at the middle: the molecule ends up in one half. Cost ~0.
3. The demon MEASURES which side it is on → acquires 1 bit (writes it down).
4. Knowing the side, it constrains the partition to move only toward the EMPTY
   half and attaches a load to it: the molecule, bouncing, pushes the piston.
   Isothermal expansion from V/2 to V → work extracted = k·T·ln2.
5. To start over, the memory must be returned to its standard state:
   ERASURE of the bit → costs k·T·ln2 (Landauer).
   Net balance: zero.

5b. How the bit becomes work (the piston), and where the energy comes from

A single bouncing molecule still exerts a pressure: on average P·V = k·T, a one-particle "ideal gas". If you let it push a piston, it does work. The problem is direction: the molecule pushes at random, and without knowing where it is you do not know on which side to collect the work.

• Work of an isothermal expansion from V/2 to V: W = k·T · ln(V / (V/2)) = k·T·ln2.
• Where does that energy come from? From the environment, as heat. The expansion is isothermal: the molecule does not cool down, it absorbs k·T·ln2 from the heat reservoir and converts all of it into work.

The role of the bit: without knowing the side, half the time you attach the load the right way (+k·T·ln2), half the time the wrong way (−k·T·ln2): net average zero. Knowing the bit lets you choose the winning configuration every time. Information is what rectifies random thermal motion into directed work (a "rectifier" of thermal noise).

Why a single reservoir (and why it matters). An ordinary heat engine (Carnot) extracts work from a temperature difference (hot reservoir + cold reservoir); the second law (Kelvin's statement) forbids extracting it from a single reservoir over a cycle. Szilárd's engine has a single reservoir, and yet it extracts k·T·ln2. How? In place of the cold reservoir there is the bit. Information plays the thermodynamic role of the second source: a resource interchangeable with a temperature gap.

Two levels not to be confused. - Energy (first law): the k·T·ln2 of work is supplied by the reservoir, as heat. It does not come out of the bit. Energy is conserved. - Possibility (second law): it is information that makes it possible to convert that heat into work in full, something otherwise forbidden with a single source.

"The work comes from the information" is correct if read as "enabled by / paid for with information", not as "the Joules come out of the bit".

5c. Erasure: why the account balances to zero

First, the distinction the whole step rests on: the engine (the molecule in the box, the working substance) and the demon's memory (the "notebook" where the bit is written) are two different objects. The bit is not the engine's molecule; erasing means returning the notebook to a blank page, without touching the engine.

The memory, in turn, is a bistable physical system; the minimal model is another molecule-in-a-box (left = "0", right = "1"). Erasing is not "reading and undoing": it is returning the register to a fixed standard state, whatever the content, with a single procedure that never looks at the value. It is a many-to-one operation (0 and 1 → 0): it is this logical irreversibility that costs.

The physical procedure (reset to "0"):

A. Remove the partition: the two possibilities merge, the molecule roams all of V.
   The register no longer remembers where it was: the information, here, is gone.
B. Compress isothermally from V to V/2, pushing the molecule into the "0" half.
   This requires work k·T·ln2 and EXPELS k·T·ln2 of heat into the environment.
C. Reinsert the partition: the molecule is certainly in the "0" half, however it
   started. Register reset.

The heat expelled in (B) is Landauer's cost. And the erased information ends up exactly there: it reappears as disorder (heat) in the environment.

The symmetry that closes the paradox. Extraction and erasure are the same isothermal process run in the two opposite directions. That is why "the account balances to zero" is not an act of faith: erasure is the energetic inverse of extraction.

Measurement ≠ erasure (Bennett, 1982). For decades the cost was believed to sit in the measurement (Szilárd, Brillouin). Bennett showed that measurement can in principle be reversible and free (it is a copy onto an empty register). It is erasure that is unavoidably dissipative, because it blindly compresses two states into one. The thermodynamic cost lives there.

"Erase now" or "consume memory that is already blank". Why must the demon erase? Because memory is finite: the next measurement needs a register in a known state. It could avoid erasing only by writing on a fresh cell every time, but blank, ordered memory is itself a low-entropy resource, prepared by someone who paid earlier. Erase now or consume ready-made order: either way, the free lunch is not there.

6. What follows: information is physical

The loop is complete. Shannon's entropy, the average uncertainty of a source of messages, and Boltzmann and Gibbs's entropy, the heat divided by the temperature of a gas, were answering the same question (how much information is missing to know the exact state of the system?), one counting messages, the other counting microstates. They were not two similar things: they were the same thing, written in different units. Landauer's principle gave the exchange rate (k·ln2), and Szilárd's engine showed it in action: a bit, in the demon's hands, becomes work; erasing it warms the environment by an exact amount.

Hence the phrase that sums it all up, owed to Rolf Landauer: information is physical. It is not an abstraction living outside the world: it has a price (k·T·ln2 per bit), a location (a physical system that records it) and a weight (the entropy it carries along). An erased bit does not vanish, it reappears as disorder in the environment. The two entropies are a single currency, counted in two different worlds.

Finally, there is a corollary that overturns the intuition about where the cost of computation lives. If the price is paid only by erasure, a computation that erases nothing can, in principle, dissipate nothing. Bennett (1973) showed that a computation can always be reorganized into logically reversible steps, which never merge two states into one: the intermediate steps are kept, the result is copied onto an empty register (a copy: free, like the demon's measurement), and the whole thing is run backwards, returning everything to its starting state without ever compressing blindly. This is reversible computing: a frontier still remote today, but the conceptual point is sharp. Landauer's limit is not a toll on computing: it is a toll on forgetting.

7. An opening: entropy that grows (toward the central limit)

There remains a second face of entropy, surfacing throughout these reflections without ever taking the stage: not just how much it is worth, but where it tends. The demon's paradox existed precisely because total entropy cannot decrease: that is the second law of thermodynamics, entropy in its most celebrated role, the arrow of time, the only law of the macroscopic world that tells before from after. And this face reaches well beyond heat.

Even in pure probability, entropy tends to grow toward a maximum. That is the central limit theorem, read with the eyes of information: summing many independent copies of the same variable, and bringing the sum back to the same scale each time, the distribution slides toward the Gaussian, which is precisely the maximum-entropy distribution at fixed variance. Entropy climbs until it reaches the maximum compatible with the constraints. The second law resurfaces where you would not expect it, in mathematics, far from any gas.

But that is another story, and it deserves a telling of its own.

See also. For a concrete application of the price of information, the energy cost of a language model.

References

• J. C. Maxwell, letter to P. G. Tait, 11 December 1867 (the demon's first appearance); then in Theory of Heat, Longmans, Green & Co., 1871, ch. XXII, section "Limitation of the Second Law of Thermodynamics".
• W. Thomson (Lord Kelvin), Kinetic Theory of the Dissipation of Energy, Nature 9, 1874 (the christening of the "demon"): doi.org/10.1038/009441c0
• M. Smoluchowski, Experimentell nachweisbare, der üblichen Thermodynamik widersprechende Molekularphänomene, Physikalische Zeitschrift 13, 1912 (the automatic demon defeated by its own fluctuations).
• L. Szilárd, Über die Entropieverminderung in einem thermodynamischen System bei Eingriffen intelligenter Wesen, Zeitschrift für Physik 53, 1929 (the one-molecule engine): doi.org/10.1007/BF01341281
• C. E. Shannon, A Mathematical Theory of Communication, Bell System Technical Journal 27, 1948 (information entropy and source coding): doi.org/10.1002/j.1538-7305.1948.tb01338.x
• L. Brillouin, Maxwell's Demon Cannot Operate: Information and Entropy. I, Journal of Applied Physics 22, 1951 (the cost placed in the measurement): doi.org/10.1063/1.1699951
• R. Landauer, Irreversibility and Heat Generation in the Computing Process, IBM Journal of Research and Development 5(3), 1961: doi.org/10.1147/rd.53.0183
• C. H. Bennett, Logical Reversibility of Computation, IBM Journal of Research and Development 17(6), 1973 (computation reorganized into reversible steps): doi.org/10.1147/rd.176.0525
• C. H. Bennett, The Thermodynamics of Computation—a Review, International Journal of Theoretical Physics 21, 1982 (reversible measurement, the cost in the reset): doi.org/10.1007/BF02084158
• R. Landauer, Information is Physical, Physics Today 44(5), 1991 (the phrase that sums it all up): doi.org/10.1063/1.881299
• S. Toyabe et al., Experimental demonstration of information-to-energy conversion and validation of the generalized Jarzynski equality, Nature Physics 6, 2010 (a Szilárd engine realized in the laboratory): doi.org/10.1038/nphys1821
• A. Bérut et al., Experimental verification of Landauer's principle linking information and thermodynamics, Nature 483, 2012 (Landauer's heat measured on the erasure of a single bit): doi.org/10.1038/nature10872
• A. R. Barron, Entropy and the Central Limit Theorem, Annals of Probability 14(1), 1986 (central-limit convergence read in entropy): doi.org/10.1214/aop/1176992632
• S. Artstein, K. Ball, F. Barthe, A. Naor, Solution of Shannon's Problem on the Monotonicity of Entropy, Journal of the American Mathematical Society 17, 2004 (the entropy of the sum grows at every step): doi.org/10.1090/S0894-0347-04-00459-X