Information and Intelligence

Motivation

Perpetual Motion and Superintelligence

1925: Motor vehicles (no perpetual motion)
2025: Promises of superintelligence singularity
Same fundamental impossibility?

Why Perpetual Motion Failed

\[\frac{\text{d}H}{\text{d}t} \geq 0\]

Entropy always increases
No motion without entropy production
No work without energy input

An Equivalent Statement for Intelligence?

Maxwell’s Demon:

“Intelligent” entity that violates 2nd law
Resolution: Landauer’s principle?
Information erasure requires energy

Implication

Intelligence has thermodynamic cost
Information processing has physical limits

Information-Theoretic Limits on Intelligence

Thermodynamics limits mechanical engines
Information theory limits information engines

Same kind of fundamental constraint

What Intelligent Systems Must Do

Acquire information (sensing)
Store information (memory)
Process information (computation)
Erase information (memory mgmt)
Act on information (output)

What is the thermodynamic cost?

Landauer’s Principle

Erasing 1 bit requires: \(Q \geq k_B T \log 2\)

Not engineering limitation
Fundamental thermodynamic bound
Entropy must go somewhere

At room temperature: \(\sim 3 \times 10^{-21}\) Joules/bit

Information

Thermalisation from Different Initial Conditions

9 balls on a \(3 \times 3\) grid;
Different starting conditions.

Initialisation: Display:

Jaynes and Maximum Entropy

Maximum Entropy Motivation

Jaynes (1957): Statistical mechanics as inference with incomplete information
Maximum entropy principle: maximise uncertainty given constraints
Avoids unwarranted assumptions beyond available data

Dice Example

Dice example: Average result 4.5 instead of 3.5
Constraints:
- \(\sum_{n=1}^6 P_n = 1\) (normalization)
- \(\sum_{n=1}^6 nP_n = 4.5\) (observed average)

Die Roll Simulation

click die or button to roll

Rolls: 0
Sample mean: —
H(p): —

Outcome weights (auto-normalised to probabilities)

The General Maximum-Entropy Formalism

\[ p_i = \frac{\exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i))}{Z(\lambda_1,\ldots,\lambda_m)} \] \[ Z(\ldots) = \sum_{i=1}^n \exp(-\lambda_1 f_1(x_i) - \ldots - \lambda_m f_m(x_i)) \] \[ \langle f_k \rangle = -\frac{\partial}{\partial \lambda_k}\log Z(\lambda_1,\ldots,\lambda_m) \quad k=1,2,\ldots,m. \]

Sampling Two Dimensional Variables

Correlation

Correlation is when two variables are dependent

Sampling Two Dimensional Variables

Independent Gaussians

Correlated Gaussians

Anticorrelated Gaussians

The Classical Observer

The Classical Observer - Correlated

The Classical Observer - Anti-correlated

Conceptually, imagine a classic physics setup where you can see inside a box: there are a number of particles moving around, sampled from random Gaussians. If you plot their velocities, they sit within a circle. In this case, the marginal \(h_i\) would be their velocity in each direction. The marginal distribution in each direction is Gaussian. Now, if the particles are correlated — everything tending to move in one diagonal direction — interestingly, the marginal entropies are the same. Although these are correlated, I’m not saying anything about the correlation when I conserve the marginal entropies, because if you look at the marginal distribution in each direction, it’s the same as in the uncorrelated case. And if they’re anti-correlated, it’s the same again. So I’m not making any statements about independence with this constraint.

The Inaccessible Game

The Munchkin Provision

Munchkin Card Game

Rules may be inconsistent … so …

Any other disputes should be settled by loud arguments, with the owner of the game having the last word.

Munckin Rules (Jackson, 2001)

A Tautology

Self-governing systems cannot refer to external arbitration.

The No-Barber Principle

Russell’s Barber Paradox:

Barber shaves all who don’t shave themselves

Does the barber shave themselves?

Paradox: Definition includes itself in scope

No External Adjudicators

Forbidden:

External observer
Pre-specified outcome space/Hamiltonian
Privileged decomposition
External time parameter

No appeal to structure outside the game

That’s the sort of thing we want to forbid. And that’s what motivates the no-barber principle. The no-barber principle basically says we want to be in a situation where there are no external adjudicators. That means no external observer, no predefined outcome space, no external time parameter, and no privileged decomposition in the game. Nothing dictated from outside. And I’ll give you a very disconnected sense of how this manifests. Imagine if you had an external time parameter in the game, and as part of that game an entity’s time parameter was manipulated so it could move in time. A time travel paradox has the same impredicative circularity feel as Russell’s paradox — went back in time, met your mother, do you continue to exist? And that seems to come from someone outside the system playing with the time. If something emerged consistently inside the system, that shouldn’t be able to happen. So the idea is to create a game where you’re not appealing to any structure outside the game.

Baez-Fritz-Leinster Characterization of Information Loss

Baez et al. (2011):

Entropy from category theory
Three axioms uniquely determine information loss
No probability needed initially

The Three Axioms

\[F(f \circ g) = F(f) + F(g)\]

Information loss is additive
Compose processes → add losses

Convex Linearity

\[F(\lambda f \oplus (1-\lambda)g) = \lambda F(f) + (1-\lambda)F(g)\]

Probabilistic mixture of processes
Linear in probability weights

Continuity

Small change in process
Small change in information loss
\(F(f)\) continuous in \(f\)

The Main Result

Three axioms \(\Rightarrow\) unique form: \[F(f) = c(H(p) - H(q))\]

Information loss = scaled entropy difference
Shannon entropy emerges from axioms
No other measure satisfies all three

The Inaccessible Game Setup

Avoid external structure.
Represent information loss
Enforce information conservation

Information Isolation

Define information loss
Isolate game from observation/interaction
No external observer can extract or inject information

Marginal Entropy Conservation

\[ \sum_{i=1}^N h_i = C \]

Isolation: cf energy conservation — but for information

The Classical Observer - Inaccessible

Joint Entropy

We don’t see see the outcome space
But we know it has a joint entropy

The \(I + H = C\) Structure

\[ \sum_{i=1}^N h_i = C \]

What does this conservation imply for dynamics?

Multi-Information: Measuring Correlation

\[ I = \sum_{i=1}^N h_i - H \]

‘Information Action’

\[ I + H = C \]

Conserved quantity splits into two parts

Analogy to classical mechanics

Energy: \(V + T = E\)
Information: \(I + H = C\)
System “rolls downhill” from correlation to disorder

Energy

Energy Constraints

Normally we derive physical laws by

Maximise entropy subject to energy conservation

The Conservation Law

Marginal Entropy Conservation

\[ \sum_{i=1}^N h_i = C \]

Entropy Constraints

Now derive game rules by

Maximise joint entropy subject to marginal entropy conservation

Energy

In certain thermodynamic limits:
- Marginal entropy conservation \(\equiv\) Energy conservation

See Lawrence (2025)

Intelligence

Thanks!

References

Baez, J.C., Fritz, T., Leinster, T., 2011. A characterization of entropy in terms of information loss. Entropy 13, 1945–1957. https://doi.org/10.3390/e13111945

Jackson, S., 2001. Munchkin. Steve Jackson Games.

Jaynes, E.T., 1957. Information theory and statistical mechanics. Physical Review 106, 620–630. https://doi.org/10.1103/PhysRev.106.620

Lawrence, N.D., 2025. The inaccessible game. https://doi.org/10.48550/arXiv.2511.06795