How much does knowing one thing tell you about another? The question has a precise answer, a single number that captures dependence of any kind. Mutual information measures relationship without assuming its shape.
The claim
Mutual information quantifies how much learning one variable reduces uncertainty about another. Zero means independence; larger values mean tighter coupling. It captures association of every form, not just the linear.
Beneath the surface
The mechanism is entropy. It compares the uncertainty in one variable alone to its uncertainty once the other is known, and the reduction is the shared information. Dependence is measured as uncertainty removed.
A reframing
The turn is generality. Unlike correlation, which sees only linear ties, mutual information detects any statistical dependence, however curved or complex. It is blind to the form and sensitive to the fact of relationship.
The trade-off
Estimating it is hard. From finite samples in many dimensions, mutual information is notoriously difficult to compute reliably. The measure is clean in theory and treacherous in practice.
Where it breaks
The implication is a universal currency for dependence. Wherever one asks how much two things are related, mutual information gives a principled answer. It is the natural measure of shared structure.
The larger point
Mutual information measures how much one variable reveals about another, capturing dependence of any form as uncertainty removed. It surpasses correlation in generality but resists reliable estimation. It is the principled currency of relationship. Seen this way, the concept is less a fact to be filed than a lens through which other facts arrange themselves. Its worth lies not in any single application but in the pattern of thought it makes available. To hold it clearly is to see a whole family of problems as variations on one theme, and to recognize the same shape recurring where it was not expected.
