The best theory is the one that, together with the data it explains, can be stated most briefly. This turns model selection into a contest of total code length. Truth is approached by economy of description.
The principle
The minimum description length principle selects the model that minimizes the combined length of describing the model and the data given the model. A complex model costs bits to specify but may cheapen the data. The best balance wins.
The mechanism
The mechanism formalizes Occam’s razor. It trades the cost of a richer model against the savings it yields in encoding the data, so overfitting is penalized automatically. Complexity must pay for itself in compression.
An unexpected turn
The turn is that learning becomes coding. To find the best explanation is to find the shortest total encoding, uniting inference and information theory. Understanding and compression merge again.
The hidden cost
The principle is elegant but hard to wield. Exact description lengths are often uncomputable, and choices of coding scheme affect the verdict. The ideal is cleaner than any practice.
The limit
The implication is a rigorous account of parsimony. Simplicity is not vague preference but measured in bits, and the trade-off between fit and complexity is made explicit. Economy of description becomes a criterion of truth.
The larger point
Minimum description length picks the model minimizing the total code for model plus data, formalizing Occam’s razor in bits. It penalizes complexity that fails to pay for itself. Learning becomes coding, and parsimony becomes measurable. Seen this way, the concept is less a fact to be filed than a lens through which other facts arrange themselves. Its worth lies not in any single application but in the pattern of thought it makes available. To hold it clearly is to see a whole family of problems as variations on one theme, and to recognize the same shape recurring where it was not expected.
