Minimum Description Length       

Information Theory Accomplishment | 1978

IBM researcher: Jorma Rissanen

Where the work was doneAlmaden Research Center

What we accomplishedIntuitively, the Minimum Description Length (MDL) principle suggests that more complex statistical models are less likely and therefore the empirical fitness of a candidate model on a sample has to be evaluated with respect to the complexity of the model. For example, a high-degree polynomial may fit a given finite set of points in the plane better than a linear function, but may not be useful for generalization because of overfitting. Rissanen's theory provides a very rigorous foundation for model selection based on the number of bits required to describe the model.

Related links: 1993 IEEE Richard W. Hamming Medal
1998 IEEE Information Theory Society Golden Jubilee Award for Technological Innovation
2006 The Kolmogorov Medal of the University of London
2009 IEEE Claude E. Shannon Award

1989 (Book): J. Rissanen, "Stochastic Complexity in Statistical Inquiry," World Scientific Publishing Company, 1989.

1978 - 1986: Selected Journal Articles

J. Rissanen, Modeling by shortest data description, Automatica 14 (1978) 465-471.

J. Rissanen, A Universal Prior for Integers and Estimation by Minimum Description Length, The Annals of Statistics 11 (1983) 416-431.

J. Rissanen, Stochastic Complexity and Modeling, The Annals of Statistics 14 (1986) 1080-1100.

See also: Algorithmic Information Theory

Image credit: Engineering and Technology History Wiki