Information Theory
Also known as: Shannon Theory
Shannon's framework quantifying information, entropy, and communication channels.
Information Theory was founded by Claude Shannon in his 1948 Bell System Technical Journal paper 'A Mathematical Theory of Communication' (substantially expanded into book form by Shannon and Warren Weaver in 1949), creating an entirely new mathematical discipline that quantifies information independent of meaning or content. Shannon's foundational results include: information measured in bits as the logarithm of the number of equally likely possibilities; entropy as the average information content of a probability distribution (H = -Σp log p); the source coding theorem (data can be compressed to its entropy but no further); the noisy channel coding theorem (reliable communication is possible at any rate below the channel capacity); mutual information as the reduction in uncertainty of one variable given another. Subsequent development has been enormous: error-correcting codes (Hamming, Reed-Solomon, LDPC, Turbo codes); Kullback-Leibler divergence; algorithmic information theory (Kolmogorov, Chaitin, Solomonoff); rate-distortion theory; network information theory; quantum information theory. The framework has had outsized impact on telecommunications (the entire digital-communication infrastructure rests on Shannon's results), data compression (ZIP, JPEG, MP3, MPEG), cryptography (Shannon's 1949 'Communication Theory of Secrecy Systems' founded modern cryptography), machine learning (cross-entropy, KL divergence, mutual-information-based feature selection), neuroscience (information-theoretic analysis of neural codes), and statistical physics (the connection between Shannon entropy and thermodynamic entropy). Shannon's framework is one of the most generative single-author intellectual achievements of the 20th century.
Core components
- Information measured in bits (log of equally likely possibilities)
- Entropy: H = -Σp log p
- Source coding theorem
- Noisy channel coding theorem
- Channel capacity
- Mutual information
- Kullback-Leibler divergence
- Error-correcting codes (Hamming, Reed-Solomon, LDPC, Turbo)
- Rate-distortion theory
- Algorithmic information theory (Kolmogorov complexity)
- Connection to statistical physics (entropy)
- Shannon's separate 1949 paper founding modern cryptography
Primary use case
Foundation of digital communication theory (the entire modern telecommunications infrastructure); data compression algorithms and standards; cryptography (Shannon's 1949 secrecy paper as founding work); machine learning (cross-entropy loss, KL divergence, information-theoretic regularization); neuroscience and computational biology (information in neural codes, genetic codes); statistical physics (Shannon entropy and thermodynamic entropy); statistical inference (MDL, AIC, mutual-information-based methods); pedagogical reference in electrical engineering, computer science, and mathematics curricula.
Common criticisms
- Shannon information measures syntactic information (bits), not semantic content — the framework deliberately abstracts from meaning, which is sometimes critiqued as insufficient for understanding 'real' information in human or biological contexts
- the assumption of stationary stochastic sources is often violated in practical applications
- channel-capacity calculations require knowledge of channel statistics that may be unavailable
- Shannon's information measures don't directly capture concepts like surprise, relevance, or meaning that some applications need
- algorithmic information theory's elegance is offset by uncomputability of Kolmogorov complexity
- quantum information theory has substantially generalized Shannon's framework, raising questions about classical-information limits
- in machine learning, mutual information estimation in high dimensions remains technically difficult
- the Shannon-Weaver 'communication model' (sender-message-receiver) has been critiqued in communication studies as oversimplifying actual human communication
- the framework's outsized success has produced 'information' as buzzword in many applied contexts where Shannon-specific concepts don't actually apply.