Bayesian Inference
Belief updating via Bayes' rule, applied across statistics, ML, and epistemology.
Bayesian Inference is the statistical and computational framework for updating probabilistic beliefs (or probability distributions over parameters or hypotheses) in light of observed data, using Bayes's theorem: posterior = likelihood × prior / evidence. The framework descends from Reverend Thomas Bayes's posthumous 1763 'An Essay towards Solving a Problem in the Doctrine of Chances' (published by Richard Price), with substantial subsequent development by Pierre-Simon Laplace (independently rediscovered and elaborated the principles in the late 18th century), Harold Jeffreys (1939, Theory of Probability — modern Bayesian foundations), Bruno de Finetti (subjective probability foundations), Leonard J. Savage (foundations of statistics, 1954), and the contemporary computational revolution through Markov Chain Monte Carlo methods (Metropolis et al. 1953, Hastings 1970, Gelfand-Smith 1990) that made Bayesian inference practical for high-dimensional problems. Bayesian Inference contrasts with Frequentist Statistics (separately enriched) on the question of what probability means: Bayesians treat probability as quantified belief that updates with evidence, while frequentists treat probability as long-run relative frequency. The framework has had outsized impact on contemporary machine learning (probabilistic graphical models, variational inference, Bayesian deep learning), scientific inference (clinical trials, particle physics, cosmology), and AI (Bayesian decision theory, probabilistic programming languages). Distinct from but philosophically connected to Bayesian Epistemology (separately enriched in philosophy batch).
Core components
- Bayes's theorem: posterior = likelihood × prior / evidence
- Prior probability distribution
- Likelihood function
- Posterior probability distribution
- Marginal likelihood (evidence)
- Conjugate priors and analytical posteriors
- Markov Chain Monte Carlo (MCMC) for high-dimensional problems
- Variational inference
- Bayesian model averaging
- Bayesian decision theory (combines posterior with utility)
- Distinction from frequentist statistics on probability interpretation
- Subjective vs objective Bayesian variants
Primary use case
Statistical inference across scientific disciplines; foundation for substantial modern machine learning (probabilistic graphical models, variational autoencoders, Bayesian neural networks); clinical trial design and analysis (adaptive trials, FDA Bayesian guidance); particle physics and cosmology (parameter estimation, model comparison); A/B testing in technology companies; foundation for probabilistic programming languages (Stan, PyMC, Pyro, Edward); reference framework in modern statistics education.
Common criticisms
- Choice of prior is contested — subjective Bayesians accept wide latitude in priors, but the resulting analyses can vary substantially with prior choice, raising concerns about analyst-dependence
- objective Bayesian attempts to specify default priors (Jeffreys priors, reference priors, maximum entropy) have not produced consensus
- computational expense remains substantial despite MCMC advances
- convergence diagnostics for MCMC are imperfect, with potential for misleading inferences from non-converged chains
- the foundational frequentist-Bayesian debate continues — frequentists argue Bayesian methods sacrifice objectivity by requiring prior specification
- integration with model-checking and goodness-of-fit assessment is less developed than in some frequentist traditions
- for some scientific contexts (regulatory clinical trials especially), Bayesian methods face institutional resistance despite mathematical equivalence under appropriate priors
- commercial 'Bayesian' implementations vary substantially in fidelity to underlying principles.
Lineage
- Parent of
- Bayesian Networks, Markov Decision Processes
- Siblings
- Frequentist Statistics, Bayesian Epistemology, Decision Theory