This is an old version, view current version.

28.1 Bayes is calibrated by construction

Suppose a Bayesian model is given in the form of a prior density $p(\theta)$ and sampling density $p(y \mid \theta).$ Now consider a process that first simulates parameters from the prior, $$\theta^{\textrm{sim}} \sim p(\theta),$$ and then simulates data given the parameters, $$y^{\textrm{sim}} \sim p(y \mid \theta^{\textrm{sim}}).$$ By the definition of conditional densities, the simulated data and parameters constitute an independent draw from the model’s joint distribution, $$(y^{\textrm{sim}}, \theta^{\textrm{sim}}) \sim p(y, \theta).$$ From Bayes’s rule, it follows that for any observed (fixed) data $y$, $$p(\theta \mid y) \propto p(y, \theta).$$ Therefore, the simulated parameters constitute a draw from the posterior for the simulated data, $$\theta^{\textrm{sim}} \sim p(\theta \mid y^{\textrm{sim}}).$$ Now consider an algorithm that produces a sequence of draws from the posterior given this simulated data, $$\theta^{(1)}, \ldots, \theta^{(M)} \sim p(\theta \mid y^{\textrm{sim}}).$$ Because $\theta^{\textrm{sim}}$ is also distributed as a draw from the posterior, the rank statistics of $\theta^{\textrm{sim}}$ with respect to $\theta^{(1)}, \ldots \theta^{(M)}$ should be uniform.

This is one way to define calibration, because it follows that posterior intervals will have appropriate coverage (A. Philip Dawid 1982; Gneiting, Balabdaoui, and Raftery 2007). If the rank of $\theta^{\textrm{sim}}$ is uniform among the draws $\theta^{(1)}, \ldots, \theta^{(M)},$ then for any 90% interval selected, the probability the true value $\theta^{\textrm{sim}}$ falls in it will also be 90%. The same goes for any other posterior interval.

References

Dawid, A Philip. 1982. “The Well-Calibrated Bayesian.” Journal of the American Statistical Association 77 (379): 605–10.

Gneiting, Tilmann, Fadoua Balabdaoui, and Adrian E Raftery. 2007. “Probabilistic Forecasts, Calibration and Sharpness.” Journal of the Royal Statistical Society: Series B (Statistical Methodology) 69 (2): 243–68.