We are interested in estimating θ given the observation x. Naturally then, any estimation strategy will be based on the posterior distribution p x θ . Furthermore, we need a criterion for assessing the quality of potential estimators.

Loss The quality of an estimate θ is measured by a real-valued loss function: L θ θ . For example, squared error or quadratic loss is simply L θ θ θ θ θ θ

Expected Loss Posterior Expected Loss: x L θ θ θ L θ θ p x θ Bayes Risk: L θ θ x θ L θ θ p x θ p x θ x L θ θ p θ x p θ x L θ θ The "best" or optimal estimator given the data x and under a specified loss is given by θ θ x L θ θ Bayes MSE BMSE θ x θ θ θ 2 p x θ p x Since p x 0 for every x , minimizing the inner integral θ θ θ 2 p x θ x L θ θ (where x L θ θ is the posterior expected loss) for each x , minimizes the overall BMSE. θ θ θ θ 2 p x θ θ θ θ θ 2 p x θ -2 θ θ θ p x θ Equating this to zero produces θ θ θ p x θ x θ The conditional mean (also called posterior mean) of θ given x! n n 1 … N x n A W n W n 0 σ 2 prior for unknown parameter A: p a U A 0 A 0 p A x 1 2 σ 2 N 2 -1 2 σ 2 n 1 N x n A 2 p x A 1 2 A 0 2 σ 2 N 2 -1 2 σ 2 n 1 N x n A 2 A A 0 A 0 1 2 A 0 2 σ 2 N 2 -1 2 σ 2 n 1 N x n A 2 A A 0 0 A A 0 Minimum Bayes MSE Estimator: A x A A a p x A A A 0 A 0 A 1 2 A 0 2 σ 2 N 2 -1 2 σ 2 n 1 N x n A 2 A A 0 A 0 1 2 A 0 2 σ 2 N 2 -1 2 σ 2 n 1 N x n A 2 Notes No closed-form estimator As A 0 , A 1 N n 1 N x n For smaller A 0 , truncated integral produces an A that is a function of x, σ 2 , and A 0 As N increases, σ 2 N decreases and posterior p x A becomes tightly clustered about 1 N x n . This implies A 1 N n x n as n (the data "swamps out" the prior)

Other Common Loss Functions

Absolute Error Loss (Laplace, 1773) L θ θ θ θ x L θ θ θ θ θ p x θ θ θ θ θ p x θ θ θ θ θ p x θ Using integration-by-parts it can be shown that θ θ θ θ p x θ y θ P x θ y θ θ θ θ p x θ y θ P x θ y where P x θ y and P x θ y are a cumulative distributions. So, x L θ θ y θ P x θ y y θ P x θ y Take the derivative with respect to θ implies P x θ θ P x θ θ which implies that the optimal θ under absolute error loss is posterior median.

'0-1' Loss L θ θ 0 θ θ 1 θ θ I { θ ̂ ≠ θ } x L θ θ x I { θ ̂ ≠ θ } x θ θ which is the probability that θ θ given x. To minimize '0-1' loss we must choose θ to be the value of θ with the highest posterior probability, which implies θ θ with the smallest probability.

The optimal estimator θ under '0-1' loss is the maximum a posteriori (MAP) estimator--the value of θ where p x θ is maximized.