Skip to content Skip to navigation

OpenStax_CNX

You are here: Home » Content » Conditional Probabilities and Bayes' Rule

Navigation

Lenses

What is a lens?

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

This content is ...

In these lenses

  • eScience, eResearch and Computational Problem Solving

    This module is included inLens: eScience, eResearch and Computational Problem Solving
    By: Jan E. OdegardAs a part of collection: "Random Processes"

    Click the "eScience, eResearch and Computational Problem Solving" link to see all content selected in this lens.

Recently Viewed

This feature requires Javascript to be enabled.
 

Conditional Probabilities and Bayes' Rule

Module by: Nick Kingsbury. E-mail the author

Summary: This module introduces conditional probabilities and Bayes' rule.

If AA and BB are two separate but possibly dependent random events, then:

  1. Probability of AA and BB occurring together = PrA,B , A B
  2. The conditional probability of AA, given that BB occurs = PrA|B B A
  3. The conditional probability of BB, given that AA occurs = PrB|A A B
From elementary rules of probability (Venn diagrams):
PrA,B=PrA|BPrB=PrB|APrA , A B B A B A B A
(1)
Dividing the right-hand pair of expressions by PrB B gives Bayes' rule:
PrA|B=PrB|APrAPrB B A A B A B
(2)
In problems of probabilistic inference, we are often trying to estimate the most probable underlying model for a random process, based on some observed data or evidence. If AA represents a given set of model parameters, and BB represents the set of observed data values, then the terms in Equation 2 are given the following terminology:
  • PrA A is the prior probability of the model AA (in the absence of any evidence);
  • PrB B is the probability of the evidence BB;
  • PrB|A A B is the likelihood that the evidence BB was produced, given that the model was AA;
  • PrA|B B A is the posterior probability of the model being AA, given that the evidence is BB.
Quite often, we try to find the model AA which maximizes the posterior PrA|B B A . This is known as maximum a posteriori or MAP model selection.

The following example illustrates the concepts of Bayesian model selection.

Example 1: Loaded Dice

Problem:

Given a tub containing 100 six-sided dice, in which one die is known to be loaded towards the six to a specified extent, derive an expression for the probability that, after a given set of throws, an arbitrarily chosen die is the loaded one? Assume the other 99 dice are all fair (not loaded in any way). The loaded die is known to have the following pmf: pL 1=0.05 pL 1 0.05 pL 2 pL 5=0.15 pL 2 pL 5 0.15 pL 6=0.35 pL 6 0.35 Here derive a good strategy for finding the loaded die from the tub.

Solution:

The pmfs of the fair dice may be assumed to be: i,i=16: pF i=16 i i 1 6 pF i 1 6 Let each die have one of two states, S=L S L if it is loaded and S=F S F if it is fair. These are our two possible models for the random process and they have underlying pmfs given by pL 1 pL 6 pL 1 pL 6 and pF 1 pF 6 pF 1 pF 6 respectively.

After NN throws of the chosen die, let the sequence of throws be ΘN = θ1 θN ΘN θ1 θN , where each θi 16 θi 1 6 . This is our evidence.

We shall now calculate the probability that this die is the loaded one. We therefore wish to find the posterior PrS=L| ΘN ΘN S L .

We cannot evaluate this directly, but we can evaluate the likelihoods, Pr ΘN | S=L S L ΘN and Pr ΘN | S=F S F ΘN , since we know the expected pmfs in each case. We also know the prior probabilities PrS=L S L and PrS=F S F before we have carried out any throws, and these are 0.010.99 0.01 0.99 since only one die in the tub of 100 is loaded. Hence we can use Bayes' rule:

PrS=L| ΘN =Pr ΘN | S=L PrS=LPr ΘN ΘN S L S L ΘN S L ΘN
(3)
The denominator term Pr ΘN ΘN is there to ensure that PrS=L| ΘN ΘN S L and PrS=F| ΘN ΘN S F sum to unity (as they must). It can most easily be calculated from:
Pr ΘN =Pr ΘN ,S=L+Pr ΘN ,S=F=Pr ΘN | S=L PrS=L+Pr ΘN | S=F PrS=F ΘN , ΘN S L , ΘN S F S L ΘN S L S F ΘN S F
(4)
so that
PrS=L| ΘN =Pr ΘN | S=L PrS=LPr ΘN | S=L PrS=L+Pr ΘN | S=F PrS=F=11+ RN ΘN S L S L ΘN S L S L ΘN S L S F ΘN S F 1 1 RN
(5)
where
RN =Pr ΘN | S=F PrS=FPr ΘN | S=L PrS=L RN S F ΘN S F S L ΘN S L
(6)
To calculate the likelihoods, Pr ΘN | S=L S L ΘN and Pr ΘN | S=F S F ΘN , we simply take the product of the probabilities of each throw occurring in the sequence of throws ΘN ΘN , given each of the two modules respectively (since each new throw is independent of all previous throws, given the model). So, after NN throws, these likelihoods will be given by:
Pr ΘN | S=L =i=1N pL θi S L ΘN i 1 N pL θi
(7)
and
Pr ΘN | S=F =i=1N pF θi S F ΘN i 1 N pF θi
(8)
We can now substitute these probabilities into the above expression for RN RN and include PrS=L=0.01 S L 0.01 and PrS=F=0.99 S F 0.99 to get the desired a posteriori probability PrS=L| ΘN ΘN S L after NN throws using Equation 5.

We may calculate this iteratively by noting that

Pr ΘN | S=L =Pr Θ N - 1 | S=L pL θn S L ΘN S L Θ N - 1 pL θn
(9)
and
Pr ΘN | S=F =Pr Θ N - 1 | S=F pF θn S F ΘN S F Θ N - 1 pF θn
(10)
so that
RN = R N - 1 pF θn pL θn RN R N - 1 pF θn pL θn
(11)
where R0 =PrS=FPrS=L=99 R0 S F S L 99 . If we calculate this after every throw of the current die being tested (i.e. as NN increases), then we can either move on to test the next die from the tub if PrS=L| ΘN ΘN S L becomes sufficiently small (say <10-4 < 10 -4 ) or accept the current die as the loaded one when PrS=L| ΘN ΘN S L becomes large enough (say >0.995 > 0.995 ). (These thresholds correspond approximately to RN >104 RN 10 4 and RN <5×10-3 RN 5-3 respectively.)

The choice of these thresholds for PrS=L| ΘN ΘN S L is a function of the desired tradeoff between speed of searching versus the probability of failure to find the loaded die, either by moving on to the next die even when the current one is loaded, or by selecting a fair die as the loaded one.

The lower threshold, p1 =10-4 p1 10 -4 , is the more critical, because it affects how long we spend before discarding each fair die. The probability of correctly detecting all the fair dice before the loaded die is reached is 1 p1 n1n p1 1 p1 n 1 n p1 , where n50 n 50 is the expected number of fair dice tested before the loaded one is found. So the failure probability due to incorrectly assuming the loaded die to be fair is approximately n p1 0.005 n p1 0.005 .

The upper threshold, p2 =0.995 p2 0.995 , is much less critical on search speed, since the loaded result only occurs once, so it is a good idea to set it very close to unity. The failure probability caused by selecting a fair die to be the loaded one is just 1 p2 =0.005 1 p2 0.005 . Hence the overall  failure  probability =0.005+0.005=0.01 overall  failure  probability  0.005 0.005 0.01

Note on computation:

In problems with significant amounts of evidence (e.g. large NN), the evidence probability and the likelihoods can both get very very small, sufficient to cause floating-point underflow on many computers if equations such as Equation 7 and Equation 8 are computed directly. However the ratio of likelihood to evidence probability still remains a reasonable size and is an important quantity which must be calculated correctly.

One solution to this problem is to compute only the ratio of likelihoods, as in Equation 11. A more generally useful solution is to compute log(likelihoods) instead. The product operations in the expressions for the likelihoods then become sums of logarithms. Even the calculation of likelihood ratios such as RN RN and comparison with appropriate thresholds can be done in the log domain. After this, it is OK to return to the linear domain if necessary since RN RN should be a reasonable value as it is the ratio of very small quantities.

Figure 1: Probabilities of the current die being the loaded one as the throws progress (20th die is the loaded one). A new die is selected whenever the probability falls below p1 p1 .
Figure 1 (figure1.png)
Figure 2: Histograms of the dice throws as the throws progress. Histograms are reset when each new die is selected.
Figure 2 (figure2.png)

Content actions

Download module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks