Skip to content Skip to navigation Skip to collection information

OpenStax_CNX

You are here: Home » Content » Applied Probability » Conditional Independence

Navigation

Table of Contents

Lenses

What is a lens?

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

This content is ...

Affiliated with (What does "Affiliated with" mean?)

This content is either by members of the organizations listed or about topics related to the organizations listed. Click each link to see a list of all content affiliated with the organization.
  • Rice Digital Scholarship

    This collection is included in aLens by: Digital Scholarship at Rice University

    Click the "Rice Digital Scholarship" link to see all content affiliated with them.

Also in these lenses

  • UniqU content

    This collection is included inLens: UniqU's lens
    By: UniqU, LLC

    Click the "UniqU content" link to see all content selected in this lens.

Recently Viewed

This feature requires Javascript to be enabled.
 

Conditional Independence

Module by: Paul E Pfeiffer. E-mail the author

Summary: The idea of stochastic (probabilistic) independence is approached as lack of conditioning: P(A|B)=P(A). This is equivalent to the product rule P(AB)=P(A)P(B). We consider an extension to conditional independence. Using the facts on repeated conditioning and the equivalent conditions for independence, we may produce a similar table of equivalent conditions for conditional independence. In a given problem, one or the other of these conditions may seem a reasonable assumption. As soon as one of these patterns is recognized, then all are equally valid assumptions. Because of its simplicity and symmetry, we take as the defining condition the product rule P(AB|C)=P(A|C)P(B|C). As in the case of simple independence, the replacement rule extends.

The idea of stochastic (probabilistic) independence is explored in the unit Independence of Events. The concept is approached as lack of conditioning: P(A|B)=P(A)P(A|B)=P(A). This is equivalent to the product rule P(AB)=P(A)P(B)P(AB)=P(A)P(B) . We consider an extension to conditional independence.

The concept

Examination of the independence concept reveals two important mathematical facts:

  • Independence of a class of non mutually exclusive events depends upon the probability measure, and not on the relationship between the events. Independence cannot be displayed on a Venn diagram, unless probabilities are indicated. For one probability measure a pair may be independent while for another probability measure the pair may not be independent.
  • Conditional probability is a probability measure, since it has the three defining properties and all those properties derived therefrom.

This raises the question: is there a useful conditional independence—i.e., independence with respect to a conditional probability measure? In this chapter we explore that question in a fruitful way.

Among the simple examples of “operational independence" in the unit on independence of events, which lead naturally to an assumption of “probabilistic independence” are the following:

  • If customers come into a well stocked shop at different times, each unaware of the choice made by the other, the the item purchased by one should not be affected by the choice made by the other.
  • If two students are taking exams in different courses, the grade one makes should not affect the grade made by the other.

Example 1: Buying umbrellas and the weather

A department store has a nice stock of umbrellas. Two customers come into the store “independently.” Let A be the event the first buys an umbrella and B the event the second buys an umbrella. Normally, we should think the events {A,B}{A,B} form an independent pair. But consider the effect of weather on the purchases. Let C be the event the weather is rainy (i.e., is raining or threatening to rain). Now we should think P(A|C)>P(A|Cc)P(A|C)>P(A|Cc) and P(B|C)>P(B|Cc)P(B|C)>P(B|Cc). The weather has a decided effect on the likelihood of buying an umbrella. But given the fact the weather is rainy (event C has occurred), it would seem reasonable that purchase of an umbrella by one should not affect the likelihood of such a purchase by the other. Thus, it may be reasonable to suppose

P ( A | C ) = P ( A | B C ) or, in another notation, P C ( A ) = P C ( A | B ) P ( A | C ) = P ( A | B C ) or, in another notation, P C ( A ) = P C ( A | B )
(1)

An examination of the sixteen equivalent conditions for independence, with probability measure P replaced by probability measure PC, shows that we have independence of the pair {A,B}{A,B} with respect to the conditional probability measure PC(·)=P(·|C)PC(·)=P(·|C). Thus, P(AB|C)=P(A|C)P(B|C)P(AB|C)=P(A|C)P(B|C). For this example, we should also expect that P(A|Cc)=P(A|BCc)P(A|Cc)=P(A|BCc), so that there is independence with respect to the conditional probability measure P(·|Cc)P(·|Cc). Does this make the pair {A,B}{A,B} independent (with respect to the prior probability measure P)? Some numerical examples make it plain that only in the most unusual cases would the pair be independent. Without calculations, we can see why this should be so. If the first customer buys an umbrella, this indicates a higher than normal likelihood that the weather is rainy, in which case the second customer is likely to buy. The condition leads to P(B|A)>P(B)P(B|A)>P(B). Consider the following numerical case. Suppose P(AB|C)=P(A|C)P(B|C)P(AB|C)=P(A|C)P(B|C) and P(AB|Cc)=P(A|Cc)P(B|Cc)P(AB|Cc)=P(A|Cc)P(B|Cc) and

P(A|C)=0.60,P(A|Cc)=0.20, P(B|C)=0.50,P(B|Cc)=0.15, with P(C)=0.30.P(A|C)=0.60,P(A|Cc)=0.20,P(B|C)=0.50,P(B|Cc)=0.15,withP(C)=0.30.
(2)

Then

P ( A ) = P ( A | C ) P ( C ) + P ( A | C c ) P ( C c ) = 0 . 3200 P ( B ) = P ( B | C ) P ( C ) + P ( B | C c ) P ( C c ) = 0 . 2550 P ( A ) = P ( A | C ) P ( C ) + P ( A | C c ) P ( C c ) = 0 . 3200 P ( B ) = P ( B | C ) P ( C ) + P ( B | C c ) P ( C c ) = 0 . 2550
(3)
P ( A B ) = P ( A B | C ) P ( C ) + P ( A B | C c ) P ( C c ) = P ( A | C ) P ( B | C ) P ( C ) + P ( A | C c ) P ( B | C c ) P ( C c ) = 0 . 1110 P ( A B ) = P ( A B | C ) P ( C ) + P ( A B | C c ) P ( C c ) = P ( A | C ) P ( B | C ) P ( C ) + P ( A | C c ) P ( B | C c ) P ( C c ) = 0 . 1110
(4)

As a result,

P ( A ) P ( B ) = 0 . 0816 0 . 1110 = P ( A B ) P ( A ) P ( B ) = 0 . 0816 0 . 1110 = P ( A B )
(5)

The product rule fails, so that the pair is not independent. An examination of the pattern of computation shows that independence would require very special probabilities which are not likely to be encountered.

Example 2: Students and exams

Two students take exams in different courses, Under normal circumstances, one would suppose their performances form an independent pair. Let A be the event the first student makes grade 80 or better and B be the event the second has a grade of 80 or better. The exam is given on Monday morning. It is the fall semester. There is a probability 0.30 that there was a football game on Saturday, and both students are enthusiastic fans. Let C be the event of a game on the previous Saturday. Now it is reasonable to suppose

P ( A | C ) = P ( A | B C ) and P ( A | C c ) = P ( A | B C c ) P ( A | C ) = P ( A | B C ) and P ( A | C c ) = P ( A | B C c )
(6)

If we know that there was a Saturday game, additional knowledge that B has occurred does not affect the lielihood that A occurs. Again, use of equivalent conditions shows that the situation may be expressed

P ( A B | C ) = P ( A | C ) P ( B | C ) and P ( A B | C c ) = P ( A | C c ) P ( B | C c ) P ( A B | C ) = P ( A | C ) P ( B | C ) and P ( A B | C c ) = P ( A | C c ) P ( B | C c )
(7)

Under these conditions, we should suppose that P(A|C)<P(A|Cc)P(A|C)<P(A|Cc) and P(B|C)<P(B|Cc)P(B|C)<P(B|Cc). If we knew that one did poorly on the exam, this would increase the likelihoood there was a Saturday game and hence increase the likelihood that the other did poorly. The failure to be independent arises from a common chance factor that affects both. Although their performances are “operationally” independent, they are not independent in the probability sense. As a numerical example, suppose

P ( A | C ) = 0 . 7 P ( A | C c ) = 0 . 9 P ( B | C ) = 0 . 6 P ( B | C c ) = 0 . 8 P ( C ) = 0 . 3 P ( A | C ) = 0 . 7 P ( A | C c ) = 0 . 9 P ( B | C ) = 0 . 6 P ( B | C c ) = 0 . 8 P ( C ) = 0 . 3
(8)

Straightforward calculations show P(A)=0.8400,P(B)=0.7400,P(AB)=0.6300P(A)=0.8400,P(B)=0.7400,P(AB)=0.6300. Note that P(A|B)=0.8514>P(A)P(A|B)=0.8514>P(A) as would be expected.

Sixteen equivalent conditions

Using the facts on repeated conditioning and the equivalent conditions for independence, we may produce a similar table of equivalent conditions for conditional independence. In the hybrid notation we use for repeated conditioning, we write

P C ( A | B ) = P C ( A ) or P C ( A B ) = P C ( A ) P C ( B ) P C ( A | B ) = P C ( A ) or P C ( A B ) = P C ( A ) P C ( B )
(9)

This translates into

P ( A | B C ) = P ( A | C ) or P ( A B | C ) = P ( A | C ) P ( B | C ) P ( A | B C ) = P ( A | C ) or P ( A B | C ) = P ( A | C ) P ( B | C )
(10)

If it is known that C has occurred, then additional knowledge of the occurrence of B does not change the likelihood of A.

If we write the sixteen equivalent conditions for independence in terms of the conditional probability measure PC(·)PC(·), then translate as above, we have the following equivalent conditions.

Table 1: Sixteen equivalent conditions
P ( A | B C ) = P ( A | C ) P ( A | B C ) = P ( A | C ) P ( B | A C ) = P ( B | C ) P ( B | A C ) = P ( B | C ) P ( A B | C ) = P ( A | C ) P ( B | C ) P ( A B | C ) = P ( A | C ) P ( B | C )
P ( A | B c C ) = P ( A | C ) P ( A | B c C ) = P ( A | C ) P ( B c | A C ) = P ( B c | C ) P ( B c | A C ) = P ( B c | C ) P ( A B c | C ) = P ( A | C ) P ( B c | C ) P ( A B c | C ) = P ( A | C ) P ( B c | C )
P ( A c | B C ) = P ( A c | C ) P ( A c | B C ) = P ( A c | C ) P ( B | A c C ) = P ( B | C ) P ( B | A c C ) = P ( B | C ) P ( A c B | C ) = P ( A c | C ) P ( B | C ) P ( A c B | C ) = P ( A c | C ) P ( B | C )
P ( A c | B c C ) = P ( A c | C ) P ( A c | B c C ) = P ( A c | C ) P ( B c | A c C ) = P ( B c | C ) P ( B c | A c C ) = P ( B c | C ) P ( A c B c | C ) = P ( A c | C ) P ( B c | C ) P ( A c B c | C ) = P ( A c | C ) P ( B c | C )
Table 2
P ( A | B C ) = P ( A | B c C ) P ( A | B C ) = P ( A | B c C ) P ( A c | B C ) = P ( A c | B c C ) P ( A c | B C ) = P ( A c | B c C ) P ( B | A C ) = P ( B | A c C ) P ( B | A C ) = P ( B | A c C ) P ( B c | A C ) = P ( B c | A c C ) P ( B c | A C ) = P ( B c | A c C )

The patterns of conditioning in the examples above belong to this set. In a given problem, one or the other of these conditions may seem a reasonable assumption. As soon as one of these patterns is recognized, then all are equally valid assumptions. Because of its simplicity and symmetry, we take as the defining condition the product ruleP(AB|C)=P(A|C)P(B|C)P(AB|C)=P(A|C)P(B|C).

Definition. A pair of events {A,B}{A,B} is said to be conditionally independent, givenC, designated {A,B} ci |C{A,B} ci |C iff the following product rule holds: P(AB|C)=P(A|C)P(B|C)P(AB|C)=P(A|C)P(B|C).

The equivalence of the four entries in the right hand column of the upper part of the table, establish

The replacement rule

If any of the pairs {A,B},{A,Bc},{Ac,B}{A,B},{A,Bc},{Ac,B}, or {Ac,Bc}{Ac,Bc} is conditionally independent, given C, then so are the others.

This may be expressed by saying that if a pair is conditionally independent, we may replace either or both by their complements and still have a conditionally independent pair.

To illustrate further the usefulness of this concept, we note some other common examples in which similar conditions hold: there is operational independence, but some chance factor which affects both.

  • Two contractors work quite independently on jobs in the same city. The operational independence suggests probabilistic independence. However, both jobs are outside and subject to delays due to bad weather. Suppose A is the event the first contracter completes his job on time and B is the event the second completes on time. If C is the event of “good” weather, then arguments similar to those in Examples 1 and 2 make it seem reasonable to suppose {A,B} ci |C{A,B} ci |C and {A,B} ci |Cc{A,B} ci |Cc. Remark. In formal probability theory, an event must be sharply defined: on any trial it occurs or it does not. The event of “good weather” is not so clearly defined. Did a trace of rain or thunder in the area constitute bad weather? Did rain delay on one day in a month long project constitute bad weather? Even with this ambiguity, the pattern of probabilistic analysis may be useful.
  • A patient goes to a doctor. A preliminary examination leads the doctor to think there is a thirty percent chance the patient has a certain disease. The doctor orders two independent tests for conditions that indicate the disease. Are results of these tests really independent? There is certainly operational independence—the tests may be done by different laboratories, neither aware of the testing by the others. Yet, if the tests are meaningful, they must both be affected by the actual condition of the patient. Suppose D is the event the patient has the disease, A is the event the first test is positive (indicates the conditions associated with the disease) and B is the event the second test is positive. Then it would seem reasonable to suppose {A,B} ci |D{A,B} ci |D and {A,B} ci |Dc{A,B} ci |Dc.

In the examples considered so far, it has been reasonable to assume conditional independence, given an event C, and conditional independence, given the complementary event. But there are cases in which the effect of the conditioning event is asymmetric. We consider several examples.

  • Two students are working on a term paper. They work quite separately. They both need to borrow a certain book from the library. Let C be the event the library has two copies available. If A is the event the first completes on time and B the event the second is successful, then it seems reasonable to assume {A,B} ci |C{A,B} ci |C. However, if only one book is available, then the two conditions would not be conditionally independent. In general P(B|ACc)<P(B|Cc)P(B|ACc)<P(B|Cc), since if the first student completes on time, then he or she must have been successful in getting the book, to the detriment of the second.
  • If the two contractors of the example above both need material which may be in scarce supply, then successful completion would be conditionally independent, give an adequate supply, whereas they would not be conditionally independent, given a short supply.
  • Two students in the same course take an exam. If they prepared separately, the event of both getting good grades should be conditionally independent. If they study together, then the likelihoods of good grades would not be independent. With neither cheating or collaborating on the test itself, if one does well, the other should also.

Since conditional independence is ordinary independence with respect to a conditional probability measure, it should be clear how to extend the concept to larger classes of sets.

Definition. A class {Ai:iJ}{Ai:iJ}, where J is an arbitrary index set, is conditionally independent, given event C, denoted {Ai:iJ} ci |C{Ai:iJ} ci |C, iff the product rule holds for every finite subclass of two or more.

As in the case of simple independence, the replacement rule extends.

The replacement rule

If the class {Ai:iJ} ci |C{Ai:iJ} ci |C, then any or all of the events Ai may be replaced by their complements and still have a conditionally independent class.

The use of independence techniques

Since conditional independence is independence, we may use independence techniques in the solution of problems. We consider two types of problems: an inference problem and a conditional Bernoulli sequence.

Example 3: Use of independence techniques

Sharon is investigating a business venture which she thinks has probability 0.7 of being successful. She checks with five “independent” advisers. If the prospects are sound, the probabilities are 0.8, 0.75, 0.6, 0.9, and 0.8 that the advisers will advise her to proceed; if the venture is not sound, the respective probabilities are 0.75, 0.85, 0.7, 0.9, and 0.7 that the advice will be negative. Given the quality of the project, the advisers are independent of one another in the sense that no one is affected by the others. Of course, they are not independent, for they are all related to the soundness of the venture. We may reasonably assume conditional independence of the advice, given that the venture is sound and also given that the venture is not sound. If Sharon goes with the majority of advisers, what is the probability she will make the right decision?

SOLUTION

If the project is sound, Sharon makes the right choice if three or more of the five advisors are positive. If the venture is unsound, she makes the right choice if three or more of the five advisers are negative. Let H=H= the event the project is sound, F=F= the event three or more advisers are positive, G=Fc=G=Fc= the event three or more are negative, and E=E= the event of the correct decision. Then

P ( E ) = P ( F H ) + P ( G H c ) = P ( F | H ) P ( H ) + P ( G | H c ) P ( H c ) P ( E ) = P ( F H ) + P ( G H c ) = P ( F | H ) P ( H ) + P ( G | H c ) P ( H c )
(11)

Let Ei be the event the ith adviser is positive. Then P(F|H)=P(F|H)= the sum of probabilities of the form P(Mk|H)P(Mk|H), where Mk are minterms generated by the class {Ei:1i5}{Ei:1i5}. Because of the assumed conditional independence,

P ( E 1 E 2 c E 3 c E 4 E 5 | H ) = P ( E 1 | H ) P ( E 2 c | H ) P ( E 3 c | H ) P ( E 4 | H ) P ( E 5 | H ) P ( E 1 E 2 c E 3 c E 4 E 5 | H ) = P ( E 1 | H ) P ( E 2 c | H ) P ( E 3 c | H ) P ( E 4 | H ) P ( E 5 | H )
(12)

with similar expressions for each P(Mk|H)P(Mk|H) and P(Mk|Hc)P(Mk|Hc). This means that if we want the probability of three or more successes, given H, we can use ckn with the matrix of conditional probabilities. The following MATLAB solution of the investment problem is indicated.

P1 = 0.01*[80 75 60 90 80];
P2 = 0.01*[75 85 70 90 70];
PH = 0.7;
PE = ckn(P1,3)*PH + ckn(P2,3)*(1 - PH)
PE =    0.9255

Often a Bernoulli sequence is related to some conditioning event H. In this case it is reasonable to assume the sequence {Ei:1in} ci |H{Ei:1in} ci |H and ci |Hc ci |Hc. We consider a simple example.

Example 4: Test of a claim

A race track regular claims he can pick the winning horse in any race 90 percent of the time. In order to test his claim, he picks a horse to win in each of ten races. There are five horses in each race. If he is simply guessing, the probability of success on each race is 0.2. Consider the trials to constitute a Bernoulli sequence. Let H be the event he is correct in his claim. If S is the number of successes in picking the winners in the ten races, determine P(H|S=k)P(H|S=k) for various numbers k of correct picks. Suppose it is equally likely that his claim is valid or that he is merely guessing. We assume two conditional Bernoulli trials:

Claim is valid:       Ten trials, probability p=P(Ei|H)=0.9p=P(Ei|H)=0.9.

Guessing at random: Ten trials, probability p=P(Ei|Hc)=0.2p=P(Ei|Hc)=0.2.

Let S=S= number of correct picks in ten trials. Then

P ( H | S = k ) P ( H c | S = k ) = P ( H ) P ( H c ) · P ( S = k | H ) P ( S = k | H c ) , 0 k 10 P ( H | S = k ) P ( H c | S = k ) = P ( H ) P ( H c ) · P ( S = k | H ) P ( S = k | H c ) , 0 k 10
(13)

Giving him the benefit of the doubt, we suppose P(H)/P(Hc)=1P(H)/P(Hc)=1 and calculate the conditional odds.

k = 0:10;
Pk1 = ibinom(10,0.9,k);    % Probability of k successes, given H
Pk2 = ibinom(10,0.2,k);    % Probability of k successes, given H^c
OH  = Pk1./Pk2;            % Conditional odds-- Assumes P(H)/P(H^c) = 1
e   = OH > 1;              % Selects favorable odds
disp(round([k(e);OH(e)]'))
           6           2      % Needs at least six to have creditability
           7          73      % Seven would be creditable,
           8        2627      % even if P(H)/P(H^c) = 0.1
           9       94585
          10     3405063

Under these assumptions, he would have to pick at least seven correctly to give reasonable validation of his claim.

Collection Navigation

Content actions

Download:

Collection as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add:

Collection to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks

Module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks