Theory

Theory 1

Conditional probability

The conditional probability of “$B$ given $A$” is defined by:

$$P[B|A]=\frac{P[B\cap A]}{P[A]}$$

This conditional probability $P[B|A]$ represents the probability of event $B$ taking place given the assumption that $A$ took place. (All within the given probability model.)

By letting the actuality of event $A$ be taken as a fixed hypothesis, we can define a conditional probability measure by plugging events into the slot of $B$:

$$P[-|A]=\frac{P[-\cap A]}{P[A]}$$

It is possible to verify each of the Kolmogorov axioms for this function, and therefore $P[-|A]$ itself defines a bona fide probability measure.

Conditioning

What does it really mean?

Conceptually, $P[B|A]$ corresponds to creating a new experiment in which we run the old experiment and record data only those times that $A$ happened. Or, it corresponds to finding ourselves with knowledge or data that $A$ happened, and we seek our best estimates of the likelihoods of other events, based on our existing model and the actuality of $A$.

Mathematically, $P[B|A]$ corresponds to restricting the probability function to outcomes in $A$, and renormalizing the values (dividing by $P[A]$) so that the total probability of all the outcomes (in $A$) is now $1$.

The definition of conditional probability can also be turned around and reinterpreted:

Multiplication rule

$$P[AB]=P[A]\cdot P[B|A]$$

“The probability of $A$ AND $B$ equals the probability of $A$ times the probability of $B$-given-$A$.”

This principle generalizes to any events in sequence:

Generalized multiplication rule

$$P[A_1{A}_2{A}_3]=P[A_1]\cdot P[A_2|A_1]\cdot P[A_3|A_1{A}_2]$$ $$P[A_1\cdots A_n]=P[A_1]\cdot P[A_2|A_1]\cdot P[A_3|A_1{A}_2]\cdots P[A_n|A_1\cdots A_{n-1}]$$

The generalized rule can be verified like this. First substitute $A_2$ for $B$ and $A_1$ for $A$ in the original rule. Now repeat, substituting $A_3$ for $B$ and $A_1{A}_2$ for $A$ in the original rule, and combine with the first one, and you find the rule for triples. Repeat again with $A_4$ and $A_1{A}_2{A}_3$, combine with the triples, and you get quadruples.

Theory 2

Law of Total Probability: 2 cases

For any events $A$ and $B$:

$$P[B]=P[A]\cdot P[B|A]+P[A^c]\cdot P[B|A^c]$$

This rule can also be called “Division into Cases.”

Interpretation: event $B$ may be divided along the lines of $A$, with some of $P[B]$ coming from the part in $A$ and the rest from the part in $A^c$.

Total Probability - Explanation

• First divide $B$ itself into parts in and out of $A$:

$$B=B\cap A\bigcup B\cap A^c$$

• These parts are exclusive, so in probability we have:

$$P[B]=P[BA]+P[B{A}^c]$$

• Use the Multiplication rule to break up $P[BA]$ and $P[B{A}^c]$:

$$\begin{array}{cc}P[BA]\gg \gg & P[A]\cdot P[B|A]\\ & \\ P[B{A}^c]\gg \gg & P[A^c]\cdot P[B|A^c]\end{array}$$

• Now substitute in the prior formula:

$$P[B]\gg \gg P[BA]+P[B{A}^c]\gg \gg P[A]\cdot P[B|A]+P[A^c]\cdot P[B|A^c]$$

This law can be generalized to any partition of the sample space $S$. A partition is a collection of events $A_i$ which are mutually exclusive and jointly exhaustive:

$$A_i\cap A_j=\varnothing ,\bigcup _i{A}_i=S$$

The generalized formulation of Total Probability for a partition is:

Law of Total Probability: $n$ cases

For a partition $A_i$ of the sample space $S$:

$$P[B]=\sum _{i=1}^{n}P[A_i]\cdot P[B|A_i]$$

By setting $n=2$ with $A_1=A$ and $A_2=A^c$, we recover the $2$-case Law from the $n$-case version of the Law.