Theory

Theory 1

Bayes’ Theorem

For any events $A$ and $B$:

$$P[B|A]=P[B]\cdot \frac{P[A|B]}{P[A]}$$

Note: Bayes’ Theorem is sometimes called Bayes’ Rule.

Bayes’ Theorem - Derivation

Start with the observation that $AB=BA$, in other words event “$A$ AND $B$” equals event “$B$ AND $A$”.

Apply the multiplication rule to each product:

$$\begin{array}{cc}P[AB]& =P[A]\cdot P[B|A]\\ & \\ P[BA]& =P[B]\cdot P[A|B]\end{array}$$

Equate them and rearrange:

$$\begin{array}{cc}P[AB]=P[BA]\gg \gg & P[A]\cdot P[B|A]=P[B]\cdot P[A|B]\\ & \\ \gg \gg & P[B|A]=P[B]\cdot \frac{P[A|B]}{P[A]}\end{array}$$

The main application of Bayes’ Theorem is to calculate $P[A|B]$ when it is easy to calculate $P[B|A]$ from the problem setup. Often this occurs in multi-stage experiments where event $A$ describes outcomes of an intermediate stage.

Note: These lecture notes use alphabetical order $A$, $B$ as a mnemonic for temporal or logical order, i.e. that $A$ comes first in time, or that $A$ is the prior conditional from which it is easy to calculate $B$.