W01 Notes

Events and outcomes

01 Theory

Theory 1

Events and outcomes – informally

• An event is a description of something that can happen.
• An outcome is a complete description of something that can happen.

All outcomes are events. An event is usually a partial description. Outcomes are events given with a complete description.

Here ‘complete’ and ‘partial’ are within the context of the probability model.

It can be misleading to say that an ‘outcome’ is an ‘observation’.

• ‘Observations’ occur in the real world, while ‘outcomes’ occur in the model.
• To the extent the model is a good one, and the observation conveys complete information, we can say ‘outcome’ for the observation.

Notice: Because outcomes are complete, no two distinct outcomes could actually happen in a run of the experiment being modeled.

When an event happens, the fact that it has happened constitutes information.

Events and outcomes – mathematically

• The sample space is the set of possible outcomes, so it is the set of the complete descriptions of everything that can happen.
• An event is a subset of the sample space, so it is a collection of outcomes.

For mathematicians: some “wild” subsets are not valid events. Problems with infinity and the continuum...

Notation

• Write $S$ for the set of possible outcomes, $s\in S$ for a single outcome in $S$.

• Write $A,B,C,\cdots \subset S$ or $A_1,A_2,A_3,\cdots \subset S$ for some events, subsets of $S$.

• Write $ℱ$ for the collection of all events. This is frequently a huge set!

• Write $|A|$ for the cardinality or size of a set $A$, i.e. the number of elements it contains.

Using this notation, we can consider an outcome itself as an event by considering the “singleton” subset $\{\omega \}\subset S$ which contains that outcome alone.

Link to original

02 Illustration

Example - Coin flipping

Coin flipping

Flip a fair coin two times and record both results.

• Outcomes: sequences, like $HH$ or $TH$.

• Sample space: all possible sequences, i.e. the set $S=\{HH,HT,TH,TT\}$.

• Events: for example:

  • $A=\{HH,HT\}=\text{“first was heads”}$
  • $B=\{HT,TH\}=\text{“exactly one heads”}$
  • $C=\{HT,TH,HH\}=\text{“at least one heads”}$

With this setup, we may combine events in various ways to generate other events:

Complex events: for example: $A\cap B=\{HT\}$, or in words:

$$\text{“first was heads”}\text{AND}\text{“exactly one heads”}=\text{“heads-then-tails”}$$

Notice that the last one is a complete description, namely the outcome $HT$. $A\cup B=\{HH,HT,TH\}$, or in words:

$$\begin{array}{c}\text{“first was heads”}\text{OR}\text{“exactly one heads”}\\ =\text{“starts with heads, else it’s tails-then-heads”}\end{array}$$ Link to original

Practice exercise

Coin flipping: counting subsets

Flip a fair coin five times and record the results.

How many elements are in the sample space? (How big is $S$?) How many events are there? (How big is $ℱ$?)

Solution

There are $2^5=32$ possible sequences, so $|S|=32$.

To count the number of possible subsets, consider that we have 32 distinct items, and a subset is uniquely determined by the binary information – for each item – of whether it is in or out. Thus there are $2^{32}$ possibilities. So $|ℱ|=2^{32}$.

Link to original

03 Theory

Theory 2

New events from old

Given two events $A$ and $B$, we can form new events using set operations:

$$\begin{array}{cc}A\cup B⟷& \text{“event }A\text{ OR }\text{event }B\text{”}\\ & \\ A\cap B⟷& \text{“event }A\text{ AND }\text{event }B\text{”}\\ & \\ A^c⟷& \text{𝐧𝐨𝐭}\text{ event }A\end{array}$$

We also use these terms for events $A$ and $B$:

• They are mutually exclusive when $A\cap B=\varnothing$, that is, they have no elements in common.

• They are collectively exhaustive $A\cup B=S$, that is, when they jointly cover all possible outcomes.

In probability texts, sometimes $A\cap B$ is written “$A\cdot B$” or even (frequently!) “$AB$”.

Rules for sets

Algebraic rules

• Associativity: $(A\cup B)\cup C=A\cup (B\cup C)$. Analogous to $(A+B)+C=A+(B+C)$.

• Distributivity: $A\cap (B\cup C)=(A\cap B)\cup (A\cap C)$. Analogous to $A(B+C)=AB+AC$.

De Morgan’s Laws

• ${(A\cup B)}^c=A^c\cap B^c$

• ${(A\cap B)}^c=A^c\cup B^c$ In other words: you can distribute “ ${}^c$ ” but must simultaneously do a switch $\cap \leftrightarrow \cup$.

Link to original

Probability models

04 Theory

Theory 1

Axioms of probability

A probability measure is a function $P:ℱ\to ℝ$ satisfying:

Kolmogorov Axioms:

• Axiom 1: $P[A]\ge 0$ for every event $A$ (probabilities are not negative!)

• Axiom 2: $P[S]=1$ (probability of “anything” happening is 1)

• Axiom 3: additivity for any countable collection of mutually exclusive events:

$$P[A_1\cup A_2\cup A_3\cup \cdots ]=P[A_1]+P[A_2]+P[A_3]+\cdots$$ $$\text{when:}{A}_i\cap A_j=\varnothing \text{for all }i\ne j$$

Notation: we write $P[A]$ instead of $P(A)$, even though $P$ is a function, to emphasize the fact that $A$ is a set.

Probability model

A probability model or probability space consists of a triple $(S,ℱ,P)$:

• $S$ the sample space

• $ℱ$ the set of valid events, where every $A\in ℱ$ satisfies $A\subset S$

• $P:ℱ\to ℝ$ a probability measure satisfying the Kolmogorov Axioms

Finitely many exclusive events

It is a consequence of the Kolmogorov Axioms that additivity also works for finite collections of mutually exclusive events:

$$\begin{array}{c}P[A\cup B]=P[A]+P[B]\\ \\ P[A_1\cup \cdots \cup A_n]=P[A_1]+\cdots +P[A_n]\end{array}$$

Inferences from Kolmogorov

A probability measure satisfies these rules. They can be deduced from the Kolmogorov Axioms.

• Negation: Can you find $P[A^c]$ but not $P[A]$? Use negation:

$$P[A]=1-P[A^c]$$

• Monotonicity: Probabilities grow when outcomes are added:

$$A\subset B\gg \gg P[A]\le P[B]$$

• Inclusion-Exclusion: A trick for resolving unions:

$$P[A\cup B]=P[A]+P[B]-P[A\cap B]$$

(even when $A$ and $B$ are not exclusive!)

Inclusion-Exclusion

The principle of inclusion-exclusion generalizes to three events:

$$\begin{array}{c}P[A\cup B\cup C]=\\ \\ P[A]+P[B]+P[C]-P[A\cap B]-P[A\cap C]-P[B\cap C]+P[A\cap B\cap C]\end{array}$$

The same pattern works for any number of events!

The pattern goes: “include singles” then “exclude doubles” then “include triples” then …

Include, exclude, include, exclude, include, …

Link to original

05 Illustration

Example - iPhones and iPads

iPhones and iPads

At Mr. Jefferson’s University, 25% of students have an iPhone, 30% have an iPad, and 60% have neither.

What is the probability that a randomly chosen student has some iProduct? (Q1)

What about both? (Q2)

Solution

(1) Set up the probability model:

A student is chosen at random:

Outcomes are chosen students.

The sample space $S$ is the set of all students. Events are subsets of $S$.

Write $O=\text{“has iPhone”}$ and $A=\text{“has iPad”}$ (regarding the chosen student).

All students are equally likely to be chosen. Therefore $P[E]=\frac{|E|}{|S|}$ for any event $E$. Therefore $P[O]=0.25$ and $P[A]=0.30$.

Furthermore, $P[O^c{A}^c]=0.60$. This states that 60% have “not iPhone AND not iPad”.

---

(2) Define the desired event:

Q1: $\text{desired event}=O\cup A$

Q2: $\text{desired event}=OA$

---

(3) Compute the probabilities:

We do not know that $O$ and $A$ are exclusive.

We could try inclusion-exclusion:

$$P[O\cup A]=P[O]+P[A]-P[OA]$$

We know $P[O]=0.25$ and $P[A]=0.30$. So this formula, with given data, RELATES Q1 and Q2. It does not solve either one by itself.

We have not yet used the information that $P[O^c{A}^c]=0.60$.

To use this, simplify it with De Morgan’s Laws:

$$P[O^c{A}^c]\gg \gg P[{(O\cup A)}^c]\gg \gg 1-P[O\cup A]$$

Therefore:

$$P[O^c{A}^c]=0.60\gg \gg P[O\cup A]=0.40$$

We have answered Q1. Recall that inclusion-exclusion relates Q1 and Q2 and solve to answer Q2:

$$\begin{array}{c}P[O\cup A]=P[O]+P[A]-P[OA]\\ \\ \gg \gg 0.40=0.25+0.30-P[OA]\\ \\ \gg \gg P[OA]=0.15\end{array}$$Link to original

Example - Lucia is Host or Player

Lucia is Host or Player

The professor chooses three students at random for a game in a class of 40, one to be Host, one to be Player, one to be Judge. What is the probability that Lucia is either Host or Player?

Solution

(1) Set up the probability model:

Label the students $1$ to $40$. Write $L$ for Lucia’s number.

Outcomes: assignments such as $(H,P,J)=(\mathrm{2,5,8})$. These are ordered triples with distinct entries in $1,2,\dots ,40$.

Sample space: $S$ is the collection of all such distinct triples

Events: any subset of $S$

Probability measure: assume all outcomes are equally likely, so $P[(i,j,k)]=P[(r,l,p)]$ for all $i,j,k,r,l,p$

In total there are $40\cdot 39\cdot 38$ triples of distinct numbers.

Therefore $P[(i,j,k)]=\frac{1}{40\cdot 39\cdot 38}$ for any specific outcome $(i,j,k)$.

Therefore $P[A]=\frac{|A|}{40\cdot 39\cdot 38}$ for any event $A$. (Recall $|A|$ is the number of outcomes in $A$.)

---

(2) Define the desired event:

We want to find $P[\text{“Lucia is Host or Player”}]$. Define $A=\text{“Lucia is Host”}$ and $B=\text{“Lucia is Player”}$. Thus:

$$A=\{(L,j,k)|\text{any }j,k\},B=\{(i,L,k)|\text{any }i,k\}$$

So, in this notation, we seek $P[A\cup B]$.

---

(3) Compute the desired probability:

Importantly, $A\cap B=\varnothing$ (mutually exclusive). There are no outcomes in $S$ in which Lucia is both Host and Player.

By additivity, we infer $P[A\cup B]=P[A]+P[B]$.

Now compute $P[A]$. There are $39\cdot 38$ ways to choose $j$ and $k$ from the students besides Lucia. Therefore $|A|=39\cdot 38$. Therefore:

$$P[A]\gg \gg \frac{|A|}{40\cdot 39\cdot 38}\gg \gg \frac{39\cdot 38}{40\cdot 39\cdot 38}\gg \gg \frac{1}{40}$$

Now compute $P[B]$. It is similar: $P[B]=\frac{1}{40}$.

Finally compute that $P[A]+P[B]=\frac{1}{20}$, so the answer is:

$$P[A\cup B]\gg \gg P[A]+P[B]\gg \gg \frac{1}{20}$$ Link to original

Conditional probability

06 Theory

Theory 1

Conditional probability

The conditional probability of “$B$ given $A$” is defined by:

$$P[B|A]=\frac{P[B\cap A]}{P[A]}$$

This conditional probability $P[B|A]$ represents the probability of event $B$ taking place given the assumption that $A$ took place. (All within the given probability model.)

By letting the actuality of event $A$ be taken as a fixed hypothesis, we can define a conditional probability measure by plugging events into the slot of $B$:

$$P[-|A]=\frac{P[-\cap A]}{P[A]}$$

It is possible to verify each of the Kolmogorov axioms for this function, and therefore $P[-|A]$ itself defines a bona fide probability measure.

Conditioning

What does it really mean?

Conceptually, $P[B|A]$ corresponds to creating a new experiment in which we run the old experiment and record data only those times that $A$ happened. Or, it corresponds to finding ourselves with knowledge or data that $A$ happened, and we seek our best estimates of the likelihoods of other events, based on our existing model and the actuality of $A$.

Mathematically, $P[B|A]$ corresponds to restricting the probability function to outcomes in $A$, and renormalizing the values (dividing by $P[A]$) so that the total probability of all the outcomes (in $A$) is now $1$.

The definition of conditional probability can also be turned around and reinterpreted:

Multiplication rule

$$P[AB]=P[A]\cdot P[B|A]$$

“The probability of $A$ AND $B$ equals the probability of $A$ times the probability of $B$-given-$A$.”

This principle generalizes to any events in sequence:

Generalized multiplication rule

$$P[A_1{A}_2{A}_3]=P[A_1]\cdot P[A_2|A_1]\cdot P[A_3|A_1{A}_2]$$ $$P[A_1\cdots A_n]=P[A_1]\cdot P[A_2|A_1]\cdot P[A_3|A_1{A}_2]\cdots P[A_n|A_1\cdots A_{n-1}]$$

The generalized rule can be verified like this. First substitute $A_2$ for $B$ and $A_1$ for $A$ in the original rule. Now repeat, substituting $A_3$ for $B$ and $A_1{A}_2$ for $A$ in the original rule, and combine with the first one, and you find the rule for triples. Repeat again with $A_4$ and $A_1{A}_2{A}_3$, combine with the triples, and you get quadruples.

Link to original

07 Illustration

Practice exercise

Simplifying conditionals inclusion

Let $A\subset B$. Simplify the following values:

$$P[A|B],P[A|B^c],P[B|A],P[B|A^c]$$

Solution

$P[A|B]$

• Definition of ‘conditional’: $P[A|B]=\frac{P[A\cap B]}{P[B]}$
• The problem assumes that $A\subset B$. Therefore $A\cap B=A$.
• Therefore, answer: $\frac{P[A]}{P[B]}$.

$P[A|B^c]$

• Definition of ‘conditional’: $P[A|B^c]=\frac{P[A\cap B^c]}{P[B^c]}$
• Since $A\subset B$, we know $A\cap B^c=\varnothing$.
• Therefore $P[A\cap B^c]=0$ and answer $=0$.

$P[B|A]$

• Definition of ‘conditional’: $P[B|A]=\frac{P[B\cap A]}{P[A]}$
• Since $A\subset B$, we have $B\cap A=A$.
• Therefore, answer $=1$.

$P[B|A^c]$

• Definition of ‘conditional’: $P[B|A^c]=\frac{P[B\cap A^c]}{P[A^c]}$
• There is no way to simplify further.
• We could write $P[A^c]=1-P[A]$ if desired.

Link to original

Example - Multiplication: draw two cards

Multiplication: draw two cards

Two cards are drawn from a standard deck (without replacement).

What is the probability that the first is a 3, and the second is a 4?

Solution

This “two-stage” experiment lends itself to a solution using the multiplication rule for conditional probability.

(1) Label events:

• Write $T$ for the event that the first card is a 3.
• Write $F$ for the event that the second card is a 4.

We seek $P[TF]$. We will use the multiplication rule:

$$P[TF]=P[T]\cdot P[F|T]$$

---

(2) Compute probabilities:

We know $P[T]=\frac{4}{52}$. (Does not depend on the second draw.)

For the conditional probability, note that if the first is a 3, then there are four remaining 4s and 51 remaining cards. Therefore:

$$P[F|T]=\frac{4}{51}$$

---

(3) Apply multiplication rule:

$$\begin{array}{c}P[TF]=P[T]\cdot P[F|T]\\ \\ \\ P[TF]=\frac{4}{52}\cdot \frac{4}{51}\gg \gg \frac{4}{13\cdot 51}\end{array}$$ Link to original

Example - Flip a coin, then roll dice

Multiplication: flip a coin, then roll dice

Flip a coin. If the outcome is heads, roll two dice and add the numbers. If the outcome is tails, roll a single die and take that number. What is the probability of getting a tails AND a number at least 3?

Solution

(1) This “two-stage” experiment lends itself to a solution using the multiplication rule for conditional probability.

Label the events of interest.

Let $H$ and $T$ be the events that the coin showed heads and tails, respectively.

Let $A_1,\dots ,A_{12}$ be the events that the final number is $1,\dots ,12$, respectively.

The value we seek is $P[T{A}_{\ge 3}]$.

---

(2) Observe known (conditional) probabilities.

We know that $P[H]=\frac{1}{2}$ and $P[T]=\frac{1}{2}$.

We know that $P[A_5|T]=\frac{1}{6}$, for example, or that $P[A_2|H]=\frac{1}{36}$.

---

(3) Apply multiplication rule:

$$P[T{A}_{\ge 3}]=P[T]\cdot P[A_{\ge 3}|T]$$

We know $P[T]=\frac{1}{2}$ and can see by counting that $P[A_{\ge 3}|T]=\frac{2}{3}$.

Therefore $P[T{A}_{\ge 3}]=\frac{1}{3}$.

Link to original

Example - Coin flipping: at least 2 heads

Coin flipping: at least 2 heads

Flip a fair coin 4 times and record the outcomes as sequences, like HHTH.

Let $A_{\ge 2}$ be the event that there are at least two heads, and $A_{\ge 1}$ the event that there is at least one heads.

First let’s calculate $P[A_{\ge 2}]$.

Define $A_2$, the event that there were exactly 2 heads, and $A_3$, the event of exactly 3, and $A_4$ the event of exactly 4. These events are exclusive, so:

$$P[A_{\ge 2}]=P[A_2\cup A_3\cup A_4]\gg \gg P[A_2]+P[A_3]+P[A_4]$$

Each term on the right can be calculated by counting:

$$\begin{array}{cc}P[A_2]& =\frac{|A_2|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0px}{}{4}{2}\right)}{16}\gg \gg \frac{6}{16}\\ & \\ P[A_3]& =\frac{|A_3|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0px}{}{4}{3}\right)}{16}\gg \gg \frac{4}{16}\\ & \\ P[A_4]& =\frac{|A_4|}{2^4}\gg \gg \frac{\left(\genfrac{}{}{0px}{}{4}{4}\right)}{16}\gg \gg \frac{1}{16}\end{array}$$

Therefore, $P[A_{\ge 2}]=\frac{11}{16}$.

Now suppose we find out that “at least one heads definitely came up”. (Meaning that we know $A_{\ge 1}$.) For example, our friend is running the experiment and tells us this fact about the outcome.

Now what is our estimate of likelihood of $A_{\ge 2}$?

The formula for conditioning gives:

$$P[A_{\ge 2}|A_{\ge 1}]=\frac{P[A_{\ge 2}\cap A_{\ge 1}]}{P[A_{\ge 1}]}$$

Now $A_{\ge 2}\cap A_{\ge 1}=A_{\ge 2}$. (Any outcome with at least two heads automatically has at least one heads.) We already found that $P[A_{\ge 2}]=\frac{11}{16}$. To compute $P[A_{\ge 1}]$ we simply add the probability $P[A_1]$, which is $\frac{4}{16}$, to get $P[A_{\ge 1}]=\frac{15}{16}$.

Therefore:

$$P[A_{\ge 2}|A_{\ge 1}]=\frac{11/16}{15/16}\gg \gg \frac{11}{15}$$ Link to original

08 Theory

Theory 2

Law of Total Probability: 2 cases

For any events $A$ and $B$:

$$P[B]=P[A]\cdot P[B|A]+P[A^c]\cdot P[B|A^c]$$

This rule can also be called “Division into Cases.”

Interpretation: event $B$ may be divided along the lines of $A$, with some of $P[B]$ coming from the part in $A$ and the rest from the part in $A^c$.

Total Probability - Explanation

• First divide $B$ itself into parts in and out of $A$:

$$B=B\cap A\bigcup B\cap A^c$$

• These parts are exclusive, so in probability we have:

$$P[B]=P[BA]+P[B{A}^c]$$

• Use the Multiplication rule to break up $P[BA]$ and $P[B{A}^c]$:

$$\begin{array}{cc}P[BA]\gg \gg & P[A]\cdot P[B|A]\\ & \\ P[B{A}^c]\gg \gg & P[A^c]\cdot P[B|A^c]\end{array}$$

• Now substitute in the prior formula:

$$P[B]\gg \gg P[BA]+P[B{A}^c]\gg \gg P[A]\cdot P[B|A]+P[A^c]\cdot P[B|A^c]$$

This law can be generalized to any partition of the sample space $S$. A partition is a collection of events $A_i$ which are mutually exclusive and jointly exhaustive:

$$A_i\cap A_j=\varnothing ,\bigcup _i{A}_i=S$$

The generalized formulation of Total Probability for a partition is:

Law of Total Probability: $n$ cases

For a partition $A_i$ of the sample space $S$:

$$P[B]=\sum _{i=1}^{n}P[A_i]\cdot P[B|A_i]$$

By setting $n=2$ with $A_1=A$ and $A_2=A^c$, we recover the $2$-case Law from the $n$-case version of the Law.

Link to original

09 Illustration

Practice exercise

Marble transferred, marble drawn

Setup:

• Bin 1 holds five red and four green marbles.
• Bin 2 holds four red and five green marbles.

Experiment:

• You take a random marble from Bin 1 and put it in Bin 2 and shake Bin 2.
• Then you draw a random marble from Bin 2 and look at it.

What is the probability that the marble you look at is red?

Solution

(1) Label events:

• Event $T_R$: a red marble is transferred.
• Event $T_G$: a green marble is transferred.
• Event $D_R$: a red marble is drawn from Bin 2.
• Event $D_G$: a green marble is drawn from Bin 2.

Answer will be $P[D_R]$.

---

(2) Apply Division into Cases:

General formula: $P[B]=P[A]\cdot P[B|A]+P[A^c]\cdot P[B|A^c]$

Insert our labels, $A\to T_R$ and $B\to D_R$ and $A^c\to T_G$. Obtain:

$$P[D_R]=P[T_R]\cdot P[D_R|T_R]+P[T_G]\cdot P[D_R|T_G]$$

---

(3) Plug in data and compute:

• Know $P[T_R]=5/9$.
• Know $P[T_G]=4/9$.
• Know $P[D_R|T_R]=5/10$.
• Know $P[D_R|T_G]=4/10$.

Therefore:

$$P[D_R]\gg \gg \frac{5}{9}\cdot \frac{5}{10}+\frac{4}{9}\cdot \frac{4}{10}\gg \gg \frac{41}{90}$$

Link to original