W11 Notes

Review: True/False

TRUE or FALSE:

(a) Suppose $\mathrm{Cov}[X,Y]=0.05$. It is possible that ${\rho }_{X,Y}=0.05$.

(b) Suppose $\mathrm{Cov}[X,Y]=0.05$. It is possible that $X$ and $Y$ have a strong linear relationship.

(c) Suppose $\mathrm{Cov}[X,Y]=0$. It is possible that $X$ and $Y$ are not independent.

(d) Suppose X and Y are not independent. It is possible that $\mathrm{Cov}[X,Y]$ is equal to 0.

(e) Suppose X and Y are independent. $\mathrm{Cov}[X,Y]$ must be equal to 0.

Review: Conditional probability

Review

Recall some items related to conditional probability.

Conditioning definition:

$$P[-|A]=\frac{P[-\cap A]}{P[A]}$$

Multiplication rule:

$$P[AB]=P[B|A]P[A]$$

Division into Cases / Total Probability:

$$P[B]=P[B|A_1]P[A_1]+\cdots +P[B|A_n]P[A_n]$$Link to original

Conditional distribution

01 Theory

Theory 1

Conditional distribution - fixed event

Suppose $X$ is a random variable, and suppose $A\subset ℝ$. The distribution of $X$ conditioned on $A$ describes the probabilities of values of $X$ given knowledge that $X\in A$.

Discrete case:

$$P_{X|A}(k)=\{\begin{array}{cc}{\displaystyle \frac{1}{P[A]}{P}_X(k)}& k\in A\\ & \\ 0& k\notin A\end{array}$$

Continuous case:

$$f_{X|A}(x)=\{\begin{array}{cc}{\displaystyle \frac{1}{P[A]}{f}_X(x)}& x\in A\\ & \\ 0& x\notin A\end{array}$$

There is also a conditional CDF, of which this conditional PDF is the derivative:

$$F_{X|A}(x)=P[X\le x|A],f_{X|A}(x)=\frac{d}{dx}{F}_{X|A}(x)$$

The Law of Total Probability has versions for distributions:

$$\begin{array}{cc}{P}_X(k)& =P_{X|A_1}(k)P[A_1]+\cdots +P_{X|A_n}(k)P[A_n]\\ & \\ f_X(x)& =f_{X|A_1}(x)P[A_1]+\cdots +f_{X|A_n}(x)P[A_n]\end{array}$$

Conditional distribution - variable event

Suppose $X$ and $Y$ are any two random variables. The distribution of $X$ conditioned on $Y$ describes the probabilities of values of $X$ in terms of $y$, given knowledge that $Y=y$.

Discrete case:

$$\begin{array}{cc}{P}_{X|Y}(k|\ell )& =P[X=k|Y=\ell ]\\ & \\ & =\frac{P_{X,Y}(k,\ell )}{P_Y(\ell )}(\text{assuming }{P}_Y(\ell )\ne 0)\end{array}$$

Continuous case:

$$f_{X|Y}(x|y)=\frac{f_{X,Y}(x,y)}{f_Y(y)}(\text{assuming }{f}_Y(y)\ne 0)$$

Remember: $P_{X,Y}(k,\ell )$ is the probability that “$X=k$ and $Y=\ell$.”

Sometimes it is useful to have the formulas rewritten like this:

$$\begin{array}{cc}{P}_{X,Y}(k,\ell )& =P_{X|Y}(k|\ell )P_Y(\ell )\\ & \\ f_{X,Y}(x,y)& =f_{X|Y}(x|y)f_Y(y)\end{array}$$

Extra - Deriving $f_{X|Y}(x|y)$

The density $f_{X|Y}$ ought to be such that $f_{X|Y}(x|y)dx$ gives the probability of $X\in [x,x+dx]$, given knowledge that $Y\in [y,y+dy]$. Calculate this probability:

$$\begin{array}{c}P[x\le X\le x+dx|y\le Y\le y+dy]\\ \\ \gg \gg \frac{P[x\le X\le x+dx,y\le Y\le y+dy]}{P[y\le Y\le y+dy]}\\ \\ \gg \gg \frac{f_{X,Y}(x,y)dxdy}{f_Y(y)dy}\\ \\ \gg \gg \frac{f_{X,Y}(x,y)}{f_Y(y)}dx\end{array}$$

Link to original

02 Illustration

Example - Conditional PMF, variable event, via joint density

Conditional PMF, variable event, via joint density

Suppose $X$ and $Y$ have joint PMF given by:

$$P_{X,Y}(k,\ell )=\{\begin{array}{cc}{\displaystyle \frac{k+\ell }{21}}& k=\mathrm{1,2,3};\ell =\mathrm{1,2}\\ 0& \text{otherwise}\end{array}$$

Find $P_{X|Y}(k|\ell )$ and $P_{Y|X}(\ell ,k)$.

Solution

Marginal PMFs:

$$P_X(k)=\frac{2k+3}{21},k=\mathrm{1,2,3}$$ $$P_Y(\ell )=\frac{\ell +2}{7},\ell =\mathrm{1,2}$$

Assuming $\ell =1$ or $2$, for each $k=1,2,3$ we have:

$$P_{X|Y}(k|\ell )=\frac{P_{X,Y}(k,\ell )}{P_Y(\ell )}\gg \gg \frac{k+\ell }{3\ell +6}$$

Assuming $k=1$, $2$, or $3$, for each $\ell =\mathrm{1,2}$ we have:

$$P_{Y|X}(\ell |k)=\frac{P_{Y,X}(\ell ,k)}{P_X(k)}\gg \gg \frac{k+\ell }{2k+3}$$ Link to original

Conditional expectation

03 Theory

Theory 1

Expectation conditioned by a fixed event

Suppose $X$ is a random variable and $A\subset ℝ$. The expectation of $X$ conditioned on $A$ describes the typical value of $X$ given the hypothesis that $X\in A$ is known.

Discrete case:

$$\begin{array}{cc}E[X|A]& =\sum _{k}k{P}_{X|A}(k)\\ & \\ E[g(X)|A]& =\sum _{k}g(k)P_{X|A}(k)\end{array}$$

Continuous case:

$$\begin{array}{cc}E[X|A]& ={\int }_{-\infty }^{+\infty }x{f}_{X|A}(x)dx\\ & \\ E[g(X)|A]& ={\int }_{-\infty }^{+\infty }g(x)f_{X|A}(x)dx\end{array}$$

Conditional variance:

$$\mathrm{Var}[X|A]=E[(X-{\mu }_{X|A}{)}^2|A]=E[X^2|A]-{\mu }_{X|A}^2$$

Division into Cases / Total Probability applied to expectation:

$$E[X]=E[X|A_1]P[A_1]+\cdots +E[X|A_n]P[A_n]$$

Linearity of conditional expectation:

$$E[a{X}_1+b{X}_2+c|Y=y]=aE[X_1|Y=y]+bE[X_2|Y=y]+c$$

Extra - Proof: Division of Expectation into Cases

We prove the discrete case only.

1. Expectation formula:

$$E[X]=\sum _{k}k{P}_X(k)$$

2. Division into Cases for the PMF:

$$P_X(k)=\sum _{i=1}^n{P}_{X|A_i}(k)P[A_i]$$

3. Substitute in the formula for $E[X]$:

$$\begin{array}{c}\sum _{k}k{P}_X(k)\gg \gg \sum _{k}k\sum _{i=1}^n{P}_{X|A_i}(k)P[A_i]\\ \\ \gg \gg \sum _{i=1}^{n}P[A_i]\sum _{k}k{P}_{X|A_i}(k)\\ \\ \gg \gg \sum _{i=1}^{n}P[A_i]E[X|A_i]\end{array}$$

---

Expectation conditioned by a variable event

Suppose $X$ and $Y$ are any two random variables. The expectation of $X$ conditioned on $Y=y$ describes the typical of value of $X$ in terms of $y$, given the hypothesis that $Y=y$ is known.

Discrete case:

$$\begin{array}{cc}E[X|Y=y]& =\sum _{k}k{P}_{X|Y}(k|y)\text{(}k\text{ over all poss. vals.)}\\ & \\ E[g(X,Y)|Y=y]& =\sum _{k}g(k,y)P_{X|Y}(k|y)\end{array}$$

Continuous case:

$$\begin{array}{cc}E[X|Y=y]& ={\int }_{-\infty }^{+\infty }x{f}_{X|Y}(x|y)dx\\ & \\ E[g(X,Y)|Y=y]& ={\int }_{-\infty }^{+\infty }g(x,y)f_{X|Y}(x|y)dx\end{array}$$

Link to original

05 Illustration

Example - Conditional PMF, fixed event, expectation

Conditional PMF, fixed event, expectation

Suppose $X$ measures the lengths of some items and has the following PMF:

$$P_X(k)=\{\begin{array}{cc}0.15& k=\mathrm{1,2,3,4}\\ 0.1& k=\mathrm{5,6,7,8}\\ 0& \text{ otherwise }\end{array}$$

Let $L=\text{“}X\ge 5\text{”}$, an event.

(a) Find the conditional PMF of $X$ given that $L$ is known.

(b) Find the conditional expected value and variance of $X$ given $L$.

Solution

(a)

Conditional PMF formula with $k\in L$ plugged in:

$$P_{X|L}(k)=\{\begin{array}{cc}{\displaystyle \frac{P_X(k)}{P[L]}}& k=\mathrm{5,6,7,8}\\ 0& \text{ otherwise }\end{array}$$

Compute $P[L]$ by adding cases:

$$P[L]=\sum _{k=5}^8{P}_X(k)\gg \gg 0.4$$

Divide nonzero PMF entries by $0.1$:

$$P_{X|L}(k)=\{\begin{array}{cc}0.25& k=\mathrm{5,6,7,8}\\ 0& \text{ otherwise }\end{array}$$

---

(b)

Find $E[X|L]$:

$$\begin{array}{c}E[X|L]=\sum _{k=5}^{8}k{P}_{X|L}(k)\\ \\ \gg \gg 5\cdot (0.25)+6\cdot (0.25)+7\cdot (0.25)+8\cdot (0.25)\\ \\ \gg \gg 6.5\min \end{array}$$

Find $E[X^2|L]$:

$$\begin{array}{c}E[X^2|L]=\sum _{k=5}^8{k}^2{P}_{X|L}(k)\\ \\ \gg \gg 5^2\cdot (0.25)+6^2\cdot (0.25)+7^2\cdot (0.25)+8^2\cdot (0.25)\\ \\ \gg \gg 43.5{\min }^2\end{array}$$

Find $\mathrm{Var}[X|L]$ using “short form” with conditioning:

$$\mathrm{Var}[X|L]=E[X^2|L]-E{[X|L]}^2\gg \gg 1.25{\min }^2$$ Link to original

04 Illustration

Example - Conditional expectations from joint density

Conditional expectations from joint density

Suppose $X$ and $Y$ are random variables with joint density given by:

$$f_{X,Y}(x,y)=\{\begin{array}{cc}{\displaystyle \frac{1}{y}{e}^{-x/y}{e}^{-y}}& x,y\in (0,\infty )\\ 0& \text{otherwise}\end{array}$$

Find $E[X|Y=y]$.

Solution

(1) Derive the marginal density $f_Y(y)$:

$$\begin{array}{c}{f}_Y(y)\gg \gg {\int }_0^{+\infty }\frac{1}{y}{e}^{-x/y}{e}^{-y}dx\\ \\ \gg \gg -e^{-x/y}{e}^{-y}{|}_{x=0}^{\infty }\gg \gg e^{-y}\end{array}$$

---

(2) Use $f_Y(y)$ to compute $f_{X|Y}(x|y)$:

$$\begin{array}{c}{f}_{X|Y}(x|y)\gg \gg \frac{f_{X,Y}(x,y)}{f_Y(y)}\\ \\ \gg \gg \frac{1}{y}{e}^{-x/y}{e}^{-y}\cdot {(e^{-y})}^{-1}\gg \gg \frac{1}{y}{e}^{-x/y}\end{array}$$

---

(3) Use $f_{X|Y}(x|y)$ to calculate expectation conditioned on the variable event:

$$\begin{array}{c}E[X|Y=y]\gg \gg {\int }_{-\infty }^{+\infty }x{f}_{X|Y}(x|y)dx\\ \\ \gg \gg {\int }_0^{\infty }\frac{x}{y}{e}^{-x/y}dx\gg \gg y\end{array}$$ Link to original

05 Theory - extra (non-examinable)

Theory 2

Expectation conditioned by a random variable

Suppose $X$ and $Y$ are any two random variables. The expectation of $X$ conditioned on $Y$ is a random variable giving the typical value of $X$ on the assumption that $Y$ has value determined by an outcome of the experiment.

$$E[X|Y]=g(Y)\text{where}g(y)=E[X|Y=y]$$

In other words, start by defining a function $g(y)$:

$$\begin{array}{cc}g:ℝ& \to ℝ\\ y& \mapsto E[X|Y=y]\end{array}$$

Now $E[X|Y]$ is defined as the composite random variable $g(Y)$.

Considered as a random variable, $E[X|Y]$ takes an outcome $s\in S$, computes $Y(s)$, sets $y=Y(s)$, then returns the expectation of $X$ conditioned on $Y=y$.

Notice that $X$ is not evaluated at $s$, only $Y$ is.

Because the value of $E[X|Y]$ depends only on $Y(s)$, and not on any additional information about $s$, it is common to represent a conditional expectation $E[X|Y]$ using only the function $g$.

---

Iterated Expectation

$$E[E[X|Y]]=E[X]$$

Proof of Iterated Expectation, discrete case

$$\begin{array}{cc}E[E[X|Y]]& =\sum _{\ell }E[X|Y=\ell ]P_Y(\ell )\\ & \\ & =\sum _{\ell }\sum _{k}k{P}_{X|Y}(k|\ell )P_Y(\ell )\\ & \\ & =\sum _{k}k\sum _{\ell }{P}_{X,Y}(k,\ell )\\ & \\ & =\sum _{k}k{P}_X(k)=E[X]\end{array}$$

Link to original

06 Illustration

Example - Conditional Expectation Variable

Sum of random number of RVs

Let $N$ denote the number of customers that enter a store on a given day.

Let $X_i$ denote the amount spent by the $i^{th}$ customer.

Assume that $E[N]=50$ and E[X_i]=\ ParseError: Unexpected character: '\' at position 8: E[X_i]=\̲8$foreach$i$.

What is the expected total spend of all customers in a day?

Solution

A formula for the total spend is $X={\sum }_{i=1}^N{X}_i$.

By Iterated Expectation, we know $E[X]=E[E[X|N]]$.

Now compute $E[X|N]$ as a function of $N$:

$$\begin{array}{cc}E[X|N=n]& \gg \gg E[\left(\sum _{i=1}^N{X}_i\right)|N=n]\\ & \\ & \gg \gg E[\left(\sum _{i=1}^n{X}_i\right)|N=n]\\ & \\ & \gg \gg \sum _{i=1}^{n}E[X_i|N=n]\\ & \\ & \gg \gg \sum _{i=1}^{n}E[X_i]\gg \gg 8n\end{array}$$

Therefore $g(n)=8n$ and $g(N)=8N$ and $E[X|N]=8N$.

Then by Iterated Expectation, E[X]=E[8N]=8E[N]=\ ParseError: Unexpected character: '\' at position 18: …X]=E[8N]=8E[N]=\̲400$.

Link to original