W07 Notes

Joint distributions

01 Theory

Theory 1

Joint distributions describe the probabilities of events associated with multiple random variables simultaneously.

In this course we consider only two variables at a time, typically called $X$ and $Y$. It is easy to extend this theory to vectors of $n$ random variables.

Joint PMF and joint PDF

Discrete joint PMF:

$$P_{X,Y}(x,y)=P[X=x,Y=y]$$

Continuous joint PDF:

$$f_{X,Y}(x,y)=\text{density at}(x,y)$$

Probabilities of events: Discrete case If $B\subset {ℝ}^2$ is a set of points in the plane, then an event $ℬ$ is formed by the set of all outcomes $s$ mapped by $X$ and $Y$ to points in $B$:

$$ℬ=\{s\in S|(X(s),Y(s))\in B\}$$

The probabilities of such events can be measured using the joint PMF:

$$P[(X,Y)\in B]=P[ℬ]=\sum _{(x,y)\in B}{P}_{X,Y}(x,y)$$

Probabilities of events: Continuous case Let $𝒱=[a,b]\times [c,d]\subset {ℝ}^2$ be the rectangular region defined by $(x,y)\in {ℝ}^2$ such that $a\le x\le b$ and $c\le y\le d$. Then:

$$P[(x,y)\in 𝒱]=P[a\le X\le b,c\le Y\le d]={\int }_c^d{\int }_a^b{f}_{X,Y}(x,y)dxdy$$

For more general regions $𝒱\subset {ℝ}^2$:

$$P[(X,Y)\in 𝒱]={\iint }_{𝒱}{f}_{X,Y}(x,y)dA$$

---

The existence of a variable $Y$ does not change the theory for a variable $X$ considered by itself.

However, it is possible to relate the theory for $X$ to the theory for $(X,Y)$, in various ways.

The simplest relationship is the marginal distribution for $X$, which is merely the distribution of $X$ itself, considered as a single random variable, but in a context where it is derived from the joint distribution for $(X,Y)$.

Marginal PMF, marginal PDF

Marginal distributions are obtained from joint distributions by summing the probabilities over all possibilities of the other variable.

Discrete marginal PMF:

$$\begin{array}{cc}{P}_X(x)& =\sum _y{P}_{X,Y}(x,y)\\ & \\ P_Y(y)& =\sum _x{P}_{X,Y}(x,y)\end{array}$$

Continuous marginal PMF:

$$\begin{array}{cc}{f}_X(x)& ={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dy\\ & \\ f_Y(y)& ={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\end{array}$$

---

Infinitesimal method

Suppose $X$ has density $f_X(x)$ that is continuous at $x_0$. Then for infinitesimal $dx$:

$$P[x_0<X\le x_0+dx]=f_X(x)dx$$

Suppose $X$ and $Y$ have joint density $f_{X,Y}(x,y)$ that is continuous at $(x_0,y_0)$. Then for infinitesimal $dx,dy$:

$$P[x_0<X\le x_0+dx,y_0<Y\le y_0+dy]=f_{X,Y}(x_0,y_0)dxdy$$

Joint densities depend on coordinates

The density $f_{X,Y}(x,y)$ in these integration formulas depends on the way $X$ and $Y$ act as Cartesian coordinates and determine differential areas $dxdy$ as little rectangles.

To find a density $f_{R,\mathrm{\Theta }}(r,\theta )$ in polar coordinates, for example, it is not enough to solve for $x(r,\theta )$ and $y(r,\theta )$ and plug into $f_{X,Y}$!

Instead, we must consider the differential area $dxdy$ vs. $drd\theta$. We find that $dxdy=rdrd\theta$.

As an example, the density of the uniform distribution on the unit disk is $f_{R,\mathrm{\Theta }}={\displaystyle \frac{r}{\pi }}$, which is not constant as a function of $r$ and $\theta$.

Extra - Joint densities may not exist

It is not always possible to form a joint PDF $f_{X,Y}$ from any two continuous RVs $X$ and $Y$.

For example, if $X=Y$, then $(X,Y)$ cannot have a joint PDF, since $P[X=Y]=1$ but the integral over the region $X=Y$ will always be 0. (The area of a line is zero.)

Link to original

02 Illustration

Example - Smaller and bigger rolls

Joint and marginal PMFs - Smaller and bigger roll

Roll two dice, and let $X$ indicate the smaller of the numbers rolled, and let $Y$ indicate the bigger number.

Make a chart showing the PMF. Compute the marginal probabilities, and write them in the margins of the chart.

Solution

Link to original

Exercise - Event probabilities by reading PMF table

Event probabilities by reading PMF table

Here is a joint PMF table:

$$\begin{array}{ccccc}{P}_{Q,G}(q,g)& g=0& g=1& g=2& g=3\\ q=0& 0.06& 0.18& 0.24& 0.12\\ q=1& 0.04& 0.12& 0.16& 0.08\end{array}$$

Using the table, compute the following event probabilities:

(a) $P[Q=0]$ $$ (b) $P[Q=G]$ $$ (c) $P[G>1]$ $$ (d) $P[G>Q]$

Link to original

Exercise - Joint and marginal PMFs - Coin flipping

Joint and marginal PMFs - Coin flipping

Flip a fair coin four times. Let $X$ measure the number of heads in the first two flips, and let $Y$ measure the total number of heads.

Make a chart showing the PMF. Compute the marginal probabilities, and write them in the margins of the chart.

Link to original

Example - Marginal and event probability from joint density

Marginal and event probability from joint density

Suppose the joint density of $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{cc}2x{e}^{x^2-y}& y>x^2,x\in [\mathrm{0,1}]\\ 0& \text{otherwise}\end{array}$$

Find (a) $f_Y(y)$ and (b) $P[Y<3X^2]$.

Solution

(a)

When $y\in [\mathrm{0,1}]$, the $x$ range is $0$ to $\sqrt{y}$:

$$\begin{array}{c}{f}_Y(y)={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\\ \\ \gg \gg {\int }_0^{\sqrt{y}}2x{e}^{x^2-y}dx\gg \gg 1-e^{-y}\text{for }0\le y\le 1\end{array}$$

When $y>1$, the $x$ range is $0$ to $1$:

$$\begin{array}{c}{f}_Y(y)={\int }_{-\infty }^{+\infty }{f}_{X,Y}(x,y)dx\\ \\ \gg \gg {\int }_0^{1}2x{e}^{x^2-y}dx\gg \gg e^{1-y}-e^{-y}\text{for }y>1\end{array}$$

Therefore:

$$f_Y(y)=\{\begin{array}{cc}0& y<0\\ 1-e^{-y}& 0\le y<1\\ (e-1)e^{-y}& 1\le y\end{array}$$

---

(b)

Find probability of the event $Y<3X^2$:

$$\begin{array}{c}P[Y<3X^2]={\int }_0^1{\int }_{x^2}^{3x^2}2x{e}^{x^2-y}dydx\\ \\ \gg \gg {\int }_0^{1}2x{e}^{x^2}(e^{-x^2}-e^{-3x^2})dx\\ \\ \gg \gg \frac{1}{2}(1+e^{-2})\end{array}$$ Link to original

Exercise - Marginals from joint density

Marginals from joint density

The joint PDF for $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{cc}6(x+y^2)/5& 0\le x,y\le 1\\ 0& \text{ otherwise }\end{array}$$

Find $f_X(x)$ and $f_Y(y)$.

Link to original

Exercise - Event probability from joint density

Event probability from joint density

The joint PDF for $X$ and $Y$ is given by:

$$f_{X,Y}(x,y)=\{\begin{array}{cc}2e^{-x}{e}^{-2y}& x,y>0\\ 0& \text{else}\end{array}$$

Compute $P[X<Y]$.

Link to original

03 Theory

Theory 2

Joint CDF

The joint CDF of $X$ and $Y$ is defined by:

$$F_{X,Y}(x,y)=P[X\le x,Y\le y]$$

We can relate the joint CDF to the joint PDF using integration:

$$F_{X,Y}(x,y)={\int }_{-\infty }^y{\int }_{-\infty }^x{f}_{X,Y}(s,t)dsdt$$

Conversely, if $X$ and $Y$ have a continuous joint PDF $f_{X,Y}(x,y)$ that is also differentiable, we can obtain the PDF from the CDF using partial derivatives:

$$f_{X,Y}(x,y)=\frac{{\partial }^2}{\partial x\partial y}{F}_{X,Y}(x,y)$$

---

There is also a marginal CDF that is computed using a limit:

$$F_X(x)=\underset{y\to +\infty }{\lim }{F}_{X,Y}(x,y)$$

This could also be written, somewhat abusing notation, as $F_X(x)=F_{X,Y}(x,+\infty )$.

Link to original

04 Illustration

Exercise - Properties of joint CDFs

Properties of joint CDFs

(a) Show with a drawing that if both $x<x^{\prime }$ and $y<y^{\prime }$, we know:

$$F_{X,Y}(x,y)\le F_{X,Y}(x^{\prime },y^{\prime })$$

(b) Explain why:

• $F_X(x)=F_{X,Y}(x,\infty )$
• $F_Y(y)=F_{X,Y}(\infty ,y)$

(c) Explain why:

• $F_{X,Y}(x,-\infty )=0$
• $F_{X,Y}(-\infty ,y)=0$

Link to original

Independent random variables

05 Theory

Theory 1

Independent random variables

Random variables $X,Y$ are independent when they satisfy the product rule for all valid subsets $B_1,B_2\subset ℝ$:

$$P[X\in B_1,Y\in B_2]=P[X\in B_1]\cdot P[Y\in B_2]$$

Since $\{X\in B_1,Y\in B_2\}=\{X\in B_1\}\cap \{Y\in B_2\}$, this definition is equivalent to independence of all events constructible using the variables $X$ and $Y$.

For discrete random variables, it is enough to check independence for simple events of type $\{X=k\}$ and $\{Y=\ell \}$ for $k$ and $\ell$ any possible values of $X$ and $Y$.

---

The independence criterion for random variables can be cast entirely in terms of their distributions and written using the PMFs or PDFs.

Independence using PMF and PDF

Discrete case:

$$P_{X,Y}(x,y)=P_X(x)\cdot P_Y(y)$$

Continuous case:

$$f_{X,Y}(x,y)=f_X(x)\cdot f_Y(y)$$

Independence via joint CDF

Random variables $X$ and $Y$ are independent when their CDFs obey the product rule:

$$F_{X,Y}(x,y)=F_X(x)\cdot F_Y(y)$$

Link to original

06 Illustration

Example - Meeting in the park

Event probability - Meeting in the park

A man and a woman arrange to meet in the park between 12:00 and 1:00pm. They both arrive at a random time with uniform distribution over that hour, and do not coordinate with each other.

Find the probability that the first person to arrive has to wait longer than 15 minutes for the second person to arrive.

Solution

Let $X$ denote the time the man arrives. Use minutes starting from 12:00, so $X\in (\mathrm{0,60})$. Let $Y$ denote the time the woman arrives, using the same interval.

The probability we seek is:

$$P[X+15<Y]+P[Y+15<X]$$

Because $X$ and $Y$ are symmetrical in probability, these terms have the same value, so we just double the first one for our answer.

Since the arrivals are independent of each other, we have $f_{X,Y}=f_X\cdot f_Y$.

Since each arrival time is uniform over the interval, we have:

$$f_X(x)=\{\begin{array}{cc}1/60& x\in (\mathrm{0,60})\\ 0& \text{otherwise}\end{array}{f}_Y(y)=\{\begin{array}{cc}1/60& y\in (\mathrm{0,60})\\ 0& \text{otherwise}\end{array}$$

Therefore the joint density is $f_{X,Y}={\left(\frac{1}{60}\right)}^2$. Calculate:

$$\begin{array}{ccc}& 2P[X+15<Y]\gg \gg 2{\iint }_{x+15<y}{f}_{X,Y}(x,y)dxdy& \text{<span class="tml-eqn"></span>}\\ & & \text{<span class="tml-eqn"></span>}\\ & \gg \gg 2{\iint }_{x+15<y}{f}_X(x)f_Y(y)dxdy\gg \gg 2{\int }_{15}^{60}{\int }_0^{y-15}{\left(\frac{1}{60}\right)}^{2}dxdy& \text{<span class="tml-eqn"></span>}\\ & & \text{<span class="tml-eqn"></span>}\\ & \gg \gg \frac{2}{(60{)}^2}{\int }_{15}^{60}y-15dy\gg \gg \frac{9}{16}& \text{<span class="tml-eqn"></span>}\end{array}$$ Link to original

Example - Uniform disk: Cartesian vs. polar

Uniform disk: Cartesian vs. polar

Suppose that a point is chosen uniformly at random on the unit disk.

(a) Let $X$ and $Y$ be the Cartesian coordinates of the chosen point. Are $X$ and $Y$ independent?

(b) Let $R$ and $\mathrm{\Theta }$ give the polar coordinates of the chosen point. Are $R$ and $\mathrm{\Theta }$ independent?

Solution

(a)

Write $f_{X,Y}$ for the joint distribution of $X$ and $Y$. We have:

$$f_{X,Y}=\{\begin{array}{cc}1/\pi & x^2+y^2\le 1\\ 0& \text{otherwise}\end{array}$$

Then computing $f_X(x)$, we obtain:

$$\begin{array}{c}{\int }_{-\sqrt{1-x^2}}^{+\sqrt{1-x^2}}\frac{1}{\pi }dy\gg \gg \frac{2}{\pi }\sqrt{1-x^2}\\ \\ \gg \gg f_X(x)=\{\begin{array}{cc}\frac{2}{\pi }\sqrt{1-x^2}& x\in [-\mathrm{1,1}]\\ 0& \text{otherwise}\end{array}\end{array}$$

By similar reasoning, $f_Y(y)=\frac{2}{\pi }\sqrt{1-y^2}$ for $y\in [-\mathrm{1,1}]$.

The product $f_X(x)f_Y(y)$ is not equal to $f_{X,Y}(x,y)$, so $X$ and $Y$ are not independent. Information about the value of $X$ does provide constraints on the possible values of $Y$, so this result makes sense.

---

(b)

To find the marginals $f_R(r)$ and $f_{\mathrm{\Theta }}(\theta )$, the standard method is to integrate the density $f_{R,\mathrm{\Theta }}$ in the opposite variables.

$f_{R,\mathrm{\Theta }}(r,\theta )$ varies!

The probability density $f_{R,\mathrm{\Theta }}(r,\theta )$ is not constant! The area of a differential sector $drd\theta$ depends on $r$.

We can take two approaches to finding the density $f_{R,\mathrm{\Theta }}$:

(i) Area of a differential sector divided by total area:

$${\displaystyle \frac{rdrd\theta }{\pi }\gg \gg \frac{r}{\pi }drd\theta }$$

So the density is $f_{R,\mathrm{\Theta }}={\displaystyle \frac{r}{\pi }}$.

(ii) Via the CDF:

Joint PDF from joint CDF: uniform distribution in polar

The region ‘below’ a given point $(r,\theta )$, in polar coordinates, is a sector with area $\frac{\theta }{2\pi }\cdot \pi r^2$. The factor $\frac{\theta }{2\pi }$ is a percentage of the circle with area $\pi r^2$.

The density is a constant $\frac{1}{\pi }$ across the disk, so the CDF at $(r,\theta )$ is this same area times $\frac{1}{\pi }$. Thus:

$$F_{R,\mathrm{\Theta }}=\frac{\theta r^2}{2\pi }$$

Then in polar coordinates the density is given by taking partial derivatives:

$$f_{R,\mathrm{\Theta }}(r,\theta )=\frac{{\partial }^2}{\partial r\partial \theta }\left(\frac{1}{2\pi }\theta r^2\right)\gg \gg \frac{r}{\pi }$$

Once we have $f_{R,\mathrm{\Theta }}$, integrate to get the marginals:

$$\begin{array}{cc}{f}_R(r)& ={\int }_{\theta =0}^{2\pi }{f}_{R,\mathrm{\Theta }}d\theta \gg \gg {\int }_0^{2\pi }\frac{r}{\pi }d\theta \gg \gg 2r\\ & \\ f_{\mathrm{\Theta }}(\theta )& ={\int }_{r=0}^1{f}_{R,\mathrm{\Theta }}dr\gg \gg {\int }_0^1\frac{r}{\pi }dr\gg \gg \frac{1}{2\pi }\end{array}$$

Check independence:

$$f_{R,\mathrm{\Theta }}=\frac{r}{\pi }\check{=}(2r)\left(\frac{1}{2\pi }\right)=f_R\cdot f_{\mathrm{\Theta }}$$

---

In this problem it is feasible to find the marginals directly, without computing the new density, only using some geometric reasoning.

Extra - Infinitesimal method for the marginals

The probability $P[R\in (r,r+dr)]$ is the area (over $\pi$) of a thickened circle with radius $r$ and thickness $dr$. The circumference of a circle at radius $r$ is $2\pi r$. So the area of the thickened circle is $2\pi rdr$. So the probability is $2rdr$. This tells us that the marginal probability density is $P_R(r)=2r$.

The probability $P[\mathrm{\Theta }\in (\theta ,\theta +d\theta )]$ is the area (over $\pi$) of a thin sector with radius 1 and angle $d\theta$. This area is $\frac{1}{2}{1}^{2}d\theta$. So the probability is $\frac{1}{2\pi }d\theta$. This tells us that the marginal probability density is $P_{\mathrm{\Theta }}(\theta )=\frac{1}{2\pi }$.

These results agree with those above from differentiating the CDF.

Link to original