Theory

Theory 1

Expectation for a function on two variables

Discrete case:

$$E[g(X,Y)]=\sum _{k,\ell }g(k,\ell )P_{X,Y}(k,\ell )\text{(sum over possible values)}$$

Continuous case:

$$E[g(X,Y)]={\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }g(x,y)f_{X,Y}(x,y)dxdy$$

These formulas are not trivial to prove, and we omit the proofs. (Recall the technical nature of the proof we gave for $E[g(X)]$ in the discrete case.)

Expectation sum rule

Suppose $X$ and $Y$ are any two random variables on the same probability model.

Then:

$$E[X+Y]=E[X]+E[Y]$$

We already know that expectation is linear in a single variable: $E[aX+b]=aE[X]+b$.

Therefore this two-variable formula implies:

$$E[aX+bY+c]=aE[X]+bE[Y]+c$$

Expectation product rule: independence

Suppose that $X$ and $Y$ are independent.

Then we have:

$$E[XY]=E[X]E[Y]$$

Extra - Proof: Expectation sum rule, continuous case

Suppose $f_X$ and $f_Y$ give marginal PDFs for $X$ and $Y$, and $f_{X,Y}$ gives their joint PDF.

Then:

$$\begin{array}{cc}E[X+Y]& \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(x+y)f_{X,Y}(x,y)dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }x{f}_{X,Y}dxdy+{\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }y{f}_{X,Y}dxdy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }x\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}dy\right)dx+{\int }_{-\infty }^{+\infty }y\left({\int }_{-\infty }^{+\infty }{f}_{X,Y}dx\right)dy\\ & \\ & \gg \gg {\int }_{-\infty }^{+\infty }x{f}_X(x)dx+{\int }_{-\infty }^{+\infty }y{f}_Y(y)dy\\ & \\ & \gg \gg E[X]+E[Y]\end{array}$$

Observe that this calculation relies on the formula for $E[g(X,Y)]$, specifically with $g(x,y)=x+y$.

Extra - Proof: Expectation product rule

$$\begin{array}{cccc}& E[XY]& \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(xy)f_{X,Y}(x,y)dxdy& \\ & & & \\ & & \gg \gg {\int }_{-\infty }^{+\infty }{\int }_{-\infty }^{+\infty }(xy)f_X(x)f_Y(y)dxdy& \text{(independence)}\\ & & & \\ & & \gg \gg {\int }_{-\infty }^{+\infty }x{f}_X(x)dx{\int }_{-\infty }^{+\infty }y{f}_Y(y)dy& \\ & & & \\ & & \gg \gg E[X]E[Y]& \end{array}$$