Theory

Theory 1

Write ${\mu }_X=E[X]$ and ${\mu }_Y=E[Y]$.

Observe that the random variables $X-{\mu }_X$ and $Y-{\mu }_Y$ are “centered at zero,” meaning that $E[X-{\mu }_X]=0=E[Y-{\mu }_Y]$.

Covariance

Suppose $X$ and $Y$ are any two random variables on a probability model. The covariance of $X$ and $Y$ measures the typical synchronous deviation of $X$ and $Y$ from their respective means.

Then the defining formula for covariance of $X$ and $Y$ is:

$$\mathrm{Cov}[X,Y]=E[(X-{\mu }_X)(Y-{\mu }_Y)]$$

There is also a shorter formula:

$$\mathrm{Cov}[X,Y]=E[XY]-E[X]E[Y]$$

To derive the shorter formula, first expand the product $(X-{\mu }_X)(Y-{\mu }_Y)$ and then apply linearity.

Notice that covariance is always symmetric:

$$\mathrm{Cov}[X,Y]=\mathrm{Cov}[Y,X]$$

The self covariance equals the variance:

$$\mathrm{Cov}[X,X]=\mathrm{Var}[X]$$

The sign of $\mathrm{Cov}[X,Y]$ reveals the correlation type between $X$ and $Y$:

Correlation	Sign
Positively correlated	$\mathrm{Cov}[X,Y]>0$
Negatively correlated	$\mathrm{Cov}[X,Y]<0$
Uncorrelated	$\mathrm{Cov}[X,Y]=0$

Correlation coefficient

Suppose $X$ and $Y$ are any two random variables on a probability model.

Their correlation coefficient is a rescaled version of covariance that measures the synchronicity of deviations:

$$\rho [X,Y]=\frac{\mathrm{Cov}[X,Y]}{{\sigma }_X{\sigma }_Y}=\mathrm{Cov}[\frac{X}{{\sigma }_X},\frac{Y}{{\sigma }_Y}]$$

The rescaling ensures:

$$-1\le {\rho }_{X,Y}\le +1$$

Covariance depends on the separate variances of $X$ and $Y$ as well as their relationship.

Correlation coefficient, because we have divided out ${\sigma }_X{\sigma }_Y$, depends only on their relationship.

Theory 2

Covariance bilinearity

Given any three random variables $X$, $Y$, and $Z$, we have:

$$\begin{array}{cc}\mathrm{Cov}[X+Y,Z]& =\mathrm{Cov}[X,Z]+\mathrm{Cov}[Y,Z]\\ & \\ \mathrm{Cov}[Z,X+Y]& =\mathrm{Cov}[Z,X]+\mathrm{Cov}[Z,Y]\end{array}$$

Covariance and correlation: shift and scale

Covariance scales with each input, and ignores shifts:

$$\mathrm{Cov}[aX+b,Y]=a\mathrm{Cov}[X,Y]=\mathrm{Cov}[X,aY+b]$$

Whereas shift or scale in correlation only affects the sign:

$$\rho [aX+b,Y]=\mathrm{sign}(a)\rho [X,Y]=\rho [X,aY+b]$$

Extra - Proof of covariance bilinearity

$$\begin{array}{cc}\mathrm{Cov}[X+Y,Z]& \gg \gg E[(X+Y-({\mu }_X+{\mu }_Y))(Z-{\mu }_Z)]\\ & \\ & \gg \gg E[(X-{\mu }_X+Y-{\mu }_Y)(Z-{\mu }_Z)]\\ & \\ & \gg \gg E[(X-{\mu }_X)(Z-{\mu }_Z)]+E[(Y-{\mu }_Y)(Z-{\mu }_Z)]\\ & \\ & \gg \gg \mathrm{Cov}[X,Z]+\mathrm{Cov}[Y,Z]\end{array}$$

Extra - Proof of covariance shift and scale rule

$$\begin{array}{cc}\mathrm{Cov}[aX+b,Y]& \gg \gg E[(aX+b)Y]-E[aX+b]E[Y]\\ & \\ & \gg \gg E[aXY+bY]-aE[X]E[Y]-E[b]E[Y]\\ & \\ & \gg \gg aE[XY]+bE[Y]-aE[X]E[Y]-bE[Y]\\ & \\ & \gg \gg a(E[XY]-E[X]E[Y])\end{array}$$

---

Independence implies zero covariance

Suppose that $X$ and $Y$ are any two random variables on a probability model.

If $X$ and $Y$ are independent, then:

$$\mathrm{Cov}[X,Y]=0$$

Proof:

We know both of these:

$$\begin{array}{ccc}& E[XY]=E[X]E[Y]& \text{(independence)}\\ & & \\ & \mathrm{Cov}[X,Y]=E[XY]-{\mu }_X{\mu }_Y& \text{(shorter form)}\end{array}$$

But $E[XY]=E[X]E[Y]={\mu }_X{\mu }_Y$, so those terms cancel and $\mathrm{Cov}[X,Y]=0$.

Sum rule for variance

Suppose that $X$ and $Y$ are any two random variables on a probability space.

Then:

$$\mathrm{Var}[X+Y]=\mathrm{Var}[X]+\mathrm{Var}[Y]+2\mathrm{Cov}[X,Y]$$

When $X$ and $Y$ are independent:

$$\mathrm{Var}[X+Y]=\mathrm{Var}[X]+\mathrm{Var}[Y]$$

Extra - Proof: Sum rule for variance

$$\begin{array}{cc}\mathrm{Var}[X+Y]& \gg \gg E\left[\right(X+Y-({\mu }_X+{\mu }_Y){)}^2]\\ & \\ & \gg \gg E\left[\right((X-{\mu }_X)+(Y-{\mu }_Y){)}^2]\\ & \\ & \gg \gg E[(X-{\mu }_X{)}^2+(Y-{\mu }_Y{)}^2+2(X-{\mu }_X)(Y-{\mu }_Y)]\\ & \\ & \gg \gg \mathrm{Var}[X]+\mathrm{Var}[Y]+2\mathrm{Cov}[X,Y]\end{array}$$

Extra - Proof that $-1\le \rho \le +1$

(1) Create standardizations:

$$\tilde{X}=\frac{X-{\mu }_X}{{\sigma }_X},\tilde{Y}=\frac{Y-{\mu }_Y}{{\sigma }_Y}$$

Now $\tilde{X}$ and $\tilde{Y}$ satisfy:

$$E[\tilde{X}]=0=E[\tilde{Y}]\text{and}\mathrm{Var}[\tilde{X}]=1=\mathrm{Var}[\tilde{Y}]$$

Observe that $\mathrm{Var}[W]\ge 0$ for any $W$. Variance can’t be negative.

---

(2) Apply the variance sum rule.

Apply to $\tilde{X}$ and $\tilde{Y}$:

$$0\le \mathrm{Var}[\tilde{X}+\tilde{Y}]=\mathrm{Var}[\tilde{X}]+\mathrm{Var}[\tilde{Y}]+2\mathrm{Cov}[\tilde{X},\tilde{Y}]$$

Simplify:

$$\begin{array}{c}\gg \gg 1+1+2\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\\ \\ \gg \gg 1+\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\end{array}$$

Notice effect of standardization:

$$\mathrm{Cov}[\tilde{X},\tilde{Y}]=\rho [X,Y]$$

Therefore $\rho [X,Y]\ge -1$.

---

(3) Modify and reapply variance sum rule.

Change to $\tilde{X}-\tilde{Y}$:

$$0\le \mathrm{Var}[\tilde{X}-\tilde{Y}]=\mathrm{Var}[\tilde{X}]+\mathrm{Var}[-\tilde{Y}]+2\mathrm{Cov}[\tilde{X},-\tilde{Y}]$$

Simplify:

$$\begin{array}{c}\gg \gg 1+1-2\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\\ \\ \gg \gg 1-\mathrm{Cov}[\tilde{X},\tilde{Y}]\ge 0\end{array}$$