Theory

Theory 1

By composing any function $g:ℝ\to ℝ$ with a random variable $X:S\to ℝ$ we obtain a new random variable $g\circ X$. This one is called a derived random variable.

Notation

The derived random variable $g\circ X$ may be written “$g(X)$”.

Expectation of derived variables

Discrete case:

$$E[g(X)]=\sum _{k}g(k)\cdot P_X(k)$$

(Here the sum is over all possible values $k$ of $X$, i.e. where $P_X(k)\ne 0$.)

Continuous case:

$$E[g(X)]={\int }_{-\infty }^{+\infty }g(x)\cdot f_X(x)dx$$

Notice: when applied to outcome $s\in S$:

• $k$ is the output of $X$
• $g(k)$ is the output of $g\circ X$

The proofs of these formulas are tricky because we must relate the PDF or PMF of $X$ to that of $g(X)$.

Proof - Discrete case - Expectation of derived variable

$$\begin{array}{cc}E[g(X)]& =\sum _{y}y\cdot P_{g(X)}(y)\\ & \\ & =\sum _{y}y\cdot \sum _{k\in g^{-1}(y)}{P}_X(k)\\ & \\ & =\sum _y\sum _{k\in g^{-1}(y)}g(k)\cdot P_X(k)\\ & \\ & =\sum _{k}g(k)\cdot P_X(k)\end{array}$$

---

Linearity of expectation

For constants $a$ and $b$:

$$E[aX+b]=aE[X]+b$$

For any $X$ and $Y$ on the same probability model:

$$E[X+Y]=E[X]+E[Y]$$

Exercise - Linearity of expectation

Using the definition of expectation, verify both linearity formulas for the discrete case.

Be careful!

Usually $E[g(X)]\ne g(E[X])$.

For example, usually $E[X\cdot X]\ne E[X]\cdot E[X]$.

We distribute $E$ over sums but not products (unless the factors are independent).

---

Variance squares the scale factor

For constants $a$ and $b$:

$$\mathrm{Var}[aX+b]=a^2\mathrm{Var}[X]$$

Thus variance ignores the offset and squares the scale factor. It is not linear!

Proof - Variance squares the scale factor

$$\begin{array}{cc}\mathrm{Var}[aX+b]& =E[{(aX+b-E[aX+b])}^2]\\ & \\ & =E[{(aX+b-a{\mu }_X-b)}^2]\\ & \\ & =E[{(aX-a{\mu }_X)}^2]\\ & \\ & =E[a^2{(X-{\mu }_X)}^2]\\ & \\ & =a^{2}E[{(X-{\mu }_X)}^2]\\ & \\ & =a^2\mathrm{Var}[X]\end{array}$$

---

Extra - Moments

The $n^{\text{th}}$ moment of $X$ is defined as the expectation of $X^n$:

Discrete case:

$$E[X^n]=\sum _k{k}^n\cdot p(k)$$

Continuous case:

$$E[X^n]={\int }_{-\infty }^{+\infty }{x}^n\cdot f(x)dx$$

A central moment of $X$ is a moment of the variable $X-E[X]$:

$$E[(X-E[X]{)}^n]$$

The data of all the moments collectively determines the probability distribution. This fact can be very useful! In this way moments give an analogue of a series representation, and are sometimes more useful than the PDF or CDF for encoding the distribution.

Theory 2

Suppose we are given the PDF $f_X(x)$ of $X$, a continuous RV.

What is the PDF $f_{g(X)}$, the derived variable given by composing $X$ with $g:ℝ\to ℝ$?

PDF of derived

The PDF of $g(X)$ is not (usually) equal to $g\circ f_X(x)$.

Relating PDF and CDF

When the CDF of $X$ is differentiable, we have:

$$\begin{array}{cc}{F}_X(x)={\int }_{-\infty }^x{f}_X(t)dt& ⟹f_X(x)=\frac{d}{dx}{F}_X(x)\\ & \\ F_{g(X)}(x)={\int }_{-\infty }^x{f}_{g(X)}(t)dt& ⟹f_{g(X)}(x)=\frac{d}{dx}{F}_{g(X)}(x)\end{array}$$

Therefore, if we know $f_X(x)$, we can find $f_{g(X)}(x)$ using a 3-step process:

(1) Integrate PDF to get CDF:

$$F_X(x)={\int }_{-\infty }^x{f}_X(t)dt$$

---

(2) Find $F_{g(X)}$, the CDF of $g(X)$, by comparing conditions:

When $g$ is monotone increasing, we have equivalent conditions:

$$\begin{array}{cc}g(X)\le x& ⟺X\le g^{-1}(x)\\ & \\ \gg \gg P[g(X)\le x]& =P[X\le g^{-1}(x)]\\ & \\ \gg \gg F_{g(X)}(x)& =F_X(g^{-1}(x))\end{array}$$

---

(3) Differentiate CDF to recover PDF:

$$\begin{array}{c}{f}_{g(X)}(x)=\frac{d}{dx}{F}_{g(X)}(x)\gg \gg \frac{d}{dx}{F}_X(g^{-1}(x))\\ \\ \gg \gg f_X(g^{-1}(x))\cdot \frac{d}{dx}(g^{-1})\end{array}$$

Alternative: Method of differentials

Change variables: The measure for integration is $f_X(x)dx$. Set $Y=X^2$ so $dy=2xdx$ and $dx=\frac{1}{2\sqrt{y}}dy$. Thus $f_X(x)dx=f_X(\sqrt{y})\frac{1}{2\sqrt{y}}dy$. So the measure of integration in terms of $y$ is $f_Y(y)=f_X(\sqrt{y})\frac{1}{2\sqrt{y}}$.

Warning: this assumes the function is one-to-one.