Theory

Theory 1 - Minimum mean square error

Suppose our problem is to estimate or guess or predict the value of a random variable $X$ in a particular run of our experiment. Assume we have the distribution of $X$. Which value do we choose as our guess?

There is no single best answer to this question. The “best guess” number depends on additional factors in the problem context.

One method is to pick a value where the PMF or PDF of $X$ is maximal. This is a value of highest probability. There may be more than one!

Another method is to pick the expected value $E[X]$. This value may be impossible!

For the normal distribution, or any symmetrical distribution, these are the same value. For most distributions, though, they are not the same.

---

Mean square error (MSE)

Given some estimate $\hat{x}\in ℝ$ for a random variable $X$, the mean square error (MSE) of $\hat{x}$ is:

$$E[(X-\hat{x}{)}^2]$$

The MSE quantifies the typical (square of the) error. Error here means the difference between the true value $X$ and the estimate $\hat{x}$.

Other error estimates are reasonable and useful in niche contexts. For example, $E[|X-\hat{x}|]$ or $\mathrm{Max}|X-\hat{x}|$. They are not frequently used, so we do not consider their theory further.

---

In problem contexts where large errors are more costly than small errors (i.e. many real problems), the most likely value of $X$ (the point with maximal PDF) may fare poorly as an estimate.

It turns out that the expected value $E[X]$ also happens to be the value that minimizes the MSE.

Expected value minimizes MSE

Given a random variable $X$, its expected value $\hat{x}=E[X]$ is the estimate of $X$ with minimal mean square error.

The MSE error for $\hat{x}=E[X]$ is:

$$E[{(X-\hat{x})}^2]=\mathrm{Var}[X]$$

Proof that $E[X]$ minimizes MSE

Expand the MSE error:

$$E[{(X-\hat{x})}^2]\gg \gg E[X^2]-2\hat{x}E[X]+{\hat{x}}^2$$

Now minimize this parabola. Differentiate:

$$\frac{d}{d\hat{x}}E[{(X-\hat{x})}^2]\gg \gg 0-2E[X]+2\hat{x}$$

Find zeros:

$$\begin{array}{c}0-2E[X]+2\hat{x}=0\\ \\ \gg \gg 2\hat{x}=2E[X]\gg \gg \hat{x}=E[x]\end{array}$$

---

When the estimate $\hat{x}$ is made in the absence of information (besides the distribution of $X$), it is called a blind estimate. Therefore, ${\hat{x}}_B=E[X]$ is the blind minimal MSE estimate, and $e_B=\mathrm{Var}[X]$ is the error of this estimate.

In the presence of additional information, for example that event $A$ is known, then the MSE estimate is ${\hat{x}}_A=E[X|A]$ and the error of this estimate is $e_{X|A}=\mathrm{Var}[X|A]$.

The MSE estimate can also be conditioned on another variable, say $Y$:

Minimal MSE of $X$ given $Y$

The minimal MSE estimate of $X$ given another variable $Y$:

$${\hat{x}}_M(y)=E[X|Y=y]$$

The error of this estimate is $\mathrm{Var}[X|Y=y]$, which equals $E[(X-{\hat{x}}_M(y){)}^2|Y=y]$.

Notice that the minimal MSE of $X$ given $Y$ can be used to define a random variable:

$${\hat{X}}_M(Y)=E[X|Y]={\hat{x}}_M(Y)$$

This is a derived variable from $Y$ given by composing $Y$ with the function ${\hat{x}}_M$.

The variable ${\hat{X}}_M(Y)$ provides the minimal MSE estimates of $X$ when experimental outcomes are viewed as providing the information of $Y$ only, and the model is used to derive estimates of $X$ from this information.

Theory 2 - Line of minimal MSE

Linear approximation is very common in applied math.

One could consider the linearization of ${\hat{x}}_M(y)$ (its tangent line at some point) instead of the exact function ${\hat{x}}_M(y)$.

One could instead minimize the MSE over all linear functions of $Y$. (Each of which is an RV.) The line with minimal MSE is called the linear estimator.

The difference here is:

• line of best fit at a single point vs.
• line of best fit over the whole range of $X$ and $Y$—weighted by likelihoods

Linear estimator: Line of minimal MSE

Let $L(y)$ be an arbitrary line $L(y)=ay+b$. Let ${\hat{X}}_L(Y)=L(Y)=aY+b$.

The mean square error (MSE) of this line $L$ is:

$$e_L(a,b)=E\left[\right(X-{\hat{X}}_L(Y){)}^2]$$

The linear estimator of $X$ in terms of $Y$ is the line $L_{\min }$ with minimal MSE, and it is:

$$L_{\min }(y)={\rho }_{X,Y}\frac{{\sigma }_X}{{\sigma }_Y}(y-{\mu }_Y)+{\mu }_X$$

The error value at the (best) linear estimator, $e_{L_{\min }}$, is:

$$e_{L_{\min }}={\sigma }_X^2(1-{\rho }_{X,Y}^2)$$

Theorem: The error variable of the linear estimator, $X-{\hat{X}}_{L_{\min }}(Y)$, is perfectly uncorrelated with $Y$.

Slope and ${\rho }_{X,Y}$

Notice:

$$\frac{{\hat{X}}_{L_{\min }}(Y)-{\mu }_X}{{\sigma }_X}={\rho }_{X,Y}\cdot \left(\frac{Y-{\mu }_Y}{{\sigma }_Y}\right)$$

Thus, for standardized variables $X$ and $Y$, it turns out ${\rho }_{X,Y}$ is the slope of the linear estimator.

In each graph, $E[X]=E[Y]=0$ and $\mathrm{Var}[X]=\mathrm{Var}[Y]=1$.

The line of minimal MSE is the “best fit” line, ${\hat{X}}_{L_{\min }}(Y)={\rho }_{X,Y}Y$.