Applied Linear Algebra - Packet 07

Packet 07

Vectors III: Orthogonality

Subspaces of ${ℝ}^n$ are extremely important. For example, if $n$ represents the number of pixels in an image, a vector $𝐯\in {ℝ}^n$ might represent an image. The number $n$ is very large, and even computers can take a (relatively) long time to perform operations on vectors in ${ℝ}^n$ such as multiplying $n\times n$ matrices to compose linear transformations.

Frequently a given dataset will belong (or ‘nearly’ belong) to a subspace of ${ℝ}^n$ of much smaller dimension. It is very useful to be able to describe and work with vectors in such a subspace in an efficient way. This typically means finding a good basis. A good basis should be easy to work with, and it should have a useful interpretation.

Orthogonal bases

An orthogonal basis $\{{𝐮}_1,\dots ,{𝐮}_k\}$ is simply a basis in which the vectors are pairwise perpendicular, meaning ${𝐮}_i\cdot {𝐮}_j=0$ for $i\ne j$. It is very easy to tell if a basis is orthogonal: just compute all pairwise dot products. This is a central reason that orthogonal bases are very popular.

If a basis is orthogonal, there is a very fast way to find the components of any given vector in this basis. Suppose $𝐯\in U\subset {ℝ}^n$, meaning $𝐯$ is a vector in the subspace $U$ of ${ℝ}^n$. Suppose that $U$ is $k$-dimensional with orthogonal basis $\{{𝐮}_1,\dots ,{𝐮}_k\}$. The components of $𝐯$ in this basis are by definition the unique coefficients $c_1,\dots ,c_k$ in the linear combination:

$$𝐯=c_1{𝐮}_1+\cdots +c_k{𝐮}_k.$$

Now, take the dot product of both sides with ${𝐮}_i$, for any given $i$, using the fact that ${𝐮}_i\cdot {𝐮}_j=0$ unless $j=i$:

$$\begin{array}{cc}{𝐮}_i\cdot 𝐯& =c_1({𝐮}_i\cdot {𝐮}_1)+\cdots +c_k({𝐮}_i\cdot {𝐮}_k)\\ & =0+\cdots +0+c_i({𝐮}_i\cdot {𝐮}_i)+0+\cdots +0,\end{array}$$

so

$$c_i=\frac{{𝐮}_i\cdot 𝐯}{{𝐮}_i\cdot {𝐮}_i}.$$

Summarizing:

Finding components in an orthogonal basis

Given $\{{𝐮}_1,\dots ,{𝐮}_k\}$ an orthogonal basis, the components of $𝐯=c_1{𝐮}_1+\cdots +c_k{𝐮}_k$ are given by the formulas:

$$c_1=\frac{{𝐮}_1\cdot 𝐯}{{𝐮}_1\cdot {𝐮}_1},c_2=\frac{{𝐮}_2\cdot 𝐯}{{𝐮}_2\cdot {𝐮}_2},\dots c_k=\frac{{𝐮}_k\cdot 𝐯}{{𝐮}_k\cdot {𝐮}_k}.$$

If $\{{𝐮}_1,\dots ,{𝐮}_k\}$ is not only orthogonal but also orthonormal, then the components are even simpler:

$$c_1={𝐮}_1\cdot 𝐯,c_2={𝐮}_2\cdot 𝐯,\dots c_k={𝐮}_k\cdot 𝐯.$$

The fact that components are so easy to calculate for orthogonal bases is one of their most important features, and frequently is the main reason they are used.

Question 07-01

Orthogonal vectors, finding components

Show that the set of vectors is orthogonal:

$${𝐮}_1=\left(\begin{array}{c}1\\ 1\\ 0\end{array}\right),{𝐮}_2=\left(\begin{array}{c}-1\\ 1\\ 2\end{array}\right),{𝐮}_3=\left(\begin{array}{c}2\\ -2\\ 2\end{array}\right),$$

and then compute the components of $\left(\begin{array}{c}1\\ 1\\ 1\end{array}\right)$ in the basis $\{{𝐮}_1,{𝐮}_2,{𝐮}_3\}$.

The above method can be written out succinctly in what is sometimes called the Fourier expansion formula:

$$𝐯=\sum _{i=1}^k\left(\frac{{𝐮}_i\cdot 𝐯}{{𝐮}_i\cdot {𝐮}_i}\right){𝐮}_i.$$

A simple observation shows another advantage of orthogonal sets of vectors – that they are automatically independent:

Orthogonal vectors are automatically independent

Suppose that $\{{𝐮}_1,\dots ,{𝐮}_k\}$ is an orthogonal set.

Assume hypothetically that $x_1{𝐮}_1+\cdots +x_k{𝐮}_k=0$.

By dotting both sides with any ${𝐮}_i$, we see that $x_i({𝐮}_i\cdot {𝐮}_i)=0$, so we must have $x_i=0$ (because ${𝐮}_i\cdot {𝐮}_i\ne 0$ since we assume ${𝐮}_i\ne 0$). This can be done for any $i$, so in fact $x_i=0$ for all $i$.

Therefore $\{{𝐮}_1,\dots ,{𝐮}_k\}$ is an independent set of vectors.

Projection

Projection and orthogonal bases Recall the projection formula:

$${\mathrm{proj}}_{{𝐮}_i}(𝐯)=\left(\frac{𝐯\cdot {𝐮}_i}{{𝐮}_i\cdot {𝐮}_i}\right){𝐮}_i.$$

So $c_i$ is just the scalar coefficient of the projection of $𝐯$ onto ${𝐮}_i$ (sometimes called the ‘scalar projection’).

Using this fact, we could equally write a vector as a decomposition according to an orthogonal basis using a sum of projections to the basis elements:

$$𝐯={\mathrm{proj}}_{{𝐮}_1}(𝐯)+\cdots +{\mathrm{proj}}_{{𝐮}_k}(𝐯).$$

Projection to a subspace The projection formula for ${\mathrm{proj}}_{𝐮}(𝐯)$ provides the closest vector to $𝐯$ that is on the line spanned by $𝐮$, and the difference $𝐯-{\mathrm{proj}}_{𝐮}(𝐯)$ is perpendicular to $𝐮$. Furthermore, ${\mathrm{proj}}_{𝐮}(𝐯)$ does not depend on the length of $𝐮$ at all, in fact it only depends on the subspace spanned by $𝐮$, which is a line. Therefore, ${\mathrm{proj}}_{𝐮}$ could really be called ${\mathrm{proj}}_L$ where $L=⟨𝐮⟩$ is the subspace spanned by $𝐮$.

It is extremely useful to be able to compute the projection of a vector $𝐯$ onto any subspace $U\subset {ℝ}^n$. This result should be the unique vector $𝐰$ inside $U$ which is closest to $𝐯$. It is also the unique vector $𝐰$ such that the displacement $𝐯-𝐰$ is perpendicular to $U$ (meaning: perpendicular to every vector in $U$).

Now suppose that we have an orthogonal basis ${𝐮}_1,\dots ,{𝐮}_k$ of $U$. We can define the projection of $𝐯\in {ℝ}^n$ onto the subspace $U$ in terms of this basis:

Projection of $𝐯$ to a subspace $U\subset {ℝ}^n$

$${\mathrm{proj}}_U(𝐯)={\mathrm{proj}}_{{𝐮}_1}(𝐯)+\cdots +{\mathrm{proj}}_{{𝐮}_k}(𝐯).$$

• Notice that ${\mathrm{proj}}_U(𝐯)$ is a linear combination of basis vectors. (Consider the separate formulas for ${\mathrm{proj}}_{{𝐮}_i}(𝐯)$.) Therefore it lies inside the subspace $U$. We might have ${\mathrm{proj}}_U(𝐯)\ne 𝐯$ though, because here we do not assume that $𝐯\in U$.

• In addition, using this formula we can deduce that the displacement vector $𝐱=𝐯-{\mathrm{proj}}_U(𝐯)$ is perpendicular to every vector in $U$: Consider the dot product against any basis vector ${𝐮}_i$:

$$\begin{array}{cc}{𝐮}_i\cdot 𝐱& ={𝐮}_i\cdot 𝐯-{𝐮}_i\cdot {\mathrm{proj}}_U(𝐯)\\ & ={𝐮}_i\cdot 𝐯-({𝐮}_i\cdot \left(\frac{𝐯\cdot {𝐮}_1}{{𝐮}_1\cdot {𝐮}_1}\right){𝐮}_1+\cdots +{𝐮}_i\cdot \left(\frac{𝐯\cdot {𝐮}_k}{{𝐮}_k\cdot {𝐮}_k}\right){𝐮}_k)\\ & ={𝐮}_i\cdot 𝐯-(\left(\frac{𝐯\cdot {𝐮}_1}{{𝐮}_1\cdot {𝐮}_1}\right)({𝐮}_i\cdot {𝐮}_1)+\cdots +\left(\frac{𝐯\cdot {𝐮}_k}{{𝐮}_k\cdot {𝐮}_k}\right)({𝐮}_i\cdot {𝐮}_k))\\ & ={𝐮}_i\cdot 𝐯-(0+\cdots +0+\left(\frac{𝐯\cdot {𝐮}_i}{{𝐮}_i\cdot {𝐮}_i}\right)({𝐮}_i\cdot {𝐮}_i)+0+\cdots +0)\\ & ={𝐮}_i\cdot 𝐯-\frac{𝐯\cdot {𝐮}_i}{({𝐮}_i\cdot {𝐮}_i)}({𝐮}_i\cdot {𝐮}_i)\\ & ={𝐮}_i\cdot 𝐯-𝐯\cdot {𝐮}_i=0.\end{array}$$

So ${𝐮}_i\cdot 𝐱=0$ for all $i$. Any element of $U$ can be written as a linear combination of the basis vectors ${𝐮}_i$, so $𝐱$ is perpendicular to any vector in $U$.

Question 07-02

Perpendicularity to a subspace

Explain and justify in more detail the last statement: “${𝐮}_i\cdot 𝐱=0$ for all $i$” implies “$𝐮\cdot 𝐱=0$ for all $𝐮\in U$.”

To summarize, orthogonal bases allow us to:

• Easily calculate the components of any vector in the orthogonal basis
• Formulate the projection of any vector onto a subspace using an orthogonal basis of the subspace

Gram-Schmidt process

In this section, we learn a procedure to create an orthogonal basis of a given subspace, starting only with some vectors that are known to generate the subspace as their span. If the original vectors are not independent, we will discover that fact along the way, and possibly wind up with a smaller number of vectors.

Let us illustrate the process with a span of three vectors $V=⟨{𝐯}_1,{𝐯}_2,{𝐯}_3⟩\subset {ℝ}^n$. We do not assume they are independent.

• Write ${𝐰}_1={𝐯}_1$.
• Exchange ${𝐯}_2$ for ${𝐰}_2={𝐯}_2-{\mathrm{proj}}_{{𝐰}_1}({𝐯}_2)$. Since ${𝐰}_2$ is a linear combination of ${𝐰}_1$ and ${𝐯}_2$, and ${𝐯}_2$ is a linear combination of ${𝐰}_1$ and ${𝐰}_2$, we know $⟨{𝐯}_1,{𝐯}_2,{𝐯}_3⟩=⟨{𝐰}_1,{𝐰}_2,{𝐯}_3⟩$. Furthermore, ${𝐰}_1$ and ${𝐰}_2$ are orthogonal.
• Now exchange ${𝐯}_3$ for ${𝐰}_3={𝐯}_3-{\mathrm{proj}}_{V_2}({𝐯}_3)$, where $V_2=⟨{𝐰}_1,{𝐰}_2⟩$. Since ${𝐰}_3$ is a linear combination of ${𝐯}_3$ and items in $V_2$, we know $⟨{𝐯}_1,{𝐯}_2,{𝐯}_3⟩=⟨{𝐰}_1,{𝐰}_2,{𝐰}_3⟩$, and ${𝐰}_3$ is orthogonal to everything in $V_2$.

We conclude with $V=⟨{𝐰}_1,{𝐰}_2,{𝐰}_3⟩$, where the ${𝐰}_i\cdot {𝐰}_j=0$ unless $i=j$. So these vectors are orthogonal.

Question 07-03

Dependence?

What happens in the above Gram-Schmidt process if $\{{𝐯}_1,{𝐯}_2,{𝐯}_3\}$ is a dependent set?

Example

Gram-Schmidt illustration for 3 vectors

Problem: Use the Gram-Schmidt process to find an orthogonal basis for the following span:

$${𝐯}_1=\left(\begin{array}{c}1\\ 1\\ 0\\ 0\end{array}\right),{𝐯}_2=\left(\begin{array}{c}0\\ 1\\ 1\\ 0\end{array}\right),{𝐯}_3=\left(\begin{array}{c}1\\ 0\\ 1\\ 1\end{array}\right).$$

Solution: We first define ${𝐰}_1={𝐯}_1$. Then compute ${\mathrm{proj}}_{{𝐰}_1}({𝐯}_2)$:

$${\mathrm{proj}}_{{𝐰}_1}({𝐯}_2)=\frac{{𝐯}_2\cdot {𝐰}_1}{{𝐰}_1\cdot {𝐰}_1}{𝐰}_1=\frac{1}{2}{𝐰}_1,$$

and therefore ${𝐰}_2={𝐯}_2-\frac{1}{2}{𝐰}_1=\left(\begin{array}{c}-1/2\\ 1/2\\ 1\\ 0\end{array}\right)$. At this point we have $⟨{𝐯}_1,{𝐯}_2,{𝐯}_3⟩=⟨{𝐰}_1,{𝐰}_2,{𝐯}_3⟩$. Since we are just trying to preserve the span, it is convenient to change the definition of ${𝐰}_2$ to $\left(\begin{array}{c}-1\\ 1\\ 2\\ 0\end{array}\right)$ by multiplying it by $2$. This won’t change the span, and we eliminate the fractions.

Finally compute ${\mathrm{proj}}_{V_2}({𝐯}_3)$. Since $V_2=⟨{𝐰}_1,{𝐰}_2⟩$, and this is actually an orthogonal basis for $V_2$, we can use the formula for $\mathrm{proj}$ that splits it into components according to the orthogonal basis:

$$\begin{array}{cc}{\mathrm{proj}}_{V_2}({𝐯}_3)& ={\mathrm{proj}}_{{𝐰}_1}({𝐯}_3)+{\mathrm{proj}}_{{𝐰}_2}({𝐯}_3)\\ & =\frac{{𝐯}_3\cdot {𝐰}_1}{{𝐰}_1\cdot {𝐰}_1}{𝐰}_1+\frac{{𝐯}_3\cdot {𝐰}_2}{{𝐰}_2\cdot {𝐰}_2}{𝐰}_2\\ & =\frac{1}{2}{𝐰}_1+\frac{1}{6}{𝐰}_2=\left(\begin{array}{c}1/3\\ 2/3\\ 1/3\\ 0\end{array}\right).\end{array}$$

Therefore ${𝐰}_3=\left(\begin{array}{c}2/3\\ -2/3\\ 2/3\\ 1\end{array}\right)$, and we can scale this up as with ${𝐰}_2$ and use instead $\left(\begin{array}{c}2\\ -2\\ 2\\ 3\end{array}\right)$. So we have an orthogonal basis:

$$⟨\left(\begin{array}{c}1\\ 1\\ 0\\ 0\end{array}\right),\left(\begin{array}{c}-1\\ 1\\ 2\\ 0\end{array}\right),\left(\begin{array}{c}2\\ -2\\ 2\\ 3\end{array}\right)⟩.$$

Exercise 07-01

Gram-Schmidt practice

Perform Gram-Schmidt to find an orthogonal basis of the subspace:

$$⟨\left(\begin{array}{c}2\\ 0\\ 1\\ 0\end{array}\right),\left(\begin{array}{c}0\\ 1\\ 1\\ 1\end{array}\right),\left(\begin{array}{c}2\\ 2\\ 3\\ 2\end{array}\right)⟩.$$

General Gram-Schmidt process

The Gram-Schmidt process can be continued in the same way for more than three vectors. Here is the general rule, in the abstract, for ${𝐯}_1,\dots ,{𝐯}_k$:

• Set ${𝐰}_1={𝐯}_1$.
• Write $V_j=⟨{𝐯}_1,\dots ,{𝐯}_j⟩$.
• Set ${𝐰}_{j+1}={𝐯}_{j+1}-{\mathrm{proj}}_{V_j}({𝐯}_{j+1})$.
  • First step: use $j=1,j+1=2$.
  • We know that $⟨{𝐰}_1,\dots ,{𝐰}_j⟩=⟨{𝐯}_1,\dots ,{𝐯}_j⟩$ by considering each formula ${𝐯}_{j+1}-{\mathrm{proj}}_{V_j}({𝐯}_{j+1})$.
  • We know that ${𝐰}_i\cdot {𝐰}_j=0$ when $i\ne j$ (orthogonality of the ${𝐰}_i$ we are building).
  • Therefore we compute using: ${\mathrm{proj}}_{V_j}({𝐯}_{j+1})={\mathrm{proj}}_{{𝐰}_1}({𝐯}_{j+1})+\cdots +{\mathrm{proj}}_{{𝐰}_j}({𝐯}_{j+1})$.
• Repeat with the next step ${𝐰}_{j+2}$.
• Produce a new list of ${𝐰}_i$ vectors with $⟨{𝐰}_1,\dots ,{𝐰}_k⟩=⟨{𝐯}_1,\dots ,{𝐯}_k⟩$.
  • If the original vectors ${𝐯}_1,\dots ,{𝐯}_k$ were independent, then we never have ${𝐰}_{j+1}=0$ for any $j$.
  • If there is some redundancy in ${𝐯}_1,\dots ,{𝐯}_k$, then at some point ${\mathrm{proj}}_{V_j}({𝐯}_{j+1})={𝐯}_{j+1}$ and ${𝐰}_{j+1}=0$.
  • We can detect this and throw out any ${𝐰}_j=0$ from the final list to obtain truly independent vectors ${𝐰}_1^{\prime },\dots ,{𝐰}_{\ell }^{\prime }$, none of which is zero.
  • The dimension of the space $⟨{𝐯}_1,\dots ,{𝐯}_k⟩$ is then $\ell$, then number of nonzero items ${𝐰}_j$ we obtained.

Application: distance from point to subspace

Many applications of linear algebra involve the measure of distance from a given vector to a subspace that does not include the vector. Typically the subspace is known as the span of a list of given vectors.

Suppose $𝐯\in {ℝ}^n$ and $U=⟨{𝐮}_1,\dots ,{𝐮}_k⟩\subset {ℝ}^n$ is a subspace, given as the span of the vectors ${𝐮}_1,\dots ,{𝐮}_k$.

Here is a standard solution to the problem of finding the distance from $𝐯$ to $U$:

• First use Gram-Schmidt to produce an orthogonal basis $\{{𝐰}_1,\dots {𝐰}_{\ell }\}$ of $U$. It’s possible that $\ell <k$.
• Now compute the projection ${\mathrm{proj}}_U(𝐯)$.
• The orthogonal part ${𝐯}_{⟂}=𝐯-{\mathrm{proj}}_U(𝐯)$ is a vector from $U$ straight to $𝐯$.
• $|{𝐯}_{⟂}|$ is the distance from $𝐯$ to $U$.

Example

Computing distance of point to subspace

Problem: Find the closest point on the subspace given by the span

$$U=⟨\left(\begin{array}{c}0\\ 2\\ 3\\ -1\end{array}\right),\left(\begin{array}{c}1\\ -2\\ 1\\ -1\end{array}\right),\left(\begin{array}{c}2\\ 0\\ 1\\ 3\end{array}\right)⟩$$

to the vector $𝐱=(-1,-\mathrm{3,2,1})$.

Solution: Label the vectors in the span ${𝐯}_1,{𝐯}_2,{𝐯}_3$, in order. Notice that they are already pairwise orthogonal, so we can skip Gram-Schmidt. Then compute ${\mathrm{proj}}_U(𝐱)$:

$$\begin{array}{cc}{\mathrm{proj}}_U(𝐱)& ={\mathrm{proj}}_{{𝐯}_1}(𝐱)+{\mathrm{proj}}_{{𝐯}_2}(𝐱)+{\mathrm{proj}}_{{𝐯}_3}(𝐱)\\ & =\frac{𝐱\cdot {𝐯}_1}{{𝐯}_1\cdot {𝐯}_1}{𝐯}_1+\frac{𝐱\cdot {𝐯}_2}{{𝐯}_2\cdot {𝐯}_2}{𝐯}_2+\frac{𝐱\cdot {𝐯}_3}{{𝐯}_3\cdot {𝐯}_3}{𝐯}_3\\ & =\frac{-1}{14}{𝐯}_1+\frac{6}{7}{𝐯}_2+\frac{3}{14}{𝐯}_3=\left(\begin{array}{c}9/7\\ -13/7\\ 6/7\\ -1/7\end{array}\right).\end{array}$$

Therefore

$$𝐱-{\mathrm{proj}}_U(𝐱)=\left(\begin{array}{c}-16/7\\ -8/7\\ 8/7\\ 8/7\end{array}\right)$$

and the length of this vector is $\sqrt{\frac{448}{49}}$, which is our answer.

Basic linear regression as projection

Linear regression: projection of dependent data vector to subspace of “best fit lines”

Basic linear regression is essentially an application of the above idea. Later we will focus on the details for a more general version; here is a preview summary:

• Data is given by vectors $𝐱$ and $𝐲$ matching input to output by the order of components. If there are $n$ data pairs $(x,y)$, then $𝐱,𝐲$ will have $n$ components.
• The line of best fit should have the form $y=mx+b$. We seek $m$ and $b$, as well as the correlation coefficient $r^2$ which expresses how close the data $𝐲$ is to the best fit line. It is specifically the sum of the square distances from each component, i.e. each data pair.
• The span $⟨𝐱,\mathrm{𝟏}⟩$ is all vectors of the form $\{m𝐱+b\mathrm{𝟏}|m,b\in ℝ\}$. These vectors encode outputs of the possible fit lines evaluated at the input data $𝐱$.
• The best fit line is $y=mx+b$ where $m𝐱+b\mathrm{𝟏}$ is the closest vector to $𝐲$ in the span $⟨𝐱,\mathrm{𝟏}⟩$.
  • Convert $⟨𝐱,\mathrm{𝟏}⟩$ to the orthogonal basis $⟨\widehat{𝐱},\mathrm{𝟏}⟩$. Here $\widehat{𝐱}=𝐱-\bar{x}\mathrm{𝟏}$ where $\bar{x}$ is the average of the components of $𝐱$.
  • Compute $𝐳={\mathrm{proj}}_{⟨𝐱,\mathrm{𝟏}⟩}(𝐲)={\mathrm{proj}}_{\widehat{𝐱}}(𝐲)+{\mathrm{proj}}_{\mathrm{𝟏}}(𝐲)$.
  • Define $m,b^{\prime }$ as the coefficients ${\mathrm{proj}}_{\widehat{𝐱}}(𝐲)=m\widehat{𝐱}$ and ${\mathrm{proj}}_{\mathrm{𝟏}}(𝐲)=b^{\prime }\mathrm{𝟏}$.
  • Therefore we have $𝐳=m\widehat{𝐱}+b^{\prime }\mathrm{𝟏}$, so then $𝐳=m𝐱+(b^{\prime }-m\bar{x})\mathrm{𝟏}$, and we have found $m$ and $b=b^{\prime }-m\bar{x}$.
• The distance $|𝐲-(m𝐱+b\mathrm{𝟏})|$ determines the correlation $r$. The absolute distance is divided by the relative length of $\widehat{𝐲}=𝐲-\bar{y}\mathrm{𝟏}$ to get a scale-invariant number. The final formula is: $$1-r^2=\frac{|𝐲-(m𝐱+b\mathrm{𝟏}){|}^2}{|𝐲-\bar{y}\mathrm{𝟏}{|}^2}.$$

Exercise 07-02

Computing a best fit line

Following the steps above, compute the best fine line (i.e. compute $m$ and $b$) for the data used in Problem 05-03:

$$(-\mathrm{3,4}),(1,-1),(0,-1),(4,-3),(-\mathrm{2,1}).$$

In Problem 05-03, you already found $\bar{x}$ and $\bar{y}$, and implicitly $\widehat{𝐱}$ and $\widehat{𝐲}$.

Derivation of correlation coefficient formula

Here we verify that

$$\frac{|𝐲-(m𝐱+b\mathrm{𝟏}){|}^2}{|𝐲-\bar{y}\mathrm{𝟏}{|}^2}=1-\frac{{(\widehat{𝐱}\cdot \widehat{𝐲})}^2}{(\widehat{𝐱}\cdot \widehat{𝐱})(\widehat{𝐲}\cdot \widehat{𝐲})}.$$

First, since $|𝐲-\bar{y}\mathrm{𝟏}{|}^2=\widehat{𝐲}\cdot \widehat{𝐲}=|\widehat{𝐲}{|}^2$, it is sufficient (confirm this for yourself!) to show that

$$\begin{array}{cc}|𝐲-(m𝐱+b\mathrm{𝟏}){|}^2& =|\widehat{𝐲}{|}^2-\frac{{(\widehat{𝐱}\cdot \widehat{𝐲})}^2}{|\widehat{𝐱}{|}^2}\\ & =|\widehat{𝐲}{|}^2-|{\mathrm{proj}}_{\widehat{𝐱}}(𝐲){|}^2.\end{array}$$

Since $m$ and $b$ are chosen to give the closest vector to $𝐲$ via projection, we know:

$$\begin{array}{cc}𝐲-(m𝐱+b\mathrm{𝟏})& =𝐲-{\mathrm{proj}}_{⟨𝐱,\mathrm{𝟏}⟩}(𝐲)\\ & =𝐲-{\mathrm{proj}}_{⟨\widehat{𝐱},\mathrm{𝟏}⟩}(𝐲)\\ & =𝐲-\frac{𝐲\cdot \widehat{𝐱}}{\widehat{𝐱}\cdot \widehat{𝐱}}\widehat{𝐱}-\frac{𝐲\cdot \mathrm{𝟏}}{\mathrm{𝟏}\cdot \mathrm{𝟏}}\mathrm{𝟏}\\ & =𝐲-\bar{y}\mathrm{𝟏}-\frac{𝐲\cdot \widehat{𝐱}}{\widehat{𝐱}\cdot \widehat{𝐱}}\widehat{𝐱}\\ & =\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(𝐲)\\ & =\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}),\end{array}$$

where we have used the fact that $\hat{y}=\frac{𝐲\cdot \mathrm{𝟏}}{\mathrm{𝟏}\cdot \mathrm{𝟏}}$, as well as the fact that $𝐲=\widehat{𝐲}+\mathrm{𝟏}$ and $\mathrm{𝟏}$ is perpendicular to $\widehat{𝐱}$ so it can be dropped from the penultimate projection. (Confirm this for yourself!)

Finally, we know that ${\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲})$ and $\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲})$ are perpendicular. (The former lies in the span of $\widehat{𝐱}$ and the latter is perpendicular to this span.) Therefore

$$\begin{array}{cc}|\widehat{𝐲}{|}^2& =|\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲})+{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}){|}^2\\ & =|\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}){|}^2+|{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}){|}^2,\\ |\widehat{𝐲}-{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}){|}^2& =|\widehat{𝐲}{|}^2-|{\mathrm{proj}}_{\widehat{𝐱}}(\widehat{𝐲}){|}^2,\end{array}$$

and by plugging in we see $|𝐲-(m𝐱+b\mathrm{𝟏}){|}^2=|\widehat{𝐲}{|}^2-|{\mathrm{proj}}_{\widehat{𝐱}}(𝐲){|}^2$.

Problems due Tuesday 12 Mar 2024 by 11:59pm

Problem 07-01

Gram-Schmidt calculation

Perform Gram-Schmidt to find an orthogonal basis for the span of the column vectors of the following matrix:

$$\left(\begin{array}{cccc}-10& 13& 7& -11\\ 2& 1& -5& 3\\ -6& 3& 13& -3\\ 16& -16& -2& 5\\ 2& 1& -5& -7\end{array}\right)$$

Problem 07-02

Distance point to plane

How far is the point $𝐱=(\mathrm{1,2,3,4})$ from the span:

$$⟨\left(\begin{array}{c}1\\ -1\\ 0\\ 0\end{array}\right),\left(\begin{array}{c}1\\ 1\\ 1\\ 2\end{array}\right)⟩?$$

Problem 07-03

Decomposition into subspace and its complement

Suppose $V\subset {ℝ}^n$ is a subspace. Prove that every vector $𝐱\in {ℝ}^n$ can be written uniquely as a sum:

$$𝐱=𝐯+𝐰$$

such that $𝐯\in V$ and $𝐰$ is perpendicular to $V$ (i.e. perpendicular to everything in $V$).

(Hint: the standard way to show an item is unique is to posit two such items with the specified property, and show that they are equal to each other.)

Problem 07-04

Best fit line, nonzero averages

Find the best fit line for these five data points, and compute the correlation coefficient:

$$(-5,-5),(-\mathrm{4,3})(-\mathrm{3,1})(-2,-3)(-\mathrm{1,4}).$$