Differential Equations - Linear Algebra Tutorial

Basic vector algebra

Vectors in ${ℝ}^n$ have $n$ real number components:

$$𝐯=(v_1,v_2,\dots ,v_n).$$

Such vectors are added componentwise, and scalars multiply every component simultaneously. All the abstract operations and properties of vectors apply to vectors in ${ℝ}^n$:

• Operations: addition and scalar multiplication,
• Properties: commutativity, associativity, distributivity, zero vector.

There are $n$ standard basis vectors:

$${𝐞}_i=(0,\dots ,0,1^{\underbrace{i^{th}}},0,\dots ,0).$$

Unique decomposition of vectors (weighted sum) using the standard basis:

$$𝐯=v_1{𝐞}_1+v_2{𝐞}_2+\cdots +v_n{𝐞}_n,𝐯=\sum _{i=1}^n{v}_i{𝐞}_i.$$

Pairs of vectors in $n$D also have dot products defined by summing component products:

$$\begin{array}{cc}𝐮\cdot 𝐯& =(u_1,\dots ,u_n)\cdot (v_1,\dots ,v_n)\\ & =u_1{v}_1+\cdots +u_n{v}_n\\ & =\sum _{i=1}^n{u}_i{v}_i.\end{array}$$

The norm of an $n$D vector is still $|𝐮|=\sqrt{𝐮\cdot 𝐮}$.

Dot product still has the meaning of “relative alignment between vectors,” and can still be used to determine the angle between vectors using the cosine formula, $𝐮\cdot 𝐯=|𝐮||𝐯|\cos \theta$. However, this angle is considerably less important in $n$D.

Spans

Linear combinations of vectors in $n$D:

$$𝐯=a_1{𝐮}_1+a_2{𝐮}_2+\cdots +a_n{𝐮}_n,a_i\in ℝ.$$

A span is a collection of all vectors which could be obtained as linear combinations of certain given vectors. For example, the collection of all possible $𝐯$ that can be written as above, for any $a_i$, in terms of the ${𝐮}_i$ given at the outset, is called either of:

$$\mathrm{span}\{{𝐮}_1,\dots ,{𝐮}_n\},⟨{𝐮}_1,\dots ,{𝐮}_n⟩.$$

It is still an important fact that a span passes through the origin $\mathrm{𝟎}=(\mathrm{0,0},\dots ,0)$. This is because linear combinations do not include a constant term, so the point $\mathrm{𝟎}$ can always be achieved by setting $a_i=0$ for all $i$.

Matrices: linear actions on vectors

Matrices represent linear transformations (‘actions’) on vectors. An action $A:{ℝ}^3\to {ℝ}^3$, for example, is linear if:

• $A(𝐮+𝐯)=A(𝐮)+A(𝐯)$
• $A(\lambda 𝐮)=\lambda (A(𝐮))$

Recall vector decomposition, $𝐱=x_1{𝐞}_1+x_2{𝐞}_2+x_3{𝐞}_3$. This gives $𝐱$ in terms of weights in the standard basis. Applying a linear transformation to this weight decomposition, we have:

$$A(𝐱)=x_{1}A({𝐞}_1)+x_{2}A({𝐞}_2)+x_{3}A({𝐞}_3).$$

Now let us write ${𝐚}_1=A({𝐞}_1)$ and ${𝐚}_2=A({𝐞}_2)$ and ${𝐚}_3=A({𝐞}_3)$. Therefore $A(𝐱)=x_1{𝐚}_1+x_2{𝐚}_2+x_3{𝐚}_3$. Then build a matrix for $A$ by putting these ‘column’ vectors ${𝐚}_i$ into a row of columns:

$$A=\left(\begin{array}{ccc}{𝐚}_1& {𝐚}_2& {𝐚}_3\end{array}\right)=\left(\begin{array}{ccc}{({𝐚}_1)}_1& {({𝐚}_2)}_1& {({𝐚}_3)}_1\\ {({𝐚}_1)}_2& {({𝐚}_2)}_2& {({𝐚}_3)}_2\\ {({𝐚}_1)}_3& {({𝐚}_2)}_3& {({𝐚}_3)}_3\end{array}\right)=\left(\begin{array}{ccc}{a}_{11}& a_{12}& a_{13}\\ a_{21}& a_{22}& a_{23}\\ a_{31}& a_{32}& a_{33}\end{array}\right).$$

(The last matrix defines the individual entries of $A$ with subscript $ij$ where $i$ is the row and $j$ is the column. Mnemonic: “downwards index goes first because it’s heavier.”) In general, the entry $a_{ij}$ is the $j^{\text{th}}$ row of the $i^{\text{th}}$ column vector of $A$.

This matrix $A$ acts upon the vector $𝐱$ by letting its ${𝐞}_i$ weights $x_i$ become the ${𝐚}_i$ weights for the new vector:

$$A𝐱=\left(\begin{array}{ccc}{𝐚}_1& {𝐚}_2& {𝐚}_3\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=x_1{𝐚}_1+x_1{𝐚}_2+x_1{𝐚}_3=\left(\begin{array}{c}{a}_{11}{x}_1+a_{12}{x}_2+a_{13}{x}_3\\ a_{21}{x}_1+a_{22}{x}_2+a_{23}{x}_3\\ a_{31}{x}_1+a_{32}{x}_2+a_{33}{x}_3\end{array}\right).$$

Determinant

The determinant of a $2\times 2$ matrix is given by the formula:

$$A=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right),\det A=\left|\begin{array}{cc}a& b\\ c& d\end{array}\right|=ad-bc.$$

The determinant of a $3\times 3$ matrix is given by the formula:

$$A=\left(\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right),\det A=\left|\begin{array}{ccc}a& b& c\\ d& e& f\\ g& h& i\end{array}\right|=a\left|\begin{array}{cc}e& f\\ h& i\end{array}\right|-b\left|\begin{array}{cc}d& f\\ g& i\end{array}\right|+c\left|\begin{array}{cc}d& e\\ g& h\end{array}\right|.$$

Higher determinants

You may notice a pattern here that hold true in general and allows calculation of $n\times n$ determinants: traverse any row or column, using these entries as weights (with alternating signs), writing a weighted sum of the smaller determinants of the minor matrices (which are given by deleting the row and column in which the given entry in our traversal process lives). This process therefore reduces an $n\times n$ determinant to a linear combination of $(n-1)\times (n-1)$ determinants, and the reduction can be iterated until the $2\times 2$ case is reached.

Independence

A collection of vectors $\{{𝐯}_1,{𝐯}_2,\dots ,{𝐯}_k\}$ is called independent when the only solution to the equation

$$x_1{𝐯}_1+x_2{𝐯}_2+\cdots +x_k{𝐯}_k=0$$

is given by setting $x_i=0$ for every $i$.

A dependency relation is any equation giving one of the vectors in terms of the others, for example ${𝐯}_2=3{𝐯}_1-4{𝐯}_3$.

A collection of vectors is independent if and only if no vector is in the span of the others, which is if and only if none of them can be written as a linear combination of the others, i.e. there are no dependency relations.

The independence or potential dependency relations among a set of vectors are best studied by putting the vectors into a matrix:

$$A=\left(\begin{array}{cccc}{𝐯}_1& {𝐯}_2& \dots & {𝐯}_k\end{array}\right).$$

The vectors are independent if and only if this matrix has rank $k$. That means: row reduction converts it to a matrix with $k$ pivots, one per column.

When $k=n$, meaning the number of vectors is the same as the number of coefficients in a vector, then $A$ is made of independent vectors if and only if it is invertible, which is if and only if it has nonvanishing determinant, i.e. $\det A\ne 0$.

Matrix algebra: composition as multiplication

Matrices can be multiplied by each other. Express $A=\left(\begin{array}{ccc}{𝐚}_1& {𝐚}_2& {𝐚}_3\end{array}\right)$ as a row of column vectors; then $BA$ is the matrix which is the row of column vectors $\left(\begin{array}{ccc}B{𝐚}_1& B{𝐚}_2& B{𝐚}_3\end{array}\right)$. In other words, $B$ simply acts upon each column of $A$ in parallel.

This rule means that $BA$ acts on a vector $𝐱$ by composition of the actions of $A$ first and then of $B$ second:

$$\begin{array}{cc}(BA)𝐱& =x_{1}B{𝐚}_1+x_{2}B{𝐚}_2+x_{3}B{𝐚}_3\\ & =B(x_1{𝐚}_1)+B(x_2{𝐚}_2)+B(x_3{𝐚}_3)\\ & =B(x_1{𝐚}_1+x_2{𝐚}_2+x_3{𝐚}_3)\\ & =B(A(𝐱)).\end{array}$$

The identity matrix $I_3=\left(\begin{array}{ccc}1& 0& 0\\ 0& 1& 0\\ 0& 0& 1\end{array}\right)$ acts by doing nothing, and therefore it does nothing in matrix multiplication: $I_{3}A=A{I}_3=A$. (For higher $n$ we have $I_{n}A=A{I}_n=A$.)

The inverse matrix, written $A^{-1}$, is a matrix which does the opposite of the action of $A$, undoing the effect of $A$. So $A{A}^{-1}𝐱=A^{-1}A𝐱=𝐱$ for every $𝐱$. In terms of matrix multiplication, $A{A}^{-1}=A^{-1}A=I_n$.

The inverse does not always exist, since some actions cannot be undone! For example the matrix $\left(\begin{array}{cc}1& 0\\ 0& 0\end{array}\right)$ sends $\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)$ to $\left(\begin{array}{c}{x}_1\\ 0\end{array}\right)$, deleting the data of $x_2$. The data cannot be recovered, so this matrix has no inverse.

Eigenvectors

The action of a diagonal matrix is to scale each row by the diagonal entry in that row:

$$\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\\ x_3\end{array}\right)=\left(\begin{array}{c}{a}_1{x}_1\\ a_2{x}_2\\ a_3{x}_3\end{array}\right).$$

This action is very easy to express algebraically in terms of standard basis vectors:

$$A{𝐞}_1=a_1{𝐞}_1,A{𝐞}_2=a_2{𝐞}_2,A{𝐞}_3=a_3{𝐞}_3.$$

In other words, standard basis vectors are sent to scalar multiples of themselves.

Whenever a vector is mapped to a multiple of itself, even when $A$ is not diagonal, the vector is called an eigenvector (“self-vector”), and the scalar is called an eigenvalue (“self-value”):

$$A𝐱=\lambda 𝐱.$$

Meaning of eigenvectors

Eigenvectors are “special” or “natural” vectors associated to the matrix $A$, since they reveal a critical proportionality between the vector entries in $𝐱$ such that $A$ does not disturb this proportionality. If an eigenvector is interpreted as a direction, then the action of $A$ simply rescales vectors in that direction. As an example, every rotation in $3$D can be written as a matrix, and an eigenvector of this matrix would point along the axis of rotation, since rotation does not change the direction of axis vectors. The eigenvalue in this case would be $\lambda =1$.

The eigenvalue equation above is equivalent to the equation $(A-\lambda I_n)𝐱=\mathrm{𝟎}$. Frequently the shorthand notation $A_{\lambda }=(A-\lambda I_n)$ is used, so the last equation becomes $A_{\lambda }𝐱=\mathrm{𝟎}$.

In linear algebra one develops further machinery to analyze equations $A𝐱=𝐛$ for any matrix $A$ and vector $𝐛$. Setting $A=A_{\lambda }$ and $𝐛=\mathrm{𝟎}$, the equation $A𝐱=𝐛$ becomes $A_{\lambda }𝐱=\mathrm{𝟎}$, so the machinery of linear algebra would be applicable here. Nonetheless, we will not learn that machinery for this ODE course. Our goal is only to see how matrix algebra techniques are applicable to solving linear systems.

Finding eigenvalues and eigenvectors

The characteristic polynomial ${\mathrm{ch}}_A(\lambda )$ of a matrix $A$ is defined as the determinant $\det A_{\lambda }$ in which $\lambda$ is considered an indefinite variable.

For example: if $A=\left(\begin{array}{cc}1& -2\\ -4& -1\end{array}\right)$ then $A_{\lambda }=\left(\begin{array}{cc}1-\lambda & -2\\ -4& -1-\lambda \end{array}\right)$ and (using the determinant formula) we have:

$${\mathrm{ch}}_A(\lambda )=\left|\begin{array}{cc}1-\lambda & -2\\ -4& -1-\lambda \end{array}\right|=(1-\lambda )(-1-\lambda )-8={\lambda }^2-9.$$

Key fact for finding eigenvalues

The eigenvalues of a matrix $A$ are the roots of its characteristic polynomial ${\mathrm{ch}}_A(\lambda )$.

In the previous example, therefore, $\lambda =\pm 3$ are the eigenvalues.

Once the eigenvalues are found by solving ${\mathrm{ch}}_A(\lambda )=0$ for $\lambda$, the solutions can be plugged back in for $\lambda$ in the equation $A_{\lambda }𝐱=\mathrm{𝟎}$, and then we can try to find the eigenvectors $𝐱$ by solving this equation. That is normally done using the linear algebra technique of “row reduction,” but in this course you will only solve such equations in the $2\times 2$ case when it is very easy: one of the rows of $A_{\lambda }$ will be a multiple of the other whenever $\lambda$ is an eigenvalue, so the equation $A_{\lambda }𝐱=\mathrm{𝟎}$ reduces to a single relation of proportionality between $x_1$ and $x_2$.

Notice that if $𝐱$ is an eigenvector, then $\alpha 𝐱$ is also an eigenvector for any scalar $\alpha \ne 0$. So really we are studying eigenlines, namely the spans of eigenvectors. The relation of proportionality between $x_1$ and $x_2$ gives the direction of this line.

For $\lambda =+3$ in the previous example, solving $A_{\lambda }𝐱=\mathrm{𝟎}$ would mean solving $\left(\begin{array}{cc}-2& -2\\ -4& -4\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$. Therefore $-x_1-x_2=0$ and thus $x_1=-x_2$, so an eigenvector is given by $\left(\begin{array}{c}1\\ -1\end{array}\right)$ and a line of eigenvectors is given by the span $\alpha \left(\begin{array}{c}1\\ -1\end{array}\right)=\left(\begin{array}{c}\alpha \\ -\alpha \end{array}\right)$ for all $\alpha$. For $\lambda =-3$ we find $\left(\begin{array}{cc}4& -2\\ -4& 2\end{array}\right)\left(\begin{array}{c}{x}_1\\ x_2\end{array}\right)=\left(\begin{array}{c}0\\ 0\end{array}\right)$, and solving this we find $2x_1=x_2$ and therefore $\left(\begin{array}{c}1\\ 2\end{array}\right)$ is an eigenvector, as is everything in the span $\left(\begin{array}{c}\alpha \\ 2\alpha \end{array}\right)$.

Matrix calculus

An important technique for differential equations that is frequently not covered in linear algebra courses is that of differentiating matrices and taking matrix powers such as $e^A$. This is important for systems of differential equations because the general form of a linear homogeneous constant coefficient system:

$${𝐱}^{\prime }(t)=A𝐱(t),𝐱(t)={𝐱}_0$$

has for its general fundamental solution the powerful formula:

$$𝐱(t)=e^{At}.$$

We unpack this formula and some related material now.

A time-varying matrix or vector may be given as an array of functions time:

$$𝐱(t)=\left(\begin{array}{c}{x}_1(t)\\ x_2(t)\\ x_3(t)\end{array}\right),A(t)=\left(\begin{array}{ccc}{a}_{11}(t)& a_{12}(t)& a_{13}(t)\\ a_{21}(t)& a_{22}(t)& a_{23}(t)\\ a_{31}(t)& a_{32}(t)& a_{33}(t)\end{array}\right).$$

Since matrices are added / subtracted / scaled by adding / subtracting / scaling their coefficients in parallel, it is easy to define derivatives (and integrals, for that matter) by coefficients:

$${𝐱}^{\prime }(t)=\left(\begin{array}{c}{x}_1^{\prime }(t)\\ x_2^{\prime }(t)\\ x_3^{\prime }(t)\end{array}\right),A^{\prime }(t)=\left(\begin{array}{ccc}{a}_{11}^{\prime }(t)& a_{12}^{\prime }(t)& a_{13}^{\prime }(t)\\ a_{21}^{\prime }(t)& a_{22}^{\prime }(t)& a_{23}^{\prime }(t)\\ a_{31}^{\prime }(t)& a_{32}^{\prime }(t)& a_{33}^{\prime }(t)\end{array}\right).$$

A ‘matrix product’ rule is a little less trivial, but also true:

$$\frac{d}{dt}(A(t)B(t))=A^{\prime }(t)B(t)+A(t)B^{\prime }(t).$$

This rule follows from the general formula (using a summation) for each single entry in the matrix product.

Now it is possible to apply any power series function to a matrix. This is because matrix multiplication (for the powers), along with scaling and summing, are enough to evaluate the series. For example, recall that

$$e^x=1+x+\frac{x^2}{2!}+\frac{x^3}{3!}+\frac{x^4}{4!}\cdots .$$

It is easy enough to plug a matrix $A$ into this formula, so we use the formula to define $e^A$:

$$e^A=I_n+A+\frac{1}{2!}{A}^2+\frac{1}{3!}{A}^3+\frac{1}{4!}{A}^4+\cdots .$$

(Does such a thing converge? It is possible to show that it does, for any $A$, but we cannot do so here.)

More generally, if $f(x)={\sum }_{i=0}^{\infty }{a}_n{x}^n$ is any power series function, then we can define the evaluation $f(A)$ where the series converges:

$$f(A)=\sum _{i=0}^{\infty }{a}_n{A}^n.$$

Now, notice a significant observation about diagonal matrices: multiplication of diagonal matrices is just multiplication of the respective diagonal entries:

$$\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)\left(\begin{array}{ccc}{b}_1& 0& 0\\ 0& b_2& 0\\ 0& 0& b_3\end{array}\right)=\left(\begin{array}{ccc}{a}_1{b}_1& 0& 0\\ 0& a_2{b}_2& 0\\ 0& 0& a_3{b}_3\end{array}\right).$$

Since the powers in a power series are defined using matrix multiplication, this tells us that powers are easy for diagonal matrices:

$${\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)}^n=\left(\begin{array}{ccc}{a}_1^n& 0& 0\\ 0& a_2^n& 0\\ 0& 0& a_3^n\end{array}\right).$$

This, in turn, implies that power series functions applied to diagonal matrices are simply given by applying the function to the diagonal entries:

$$f(A)=f(\left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right))=\left(\begin{array}{ccc}f(a_1)& 0& 0\\ 0& f(a_2)& 0\\ 0& 0& f(a_3)\end{array}\right)$$

so in particular:

$$e^A=\exp A=\exp \left(\begin{array}{ccc}{a}_1& 0& 0\\ 0& a_2& 0\\ 0& 0& a_3\end{array}\right)=\left(\begin{array}{ccc}{e}^{a_1}& 0& 0\\ 0& e^{a_2}& 0\\ 0& 0& e^{a_3}\end{array}\right).$$

(Sometimes “$\exp x$” or “$\exp A$” is used instead of $e^x$ or $e^A$ when the notation is more convenient.)

Non-diagonal matrix powers

If $A$ is non-diagonal, then computing $e^A$ can be extremely challenging! It is necessary to develop an efficient way to find higher matrix powers. There are some tools for this in linear algebra: basically you change coordinates so that the matrix appears to be diagonal or almost diagonal, and you power it up in the new coordinates, and then change back to the original coordinates.

The matrix exponential has some nice properties that follow from the power series formula:

• $e^{0I_n}=I_n$
• $e^{A+B}=e^A{e}^B$ (When $AB=BA$, but not necessarily when $AB\ne BA$!)
• $e^{At+As}=e^{At}{e}^{As}$ (Always.)
• $(e^A{)}^{-1}=e^{-A}$
• $e^{r{I}_n}=e^r{I}_n$

Finally, consider differentiation. Using the power series definition for $e^{At}$, we discover that:

$$\begin{array}{cc}\frac{d}{dt}\left(e^{At}\right)& =\frac{d}{dt}(I+At+A^2\frac{t^2}{2!}+A^3\frac{t^3}{3!}+\cdots )\\ & =A+A^{2}t+A^3\frac{t^2}{2!}+A^4\frac{t^3}{3!}+\cdots \\ & =A(I+At+A^2\frac{t^2}{2!}+A^3\frac{t^3}{3!}+\cdots )\\ & =A{e}^{At}.\end{array}$$

Therefore:

Matrix exponential solves ODE

The function $X(t)=e^{At}$ solves the matrix ODE:

$$X^{\prime }(t)=AX(t).$$

Given the way matrix multiplication works, this formula means (with $3\times 3$ matrices) that we have $3$ distinct vector solutions to the ODE, namely:

$$\left(\begin{array}{ccc}{𝐱}_1^{\prime }(t)& {𝐱}_2^{\prime }(t)& {𝐱}_3^{\prime }(t)\end{array}\right)=\left(\begin{array}{ccc}A({𝐱}_1(t))& A({𝐱}_2(t))& A({𝐱}_3(t))\end{array}\right).$$

Equating the corresponding columns, this means:

$$\begin{array}{cc}{𝐱}_1^{\prime }(t)& =A{𝐱}_1(t)\\ {𝐱}_2^{\prime }(t)& =A{𝐱}_2(t)\\ {𝐱}_3^{\prime }(t)& =A{𝐱}_3(t).\end{array}$$

Notice that at each given time $t$, the vectors ${𝐱}_1,{𝐱}_2,{𝐱}_3$ are linearly independent! This is because they are the column vectors of the matrix $X=e^{At}$, and this entity is invertible because its inverse is always given by the matrix $e^{-At}$.

In conclusion, by using matrix exponentials we have a remarkably efficient way to generate and describe a set of $n$ independent solutions to the ODE given by ${𝐱}^{\prime }(t)=A𝐱(t)$.