When we apply a matrix to a vector (i.e. multiply them
together), the vector is transformed. An interesting question
to ask ourselves is whether there are any particular
combinations of such a matrix and vector whose result is a new
vector that is proportional to the original vector. In math
terminology, this question can be posed as follows: if we have a
matrix AA:
ℝn→ℝn
→
ℝ
n
ℝ
n
,
does there exist a vector
x∈ℝn
x
ℝ
n
and a scalar
λ∈C
λ
such that
Ax=λx
A
x
λ
x
?
If so, then the complexity of
Ax
A
x
is reduced. It no longer must be thought of as a matrix
multiplication; instead, applying AA to xx
has the simple effect of linearly scaling xx by some scalar
factor λλ.
In this situation, where
Ax=λx
A
x
λ
x
,
λλ is known as an eigenvalue and xx
is its associated eigenvector. For a certain matrix, each one
of its eigenvectors is associated with a particular (though not
necessarily unique) eigenvalue. The word "eigen" is German and
means "same"; this is appropriate because the vector xx
after the matrix multiplication is the same as the original
vector xx,
except for the scaling factor. The following two examples give actual possible values for the matrices, vectors, and values discussed in general terms above.
(
1-1
-11
)11=011
1-1
-11
1
1
0
1
1
(1)Here,
11
1
1
is the eigenvector and 00
is its associated eigenvalue.
(
21
12
)11=311
21
12
1
1
3
1
1
(2)In this second example,
11
1
1
is again the eigenvector but the eigenvalue is now 33.
Now we'd like to develop a method of finding the eigenvalues and eigenvectors of a matrix. We start with what is basically the defining equation behind this whole idea:
Next, we move the
λx
λ
x
term to the left-hand side and factor:
(A−λI)x=0
A
λ
I
x
0
(4)Here's the important rule to remember: there exists
x≠0
x
0
satisfying the equation if and only if
det(A−λI)=0
A
λ
I
0
.
So, to find the eigenvalues, we need to solve this determinant equation.
Given the matrix AA,
solve for λλ in
det(A−λI)=0
A
λ
I
0
.
det(A−λI)=det(
2−λ1
12−λ
)=λ−22−1=λ2−4λ+3=0
A
λ
I
2
λ
1
1
2
λ
λ
2
2
1
λ
2
4
λ
3
0
(6)After finding the eigenvalues, we need to find the
associated eigenvectors. Looking at the defining equation, we see that the
eigenvector xx
is annihilated by the matrix
A−λI
A
λ
I
.
So to solve for the eigenvectors, we simply find the kernel (nullspace) of
A−λI
A
λ
I
using the two eigenvalues we just calculated. If we did this for the example above, we'd find that the eigenvector associated with
λ=3
λ
3
is
11
11
and the eigenvector associated with
λ=1
λ
1
is
1-1
1-1
.
You may be wondering why eigenvalue decomposition is useful. It seems at first glance that it is only helpful in determining the effect a matrix has on a certain small subset of possible vectors (the eigenvectors). However, the benefits become clear when you think about how many other vectors can be looked at from an eigenvalue perspective by decomposing them into components along the available eigenvectors. For instance, in the above example, let's say we wanted to apply
AA
to the vector
20
20
.
Instead of doing the matrix multiply (admittedly not too difficult in this case), the vector
20
20
could be split into components in the direction of the eigenvalues:
20=11+1-1
2
0
1
1
1
-1
(8)Now, each of these components could be scaled by the appropriate eigenvalue and then added back together to form the net result.
Once we have determined the eigenvalues of a particular matrix, we can start to discuss them in terms of their multiplicity. There are two types of eigenvalue multiplicity: algebraic multiplicity and geometric multiplicity.
- Definition 1: Algebraic Multiplicity
The number of repetitions of a certain eigenvalue. If, for a certain matrix,
λ=334
λ
3
3
4
,
then the algebraic multiplicity of 33 would be 22
(as it appears twice) and the algebraic multiplicity of 44
would be 11
(as it appears once). This type of multiplicity is normally
represented by the Greek letter αα,
where
αλi
α
λi
represents the algebraic multiplicity of
λiλi.
- Definition 2: Geometric Multiplicity
A particular eigenvalue's geometric multiplicity is defined as the dimension of the nullspace of
λI−A
λ
I
A
.
This type of multiplicity is normally represented by the Greek letter
γγ, where
γλi
γ
λi
represents the geometric multiplicity of
λiλi.
Here are some helpful facts about certain special cases of matrices.
A matrix
AA
is full rank if
detA≠0
A
0
.
However, if
λ=0
λ
0
then
det(λI−A)=0
λ
I
A
0
.
This tells us that
detA=0
A
0
.
Therefore, if a matrix has at least one eigenvalue equal to
00, then it cannot have full rank. Specifically, for an
nn-dimensional square matrix:
- When one eigenvalue equals 00,
rankA=n−1
rank
A
n
1
- When multiple eigenvalues equal 00
rankA=n−γ0
rank
A
n
γ
0
.
This property holds even if there are other non-zero eigenvalues
A symmetric matrix is one whose transpose is equal to itself
(
A=AT
A
A
).
These matrices (represented by A
below) have the following properties:
- Its eigenvalues are real.
- Its eigenvectors are orthogonal.
- They are always diagonalizable.
