Skip to content Skip to navigation

Connexions

You are here: Home » Content » Review of Linear Algebra

Navigation

Recently Viewed

This feature requires Javascript to be enabled.

Review of Linear Algebra

Module by: Clayton Scott. E-mail the author

User rating (How does the rating system work?)
Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

:
(0 ratings)

Note: Your browser may not currently support MathML. See our browser support page for additional details. You can always view the correct math in the PDF version.

Vector spaces are the principal object of study in linear algebra. A vector space is always defined with respect to a field of scalars.

Fields

A field is a set FF equipped with two operations, addition and mulitplication, and containing two special members 0 and 1 ( 0≠1 0 1 ), such that for all abc∈F a b c F

    1. a+b∈F a b F
    2. a+b=b+a a b b a
    3. ( a + b ) + c = a + ( b + c ) ( a + b ) + c a + ( b + c )
    4. a+0=a a 0 a
    5. there exists -a a such that a+-a=0 a a 0
    1. ab∈F a b F
    2. ab=ba a b b a
    3. abc=abc a b c a b c
    4. a · 1 =a a · 1 a
    5. there exists a-1 a such that aa-1=1 a a 1
  3. ab+c=ab+ac a b c a b a c
More concisely
  1. FF is an abelian group under addition
  2. FF is an abelian group under multiplication
  3. multiplication distributes over addition

Examples

ℚ, ℝ, ℂ

Vector Spaces

Let FF be a field, and VV a set. We say VV is a vector space over F F if there exist two operations, defined for all a∈F a F , u∈V u V and v∈V v V :

  • vector addition: (uu, vv) → u+v∈V u v V
  • scalar multiplication: (aa,vv) → av∈V a v V
and if there exists an element denoted 0∈V 0 V , such that the following hold for all a∈F a F , b∈F b F , and u∈V u V , v∈V v V , and w∈V w V
    1. u + ( v + w ) = ( u + v ) + w u + ( v + w ) ( u + v ) + w
    2. u+v=v+u u v v u
    3. u+0=u u 0 u
    4. there exists -u u such that u+-u=0 u u 0
    1. au+v=au+av a u v a u a v
    2. a+bu=au+bu a b u a u b u
    3. abu=abu a b u a b u
    4. 1 · u =u 1 · u u
More concisely,
  1. VV is an abelian group under plus
  2. Natural properties of scalar multiplication

Examples

  • ℝN N is a vector space over ℝ
  • ℂN N is a vector space over ℂ
  • ℂN N is a vector space over ℝ
  • ℝN N is not a vector space over ℂ
The elements of VV are called vectors.

Euclidean Space

Throughout this course we will think of a signal as a vector x= x 1 x 2 ⋮ x N = x 1 x 2 … x N T x x 1 x 2 ⋮ x N x 1 x 2 … x N The samples x i x i could be samples from a finite duration, continuous time signal, for example.

A signal will belong to one of two vector spaces:

Real Euclidean space

x∈ℝN x N (over ℝ)

Complex Euclidean space

x∈ℂN x N (over ℂ)

Subspaces

Let VV be a vector space over FF.

A subset S⊆V S V is called a subspace of VV if SS is a vector space over FF in its own right.

Example 1

V=ℝ2 V 2 , F=ℝ F , S=any line though the origin S any line though the origin .

Figure 1: SS is any line through the origin.
Figure 1 (subspace.png)

Are there other subspaces?

Theorem 1

S⊆V S V is a subspace if and only if for all a∈F a F and b∈F b F and for all s∈S s S and t∈S t S , as+bt∈S a s b t S

Linear Independence

Let u 1 , … , u k ∈V u 1 , … , u k V .

We say that these vectors are linearly dependent if there exist scalars a 1 , … , a k ∈F a 1 , … , a k F such that

∑i=1k a i ui=0 i 1 k a i u i 0 (1)
and at least one a i ≠0 a i 0 .

If Equation 1 only holds for the case a 1 =…= a k =0 a 1 … a k 0 , we say that the vectors are linearly independent.

Example 2

11-12−2-230+1-57-2=0 1 1 -1 2 2 -2 3 0 1 -5 7 -2 0 so these vectors are linearly dependent in ℝ3 3 .

Spanning Sets

Consider the subset S=v1v2…vk S v 1 v 2 … v k . Define the span of SS < S > ≡spanS≡{∑i=1k a i vi| a i ∈F} < S > span S i 1 k a i v i a i F

Fact: < S > < S > is a subspace of VV.

Example 3

V=ℝ3 V 3 , F=ℝ F , S=v1v2 S v 1 v 2 , v1=100 v 1 1 0 0 , v2=010 v 2 0 1 0 ⇒ < S > =xy-plane < S > xy-plane .

Figure 2: < S > < S > is the xy-plane.
Figure 2 (span.png)

Aside

If SS is infinite, the notions of linear independence and span are easily generalized:

We say SS is linearly independent if, for every finite collection u1 , … , uk ∈S u 1 , … , u k S , (kk arbitrary) we have ∑i=1k a i ui=0⇒∀i: a i =0 i 1 k a i u i 0 i a i 0 The span of SS is < S > ={∑i=1k a i ui| a i ∈F∧ui∈S∧k<∞} < S > i 1 k a i u i a i F u i S k

Note:

In both definitions, we only consider finite sums.

Bases

A set B⊆V B V is called a basis for VV over FF if and only if

  1. B B is linearly independent
  2. < B > =V < B > V
Bases are of fundamental importance in signal processing. They allow us to decompose a signal into building blocks (basis vectors) that are often more easily understood.

Example 4

VV = (real or complex) Euclidean space, ℝN N or ℂN N . B=e1…eN≡standard basis B e 1 … e N standard basis ei=0⋮1⋮0 e i 0 ⋮ 1 ⋮ 0 where the 1 is in the ith i th position.

Example 5

V=ℂN V N over ℂ. B=u1…uN B u 1 … u N which is the DFT basis. uk=1ⅇ-ⅈ2πkN⋮ⅇ-ⅈ2πkNN−1 u k 1 2 k N ⋮ 2 k N N 1 where ⅈ=-1 -1 .

Key Fact

If BB is a basis for VV, then every v∈V v V can be written uniquely (up to order of terms) in the form v=∑i=1N a i vi v i 1 N a i v i where a i ∈F a i F and vi∈B v i B .

Other Facts

  • If SS is a linearly independent set, then SS can be extended to a basis.
  • If < S > =V < S > V , then SS contains a basis.

Dimension

Let VV be a vector space with basis BB. The dimension of VV, denoted dimV dim V , is the cardinality of BB.

Theorem 2

Every vector space has a basis.

Theorem 3

Every basis for a vector space has the same cardinality.

⇒dimV dim V is well-defined.

If dimV<∞ dim V , we say VV is finite dimensional.

Examples

Table 1
vector space field of scalars dimension
ℝN N ℝ
ℂN N ℂ
ℂN N ℝ

Every subspace is a vector space, and therefore has its own dimension.

Example 6

Suppose S=u1…uk⊆V S u 1 … u k V is a linearly independent set. Then dim < S > = dim < S >

Facts

  • If SS is a subspace of VV, then dimS≤dimV dim S dim V .
  • If dimS=dimV<∞ dim S dim V , then S=V S V .

Direct Sums

Let VV be a vector space, and let S⊆V S V and T⊆V T V be subspaces.

We say VV is the direct sum of SS and TT, written V=S⊕T V ⊕ S T , if and only if for every v∈V v V , there exist unique s∈S s S and t∈T t T such that v=s+t v s t .

If V=S⊕T V ⊕ S T , then TT is called a complement of SS.

Example 7

V= C ′ = { f : ℝ → ℝ | f is continuous } V C ′ { f : → | f is continuous } S= even funcitons in C ′ S even funcitons in C ′ T= odd funcitons in C ′ T odd funcitons in C ′ ft=12ft+f-t+12ft−f-t f t 1 2 f t f t 1 2 f t f t If f=g+h= g ′ + h ′ f g h g ′ h ′ , g∈S g S and g ′ ∈S g ′ S , h∈T h T and h ′ ∈T h ′ T , then g− g ′ = h ′ −h g g ′ h ′ h is odd and even, which implies g= g ′ g g ′ and h= h ′ h h ′ .

Facts

  1. Every subspace has a complement
  2. V=S⊕T V ⊕ S T if and only if
    1. S⋂T=0 S T 0
    2. < S , T > =V < S , T > V
  3. If V=S⊕T V ⊕ S T , and dimV<∞ dim V , then dimV=dimS+dimT dim V dim S dim T

Proofs

Invoke a basis.

Norms

Let VV be a vector space over FF. A norm is a mapping V→F → V F , denoted by ∥·∥ · , such that forall u∈V u V , v∈V v V , and λ∈F λ F

  1. ∥u∥>0 u 0 if u≠0 u 0
  2. ∥λu∥=|λ|∥u∥ λ u λ u
  3. ∥u+v∥≤∥u∥+∥v∥ u v u v

Examples

Euclidean norms:

x∈ℝN x N : ∥x∥=∑i=1N x i 212 x i 1 N x i 2 1 2 x∈ℂN x N : ∥x∥=∑i=1N| x i |212 x i 1 N x i 2 1 2

Induced Metric

Every norm induces a metric on VV duv≡∥u−v∥ d u v u v which leads to a notion of "distance" between vectors.

Inner products

Let V V be a vector space over FF, F=ℝ F or ℂ. An inner product is a mapping V×V → F V V → F , denoted <·,·> · · , such that

  1. <v,v>≥0 v v 0 , and <v,v>=0⇔v=0 ⇔ v v 0 v 0
  2. <u,v>=<v,u>¯ u v v u
  3. <au+bv,w>=a<u,w>+b<v,w> a u b v w a u w b v w

Examples

ℝN N over ℝ: <x,y>=xTy=∑i=1N x i y i x y x y i 1 N x i y i

ℂN N over ℂ: <x,y>=xHy=∑i=1N x i ¯ y i x y x y i 1 N x i y i

If x= x 1 … x N T∈ℂ x x 1 … x N , then xH≡ x 1 ¯⋮ x N ¯T x x 1 ⋮ x N is called the "Hermitian," or "conjugate transpose" of xx.

Triangle Inequality

If we define ∥u∥=<u,u> u u u , then ∥u+v∥≤∥u∥+∥v∥ u v u v Hence, every inner product induces a norm.

Cauchy-Schwarz Inequality

For all u∈V u V , v∈V v V , |<u,v>|≤∥u∥∥v∥ u v u v In inner product spaces, we have a notion of the angle between two vectors: ∠uv=arccos<u,v>∥u∥∥v∥∈02π ∠ u v u v u v 0 2

Orthogonality

uu and vv are orthogonal if <u,v>=0 u v 0 Notation: u⊥v ⊥ u v .

If in addition ∥u∥=∥v∥=1 u v 1 , we say uu and vv are orthonormal.

In an orthogonal (orthonormal) set, each pair of vectors is orthogonal (orthonormal).

Figure 3: Orthogonal vectors in ℝ2 2 .
Figure 3 (orthogonal.png)

Orthonormal Bases

An Orthonormal basis is a basis vi v i such that <vi,vi>= δ i ​ j =1ifi=j0ifi≠j v i v i δ i ​ j 1 i j 0 i j

Example 8

The standard basis for ℝN N or ℂN N

Example 9

The normalized DFT basis uk=1N1ⅇ-ⅈ2πkN⋮ⅇ-ⅈ2πkNN−1 u k 1 N 1 2 k N ⋮ 2 k N N 1

Expansion Coefficients

If the representation of vv with respect to vi v i is v=∑ a i vi v i a i v i then a i =<vi,v> a i v i v

Gram-Schmidt

Every inner product space has an orthonormal basis. Any (countable) basis can be made orthogonal by the Gram-Schmidt orthogonalization process.

Orthogonal Compliments

Let S⊆V S V be a subspace. The orthogonal compliment SS is S ⊥ ={u|u∈V∧<u,v>=0∧∀v:v∈S} S ⊥ u u V u v 0 v v S S ⊥ S ⊥ is easily seen to be a subspace.

If dimv<∞ dim v , then V=S⊕ S ⊥ V ⊕ S S ⊥ .

Aside:

If dimv=∞ dim v , then in order to have V=S⊕ S ⊥ V ⊕ S S ⊥ we require VV to be a Hilbert Space.

Linear Transformations

Loosely speaking, a linear transformation is a mapping from one vector space to another that preserves vector space operations.

More precisely, let VV, WW be vector spaces over the same field FF. A linear transformation is a mapping T : V → W T : V → W such that Tau+bv=aTu+bTv T a u b v a T u b T v for all a∈F a F , b∈F b F and u∈V u V , v∈V v V .

In this class we will be concerned with linear transformations between (real or complex) Euclidean spaces, or subspaces thereof.

Image

imageT={w|w∈W∧ Tv=w for some v } T w w W T v w for some v

Nullspace

Also known as the kernel: kerT={v|v∈V∧Tv=0} ker T v v V T v 0

Both the image and the nullspace are easily seen to be subspaces.

Rank

rankT=dimimageT rank T dim T

Nullity

nullT=dimkerT null T dim ker T

Rank plus nullity theorem

rankT+nullT=dimV rank T null T dim V

Matrices

Every linear transformation TT has a matrix representation. If T : 𝔼 N → 𝔼 M T : 𝔼 N → 𝔼 M , 𝔼=ℝ 𝔼 or ℂ, then TT is represented by an M×N M N matrix A= a 1 ​ 1 … a 1 ​ N ⋮⋱⋮ a M ​ 1 … a M ​ N A a 1 ​ 1 … a 1 ​ N ⋮ ⋱ ⋮ a M ​ 1 … a M ​ N where a 1 ​ i … a M ​ i T=Tei a 1 ​ i … a M ​ i T e i and ei=0…1…0T e i 0 … 1 … 0 is the ith i th standard basis vector.

Aside:

A linear transformation can be represented with respect to any bases of 𝔼N 𝔼 N and 𝔼M 𝔼 M , leading to a different AA. We will always represent a linear transformation using the standard bases.

Column span

colspanA= < A > =imageA colspan A < A > A

Duality

If A : ℝ N → ℝ M A : N → M , then ker⊥A=imageAT ker A ⊥ A

Figure 4
Figure 4 (dual.png)

If A : ℂ N → ℂ M A : N → M , then ker⊥A=imageAH ker A ⊥ A

Inverses

The linear transformation/matrix AA is invertible if and only if there exists a matrix BB such that AB=BA=I A B B A I (identity).

Only square matrices can be invertible.

Theorem 4

Let A : 𝔽 N → 𝔽 N A : 𝔽 N → 𝔽 N be linear, 𝔽=ℝ 𝔽 or ℂ. The following are equivalent:

  1. AA is invertible (nonsingular)
  2. rankA=N rank A N
  3. nullA=0 null A 0
  4. detA≠0 A 0
  5. The columns of AA form a basis.

If A-1=AT A A (or AH A in the complex case), we say AA is orthogonal (or unitary).

Content actions

Give Feedback:

E-mail the module author | Rate module ( How does the rating system work?)

Rating system

Ratings

Ratings allow you to judge the quality of modules. If other users have ranked the module then its average rating is displayed below. Ratings are calculated on a scale from one star (Poor) to five stars (Excellent).

How to rate a module

Hover over the star that corresponds to the rating you wish to assign. Click on the star to add your rating. Your rating should be based on the quality of the content. You must have an account and be logged in to rate content.

(0 ratings)

Download:

Module PDF

Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections directly in Connexions. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need a Connexions account to use 'My Favorites'.

| A lens (?)

Definition of a lens

Lenses

A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual Connexions member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks