Connexions

You are here: Home » Content » Review of Linear Algebra
Content Actions
Lenses

What is a lens?

Lenses

A lens is a custom view of Connexions content. You can think of it as a fancy kind of list that will let you see Connexions through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to Connexions materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual Connexions member, a community, or a respected organization.

This content is ...
Affiliated with (?)
This content is either by members of the organizations listed or about topics related to the organizations listed. Click each link to see a list of all content affiliated with the organization.

  • This module is included inLens: Rice University OpenCourseWare
    By: OpenCourseWare ConsortiumAs a part of collection:"Statistical Signal Processing"

    Click the "Rice University OCW" link to see all content affiliated with them.

    Rice University OCW
Tags

(?)

These tags come from the endorsement, affiliation, and other lenses that include this content.

Review of Linear Algebra

Module by: Clayton Scott

Vector spaces are the principal object of study in linear algebra. A vector space is always defined with respect to a field of scalars.

Fields

A field is a set FF equipped with two operations, addition and mulitplication, and containing two special members 0 and 1 ( 0≠1 0 1 ), such that for all abc∈F a b c F
    1. a+b∈F a b F
    2. a+b=b+a a b b a
    3. ( a + b ) + c = a + ( b + c ) ( a + b ) + c a + ( b + c )
    4. a+0=a a 0 a
    5. there exists -a a such that a+-a=0 a a 0
    1. ab∈F a b F
    2. ab=ba a b b a
    3. abc=abc a b c a b c
    4. a · 1 =a a · 1 a
    5. there exists a-1 a such that aa-1=1 a a 1
  3. ab+c=ab+ac a b c a b a c
More concisely
  1. FF is an abelian group under addition
  2. FF is an abelian group under multiplication
  3. multiplication distributes over addition

Examples

ℚ, ℝ, ℂ

Vector Spaces

Let FF be a field, and VV a set. We say VV is a vector space over F F if there exist two operations, defined for all a∈F a F , u∈V u V and v∈V v V :
  • vector addition: (uu, vv) → u+v∈V u v V
  • scalar multiplication: (aa,vv) → av∈V a v V
and if there exists an element denoted 0∈V 0 V , such that the following hold for all a∈F a F , b∈F b F , and u∈V u V , v∈V v V , and w∈V w V
    1. u + ( v + w ) = ( u + v ) + w u + ( v + w ) ( u + v ) + w
    2. u+v=v+u u v v u
    3. u+0=u u 0 u
    4. there exists -u u such that u+-u=0 u u 0
    1. au+v=au+av a u v a u a v
    2. a+bu=au+bu a b u a u b u
    3. abu=abu a b u a b u
    4. 1 · u =u 1 · u u
More concisely,
  1. VV is an abelian group under plus
  2. Natural properties of scalar multiplication

Examples

  • ℝN N is a vector space over ℝ
  • ℂN N is a vector space over ℂ
  • ℂN N is a vector space over ℝ
  • ℝN N is not a vector space over ℂ
The elements of VV are called vectors.

Euclidean Space

Throughout this course we will think of a signal as a vector x= x 1 x 2 ⋮ x N = x 1 x 2 … x N T x x 1 x 2 ⋮ x N x 1 x 2 … x N The samples x i x i could be samples from a finite duration, continuous time signal, for example.
A signal will belong to one of two vector spaces:

Real Euclidean space

x∈ℝN x N (over ℝ)

Complex Euclidean space

x∈ℂN x N (over ℂ)

Subspaces

Let VV be a vector space over FF.
A subset S⊆V S V is called a subspace of VV if SS is a vector space over FF in its own right.
Example 1 
V=ℝ2 V 2 , F=ℝ F , S=any line though the origin S any line though the origin .
subspace.png
Figure 1: SS is any line through the origin.
Are there other subspaces?
theorem 1 
S⊆V S V is a subspace if and only if for all a∈F a F and b∈F b F and for all s∈S s S and t∈S t S , as+bt∈S a s b t S

Linear Independence

Let u 1 , … , u k ∈V u 1 , … , u k V .
We say that these vectors are linearly dependent if there exist scalars a 1 , … , a k ∈F a 1 , … , a k F such that
∑i=1k a i ui=0 i 1 k a i u i 0 (1)
and at least one a i ≠0 a i 0 .
If Equation 1 only holds for the case a 1 =…= a k =0 a 1 … a k 0 , we say that the vectors are linearly independent.
Example 2 
11-12-2-230+1-57-2=0 1 1 -1 2 2 -2 3 0 1 -5 7 -2 0 so these vectors are linearly dependent in ℝ3 3 .

Spanning Sets

Consider the subset S=v1v2…vk S v 1 v 2 … v k . Define the span of SS < S > ≡spanS≡{∑i=1k a i vi| a i ∈F} < S > span S i 1 k a i v i a i F
Fact: < S > < S > is a subspace of VV.
Example 3 
V=ℝ3 V 3 , F=ℝ F , S=v1v2 S v 1 v 2 , v1=100 v 1 1 0 0 , v2=010 v 2 0 1 0 ⇒ < S > =xy-plane < S > xy-plane .
span.png
Figure 2: < S > < S > is the xy-plane.

Aside

If SS is infinite, the notions of linear independence and span are easily generalized:
We say SS is linearly independent if, for every finite collection u1 , … , uk ∈S u 1 , … , u k S , (kk arbitrary) we have ∑i=1k a i ui=0⇒∀i: a i =0 i 1 k a i u i 0 i a i 0 The span of SS is < S > ={∑i=1k a i ui| a i ∈F∧ui∈S∧k<∞} < S > i 1 k a i u i a i F u i S k
Note: In both definitions, we only consider finite sums.

Bases

A set B⊆V B V is called a basis for VV over FF if and only if
  1. B B is linearly independent
  2. < B > =V < B > V
Bases are of fundamental importance in signal processing. They allow us to decompose a signal into building blocks (basis vectors) that are often more easily understood.
Example 4 
VV = (real or complex) Euclidean space, ℝN N or ℂN N . B=e1…eN≡standard basis B e 1 … e N standard basis ei=0⋮1⋮0 e i 0 ⋮ 1 ⋮ 0 where the 1 is in the ith i th position.
Example 5 
V=ℂN V N over ℂ. B=u1…uN B u 1 … u N which is the DFT basis. uk=1ⅇ-ⅈ2πkN⋮ⅇ-ⅈ2πkNN-1 u k 1 2 k N ⋮ 2 k N N 1 where ⅈ=-1 -1 .

Key Fact

If BB is a basis for VV, then every v∈V v V can be written uniquely (up to order of terms) in the form v=∑i=1N a i vi v i 1 N a i v i where a i ∈F a i F and vi∈B v i B .

Other Facts

  • If SS is a linearly independent set, then SS can be extended to a basis.
  • If < S > =V < S > V , then SS contains a basis.

Dimension

Let VV be a vector space with basis BB. The dimension of VV, denoted dimV dim V , is the cardinality of BB.
theorem 2 
Every vector space has a basis.
theorem 3 
Every basis for a vector space has the same cardinality.
⇒dimV dim V is well-defined.
If dimV<∞ dim V , we say VV is finite dimensional.

Examples

vector space field of scalars dimension
ℝN N ℝ
ℂN N ℂ
ℂN N ℝ
Every subspace is a vector space, and therefore has its own dimension.
Example 6 
Suppose S=u1…uk⊆V S u 1 … u k V is a linearly independent set. Then dim < S > = dim < S >
    Facts
  • If SS is a subspace of VV, then dimS≤dimV dim S dim V .
  • If dimS=dimV<∞ dim S dim V , then S=V S V .

Direct Sums

Let VV be a vector space, and let S⊆V S V and T⊆V T V be subspaces.
We say VV is the direct sum of SS and TT, written V=S⊕T V ⊕ S T , if and only if for every v∈V v V , there exist unique s∈S s S and t∈T t T such that v=s+t v s t .
If V=S⊕T V ⊕ S T , then TT is called a complement of SS.
Example 7 
V= C ′ = { f : ℝ → ℝ | f is continuous } V C ′ { f : → | f is continuous } S= even funcitons in C ′ S even funcitons in C ′ T= odd funcitons in C ′ T odd funcitons in C ′ ft=12ft+f-t+12ft-f-t f t 1 2 f t f t 1 2 f t f t If f=g+h= g ′ + h ′ f g h g ′ h ′ , g∈S g S and g ′ ∈S g ′ S , h∈T h T and h ′ ∈T h ′ T , then g- g ′ = h ′ -h g g ′ h ′ h is odd and even, which implies g= g ′ g g ′ and h= h ′ h h ′ .

Facts

  1. Every subspace has a complement
  2. V=S⊕T V ⊕ S T if and only if
    1. S⋂T=0 S T 0
    2. < S , T > =V < S , T > V
  3. If V=S⊕T V ⊕ S T , and dimV<∞ dim V , then dimV=dimS+dimT dim V dim S dim T

Proofs

Invoke a basis.

Norms

Let VV be a vector space over FF. A norm is a mapping V→F → V F , denoted by ∥·∥ · , such that forall u∈V u V , v∈V v V , and λ∈F λ F
  1. ∥u∥>0 u 0 if u≠0 u 0
  2. ∥λu∥=|λ|∥u∥ λ u λ u
  3. ∥u+v∥≤∥u∥+∥v∥ u v u v

Examples

Euclidean norms:
x∈ℝN x N : ∥x∥=∑i=1N x i 212 x i 1 N x i 2 1 2 x∈ℂN x N : ∥x∥=∑i=1N| x i |212 x i 1 N x i 2 1 2

Induced Metric

Every norm induces a metric on VV duv≡∥u-v∥ d u v u v which leads to a notion of "distance" between vectors.

Inner products

Let V V be a vector space over FF, F=ℝ F or ℂ. An inner product is a mapping V×V → F V V → F , denoted <·,·> · · , such that
  1. <v,v>≥0 v v 0 , and <v,v>=0⇔v=0 ⇔ v v 0 v 0
  2. <u,v>=<v,u>¯ u v v u
  3. <au+bv,w>=a<u,w>+b<v,w> a u b v w