# Connexions

You are here: Home » Content » Linear Vector Spaces

### Recently Viewed

This feature requires Javascript to be enabled.

# Linear Vector Spaces

Module by: Don Johnson. E-mail the author

Summary: Introduces tools and formulas to use when dealing with Linear Vector Spaces. Topics covered include: linear vector spaces, inner product spaces, norm, Schwarz inequality, and distance between two vectors

One of the more powerful tools in statistical communication theory is the abstract concept of a linear vector space. The key result that concerns us is the representation theorem: a deterministic time function can be uniquely represented by a sequence of numbers. The stochastic version of this theorem states that a process can be represented by a sequence of uncorrelated random variables. These results will allow us to exploit the theory of hypothesis testing to derive the optimum detection strategy.

## Basics

### definition 1

A linear vector space S S is a collection of elements called vectors having the following properties:

1. The vector-addition operation can be defined so that if xyzS x y z S :
• (x+y)S x y S (the space is closed under addition)
• x+y=y+x x y y x (Commutivity)
• x + y +z=x+ y + z x + y z x y + z (Associativity)
• The zero vector exists and is always an element of S S. The zero vector is defined by x+0=x x 0 x .
• For each xS x S , a unique vector x x is also an element of S S so that x+x=0 x x 0 , the zero vector.
2. Associated with the set of vectors is a set of scalars which constitute an algebraic field. A field is a set of elements which obey the well-known laws of associativity and commutivity for both addition and multiplication. If a a, b b are scalars, the elements x x, y y of a linear vector space have the properties that:
• ax a x (multiplication by scalar a a) is defined and axS a x S .
• a b x = a b x a b x a b x .
• If "1" and "0" denotes the multiplicative and additive identity elements respectively of the field of scalars; then 1x=x 1 x x and 0x=0 0 x 0
• a(x+y)=ax+ay a x y a x a y and (a+b)x=ax+bx a b x a x b x .

There are many examples of linear vector spaces. A familiar example is the set of column vectors of length N N. In this case, we define the sum of two vectors to be:

x 1 x 2 x N + y 1 y 2 y N = x 1 + y 1 x 2 + y 2 x N + y N x 1 x 2 x N y 1 y 2 y N x 1 y 1 x 2 y 2 x N y N
(1)
and scalar multiplication to be a x 1 x 2 x N T=a x 1 a x 2 a x N T a x 1 x 2 x N a x 1 a x 2 a x N . All of the properties listed above are satisfied.

A more interesting (and useful) example is the collection of square integrable functions. A square-integrable function xt x t satisfies:

T i T f |xt|2d t < t T i T f x t 2
(2)
One can verify that this collection constitutes a linear vector space. In fact, this space is so important that it has a special name - L 2 T i T f L 2 T i T f (read this as el-two); the arguments denote the range of integration.

### definition 2

Let SS be a linear vector space. A subspace 𝒯𝒯 of SS is a subset of SS which is closed. In other words, if xy𝒯 x y 𝒯 , then xyS x y S and all elements of 𝒯𝒯 are elements of SS, but some elements of SS are not elements of 𝒯𝒯. Furthermore, the linear combination (ax+by)𝒯 a x b y 𝒯 for all scalars aa, bb. A subspace is sometimes referred to as a closed linear manifold.

## Inner Product Spaces

A structure needs to be defined for linear vector spaces so that definitions for the length of a vector and for the distance between any two vectors can be obtained. The notions of length and distance are closely related to the concept of an inner product.

### definition 3

An inner product of two real vectors xyS x y S , is denoted by x,y x y and is a scalar assigned to the vectors x x and y y which satisfies the following properties:

1. x,y=y,x x y y x
2. ax,y=a(x,y) a x y a x y , a a is a scalar
3. x+y,z=x,z+y,z x y z x z y z , z z a vector.
4. x,x>0 x x 0 unless x=0 x 0 . In this case, x,x=0 x x 0 .

As an example, an inner product for the space consisting of column matrices can be defined as x,y=xTy=i=1N x i y i x y x y i 1 N x i y i The reader should verify that this is indeed a valid inner product (i.e., it satisfies all of the properties given above). It should be noted that this definition of an inner product is not unique: there are other inner product definitions which also satisfy all of these properties. For example, another valid inner product is x,y=xTKy x y x K y where KK is an N x N N x N positive-definite matrix. Choices of the matrix KK which are not positive definite do not yield valid inner products (property 4 is not satisfied). The matrix KK is termed the kernel of the inner product. When this matrix is something other than an identity matrix, the inner product is sometimes written as x , y K x , y K to denote explicitly the presence of the kernel in the inner product.

### definition 4

The norm of a vector xS x S is denoted by x x and is defined by:

x=x,x1/2 x x x 12
(3)

Because of the properties of an inner product, the norm of a vector is always greater than zero unless the vector is identically zero. The norm of a vector is related to the notion of the length of a vector. For example, if the vector x x is multiplied by the constant scalar a a, the norm of the vector is also multiplied by a a. ax=ax,ax1/2=ax a x a x a x 12 a x In other words, "longer" vectors ( a>1 a 1 ) have larger norms. A norm can also be defined when the inner product contains a kernel. In this case, the norm is written xK K x for clarity.

### definition 5

An inner product space is a linear vector space in which an inner product can be defined for all elements of the space and a norm is given by Equation 3. Note in particular that every element of an inner product space must satisfy the axioms of a valid inner product.

For the space S S consisting of column matrices, the norm of a vector is given by (consistent with the first choice of an inner product) x=i=1N x i 21/2 x i 1 N x i 2 12 This choice of a norm corresponds to the Cartesian definition of the length of a vector.

One of the fundamental properties of inner product spaces is the Schwarz inequality

|x,y|xy x y x y
(4)
This is one of the most important inequalities we shall encounter. To demonstrate this inequality, consider the norm squared of x+ay x a y . x+ay2=x+ay,x+ay=x2+2a(x,y)+a2y2 x a y 2 x a y x a y x 2 2 a x y a 2 y 2 Let a=x,yy2 a x y y 2 . In this case: x+ay2=x22|x,y|2y2+|x,y|2y4y2=x2|x,y|2y2 x a y 2 x 2 2 x y 2 y 2 x y 2 y 4 y 2 x 2 x y 2 y 2 As the left hand side of this result is non-negative, the right-hand side is lower-bounded by zero. The Schwarz inequality is thus obtained. Note that the equality occurs only when x=(ay) x a y , or equivalently when x=cy x c y , where c c is any constant.

### definition 6

Two vectors are said to be orthogonal if the inner product of the vectors is zero: x,y=0 x y 0 .

Consistent with these results is the concept of the "angle" between two vectors. The cosine of this angle is defined by: cos x , y =x,yxy x , y x y x y Because of the Schwarz inequality, |cos x , y |1 x , y 1 . The angle between the orthogonal vectors is ±π2 ± 2 and the angle between vectors satisfying the Schwarz inequality with equality xy x y is zero (the vectors are parallel to each other).

### definition 7

The distance between two vectors is taken to be the norm of the difference of the vectors. dxy=xy d x y x y

In our example of the normed space of column matrices, the distance between xx and yy would be xy=i=1N x i y i 21/2 x y i 1 N x i y i 2 12 which agrees with the Cartesian notion of distance. Because of the properties of the inner product, this distance measure (or metric) has the following properties:

• dxy=dyx d x y d y x (Distance does not depend on how it is measured.)
• (dxy=0)(x=y) d x y 0 x y (Zero distance means equality)
• dxzdxy+dyz d x z d x y d y z (Triangle inequality)
We use this distance measure to define what we mean by convergence. When we say the sequence of vectors x n x n converges to xx ( x n x x n x ), we mean limit  nn x n x=0 n n x n x 0

## Content actions

PDF | EPUB (?)

### What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

#### Definition of a lens

##### Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

##### What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

##### Who can create a lens?

Any individual member, a community, or a respected organization.

##### What are tags?

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks