Skip to content Skip to navigation Skip to collection information

Connexions

You are here: Home » Content » A First Course in Electrical and Computer Engineering » Linear Algebra: Inner Product and Euclidean Norm

Navigation

Table of Contents

Lenses

What is a lens?

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

This content is ...

Endorsed by Endorsed (What does "Endorsed by" mean?)

This content has been endorsed by the organizations listed. Click each link for a list of all content endorsed by the organization.
  • IEEE-SPS

    This collection is included inLens: IEEE Signal Processing Society Lens
    By: IEEE Signal Processing Society

    Click the "IEEE-SPS" link to see all content they endorse.

  • College Open Textbooks display tagshide tags

    This collection is included inLens: Community College Open Textbook Collaborative
    By: CC Open Textbook Collaborative

    Comments:

    "Reviewer's Comments: 'I recommend this book as a "required primary textbook." This text attempts to lower the barriers for students that take courses such as Principles of Electrical Engineering, […]"

    Click the "College Open Textbooks" link to see all content they endorse.

    Click the tag icon tag icon to display tags associated with this content.

Affiliated with (What does "Affiliated with" mean?)

This content is either by members of the organizations listed or about topics related to the organizations listed. Click each link to see a list of all content affiliated with the organization.
  • Bookshare

    This collection is included inLens: Bookshare's Lens
    By: Bookshare - A Benetech Initiative

    Comments:

    "Accessible versions of this collection are available at Bookshare. DAISY and BRF provided."

    Click the "Bookshare" link to see all content affiliated with them.

  • NSF Partnership display tagshide tags

    This collection is included inLens: NSF Partnership in Signal Processing
    By: Sidney Burrus

    Click the "NSF Partnership" link to see all content affiliated with them.

    Click the tag icon tag icon to display tags associated with this content.

  • Featured Content display tagshide tags

    This collection is included inLens: Connexions Featured Content
    By: Connexions

    Comments:

    "A First Course in Electrical and Computer Engineering provides readers with a comprehensive, introductory look at the world of electrical engineering. It was originally written by Louis Scharf […]"

    Click the "Featured Content" link to see all content affiliated with them.

    Click the tag icon tag icon to display tags associated with this content.

Also in these lenses

  • UniqU content

    This collection is included inLens: UniqU's lens
    By: UniqU, LLC

    Click the "UniqU content" link to see all content selected in this lens.

  • Evowl

    This collection is included inLens: Rice LMS's Lens
    By: Rice LMS

    Comments:

    "Language: en"

    Click the "Evowl" link to see all content selected in this lens.

  • Lens for Engineering

    This module and collection are included inLens: Lens for Engineering
    By: Sidney Burrus

    Click the "Lens for Engineering" link to see all content selected in this lens.

Recently Viewed

This feature requires Javascript to be enabled.

Tags

(What is a tag?)

These tags come from the endorsement, affiliation, and other lenses that include this content.
 

Linear Algebra: Inner Product and Euclidean Norm

Module by: Louis Scharf. E-mail the author

Note:

This module is part of the collection, A First Course in Electrical and Computer Engineering. The LaTeX source files for this collection were created using an optical character recognition technology, and because of this process there may be more errors than usual. Please contact us if you discover any errors.

The inner product (x,y)(x,y) between vectors x x and y y is a scalar consisting of the following sum of products:

(x,y)=x1y1+x2y2+x3y3+⋯+xnyn.(x,y)=x1y1+x2y2+x3y3+⋯+xnyn.
(1)

This definition seems so arbitrary that we wonder what uses it could possibly have. We will show that the inner product has three main uses:

  1. computing length or “norm”,
  2. finding angles between vectors and checking for “orthogonality”, and
  3. computing the “component of one vector along another” (projection).

Since the inner product is so useful, we need to know what algebraic operations are permitted when we are working with inner products. The following are some properties of the inner product. Given x,y,z∈Rnx,y,z∈Rn and a∈Ra∈R,

  1. (x,y)=(y,x)(x,y)=(y,x);
  2. (ax,y)=a(x,y)=(x,ay)(ax,y)=a(x,y)=(x,ay); and
  3. (x,y+z)=(x,y)+(x,z)(x,y+z)=(x,y)+(x,z).

Exercise 1

Prove the three preceding properties by using the definition of inner product. Is the equation x(y,z)=(x,y)zx(y,z)=(x,y)z also a true property? Prove or give a counterexample.

Euclidean Norm. Sometimes we want to measure the length of a vector, namely, the distance from the origin to the point specified by the vector's coordinates. A vector's length is called the norm of the vector. Recall from Euclidean geometry that the distance between two points is the square root of the sum of the squares of the distances in each dimension. Since we are measuring from the origin, this implies that the norm of the vector x is

∥ x ∥ = x 1 2 + x 2 2 + + x n 2 . ∥ x ∥ = x 1 2 + x 2 2 + + x n 2 .
(2)

Notice the use of the double vertical bars to indicate the norm. An equivalent definition of the norm, and of the norm squared, can be given in terms of the inner product:

∥ x ∥ = ( x , x ) ∥ x ∥ = ( x , x )
(3)

or

∥ x ∥ 2 = ( x , x ) . ∥ x ∥ 2 = ( x , x ) .
(4)

Example 1

The Euclidean norm of the vector

x = 1 3 5 - 2 , x = 1 3 5 - 2 ,
(5)

is ∥x∥=12+32+52+(-2)2=39.∥x∥=12+32+52+(-2)2=39.

An important property of the norm and scalar product is that, for any x∈Rnx∈Rn and a∈Ra∈R,

||ax||=|a|||x||.||ax||=|a|||x||.
(6)

So we can take a scalar multiplier outside of the norm if we take its absolute value.

Exercise 2

Prove Equation 6.

Cauchy-Schwarz Inequality. Inequalities can be useful engineering tools. They can often be used to find the best possible performance of a system, thereby telling you when to quit trying to make improvements (or proving to your boss that it can't be done any better). The most fundamental inequality in linear algebra is the Cauchy-Schwarz inequality. This inequality says that the inner product between two vectors x x and y y is less than or equal (in absolute value) to the norm of x x times the norm of y y, with equality if and only if y=αxy=αx:

|(x,y)|≤||x||||y||.|(x,y)|≤||x||||y||.
(7)

To prove this theorem, we construct a third vector z=λx-yz=λx-y and measure its norm squared:

| | λ x - y | | 2 = ( λ x - y , λ x - y ) = λ 2 | | x | | 2 - 2 λ ( x , y ) + | | y | | 2 ≥ 0 . | | λ x - y | | 2 = ( λ x - y , λ x - y ) = λ 2 | | x | | 2 - 2 λ ( x , y ) + | | y | | 2 ≥ 0 .
(8)

So we have a polynomial in λ λ that is always greater than or equal to 0 (because every norm squared is greater than or equal to 0). Let's assume that x x and y y are given and minimize this norm squared with respect to λ λ. To do this, we take the derivative with respect to λ λ and equate it to 0:

2λ||x||2-2(x,y)=0⇒λ=(x,y)||x||22λ||x||2-2(x,y)=0⇒λ=(x,y)||x||2
(9)

When this solution is substituted into the formula for the norm squared in Equation 8, we obtain

[ ( x , y ) | | x | | 2 ] 2 | | x | | 2 - 2 ( x , y ) | | x | | 2 ( x , y ) + | | y | | 2 ≥ 0 , [ ( x , y ) | | x | | 2 ] 2 | | x | | 2 - 2 ( x , y ) | | x | | 2 ( x , y ) + | | y | | 2 ≥ 0 ,
(10)

which simplifies to

-(x,y)2||x||2+||y||2≥0⇒(x,y)2≤||x||2||y||2.-(x,y)2||x||2+||y||2≥0⇒(x,y)2≤||x||2||y||2.
(11)

The proof of the Cauchy-Schwarz inequality is completed by taking the positive square root on both sides of Equation 11. When y=αxy=αx, then

( x , y ) 2 = ( x , α x ) 2 = [ | α | ( x , x ) ] 2 = ( | α | | | x | | 2 ) 2 = ( | α | 2 | | x | | 2 ) | | x | | 2 = ( α x , α x ) | | x | | 2 = ( y , y ) | | x | | 2 = | | y | | 2 | | x | | 2 , ( x , y ) 2 = ( x , α x ) 2 = [ | α | ( x , x ) ] 2 = ( | α | | | x | | 2 ) 2 = ( | α | 2 | | x | | 2 ) | | x | | 2 = ( α x , α x ) | | x | | 2 = ( y , y ) | | x | | 2 = | | y | | 2 | | x | | 2 ,
(12)

which shows that equality holds in Equation 7 when y y is a scalar multiple of x x.

Exercise 3

Use the Cauchy-Schwarz inequality to prove the triangle inequality, which states

| | x + y | | ≤ | | x | | + | | y | | . | | x + y | | ≤ | | x | | + | | y | | .
(13)

Explain why this is called the triangle inequality.

Collection Navigation

Content actions

Download:

Collection as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Module as:

PDF | EPUB (?)

What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

Downloading to a reading device

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

| More downloads ...

Add:

Collection to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks

Module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

Definition of a lens

Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

Who can create a lens?

Any individual member, a community, or a respected organization.

What are tags? tag icon

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks