# Connexions

You are here: Home » Content » Orthogonal Projections and the Orthogonality Principle

### Recently Viewed

This feature requires Javascript to be enabled.

# Orthogonal Projections and the Orthogonality Principle

Module by: Richard Baraniuk. E-mail the author

A key problem in DSP:

Given a signal y y find the signal x x from a subspace SS that is the closest to y y.

## Example 1

y y is a noisy signal and all xS x S are smooth.

Finding an x x close to y y will approximate it by a noise-free version.

Geometry of the situation via linear algebra. xS(x= k =0M1 α k b k ) x S x k 0 M 1 α k b k where b k b k are a basis for the subspace S S (We assume it is an ONB).

In general, y y is not S S

The goal is to find xS x S close to y y, which is equivalent to finding α k α k .

We can also measure how close x x is to y y by l 2 l 2 norm error in RN N : yx22= n =0N1ynxn2 2 y x 2 n 0 N 1 y n x n 2 Define e=yx e y x as the error vector and minimize l 2 l 2 strength of e e.

## Geometry

### Exercise 1

How close to choose - to minimize x2 2 x ?

#### Solution

Choose x x so that error e e is ⊥ to x x and ⊥ to the entire subspace S S.

### Orthogonality Principle

e2 2 e is minimum when e,x=0 e x 0 Also have s ,sS:e,s=0 s s S e s 0

#### Note:

Linear algebra lets us convert a distance problem (minimize l 2 l 2 distance between 2 vectors) into an angle problem (minimize the inner product between error and subspace).

We perform an orthogonal projection of y y onto S S to obtain x x closest to y y in S S.

Now apply the OP to find the optimal approximant xS x S to y y.

Recall: yCn y n x(SCN) x S N x= k =0M1 α k b k x k 0 M 1 α k b k where b k b k is an ONB for S S.

## Purpose

The goal is that given y y, we must find α k α k 's.

### Solution

Apply OP: Define e=yx=y k = α k b k e y x y k α k b k Then: 0=e,x=y k' = α k' b k' , k' = α k b k 0 e x y k' α k' b k' k' α k b k by linearity of OP,

0=(y, k' = α k b k )( k' = α k' b k' , k' = α k b k )= k' = α k ¯(y, b k ) k' = α k ¯( k' = α k' b k' , b k )= k' = α k ¯((y, b k )( k' = α k' b k' , b k ))= k' = α k ¯((y, b k ) k = α k' ( b k' , b k ))= k' = α k ¯((y, b k ) α k ) 0 y k' α k b k k' α k' b k' k' α k b k k' α k y b k k' α k k' α k' b k' b k k' α k y b k k' α k' b k' b k k' α k y b k k α k' b k' b k k' α k y b k α k
(1)
There are 2 ways to set to zero:
1. All α k =0 α k 0 .
2. (y, b k ) α k =0 y b k α k 0 for all kk chosen.
The solution to (2) is simple: (y, b k ) α k =0 y b k α k 0 or α k =y, b k α k y b k .

### Summary

The optimal l 2 l 2 approximation x= k = α k b k x k α k b k to y y has α k =y, b k α k y b k when b k b k are an ONB for the subspace.

#### Projections

In Figure 5, e=yx e y x , e,x=0 e x 0 , e,S=0 e S 0 x= k =01 α k b k x k 0 1 α k b k where b k b k is an ONB for SS α k =y, b k α k y b k e2 2 e is minimized.

##### Example 2

R2 2 . Find the closest x x to yR2 y 2 where x x lies on a 45° angled line.

1. Find an ONB for SS, S= a ,aR:x=aa S a a x a a . ONB: b 0 =112 b 0 1 1 2 .
2. Optimal x= k =00 α k b k = α 0 b 0 x k 0 0 α k b k α 0 b 0 α 0 =y, b 0 α 0 y b 0
For example, in Figure 7, y=37 y 3 7 α 0 =12( 11 )37=102 α 0 1 2 1 1 3 7 10 2 x=102112=55 x 10 2 1 1 2 5 5

##### Example 3

Given an ecg signal, yCN y N that is a smooth signal contaminated by noise

we would like to make an estimate y ̂ y ̂ that is smooth.
• Step1: Signal model: y=x+n y x n xS x S , subspace of smooth functions. Find an appropriate subspace. For example, xS x S spanned by the 1st M≪N DCT basis vectors (lowest frequencies). x= k =0M1 α k b k x k 0 M 1 α k b k Noise model: Assume for example, that n n is zero-mean Gaussian noise (i.i.d.).
• Step2: ⊥ project y y onto SS to get estimate y ̂ y ̂ (In statistical DSP, this is called the inear minimum mean squared error estimate - LMMSE est.). y ̂ = k =0N1 β k b k y ̂ k 0 N 1 β k b k β k =y, b k β k y b k

##### Exercise 2
###### 2.a)

Why is y ̂ x y ̂ x ?

###### 2.b)

Is y ̂ y ̂ closer to xx than yy is?

###### 2.c)

Using the DCT basis, could you solve this problem in matlab?

###### 2.d)

How do you choose the subspace and the ONB?

## Content actions

PDF | EPUB (?)

### What is an EPUB file?

EPUB is an electronic book format that can be read on a variety of mobile devices.

For detailed instructions on how to download this content's EPUB to your specific device, click the "(?)" link.

### Add module to:

My Favorites (?)

'My Favorites' is a special kind of lens which you can use to bookmark modules and collections. 'My Favorites' can only be seen by you, and collections saved in 'My Favorites' can remember the last module you were on. You need an account to use 'My Favorites'.

| A lens I own (?)

#### Definition of a lens

##### Lenses

A lens is a custom view of the content in the repository. You can think of it as a fancy kind of list that will let you see content through the eyes of organizations and people you trust.

##### What is in a lens?

Lens makers point to materials (modules and collections), creating a guide that includes their own comments and descriptive tags about the content.

##### Who can create a lens?

Any individual member, a community, or a respected organization.

##### What are tags?

Tags are descriptors added by lens makers to help label content, attaching a vocabulary that is meaningful in the context of the lens.

| External bookmarks