Now we consider inner product spaces with nice convergence
properties that allow us to define countably-infinite orthonormal
bases.
-
A Hilbert space is a
complete inner product space. A
complete space is one where all
Cauchy sequences
converge to some vector within the
space. For sequence
xn
xn
to be Cauchy, the distance between its
elements must eventually become arbitrarily small:
∃
N
ε
:∥x⇀n−x⇀m∥<ε ,
(n≥
N
ε
)∧(m≥
N
ε
)
,
ε>0
ε
ε
0
N
ε
n
m
n
N
ε
m
N
ε
x
n
x
m
ε
For a sequence
xn
xn
to be convergent to x, the distance
between its elements and x⇀x must eventually become
arbitrarily small:
∃
N
ε
:∥x⇀n−x∥<ε⇀ ,
n≥
N
ε
,
ε>0
ε
ε
0
N
ε
n
n
N
ε
x
n
x
ε
Examples are listed below (assuming the usual inner products):
-
V=RN
V
N
-
V=CN
V
N
-
V=
l
2
V
l
2
(i.e., square summable sequences)
-
V=
ℒ
2
V
ℒ
2
(i.e., square integrable functions)
-
We will always deal with separable Hilbert
spaces, which are those that have a countable orthonormal (ON) basis. A countable
orthonormal basis for
VV is a countable orthonormal set
S=x⇀k
S
x
k
such that every vector in
VV can be represented as a linear
combination of elements in SS:
∃
α
k
:y⇀=∑k
α
k
x⇀k ,
y⇀∈V
y
y
V
α
k
y
k
k
α
k
x
k
Due to the orthonormality of SS, the basis coefficients are given
by
α
k
=x⇀k · y⇀
α
k
x
k
y
We can see this via:
x⇀k · y⇀=x⇀k · limit
n
→
∞
∑
i
=0n
α
i
x⇀i=limit
n
→
∞
x⇀k · ∑
i
=0n
α
i
x⇀i=limit
n
→
∞
∑
i
=0n
α
i
x⇀k · x⇀i=
αk
x
k
y
x
k
n
i
0
n
α
i
x
i
n
x
k
i
0
n
α
i
x
i
n
i
0
n
α
i
x
k
x
i
αk
where
δk−i=x⇀k · x⇀i
δ
k
i
x
k
x
i
(where the second equality invokes the continuity of
the inner product). In finite
nn-dimensional spaces
(e.g.,
Rn
n
or
Cn
n
), any nn-element ON set
constitutes an ON basis. In infinite-dimensional spaces, we
have the following equivalences:
-
x⇀0x⇀1x⇀2…
x
0
x
1
x
2
…
is an ON basis
-
If
x⇀i · y⇀=0
x
i
y
0
for all ii, then
y⇀=0
y
0
-
∥y⇀∥2=∑i|x⇀i · y⇀|2 ,
y⇀∈V
y
y
V
y
2
i
i
x
i
y
2
(Parseval's theorem)
- Every
y⇀∈V
y
V
is a limit of a sequence of vectors in
spanx⇀0x⇀1x⇀2…
span
x
0
x
1
x
2
…
-
Rn
n
:
e⇀0…e⇀
N
-
1
e0
…
e
N
-
1
for
e⇀i=(
0…010…0
)T
ei
0
…
0
1
0
…
0
with "1" in the
ith
ith
position
-
Cn
n
: same as
Rn
n
-
l2
l2:
δi
n
i∈Z
δi
n
i
, for
δi
n≔δn−i
≔
δi
n
δ
n
i
(all shifts of the Kronecker sequence)
-
ℒ2
ℒ2: to be constructed using wavelets ...
-
Say SS is a subspace
of Hilbert space VV. The orthogonal complement
of S in V, denoted
S⊥
S⊥, is the subspace defined by the set
x⇀∈V
x⇀⊥S
x
V
⊥
x
S
. When SS is
closed, we can write
V=S⊕S⊥
V
S
S⊥
-
The orthogonal projection of y onto S, where
SS is a closed subspace
of VV, is
y
̂
=∑
i
ix⇀i · y⇀x⇀i
y
̂
i
i
x
i
y
x
i
s.t.
x⇀i
x
i
is an ON basis for SS. Orthogonal projection yields
the best approximation of y⇀y in SS:
y
̂
=argmin
x⇀∈S
∥y⇀−x⇀∥
y
̂
argmin
x
S
y
x
The approximation error
e⇀≔y⇀−
y
̂
≔
e
y
y
̂
obeys the orthogonality principle:
e⇀⊥S
⊥
e
S
We illustrate this concept using
V=R3
V
3
(Figure 1) but stress that
the same geometrical interpretation applies to any Hilbert
space.
A proof of the orthogonality principle is:
e⇀⊥S⇔e⇀ · x⇀i=0
⇔
⊥
e
S
i
e
x
i
0
y⇀−
y
̂
· x⇀i=0
y
y
̂
x
i
0
y⇀ · x⇀i=
y
̂
· x⇀i=∑
j
jx⇀j · y⇀x⇀j · x⇀i=∑
j
jx⇀j · y⇀*x⇀j · x⇀i=∑
j
jy⇀ · x⇀j
δ
i
−
j
=y⇀ · x⇀i
y
x
i
y
̂
x
i
j
j
x
j
y
x
j
x
i
j
j
x
j
y
x
j
x
i
j
j
y
x
j
δ
i
−
j
y
x
i
(1)
