Suppose our inner product space V=RMV=RM or CMCM with the standard inner
product (which induces the ℓ2ℓ2-norm).
Re-examining what we have just derived, we can write our approximation
x^=Px=Vcx^=Px=Vc, where VV is an M×NM×N matrix given by
V
=
⋮
⋮
⋮
v
1
v
2
⋯
v
N
⋮
⋮
⋮
V
=
⋮
⋮
⋮
v
1
v
2
⋯
v
N
⋮
⋮
⋮
(1)and cc is an N×1N×1 vector given by
c
1
c
2
⋮
c
N
.
c
1
c
2
⋮
c
N
.
(2)Given x∈RMx∈RM (or CMCM), our search for the closest approximation
can be written as
min
c
x
-
V
c
2
min
c
x
-
V
c
2
(3)or as
min
c
,
e
e
2
2
subjectto
x
=
V
c
+
e
min
c
,
e
e
2
2
subjectto
x
=
V
c
+
e
(4)Using VV, we can replace G=VHVG=VHV and b=VHxb=VHx. Thus, our solution can be written as
c
=
(
V
H
V
)
-
1
V
H
x
,
c
=
(
V
H
V
)
-
1
V
H
x
,
(5)which yields the formula
x
^
=
V
(
V
H
V
)
-
1
V
H
x
.
x
^
=
V
(
V
H
V
)
-
1
V
H
x
.
(6)The matrix V†=(VHV)-1VHV†=(VHV)-1VH is known as the “pseudo-inverse.” Why the name “pseudo-inverse”? Observe that
V
†
V
=
(
V
H
V
)
-
1
V
H
V
=
I
.
V
†
V
=
(
V
H
V
)
-
1
V
H
V
=
I
.
(7)Note that x^=VV†xx^=VV†x. We can verify that VV†VV† is a projection matrix since
V
V
†
V
V
†
=
V
(
V
H
V
)
-
1
V
H
V
(
V
H
V
)
-
1
V
H
=
V
(
V
H
V
)
-
1
V
H
=
V
V
†
V
V
†
V
V
†
=
V
(
V
H
V
)
-
1
V
H
V
(
V
H
V
)
-
1
V
H
=
V
(
V
H
V
)
-
1
V
H
=
V
V
†
(8)
Thus, given a set of NN linearly independent vectors in RMRM or CMCM (N<MN<M), we can use the pseudo-inverse to project any vector onto the subspace
defined by those vectors. This can be useful any time we have a problem of the form:
x
=
V
c
+
e
x
=
V
c
+
e
(9)where xx denotes a set of known “observations”, VV is a set of known “expansion vectors”, cc are the unknown coefficients, and ee represents an unknown “noise” vector. In this case, the least-squares estimate is given by
c
=
V
†
x
,
x
^
=
V
V
†
x
.
c
=
V
†
x
,
x
^
=
V
V
†
x
.
(10)
