Gauss-Markov Theorem and Wiener Filtering 1.6 2003/07/16 11:11:44 GMT-5 2003/12/15 15:29:29.673 US/Central Rob "The Kid" Nowak nowak@rice.edu Clayton Scott cscott@rice.edu Rob "The Kid" Nowak nowak@rice.edu Clayton Scott cscott@rice.edu Elizabeth Gregory lizzardg@rice.edu Jeffrey Silverman jsilv@rice.edu Let x and y be jointly Gaussian distributed: x y m x m y R xx R xy R yx R yy Then the conditional distribution of y given x is y | x m y R yx R xx x m x Q where Q R yy R yx R xx R xy We know that the conditional mean y m y R yx R xx x m x is the best estimate of y give x in a (mean) squared error sense. x y W where y 0 R yy and W 0 R WW and y and W are independent. x y 0 0 R yy R WW R yy R yy R yy y | x R yx R xx x R yy R yx R xx R xy y R yy R yy R WW x H x where H is the Wiener filter. Minimum MSE estimator of y given x. R yx R yy and R xx R yy R WW .

Direct Optimization x y W y G x H G y G x y G x where y G x y G x is the MSE. MS y G x y G x tr y G x y G x tr y G x y G x Minimizing MSE is equivalent to minimizing ε 2 y G x y G x R yy G R xy R yx G G R xx G Taking the derivative with repsect to G G ε 2 -2 R yx 2 G R xx 0 This implies H R yx R xx R yy R yy R WW

Geometrical Interpretation y y y y , where y y is the error. The error and the estimate are statistically orthogonal to each other.

Orthogonality Condition The optimal Wiener filter H R yy R yy R WW satisfies the following condition y y y 0 y y y y x H H x x H R yx H H R xx H R yy H H R yy R WW H R yy R yy R WW R yy R yy R yy R WW R yy R WW R yy R WW R yy 0

The Classical Wiener Filter

y t is a stationary random signal, w t is stationary random noise, and the filter is H

We want to find H ω that minimizes the MSE ε 2 y t y t 2 R ŷ ŷ 0 2 R y ŷ 0 R yy 0 where R yy τ y t y t τ . We can express the MSE in the frequency domain by noting that R yy 0 1 2 ω S yy ω where S yy ω is the power spectrum of y t . R yy τ and S yy ω are FT pairs: S yy ω τ R yy τ ω τ R yy τ 1 2 ω S yy ω ω τ

Random Signal Response of Linear Systems

x t is a stationary random process and H is a linear time-invariant system.

y t u h u x t u y t u h u x t u m x u h u since x t u is a constant. In particular, if x t is zero-mean then so is y t .

Autocorrelation of Output Process R ŷ ŷ τ y t y t τ s h s x t s u h u x t τ u u s x t s x t τ u h s h u u s R xx τ s u h s h u R xx τ h τ h τ

Power Spectrum S ŷ ŷ ω S xx ω H ω H ω S xx ω H ω 2

Cross-Correlation and Cross-Spectrum R ŷ x τ y t x t τ s h s x t s x t τ s h s R xx τ s R xx τ h τ This implies S ŷ x ω S xx ω H ω R ŷ ŷ 0 1 2 ω H ω 2 S xx ω R y ŷ 0 1 2 ω H ω S yx ω S xx ω S yy ω S ww ω S yx ω S yy ω since y t and w t are independent. Thus, the expression for the MSE becomes ε 2 1 2 ω H ω 2 S yy ω S ww ω 2 H ω S yy ω S yy ω ε 2 is minimized by minimizing the integrand each frequency ω. This implies H ω S yy ω S ww ω S yy ω H ω S yy ω S yy ω S ww ω

Comparison

Signal Vector (discrete-time) H R yy R yy R ww Here H is a matrix, R yy is the signal "power", and R ww is the noise "power".

Classical (continuous-time) H ω S yy ω S yy ω S ww ω S yy ω S yy ω S ww ω Which means that the Wiener Filter is defined as the ratio of signal power to the sum of signal power and noise power.