Since a quantized coefficient is fixed for all time, we treat it
differently than data quantization. The fundamental question
is: how much does the quantization affect the frequency
response of the filter?
The quantized filter frequency response is
DTFTh⇀Q=DTFTh⇀inf. prec.+e=
H
inf. prec.
w+
H
e
w
DTFT
h
Q
DTFT
h
inf. prec.
e
H
inf. prec.
w
H
e
w
Assuming the quantization model is correct,
H
e
w
H
e
w
should be fairly random and white, with the error
spread fairly equally over all frequencies
w∈
−π
π
w
; however, the randomness of this error destroys any
equiripple property or any infinite-precision optimality of a filter.
What quantization scheme minimizes the
L
2
L
2
quantization error in frequency (minimizes
∫−ππ|Hw−
H
Q
w|2d
w
w
H
w
H
Q
w
2
)? On average, how big is this error?
Ideally, if one knows the coefficients are to be quantized to
BB bits, one should incorporate
this directly into the filter design problem, and find the
MM
BB-bit binary fractional
coefficients minimizing the maximum deviation
(
L
∞
L
∞
error). This can be done, but it is an integer
program, which is known to be np-hard (i.e., requires almost a
brute-force
search). This is so expensive computationally that it's
rarely done. There are some sub-optimal methods that are
much more efficient and usually produce pretty good results.
