Quantization Error in FIR Filters 1.1 2003/07/16 16:46:38 GMT-5 2004/12/29 17:28:10.555 US/Central Douglas L. Jones dl-jones@uiuc.edu Douglas L. Jones dl-jones@uiuc.edu Kyle Clarkson kclarks@rice.edu coefficient quantization data quantization direct form FIR filters quantization error transpose form FIR filters suffer from both data and coefficient quantization; each has different effects. Double-precision accumulation inside the FIR filter structure greatly reduces the data quantization error. In digital filters, both the data at various places in the filter, which are continually varying, and the coefficients, which are fixed, must be quantized. The effects of quantization on data and coefficients are quite different, so they are analyzed separately.

Data Quantization Typically, the input and output in a digital filter are quantized by the analog-to-digital and digital-to-analog converters, respectively. Quantization also occurs at various points in a filter structure, usually after a multiply, since multiplies increase the number of bits.

Direct-form Structures There are two common possibilities for quantization in a direct-form FIR filter structure: after each multiply, or only once at the end.

Single-precision accumulate; total variance M Δ 2 12 Double-precision accumulate; variance Δ 2 12

In the latter structure, a double-length accumulator adds all 2 B 1 bits of each product into the accumulating sum, and truncates only at the end. Obviously, this is much preferred, and should always be used wherever possible. All DSP microprocessors and most general-pupose computers support double-precision accumulation.

Transpose-form Similarly, the transpose-form FIR filter structure presents two common options for quantization: after each multiply, or once at the end.

Quantize at each stage before storing intermediate sum. Output variance M Δ 2 12 or Store double-precision partial sums. Costs more memory, but variance Δ 2 12

The transpose form is not as convenient in terms of supporting double-precision accumulation, which is a significant disadvantage of this structure.

Coefficient Quantization Since a quantized coefficient is fixed for all time, we treat it differently than data quantization. The fundamental question is: how much does the quantization affect the frequency response of the filter? The quantized filter frequency response is DTFT h Q DTFT h inf. prec. e H inf. prec. w H e w Assuming the quantization model is correct, H e w should be fairly random and white, with the error spread fairly equally over all frequencies w ; however, the randomness of this error destroys any equiripple property or any infinite-precision optimality of a filter. What quantization scheme minimizes the L 2 quantization error in frequency (minimizes w H w H Q w 2 )? On average, how big is this error? Ideally, if one knows the coefficients are to be quantized to B bits, one should incorporate this directly into the filter design problem, and find the M B-bit binary fractional coefficients minimizing the maximum deviation ( L ∞ error). This can be done, but it is an integer program, which is known to be np-hard (i.e., requires almost a brute-force search). This is so expensive computationally that it's rarely done. There are some sub-optimal methods that are much more efficient and usually produce pretty good results.