What value of ββ is required so that the output of an FIR filter cannot overflow ( |y⁢n|≤1   n y n 1 , |x⁢n|≤1   n x n 1 )? |y⁢n|=|∑ k =0M−1h⁢k⁢β⁢x⁢n−k|≤∑ k =0M−1|h⁢k|⁢|β|⁢|x⁢n−k|≤β⁢∑ k =0M−1|h⁢k|⁢1 y n k 0 M 1 h k β x n k k 0 M 1 h k β x n k β k M 1 0 h k 1 ⇓ ⇓ β<∑ k =0M−1|h⁢k| β k M 1 0 h k Alternatively, we can incorporate the scaling directly into the filter, and require that ∑ k =0M−1|h⁢k|<1 k M 1 0 h k 1 to prevent overflow.

To prevent the output from overflowing in an IIR filter, the condition above still holds: ( M=∞ M ) |y⁢n|<∑ k =0∞|h⁢k| y n k 0 h k so an initial scaling factor β<1∑ k =0∞|h⁢k| β 1 k 0 h k can be used, or the filter itself can be scaled.

However, it is also necessary to prevent the states from overflowing, and to prevent overflow at any point in the signal flow graph where the arithmetic hardware would thereby produce errors. To prevent the states from overflowing, we determine the transfer function from the input to all states ii, and scale the filter such that ∑ k =0∞| h i ⁢k|≤1   i k 0 h i k 1

Although this method of scaling guarantees no overflows, it is often too conservative. Note that a worst-case signal is x⁢n=sign⁢h⁢−n x n sign h n ; this input may be extremely unlikely. In the relatively common situation in which the input is expected to be mainly a single-frequency sinusoid of unknown frequency and amplitude less than 1, a scaling condition of |H⁢w|≤1   w H w 1 is sufficient to guarantee no overflow. This scaling condition is often used. If there are several potential overflow locations ii in the digital filter structure, the scaling conditions are | H i ⁢w|≤1   i w H i w 1 where H i ⁢w H i w is the frequency response from the input to location ii in the filter.

Even this condition may be excessively conservative, for example if the input is more-or-less random, or if occasional overflow can be tolerated. In practice, experimentation and simulation are often the best ways to optimize the scaling factors in a given application.

For filters implemented in the cascade form, rather than scaling for the entire filter at the beginning, (which introduces lots of quantization of the input) the filter is usually scaled so that each stage is just prevented from overflowing. This is best in terms of reducing the quantization noise. The scaling factors are incorporated either into the previous or the next stage, whichever is most convenient.

Some heurisitc rules for grouping poles and zeros in a cascade implementation are:

Leland B. Jackson has an excellent intuitive discussion of finite-precision problems in digital filters. The book by Roberts and Mullis is one of the most thorough in terms of detail.