Hamming distance

Consider two distinct five digit codewords C1 = 00000 and C2 = 00011. These have a binary digit difference (or Hamming distance) of 2 in the last two digits. The minimum distance in binary digits between any two codewords is known as the minimum Hamming distance, D min D min

For block codes the minimum Hamming distance or the smallest difference between the digits for any two codewords in the complete code set, D min D min , is the property which controls the error correction performance. We can thus calculate the error detecting and correcting power of a code from the minimum distance in bits between the codewords.

Thus for a code with a minimum distance D min D min = 3 then this code can be used to correct:

Figure 1: Relationship between Dmin and error detection OR correction capability (but not both simultaneously)

Note in the earlier example of two five digit codewords C1 = 00000 and C2 = 00011 which had a Hamming distance of 2 there is only one codeword (e.g. A = 00001 or B = 00010) which lies inbetween these two codewords. Now if there was an error result (e.g. A = 00001) we cannot tell whether it came from C1 or C2 so we can thus only used this to detect that an error has occurred.

If the two five digit codewords had been C1 = 00000 and C2 = 00111, which have a Hamming distance of 3, there are then two words which lie inbetween these codewords (e.g. A = 00001 and B = 00011) and these can thus be used EITHER to detect two errors without any correction capability OR if detection is not required they can used to correct a single error (e.g. C1 = 00000 distorted into A = 00001 or C2 = 00111 distorted into B = 00011), Figure 1.

When performing the correction operation we require to insert the decision boundary as shown in Example 1 below. If in this example we had wished to perform detection only as shown in the lower part of Figure 1 then we would ignore whether the received code A = 00001 resulted from a single error from a C1 transmission or a double error from a C2 code transmission and only identify it as a detected error.

Example 1 – error correction

C1 = 00000

A = 00001

---------------

B = 00011

C2 = 00111

This explains further the detailed operation of the equations in Figure 1 where detection only operation does not require the decision boundary to aid identification of the origination of the error.

Block error probability and correction capability

If we have an error correcting code which can correct R’ errors, than the probability of a codeword not being correctable is the probability of having more than R’ errors in n digits. The probability of having more than R’ errors is given in Figure 2. We can this calculate this probability by summing all the induvidual error probabilities up to and including R' errors in the block.

Figure 2: Correction of more than R' errors in an n digit block

The probability of j errors occurring in an n digit codeword is given in Figure 3. P e P e is the probability of error in a single binary digit and n is the block length. Figure 3 also shows how to calculate the nCj term representing all the possible number of ways or error positions that j errors can occur within a block of length n binary digits.

Figure 3: Probability of j errors occurring an an n-digit codeword

Example 2: If we have an error correcting code which can correct 3 errors within a block length n of 10, what is the probability that the code cannot correct a received block if the per digit error probability is P e P e = 0.01?

Solution: The code cannot correct the received block if there are more than 3 errors. Thus:

P > 3 errors = 1 - P(0 errors) - P(1 error) - P(2 errors) - P(3 errors).

Figure 4 shows the component parts of this calculation.

Figure 4: Calculation of zero, 1, 2 and 3 errors in a 10-digit codeword with a per-digit P e P e of 0.01

Thus the probability that the code cannot correct a received block is then:

1 - 0.9043821 - 0.0913517 - 0.0041523 - 0.0001118 = 0.0000021.

This illustrates that the very low overall error remaining after correction of three errors is much less than the original probability of error in a single bit, P e P e = 0.01. Note also the need for high precision arithmetic (it may be an eight digit calculator is not good enough to calculate the answer to more than 1 significant figure). Note also in Figure 4 the much lower probability of t + 1 errors occuring, compared to t errors, as is implied in FECC.

Group codes

Group codes are a special kind of block codes. They comprise a set of codewords, C1 … CN, which contain the all zeros codeword (e.g. 00000) and exhibit a special property called closure. This property means that if any two valid codewords are subject to a bit wise EX – OR operation then they will produce another valid codeword in the set.

The closure property means that to find the minimum Hamming distance, see below, all that is required is to compare all the remaining codewords in the set with the all zeros codeword instead of comparing all the possible pairs of codewords.

The saving gets bigger the longer the codeword. For example a code set with 100 codewords will require 100 comparisons for a Group code design, compared with 100+99+98+…+2+1, for a non-group code!

In Group codes the D min D min calculation is further simplified into calculating the minimum codeword weight or minimum number of 1 digits in a codeword in the set.

Nearest neighbour decoding

Nearest neighbour decoding assumes that the codeword nearest in Hamming distance to the received word is what was transmitted, as shown in Example 1 above. This inherently contains the assumption that the probability of a small number of t errors is greater than the probability of the larger number of t+1 errors, i.e that P e P e is small.

A nearest neighbour decoding table for a (n, k) = (5, 2) i.e. a 5-digit group code is shown in Figure 5. Recall that for an n = 5 bit codeword there are 2 5 2 5 = 32 unique patterns generated by all the possible combinations of the 5 digits.

Figure 5: Nearest neighbour decoding table for 5 bit code with 4 codewords implying 2 information bits

Figure 5 starts by forming a table with the 4 codewords across the top row. All the single error patterns, which each only differ by one bit from each of the transmitted codewords, can be readily and uniquely assigned back to an error free codeword. Thus the next 5 rows represent these single errors in position 1 through 5 in each of the 4 codewords. Now we have a table up to this point with a total of 4 x 6 = 24 unique entries. Therefore this code is capable of correcting all these single errors.

There are also eight remaining codes or table entries as 32 - 24 = 8 and these represent double error patterns which, as can be seen, lie an equal Hamming distance from at least 2 of the initial 4 codewords in the top row. Note for example errors in the first two digits of the 00000 codeword result in us receiving 11000. However data bit pattern is identified here in Figure 5 as a single error from codeword 11100 as we assume that 1 error is a much more likely occurence than two errors!

These represent some of the double error patterns, which can thus be detected here, but they cannnot be corrected as all the possible double error patterns do not have a unique representation in Figure 5.

Soft decision decoding

Nearest neighbour decoding can also be done on a soft decision basis, with real non-binary numbers from the receiver. The nearest Euclidean distance (nearest to these 5 codewords in terms of a 5-D geometry) is then used and this gives a considerable performance increase over the hard decision decoding described here.

Hamming bound

This defines mathematically the error correcting performance of a block coder. The upper bound on the performance of block codes is given by the Hamming bound, some times called the sphere packing bound. If we are trying to create a code to correct t errors with a block length of n with k information digits, then Figure 6 shows the Hamming bound equation.

Figure 6: Hamming bound calculation for (n, k) block code to establish number of terms which can be included in the denominator and hence arrive at the codes error correcting power t

Here the denominator terms, which are represented by the binomial coefficients, represent the number of possible patterns or positions in which 1, 2, ..., t errors can occur in an n-bit codeword.

Note the relationship between the decoding table in Figure 5 and the Hamming Bound equation in Figure 6. The 2 k 2 k = 4 left hand entry represents the number of transmitted codewords or columns in the table. The numerator 2 n 2 n = 32 represents the total possible number of unique entries in the table. The demoninator represents the number of rows which can be accommodated within the table. Here the first denominator term (1) represents the first row (i.e. the transmitted codewords) and the second term (n) the 5 single error patterns. Subsequent terms then represent all the possible double, triple error patterns, etc. The denominator has to be sized or restricted to t to ensure the inequality and this gives or defines the error correction capability as t.

If the equation in Figure 6 is satisfied then the design of suck an (n, k) code is possible with the error correcting power of t. If the equation is not satisfied, then we must be less ambitious by reducing t or k (for the same block length n) or increasing n (while maintaining t and k).

Example 2

Comment on the ability of a (5, 2) code to correct 1 error and the possibility of a (5, 2) code to correct 2 errors?

Solution

For single error: k = 2, n = 5 and t = 1, leads to the case summarized in Figure 7.

Figure 7: Calculation to assess whether (5, 2) block code can correct t = 1 error - Answer yes

which is true so such a code design is possible.

However if we try to design a (5, 2) code to correct 2 errors we have k = 2, n = 5 and t = 2, which is summarized in Figure 8.

Figure 8: Calculation to assess whether (5, 2) block code can correct t = 2 errors - Answer no

This result is false or cannot be satisfied and thus this short code cannot be designed with a t = 2 error correcting power or capability.

This provides further mathematical derivation for the error correcting performance limits of the nearest neighbour decoding table shown previously in Figure 5 where we could correct all single error patterns but we could not correct all the possible double error patterns.

A full decoding table is not required to be created as, through checking the Hamming bound, one can identify the required block size and number of parity check bits which are required for a given error correction capability in a block or group coder design.

Figure 9 shows the performance of various BLOCK codes, all of rate ½, whose performance progressively improves as the block length increases from 7 to 511, even for the same coding rate of ½.

The power of these forward error correcting codes (FECC) is quantified as the coding gain, i.e. the reduction in the required E b N 0 E b N 0 ratio or energy required to transmit each bit divided by the spectral noise density, for a given bit error ratio or error probability.

For example in Figure 9 the (31, 16) code has a coding gain over the uncoded case of around 1.8 dB at a P b P b of 10 -5 10 -5 .

Figure 9: Error performance of 1/2 rate block coders with differing block lengths