Error Correcting Codes: Basic Notions and Gaussian Elimination, Assignments of Algebra

Download Error Correcting Codes: Basic Notions and Gaussian Elimination and more Assignments Algebra in PDF only on Docsity! Chapter 2 Error Correcting Codes The identification number schemes we discussed in the previous chapter give us the ability to determine if an error has been made in recording or transmitting information. However, they are limited in two ways. First, each allows detection of an error in just one digit, expect for some special types of errors, such as interchanging digits. Second, they provide no way to recover the intended information. By making use of more sophisticated ideas and mathematical concepts, we will study methods of encoding and transmitting information that allow us to both detect and correct errors. There are many places that use these so-called error correcting codes, from transmitting photographs from planetary probes to playing of compact discs and dvd movies. 2.1 Basic Notions To discuss error correcting codes, we need first to set the context and define some terms. We work throughout in binary; that is, we will work over Z2. To simplify notation, we will write the two elements of Z2 as 0 and 1 instead of as 0 and 1. If n is a positive integer, then the set Zn2 is the set of all n-tuples of Z2-entries. Elements of Zn2 are called words, or words of length n. A code of length n is a nonempty subset of Zn2 . We will refer to elements of a code as codewords. For convenience we will write elements of Zn2 either with the usual notation, or as a concatenation of digits. For instance, we will write (0, 1, 0, 1) and 0101 for the same 4-tuple. We can equip Zn2 with an operation of addition by using point-wise addition. That is, we define (a1, . . . , an) + (b1, . . . , bn) = (a1 + b1, . . . , an + bn). Note that, as a consequence of the facts that 0 + 0 = 0 = 1 + 1 in Z2, we have a + a = 0 for any a ∈ Zn2 , where 0 is the vector (0, . . . , 0) consisting of all zeros. Example 2.1. The set {01, 10, 11} is a code of length 2, and {0000, 1010, 0101, 1111} is a code of length 4. 19 20 Chapter 2. Error Correcting Codes Let w = a1 · · · an be a word of length n. Then the weight of w is the number of digits of w equal to 1. We denote the weight of w by wt(w). There are some obvious consequences of this definition. First of all, wt(w) = 0 if and only if w = 0. Second, wt(w) is a nonnegative integer. A more sophisticated fact about weight is its relation with addition. If v, w ∈ Zn2 , then wt(v +w) ≤ wt(v)+wt(w). To see why this is true, we write xi for the i-th component of a word x. The weight of x is then given by the equation wt(x) = |{i : 1 ≤ i ≤ n, xi = 1}|. Using this description of weight, we note that (v + w)i = vi + wi. Therefore, if (v + w)i = 1, then either vi = 1 or wi = 1 (but not both). Therefore, {i : 1 ≤ i ≤ n, (v + w)i = 1} ⊆ {i : vi = 1} ∪ {i : wi = 1} . Since |A ∪B| ≤ |A|+ |B| for any two finite sets A, B, the inclusion above yields wt(v+w) ≤ wt(v) + wt(w), as desired. From idea of weight we can define the notion of distance on Zn2 . If v, w are words, then we set the distance D(v, w) between v and w to be D(v, w) = wt(v + w). Alternatively, D(v, w) is equal to the number of positions in which v and w differ. The function D shares the basic properties of distance in Euclidean space R3. More precisely, it satisfies the properties of the following lemma. Lemma 2.2. The distance function D defined on Zn2 × Zn2 satisfies 1. D(v, v) = 0 for all v ∈ Zn2 ; 2. for any v, w ∈ Zn2 , if D(v, w) = 0, then v = w; 3. D(v, w) = D(w, v) for any v, w ∈ Zn2 ; 4. triangle inequality: D(v, w) ≤ D(v, u) + D(u, w) for any u, v, w ∈ Zn2 . Proof. Since v + v = 0, we have D(v, v) = wt(v + v) = wt(0) = 0. This proves (1). We note that 0 is the only word of weight 0. Thus, if D(v, w) = 0, then wt(v + w) = 0, which forces v + w = 0. However, adding w to both sides yields v = w, and this proves (2). The equality D(v, w) = D(w, v) is obvious since v+w = w+v. Finally, we prove (4), the only non-obvious statement, with a cute argument. Given u, v, w ∈ Zn2 , we have, from the definition and the fact about weight given above, D(v, w) = wt(v + w) = wt((v + u) + (u + w)) ≤ wt(v + u) + wt(u + w) = D(v, u) + D(u, w). 2.2. Gaussian Elimination 23 the column matrix of variables. For example, the system 2x + 3y − z = 1 x− y + 5z = 2 is equivalent to the matrix equation ( 2 3 −1 1 −1 5 )  xy z  = ( 1 2 ) . The primary matrix-theoretic method for solving such a system is to perform Gaussian elimination on the augmented matrix, that matrix that adds to the coefficient matrix one column at the right equal to the column on the right side of the equation. Recall Gaussian elimination performs operations on the rows of a matrix in order to replace the matrix by one in which the solution to the system can be found easily. There are three such row operations: • multiply or divide a row by a nonzero scalar, • interchange two rows, • add a multiple of one row to another row. It is likely that in all your work with matrices, the entries of the matrices were real numbers. However, to perform the row operations, all you need is to be able to add, subtract, multiply, and divide the entries. In many situations, matrices arise whose entries are not real numbers. For coding theory we need to work with matrices whose entries lie in Z2 = { 0, 1 } . Within this set we can add, subtract, multiply, and divide just as if we had real numbers. Furthermore, all the theorems of linear algebra have analogues to the setting where entries lie in Z2. In fact, we will generalize the idea of linear algebra later on to include many more sets of scalars. Again, all we need is to be able to perform the four arithmetic operations on the scalars, and we need properties analogous to those that hold for real number arithmetic. Recall that the only symbolic difference between Z2 arithmetic and ordinary arithmetic of these symbols is that 1 + 1 = 0 in Z2. Note that the first of the three row operations listed above is not useful; multiplying a row by 1 does not affect the row, so is an operation that is not needed. Also, the third operation in the case of Z2 reduces to adding one row to another. Before working some examples, we recall what it means for a matrix to be in row reduced echelon form. Definition 2.10. A matrix A is in row reduced echelon form if 1. the first nonzero entry of any row is 1. This entry is called a leading 1; 2. If a column contains a leading 1, then all other entries of the column are 0; 24 Chapter 2. Error Correcting Codes 3. If i > j, and if row i and row j each contain a leading 1, then the column containing the leading 1 of row i is further to the right than the column containing the leading 1 of row j. To help understand Condition 3 of the definition, the leading 1’s go to the right as you go from top to bottom in the matrix. We now give several examples of reducing matrices with Z2 entries to echelon form. In each example once we have the matrix in row reduced echelon form, the leading 1’s are marked in boldface. In understanding the computations below, recall that since −1 = 1 in Z2, subtraction and addition are the same operation. Example 2.11. Consider the matrix 1 0 0 11 1 0 1 0 1 1 1  . We reduce the matrix with the following steps. You should determine which row operation was done in each step. 1 0 0 11 1 0 1 0 1 1 1  =⇒  1 0 0 10 1 0 0 0 1 1 1  =⇒  1 0 0 10 1 0 0 0 0 1 1  . Example 2.12. Consider the matrix 1 1 0 0 1 0 1 0 1 0 0 1 0 1 1 1 1 0 0 0 0 1 0 1  . To reduce this matrix, we can do the following steps. 1 1 0 0 1 0 1 0 1 0 0 1 0 1 1 1 1 0 0 1 1 0 1 1  =⇒  1 1 0 0 1 0 0 1 1 0 1 1 0 1 1 1 1 0 0 1 1 0 1 1  =⇒  1 0 1 0 0 1 0 1 1 0 1 1 0 1 1 1 1 0 0 1 1 0 1 1  =⇒  1 0 1 0 0 1 0 1 1 0 1 1 0 0 0 1 0 1 0 1 1 0 1 1  =⇒  1 0 1 0 0 1 0 1 1 0 1 1 0 0 0 1 0 1 0 0 0 0 0 0  . Example 2.13. To reduce the matrix 0 0 0 1 1 1 10 1 1 0 0 1 1 1 0 1 0 1 0 1  2.2. Gaussian Elimination 25 we can apply the following single row operation. 0 0 0 1 1 1 10 1 1 0 0 1 1 1 0 1 0 1 0 1  =⇒  1 0 1 0 1 0 10 1 1 0 0 1 1 0 0 0 1 1 1 1  . We now recall why having a matrix in row reduced echelon form will give us the solution to the corresponding system of equations. The row operations on the augmented matrix corresponds to performing various algebraic manipulations to the equations, such as inter- changing equations. So, the system of equations corresponding to the reduced matrix is equivalent to the original system; that is, the two systems have exactly the same solutions. Example 2.14. Consider the system of equations x = 1 x + y = 1 y + z = 1. This system has augmented matrix  1 0 0 11 1 0 1 0 1 1 1  , and reducing this matrix yields  1 0 0 10 1 0 0 0 0 1 1  . This new matrix corresponds to the system of equations x = 1, y = 0, z = 1. Thus, we have already the solution to the original system. Example 2.15. The augmented matrix 1 1 0 0 1 0 1 0 1 0 0 1 0 1 1 1 1 0 0 1 1 0 1 1  28 Chapter 2. Error Correcting Codes the dimension of the row space is the number of leading 1’s. Thus, an alternative definition of the rank of a matrix is that it is equal to the number of leading 1’s in the row reduced echelon form obtained from the matrix. The kernel, or nullspace, of a matrix A is the set of all solutions to the homogeneous equation AX = 0. To help understand this example, consider the Hamming matrix H of the previous example. Example 2.17. The solution to the homogeneous equation HX = 0 from the previous example is x1 = x3 + x5 + x7, x2 = x3 + x6 + x7, x4 = x5 + x6 + x7, x3, x5, x6, x7 are arbitrary. For each arbitrary variable we can set it equal to 1 and all other arbitrary variables equal to 0. The resulting vector will be a solution to HX = 0. If we do this for each arbitrary variable, we will have a basis for the nullspace. Doing this, we get the four vectors 1 1 1 0 0 0 0  ,  1 0 0 1 1 0 0  ,  0 1 0 1 0 1 0  ,  1 1 0 1 0 0 1  These vectors do form a basis for the nullspace of H since the general solution of HX = 0 is x1 x2 x3 x4 x5 x6 x7  =  x3 + x5 + x7 x3 + x6 + x7 x3 x5 + x6 + x7 x5 x6 x7  = x3  1 1 1 0 0 0 0  + x5  1 0 0 1 1 0 0  + x6  0 1 0 1 0 1 0  + x7  1 1 0 1 0 0 1  . From this equation, we see that every solution is a linear combination of the four specific solutions written above, and a little work will show that every solution can be written in a unique way as a linear combination of these vectors. For example, one can check that (0, 1, 1, 1, 1, 0, 0) is a solution to the system HX = 0, and that to write this vector as a linear combination of the four given vectors, we must have x3 = x5 = 0 and x6 = x7 = 0, 2.3. The Hamming Code 29 and so  0 1 1 1 1 0 0  =  1 1 1 0 0 0 0  +  1 0 0 1 1 0 0  is a sum of two of the four given vectors, and can be written in no other way in terms of the four. This example indicates the following general fact that for a homogeneous system AX = 0, that the number of variables not corresponding to leading 1’s is equal to the dimension of the nullspace of A. Let us call these variables leading variables. If we reduce A, the leading variables can be solved in terms of the other variables, and these other variables are all arbitrary; we call them free variables. By mimicing the example above, any solution can be written uniquely in terms of a set of solutions, one for each free variable. This set of solutions is a basis for the nullspace of A; therefore, the number of free variables is equal to the dimension of the nullspace. Every variable is then either a leading variable or a free variable. The number of variables is the number of columns of the matrix. This observation leads to the rank-nullity theorem. The nullity of a matrix A is the dimension of the nullspace of A. Theorem 2.18 (Rank-Nullity). Let A be an n×m matrix. Then m is equal to the sum of the rank of A and the nullity of A. The point of this theorem is that once you know the rank of A, the nullity of A can be immediately calculated. The number of solutions to AX = 0 can then be found. In coding theory this will allow us to determine the number of codewords in a given code. 2.3 The Hamming Code The Hamming code, discovered independently by Hamming and Golay, was the first example of an error correcting code. Let H =  0 0 0 1 1 1 10 1 1 0 0 1 1 1 0 1 0 1 0 1  be the Hamming matrix, described in Example 2.16 above. Note that the columns of this matrix give the base 2 representation of the integers 1-7. The Hamming code C of length 7 is the nullspace of H. More precisely, C = { v ∈ K7 : HvT = 0 } . 30 Chapter 2. Error Correcting Codes Also by Gaussian elimination, we can solve the linear equation Hx = 0, and we get the solution  x1 x2 x3 x4 x5 x6 x7  = x3  1 1 1 0 0 0 0  + x5  1 0 0 1 1 0 0  + x6  0 1 0 1 0 1 0  + x7  1 1 0 1 0 0 1  . Therefore, C has dimension 4, and the set {1110000, 1001100, 0101010, 1101001} forms a basis for C. If we write out all 16 codewords in C, we will see that the distance of C is 3. Thus, C is a (7, 4, 3)-code. It will then correct 1 error. The code C has a particularly elegant decoding algorithm, which we now describe. Let e1, . . . , e7 be the standard basis for K 7. We point out a simple fact of matrix multiplication: HeTi is equal to the i-th column of H. Moreover, we note that the 7 nonzero vectors in K 3 are exactly the 7 columns of H. Suppose that v is a codeword that is transmitted as a word w 6= v. Suppose that exactly one error has been made in transmission. Then w = v + ei for some i. However, we do not yet know i, so we cannot yet determine v from w. However, HwT = H(v + ei) T = HvT + HeTi = He T i , and HeTi is the i-th column of H, as we pointed out above. Therefore, we can determine i by computing HwT and determining which column of H is this. The Hamming code C has an additional property: every word is within 1 of a codeword. To see this, suppose that w is a word. If HwT = 0, then w is a codeword. If not, then HwT is a nonzero 3-tuple. Therefore, it is equal to a column of H; say that HwT is equal to the i-th column of H. Then HwT = HeTi , so H(w T + eTi ) = 0. Therefore, w + ei := v ∈ C. The word v is then a codeword a distance of 1 from w. A code that corrects t errors and for which every word is within t of some codeword is called perfect. Such codes are particularly nice, in part because a decoding procedure will always return a word. Later we will see some important codes that are not perfect.