Error-Correcting Codes Lab 05, Lecture Slide - Engineering, Slides of Applied Mathematics

Download Error-Correcting Codes Lab 05, Lecture Slide - Engineering and more Slides Applied Mathematics in PDF only on Docsity! 18.413: Error-Correcting Codes Lab February 19, 2004 Lecture 5 Lecturer: Daniel A. Spielman 5.1 Parity Continued 5.2 The Gaussian Distribution One of the most natural distribution is the Gaussian distribution. A Gaussian random variable with mean µ and standard deviation σ has probability density function p(x) = 1√ 2πσ e −(x−µ)2 2σ2 . For those who haven’t learned about probability density functions before, this means that the probability that x lies between x0 and x1 is∫ x1 x=x0 p(x) dx. You can verify that ∫ ∞ x=−∞ p(x) dx = 1. The standard deviation tells you how far from the mean x is likely to lie. In particular, one can prove that P [x− µ > kσ] < e −−k 2 2 √ 2πk , and that this bound is very close to tight for k ≥ 2. For σ = 1 and µ = 0, this bound indicates that the probability that the Gaussian lies between −2 and 2 is at least 0.946, whereas the actual probability is about 0.956. If g is a Gaussian random variable of mean 0 and standard deviation 1, then σg + µ is a Gaussian random variable of mean µ and standard deviation σ. Often, you will see a Gaussian described by its variance. It’s variance is the square of its standard deviation. Gaussian random variables arise as the limit of binomial random variables. That is, if we let x1, . . . , xn be indepent random variables uniformly distributed in {1,−1}, and let X = ∑ xi/ √ n, then as n grows large, X approaches the Gaussian distribution. We will now sketch a proof of this. In particular, we consider the probability that X = c √ n. From counting, we can determine that this is P [ X = c √ n ] = 2−n ( n n/2− c √ n/2 ) . 5-1 Lecture 5: February 19, 2004 5-2 We will show that this is approximately equal to∫ c+1/√n g=c−1/ √ n p(g) dg ≈ 2p(c)/ √ n = √ 2 πn e −c2 2 . Applying Stirling’s formula, we find that 2−n ( n n/2− k ) = 2−n n! (n/2− k)!(n/2 + k)! ≈ 2−n  (ne )n( n/2−k e )n/2−k ( n/2+k e )n/2+k ( √2πn√ 2π(n/2− k) √ 2π(n/2 + k) ) = ( (n)n (n− 2k)n/2−k (n + 2k)n/2+k )( √ 2πn√ 2π(n/2− k) √ 2π(n/2 + k) ) = ( (n)n−2k (n2 − 4k2)n/2−k )( (n)2k (n + 2k)2k )( √ 2πn√ 2π(n/2− k) √ 2π(n/2 + k) ) We now evaluate each of the three terms in this product individually, substituting in k = (c/2) √ n. First, we find (n)n−2k (n2 − 4k2)n/2−k = (n)n−c √ n (n2 − c2n)n/2−(c/2) √ n = ( 1 (1− c2/n) )n/2−(c/2)√n ≈ ( 1 + c2 n )n/2−(c/2)√n ≈ ec2/2, as (1 + 1/k)k ≈ e. To evalue the other term, we compute (n)2k (n + 2k)2k = (n)c √ n (n + c √ n)c √ n = ( 1 1 + c/ √ n )c√n ≈ ( 1− c/ √ n )c√n ≈ e−c2 , as (1− 1/k)k ≈ e−1. Finally, we find √ 2πn√ 2π(n/2− k) √ 2π(n/2 + k) ≈ √ 2 πn , Lecture 5: February 19, 2004 5-5 We usually understand this code by writing the bits in a matrix, like this x1 x2 x5 x3 x4 x6 x7 x8 x9, and observing that each row and column must have parity 0. This is why we call it a product code: each row and each column looks like a code (we’ll see more of these later). If we flip one bit in a codeword of this code, then the parity constraints on the row and colum containing the flipped bit will be violated, indicating where the error is. If two bits are flipped, you might be able to detect that an error has occured, but won’t be able to figure out where it is. That is, if you only have bits and not and confidence information. If in addition to bit you have information for each bit indicating how confident you are that the bit is 0 or 1 (that is, if you have the full information provided by the channel), then you can decode in one of the ways discussed in the last lecture: either maximum likelihood decoding to minimize word error or bit error. In Small Project 2, we will minimize the bit error rate, denoted (BER). 5.5 BER I’ll now define the bit error rate (BER) precisely. It depends upon the coding scheme, the channel, and the decoding scheme. Let w1, . . . , wk be the bits in the message to be sent. We assume that they are encoded as x1, . . . , xn, and received by the decoder as y1, . . . , yn. We then let z1, . . . , zk be the outputs of the decoder, which we now force to output a 0 or 1 for each bit. The bit error rate is then E [ (1/k) ∑ i P [wi 6= zi] ] . Empircally, you can compute this by averaging (1/k) ∑ i [wi 6= zi] over many trials. 5.6 Heuristic Decoding of the Parity Product Code I will now describe a heuristic decoding algorithm for the code we will examine on Small Project 2. This algorithm will be less accurate and take longer to run than the ideal algorithm. However, for larger codes the natural extension of this algorithm will be practical and the ideal algorithm will be impractical. For now, I will assume that we are just trying to compute x1. Here is the algorithm: • compute Pext [x3 = 1|y4, y6]. Lecture 5: February 19, 2004 5-6 • from the previous calculation, obtain q3 = Ppost [x3 = 1|y3, y4, y6]. • similarly, compute q7 = Ppost [x7 = 1|y7, y8, y9]. • Now, we will to compute Pext [x1 = 1|y3, y4, y6, y7, y8, y9]. We do this by observing that x1 = x3 ⊕ x7, and use the parity formula to obtain the extrinsic probability that x1 = 1 given our estimates of the probabiliites that x3 and x7 equal 1 obtained above, q3 and q7. • We then compute Pext [x1 = 1|y2, y4, y8, y5, y6, y9], working by columns instead of rows. • We then treat Pext [x1 = 1|y3, y4, y6, y7, y8, y9] , Pext [x1 = 1|y2, y4, y8, y5, y6, y9] , and Pext [x1 = 1|y1] as three independent estimators of x1, and combine these estimations as discussed for the repition code. This is a heuristic algorithm as the three estimators are not actually independent. 5.7 Confidence Intervals Whenever we plot data, we should provide confidence intervals on the data in the plot. The goal of a confidence interval is to indicate the range of values that could reasonbly explain your output. It is standard to give a 95% confidence interval. That is, a range such that with probability 95% the actual value lies within the confidence interval. For this project, we will just be dealing with binary variables—either “yes” or “no”, and estimating the probability of a “yes”. In this case, obtaining confidence intervals is reasonably simple. Let’s assume that we do n experiments and r of them come out “yes”. Then, we estimate the probability of “yes” to be p̂ = r/n. For reasonably large n, and not unreasonably small or large p, if we draw the confidence interval from p̂e−2 √ (1−p̂)/p̂n to p̂e2 √ (1−p̂)/p̂n, then with probability 95% the actual p should lie in this interval. If you have r = 0, then we set the bottom of the interval to 0 and the top to 1− e(−2/n). However, I should warn you that, while this is a very well accepted way of drawing the confidence intervals, it does not always get you 95% confidence. But, it takes strange values to get much below 90%. Yes, I am fudging here. I can give you papers on this topic if you would like to learn more. For this class, I will be happy if you draw your confidence intervals using this rule. You should expect your data to fluctuate quite a bit within the inner half of the confidence interval. As pne±2 √ (1−p)/pn ≈ pn ± 2 √ p(1− p)n, this means that you should only expect to get relative accuracy around √ p(1− p)n/pn ≈ 1/√pn. For example, if p ≈ 1/100 and you run n = 100, 000, 000 experiments, you should only expect to get about 3 digits of accuracy. Lecture 5: February 19, 2004 5-7 5.8 How big should N be? Let’s say that we are trying to estimate the BER, but the BER is quite small. You might want to know how many trials you need to run to estimate the BER reasonably. Rather than fixing in advance how many trials you will run, you can run until you see some fixed number of errors. For crude data, just to get general order of magnitude, 10 observations would be reasonable. If you want to get a digit of accuracy, then you need 100 observations. The formula given in the previous section will give you a resonable confidence interval. 5.9 Plotting in Matlab To make many plots appear on the same set of axes, type hold on after you plot. Here is an example of how to plot a confidence interval on some imaginary data. In this case, I’ve made the confidence interval go from y (i) - sig to y (i) + sig, just to show you how the graphics should look. x = [1:10]; y = x.^(1.5); plot(x,y) hold on plot(x,y,’o’) i = 4; sig = 3; plot([x(i),x(i)],[y(i)-sig,y(i)+sig],’+’)