Probability, Statistics and Information Theory, Summaries of Probability and Statistics

Download Probability, Statistics and Information Theory and more Summaries Probability and Statistics in PDF only on Docsity! Probability, Statistics and Information Theory Contents 1 Introduction to Source Coding 2 Expected Length of a Source Code 3 Types Of Codes 3.1 Non-SingularCodes .....-..00 02200 3.2 Uniquely DecodableCodes .......++++55 3.3 PrefixCodes .. 2... eee ee es 4 Kraft’s Inequality 4.1 Proof of Kraft’s inequality. ............. 5 Optimal Codes our o on 10 1 Introduction to Source Coding Nowadays many digital streams contain a lot of redundant information. For example, an image file can contain more 0’s then 1’s: 0,0, 0, 1, 0,0, 1, 1, 0,0, 0, 1,0, 1,0 We wish to squeeze out the redundant information to minimize the amount of data to be stored and transmitted. This is where source coding comes into picture. Source coding deals with how to represent this information as accurately as possible and using as few bits as possible. This process is called as compression. Formally a Source Code C for a random variable X is defined as a mapping C:X > {0,1}* where ¥ is a sequence of symbols from an information source and {0,1}* is the set of all finite bit strings containing 0’s and 1's. There are two main types of source coding: 1. Lossless: In this case the source symbols can be exactly recovered from the binary bits. 2. Lossy: In this case the source symbols are recovered from the binary bits but with some distortion. Few of the commonly used Source Coding techniques are as follows: + Huffman Coding + Lempel-Ziv Coding + Fano Coding + Arithmetic Coding * Shannon Coding 3.2 Uniquely Decodable Codes To understand what are uniquely decodable codes we need to understand what is meant by code extension. Mathematically the extension C* of a code C is defined as a mapping C*: X* + {0,1}* (where %* is the set of all finite length strings of the set .”) defined by, C21 2%3.....Ln) = C(a1)C(x2)C(a3)....C (an) (2) where the right hand side represents concatenation. For example, let V = {1,2} and consider the code C given by C(1) = 00 and C(2) = 10. Then its extension would be a mapping from 7* = {1,2}* to {0, 1}* obtained by replacing each 1 by 00 and each 2 by 10. So C(12) = C(1)C(2) = 0010 and C(121) = C(1)C(2)C(1) = 001000 Now let us understand what is meant by Uniquely decodable codes. A code is said to be Uniquely Decodable if its extension is non-singular i.e. we should not get the same codeword if we use two different permutations from the set **. For example, consider the following mapping from the set {1,2,3,4} to the set {0,1}*, Symbol | Codeword X=1 00 X=2 100 X=3 1 X=4 01 We can see that each element of 1 has a unique codeword associated with it and hence it is non-singular. However, it is not uniquely decodable because the sequence 100 could represent either the symbol 2 or the con- catenation of codewords for symbols 3 and 1 (C(31) = C(3)C(1) = 100). 3.3 Prefix Codes Prefix codes (or sometimes referred to as instantaneous codes) are such codes wherein no codeword is a prefix of any other codeword. They are also called instantaneous codes because of the fact that they can be parsed into codewords without the need for lookahead. Consider the mes- sage 10000001 that is encoded using the codeword set {1, 1000, 00}. For this message the first codeword is 1, but this cannot be determined until the 8th message bit is received. So this code cannot be considered as an instantaneous code. Alternatively prefix codes can also be defined as those codes which re- quire no special markers/characters (such as commas, dots) to identify each word in an accurate sequence. For example, consider the following mapping from the set {1,2,3,4} to the set {0,1}*, Symbol | Codeword X=1 01 X=2 000 X=3 10 X=4 111 Here we can see that no codeword is a prefix of another codeword and hence the above code is a prefix code. Prefix Code C Uniquely Decodable Codes © Non-Singular Codes © All Codes Example-3: Classify the following codes as non-singular, uniquely decodable or prefix codes. xX A B Cc X=1] 00 1 01 X=2] 11 | 01 | 101 X=3] 001 1 | 110 X=4] 110 | 001 | 1001 X=5/1010/010] 010 Solution: Here for the mapping A, we can see that all codes are unique and hence A is a non-singular codes. However we can see that 00, the codeword for X = 1 is a prefix for 001, the codeword for X = 3, hence this is not a prefix code. Also for the mapping A the extension of the codewords is also non-singular and hence it is uniquely decodable as well. For the mapping B, we can see that the codeword for X = 1 and X = 3 are the same and hence mapping B is not non-singular. Using the above set relation we can say that mapping B is also not uniquely decodable and not a prefix code. For the mapping C, we can see that all the codewords are unique and hence the mapping is non-singular. Also no codeword is a prefix of any of the other codewords and so it is a prefix code. Since the code is is a prefix code so it is also uniquely decodable as well. 4 Kraft’s Inequality In coding theory, Kraft’s Inequality gives a necessary and sufficient condi- tion for the existence of a prefix code. Mathematically itis stated as follows, 5 Optimal Codes Optimal codes are those prefix codes for which the expected length is minimum i.e. we need to minimize L(C) =) pili (12) over all integers 11, lz, Is, ...., ln satisfying the Kraft’s inequality So2t<1 (13) i=1 Lagrange Multipliers Lagrange Multipliers are used to find the extremum of a multivariate function f when there is some constraint on the input values of the func- tion. Mathematically we define a Lagrangian function F'(Y, \) as follows, FIY,A) = (VY) — Ag(¥) (14) where Y is the set of input parameters, \ is the lagrange multiplier and g(Y) is the constraint function. Assumptions for the proof: (i). The length of the codewords need not necessarily be integers. (ii). Assume equality for equation (13). Applying the definition of Lagrange Multipliers to find the minimum expected length we get, F= So pili- AS 62% (15) a i=l 10 Partial differentiation with respect to /; gives, OF — =p,— 2" In2 (16) Ol; To find the extremum we equate the above equation to 0 pi — A2" In2 =0 (17) On rearrangement we get, _L Pi gt = 18 Aln (18) Substituting the above value in equation (13) we get, Di 1 ind And | (19) So we get the value of \ as log, e. Substituting the value of in (18) we get p; = 2-"' i.e. 1;* = — log, p;. So now the expected codeword length can be calculated as follows, L*= > Pili® =— DPi logypi= H(X) (20) where H(X) denotes the entropy of the random variable X. So from the above proof we can conclude that the expected length of a binary prefix code for a random variable X is greater than or equal to its entropy i.e. H(X). 11 Example-5: Given a source with 5 symbols s1, 52, 53, 54, 85 with probabili- ties 0.25, 0.25, 0.25, 0.125, 0.125 respectively. What is the optimal length of the code? Solution: The optimal length of the code can be calculated using the formula stated in (20). Substituting the values of the probabilities in the formula we get, L* = —S~ pjlogy pi (21) L* = (0.25)(2) + (0.25)(2) + (0.25)(2) + (0.125)(3) + (0.125)(3) So the optimal length of the codeword having the above probabilities would be 2.25. 12