Information Theory with Applications: Homework Set 2 - Fall 2008, Assignments of Mathematics

Download Information Theory with Applications: Homework Set 2 - Fall 2008 and more Assignments Mathematics in PDF only on Docsity! Information Theory with Applications MATH 6397 – Fall 2008 October 13, 2008 Homework Set 2, due Thu Oct 16, 2008 Unless noted, exercises are taken from the textbook. Additional hints may be available there. 1. p. 115 Ex. 3.4 Find an example of a regular (invertible) K-ary vari- able length code for a discrete memoryless source {Xj} with marginal Q on a finite non-empty alphabet A such that E[`(X1)] < HK(Q). 2. p. 117 Ex. 3.10 Let T be the code tree for a prefix code which encodes a discrete memoryless source with marginal Q on a finite alphabet A. Label the nodes in the tree by the sum of the probabilities of the descendant leaves, as discussed in class. Prove the following: Denumerate the nodes in an arbitrary fashion. Let the chil- dren of node j be i1, i2, . . . ik and their respective proba- bilities qi1 , qi2 , dots qik . If qj 6= 0, define the conditional probability measure Qj supported on the children by nor- malizing their probabilities, q′i1 = qi1/qj , q ′ i2 = qi2/qj , etc., then H(Q) = #nodes∑ j=1 qjH(Qj) . 3. p. 117 Ex. 3.11 With the same notation as in the preceding exercise, show that if T is an optimal code tree (average code length assumes the minimum value) then for any two nodes i and j and their levels (root has level zero) li and lj , −1− qi qj ≤ li − lj ≤ 1 + qj qi . 1