Elements of Information Theory: Introduction - Lecture Slides | EE T220, Study notes of Microelectronic Circuits

Download Elements of Information Theory: Introduction - Lecture Slides | EE T220 and more Study notes Microelectronic Circuits in PDF only on Docsity! January 13, 2009 WVU EE 568 1 Elements of Information Theory: Introduction Course number: EE 568 Instructor: Natalia Schmid Concepts • Information Theory (IT) is a subject suggesting a common mathematical approach to the study of data collection and manipulation of information. • It provides theoretical basis for activities such as – Observation, measurement, – Data compression, data storage – Communication – Estimation – Decision making – Pattern recognition and learning January 13, 2009 WVU EE 568 2 Concepts • Increasing System Complexity • Performance requirements • Mathematical Models • IT is the study of how the laws of probability, and mathematics in general, describe limits on the design of information processing and transmission systems . • (ex. IT tells how one may design strategies in a communication system by using the laws of large numbers to overcome the noise imposed by nature and the following errors that may occur in the system). January 13, 2009 WVU EE 568 3 Communication Channel • The primary scope of the IT are communication systems. • Two fundamental questions: – What is the data compression limit? – What is the transmission limit in a noisy media? January 13, 2009 WVU EE 568 4 January 13, 2009 WVU EE 568 9 Example: CT Imaging Ex.: Brachytherapy FBP Model-Based Approach A more precise probabilistic model (Poisson distribution with unknown mean) will provide better estimation results. January 13, 2009 WVU EE 568 10 Example: Radar http://www.sandia.gov/RADAR/whatis.html Dimensions: (1)Range: Send pulse and listen to echo. Resolution ~ pulse width. (2) Azimuth: Require large antenna (hundreds of meters). Synthetic Aperture: Fine resolution is synthesized while airplane is flying. January 13, 2009 WVU EE 568 11 Example: ATR SAR images displayed as variance images Targets Given orientation, complex Gaussian model provides a good fit to SAR data and high recognition performance. January 13, 2009 WVU EE 568 12 Information Theory • Claude Shannon (1948) • Mathematical Theory about limits of Communication Theory • Characteristics: – Emphasis on probability theory – Optimization Theory – Concerns with encoder and decoder – Proves existence of encoders and decoders – Limits but no indication how to achieve them (involves computationally complex procedures to prove results). January 13, 2009 WVU EE 568 13 Information Theory • Deals with theory = mathematical models • Appropriate way: Physical Model Mathematical Model ? • Our approach: - Study a family of simple sources and channels - More complicated • Theory is Useful: - Framework to model real sources and channels - Points to types of tradeoffs (in encoders/decoders) January 13, 2009 WVU EE 568 14 Source and Channel Coding • Study source and channel models separately Source Source encoder Channel encoder Channel Binary data Binary data Channel decoder Source decoder Destination Source encoder: binary representation; number of bits Channel encoder: reliable communication and reproduction January 13, 2009 WVU EE 568 19 Capacity • Discrete channel capacity (Ch. 8, CT’91) • Continuous channel capacity (Ch. 10, CT’91) Capacity: maximum average number of bits that can be transmitted with probability close to zero. • Capacity of continuous channel can be approached (proper modulator and demodulator) ex.: turbo coding/decoding • Channel Coding Theorem: If data are communicated at R<C, then the input sequence can be reconstructed with asymptotically small P(error). • Joint Source-Channel Coding: If H<C, then the source sequence can be reconstructed on the output of decoder with arbitrary small P(error). January 13, 2009 WVU EE 568 20 Modern Information Theory Probability and Statistics Decision Theory EconomicsPhysics (AEP) Communication Theory Computer Science Information Theory Kullbuck, Sanov, Chernoff Shannon, Huffman, Gallager, Cover Breiman, Cover, Algoet, Barron Kolmogorov, Chaitin Shannon, McMillan, Breiman Sanov, Csiszar IT contributes and gets contribution from: January 13, 2009 WVU EE 568 21 AEP • Law of Large Numbers (EE513) applied to the sequence: ∑ = − n k kXp n 1 )(log 1 • Applications: efficient (elegant) proves of source and channel coding theorems • Typical Set: set of long sequences with probability close to and cardinality close to nH− 2 nH 2 January 13, 2009 WVU EE 568 22 Entropy Rates • AEP results are established for i.i.d. sequences. • If variables X(1), X(2) , …. are not i.i.d., define entropy rate ),,,( 1 lim 21 n n XXXH n K ∞→ January 13, 2009 WVU EE 568 23 IT and Statistics Sanov’s Theorem (Large Deviations): Given sequences         ∑ = n k kX n 1 1 By the Law of Large Numbers ][ 1 1 1 XEX n Pn k k →         ∑ = ...}{ diiX k ∝ What is the probability         >−∑ = ε][ 1 1 1 XEX n P n k k for large n? It converges exponentially fast to zero. Sanov’s Theorem finds the exponent. January 13, 2009 WVU EE 568 24 IT and Statistics Hypothesis Testing Problem: R is an observed random variable.    22 11 ~: ~: pRH pRH • Optimal test statistics (EE591K): likelihood ratio (Neyman-Pearson, minimum probability of error) • Conditional probabilities of error: False Alarm and Missed Detection • Sanov’s theorem provides the rates of convergence for the FA and MD