6. Claude Shannon, widely credited as the founder of information theory, introduced the concept of information...

Transcribed Image Text:

6. Claude Shannon, widely credited as the founder of information theory, introduced the concept of information entropy in his 1948 paper "A Mathematical Theory of Communication". If X is a discrete random variable having support Sx and PDF f, then the entropy (measured in bits) of X is log Hx - E [1062 (IX))] [log2 = Entropy gives a measurement of expected level of surprise in the possible outcomes of X and acts as a bound on performance of lossless compression. (a) Find the entropy of the outcome of flipping a fair coin. (b) Find the entropy of the outcome of rolling a fair, 6-sided die. In(O) (c) Find the entropy of X ~Poi(2). (Hint: log2() = In(2) 6. Claude Shannon, widely credited as the founder of information theory, introduced the concept of information entropy in his 1948 paper "A Mathematical Theory of Communication". If X is a discrete random variable having support Sx and PDF f, then the entropy (measured in bits) of X is log Hx - E [1062 (IX))] [log2 = Entropy gives a measurement of expected level of surprise in the possible outcomes of X and acts as a bound on performance of lossless compression. (a) Find the entropy of the outcome of flipping a fair coin. (b) Find the entropy of the outcome of rolling a fair, 6-sided die. In(O) (c) Find the entropy of X ~Poi(2). (Hint: log2() = In(2)