Please answer question $4$

$(4)$Answer why entropy is maximized in a uniform distribution. $(25$points$)$

Self$-$Information

In information theory, the entropy of a random variable is the average level of "information",

"surprise", or "uncertainty" inherent to the variable's possible outcomes.

The self$-$information is a measure of the information content associated with the outcome of a

random variable. The self$-$information of an event $x=x$ is defined as:

$I(x)=-lo{g}_{2}P(x=x)$

The choice of base for log$,$ the logarithm, varies for different applications. Base $2$ gives the unit of

bits. We can quantify the amount of uncertainty in an entire probability distribution using the

Shannon entropy.

Shannon Entropy

Given a discrete random variable $x,$ with possible outcomes $x_1,$dots,$x_n,$ which occur with probability

$P(x=x_1),$dots,$P(x=x_n)$ the entropy of $x$ is formally defined as:

$H(x)=-{\displaystyle \underset{i=1}{\overset{n}{}}}P(x=x_i)lo{g}_{2}P(x=x_i)$

where ${\displaystyle \underset{?}{\overset{?}{}}}$ denotes the sum over the variable's possible values. An equivalent definition of entropy is

the expected value of self$-$information of a variable.

Problem:

Study Shannon Entropy yourself in more detail and calculate the entropy of two random

variables $x$ and $Y,$ respectively.

Answer why entropy is maximized in a uniform distribution. $(25$ points$)$