Exercises

We can explore the connection between exponential families and the softmax in some more depth.

Compute the second derivative of the cross$-$entropy loss $l(y,$hat$(y))$ for the softmax.

Compute the variance of the distribution given by softmax $(0)$ and show that it matches the second derivative compt

Assume that we have three classes which occur with equal probability, i$.$e$.,$ the probability vector is $(\frac{1}{3},\frac{1}{3},\frac{1}{3}).$

What is the problem if we try to design a binary code for it$?$

Can you design a better code? Hint: what happens if we try to encode two independent observations? What if we ence

Softmax is a misnomer for the mapping introduced above $($but everyone in deep learning uses it$).$ The real softmax is defined $log(exp(a)+$exp$(b)).$

Prove that RealSoftMax $(a,b)>$max$(a,b).$

Prove that this holds for ${}^{-1}$ RealSoftMax $(a,b),$ provided that $>0.$

Show that for $$ we have ${}^{-1}$ RealSoftMax $(a,b)$max$(a,b).$

What does the soft$-$min look like?

Extend this to more than two numbers.

Discussions