Exercise 2 on the Cross-entropy cost function in NNDL 3

FromChapter 3 of NNDL, answer thesecond question only, which starts like:

"In the single-neuron discussion at the start of this section, I argued that the cross-entropy is small if(z)y

(z)yfor all training inputs..."

For this problem, you can assumeyas a single neuron (in the output layer) or a vector (i.e., the whole output layer).

• If you assume the former, you can do a rigorous proof by using calculus and minimizing the derivative of the function. But if you are not comfortable with calculus,you can pick at least three values fory(between 0 and 1), and for each value ofy, you should compute the Cross-entropy value using thatyand several varying values ofa(e.g. 0.1, 0.2, 0.3,..., 0.9).
• If you assume the latter, you can do a formal proof by calculus as well (although the derivative function will have several variables), but I recommend using Information Theory to prove the formula will minimize whenyandaare equal.