Question: Consider again Example 9.4, where we used a softmax output function (S_{L}) in conjunction with the cross-entropy loss: (C(boldsymbol{theta})=-ln g_{y+1}(boldsymbol{x} mid boldsymbol{theta})). Find formulas for
Consider again Example 9.4, where we used a softmax output function \(S_{L}\) in conjunction with the cross-entropy loss: \(C(\boldsymbol{\theta})=-\ln g_{y+1}(\boldsymbol{x} \mid \boldsymbol{\theta})\). Find formulas for \(\frac{\partial C}{\partial \boldsymbol{g}}\) and \(\frac{\partial \boldsymbol{S}_{L}}{\partial \boldsymbol{z}_{L}}\). Hence, verify that:
\[ \begin{equation*} \frac{\partial \boldsymbol{S}_{L}}{\partial \boldsymbol{z}_{L}} \frac{\partial C}{\partial \boldsymbol{g}}=\boldsymbol{g}(\boldsymbol{x} \mid \boldsymbol{\theta})-\boldsymbol{e}_{y+1} \tag{333} \end{equation*} \]
where \(\boldsymbol{e}_{i}\) is the unit length vector with an entry of 1 in the \(i\)-th position.
Step by Step Solution
3.47 Rating (150 Votes )
There are 3 Steps involved in it
Direc... View full answer
Get step-by-step solutions from verified subject matter experts
