Theory

$(10$ points$)$ Given $H=\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]$ as an input, compute the gradients of the output with

respect. to this input for the following activation layers.

$($a$)$ A ReLU layer

$($b$)$ A Softmax layer

$($c$)$ A Logistic Sigmoid Layer

$($d$)$ A Tanh Layer

$($e$)$ A Linear Layer

$(2$ points$)$ Given $H=\left[\begin{array}{ccc}1& 2& 3\\ 4& 5& 6\end{array}\right]$ as an input, compute the gradient of the output a fully

connected layer with regards to this input if the fully connected layer has weights of

$W=\left[\begin{array}{cc}1& 2\\ 3& 4\\ 5& 6\end{array}\right]$

as biases $b=\left[\begin{array}{cc}-1& 2\end{array}\right]$

$(2$ points$)$ Given target values of

$Y={\left[\begin{array}{c}0\\ 1\end{array}\right]}_{\text{a}\text{n}\text{d}\text{e}\text{s}\text{t}\text{i}\text{m}\text{a}\text{t}\text{e}\text{d}\text{v}\text{a}\text{l}\text{u}\text{e}\text{s}of}$hat$(Y)={\left[\begin{array}{c}0\mathrm{.}2\\ 0\mathrm{.}3\end{array}\right]}_{\text{c}\text{o}\text{m}\text{p}\text{u}\text{t}\text{e}\text{t}\text{h}\text{e}}$

loss

for:

$($a$)$ A squared error objective function

$($b$)$ A log loss $($negative log$-$likelihood$)$ objective function$)$