$($a$)$ Figure $1$ below plots the loss when batch gradient descent is used for training. Which

optimizers plots the loss in figure $2$ and $3.[1]$

Figure $1$

$($b$)$ In figure $2,$ assume that the learning rate used was $0\mathrm{.}5.$ Redraw the plot to show the

effect of increasing the learning rate to $2$ and decreasing the learning rate to $0\mathrm{.}01.[2]$

$($c$)$ You are given a simple neural network with a single hidden layer containing two neurons,

and an output layer containing one neuron. All neurons use the sigmoid activation

function. The weights and biases of the network are as follows: $[3]$

$$ Weights from input to hidden layer: w$1=0\mathrm{.}5,$w$2=0\mathrm{.}6$

$$ Biases in hidden layer: b$1=0\mathrm{.}1,$ b$2=-0\mathrm{.}2$

$$ Weights from hidden layer to output: w$3=0\mathrm{.}7,$ w$4=-0\mathrm{.}8$

$$ Bias in output layer: b$3=0\mathrm{.}3$

Given an input x $=0\mathrm{.}75,$ calculate the output of the network and mean squared loss

if the desired output is $1\mathrm{.}25.$ Use the sigmoid activation function. What will be the

effect in the loss if the loss function used is

L$($w$,$ b$)=1$

$2($d $$y$)2+||$w$2||$