Exercise $1$

The figure below shows a multiple layer neural network deployed to predict the outcome of a $3-$class categorical target

variable $y.$ The activation function in both hidden layers is the rectified linear unit $($ReLU$)$ and the output layer uses

a softmax output activation function. The weight and bias parameters are shown along the edges in the network. The

numbers in the output nodes denote the class label corresponding to each node.

Input layer

Hidden layer $1$

Hidden layer $2$

Output layer

Consider a generic input observation vector $x=(x_1,x_2).$ Perform a full forward propagation calculation through

the network for the input observation vector $(0\mathrm{.}1,-0\mathrm{.}7).$

$(20$ marks$)$

What is the effect of removing the bias parameters from the first layer of this network? Justify your answer.

$(10$ marks$)$

The network is trained focusing on fine tuning of a particular hyperparameter. The figures below $($in the next

page$)$ report the loss learning curves corresponding to three different values of this hyperparameter. The value

of this hyperparameter is increasing going from left to right. What is most likely this hyperparameter and how

does it affect the learning and validation processes? Comment briefly.

The table below reports the outputs produced by the network and the associated labels of the target variable

for a sample of $4$ observations. Compute the average loss and the classification accuracy for this sample.