a$)$ In a simple Multi$-$Layer$-$Perceptron neural network model with $8$ neurons in the input layer, $5$ neurons in the hidden layer and $1$ neuron in the output layer. What is the size of the weight matrices between hidden output layer and input hidden layer?

$20\%$

b$)$ Assume a simple Feed Forward model with $3$ neurons and inputs $=1,2,3.$ The weights to the input neurons are $4,5$ and $6$ respectively. Assume the activation function is a linear constant value of $3.$ What will be the output?

$20\%$

c$)$ What steps can we take to prevent overfitting in a Neural Network?

$20\%$

d$)$ The recurrent neural network shown in Figure Q$2$ has $1$ hidden layer.

The weights on the connection between nodes is denoted by $w$ as indicated in the following matrices:

$w_{12}=\left[\begin{array}{cc}0\mathrm{.}4& 0\mathrm{.}1\\ -0\mathrm{.}6& 0\mathrm{.}6\\ 2\mathrm{.}0& 1\mathrm{.}0\end{array}\right],w_{23}=\left[\begin{array}{cc}0\mathrm{.}1& 0\mathrm{.}5\\ 0\mathrm{.}4& 0\mathrm{.}2\end{array}\right],w_f=\left[\begin{array}{c}0\mathrm{.}01\\ -0\mathrm{.}02\end{array}\right]$

Assume the hidden and output layers neurons have a sigmoid activation function given as follows:

$f(x)=\frac{1}{1+\text{e}\text{x}\text{p}(-x)}$

If the network is tested with an input vector $x(1)=[1\mathrm{.}0,2\mathrm{.}0,0\mathrm{.}1],$ and $x(2)=[2\mathrm{.}0,2\mathrm{.}2,0\mathrm{.}4],$ calculate the predicted outputs of the network $Y^{\text{'}}$ and $t=1$ and $t=2$;

$40$

$3.$

a$)$ Explain the meaning of Leaky ReLU activation function used in Deep Neural Networks. Give the mathematical expression of this function, and its graphical representation.