Problem $1.$ Consider a simple two$-$layer neural network with one input layer with two nodes, one hidden

layer with two nodes, and one output layer with a single node which we saw in the previous assignment. The

activation function for the first layer is ReLU and for the second layer is a sigmoid function s$($z$)=1/(1+$e

$$z

$).$

The information in the network propagates by the following rules:

$1.$ At the input later, you are given a $2-$dimensional vector x

$[1]$

$.$

$2.$ Then the $2-$dimensional vector z

$[2]$ is computed by multiplying by $2\backslash$times $2$ matrix W$[2]$ and adding a

$2-$dimensional vector of biases b

$[2]$:

z

$[2]=$ W$[2]$x

$[1]+$ b

$[2]$

$.$

$3.$ Then the values of the second layer x

$[2]$ are computed by applying applying the activation function

s$($z$)$ componentwise to the vector z

$[2]$:

x

$[2]=$ ReLU$($z

$[2]).$

$4.$ Then a scalar value z

$[3]$ is computed by multiplying by $1\backslash$times $2$ matrix W$[3]$ and adding a scalar bias b

$[3]$:

z

$[3]=$ W$[3]$x

$[2]+$ b

$[2]$

$.$

$5.$ Finally, the output value x

$[3]$ is computed by applying applying the activation function s$($z$)$ to the

variable z

$[3]$:

x

$[3]=$ s$($z

$[3]).$

You are given the current values of the parameters and the gradients of the loss function:

W$[2]=$

$31$

$02$

$,$ b$[2]=$

$1$

$0$

$,$

W$[3]=$

$11$

$,$ b$[3]=1/2,$

$$C

$$W$[2]=$

$0\mathrm{.}30\mathrm{.}2$

$0\mathrm{.}10\mathrm{.}2$

$,$

$$C

$$b$[2]=$

$0\mathrm{.}1$

$0\mathrm{.}2$

$,$

$$C

$$W$[3]=$

$0\mathrm{.}10\mathrm{.}1$

$,$

$$C

$$b$[3]=0\mathrm{.}1.$

$1$

You want to do one step of the gradient descent method with the learning rate k

$[2]=1\mathrm{.}0$ for the parameters

in the $[2]-$layer, and the learning rate k

$[3]=0\mathrm{.}5$ for the parameters in $[3]-$layer.

$1.$ What are the new values of weights and biases W$[2]$

$,$ b

$[2]$

$,$ W$[3]$

$,$ b

$[3]?$

$2.$ What is the new prediction of the network for the initial input value x

$[1]=[1,1]$T

$?$

How to find the new weights and biases