Neural Networks. $(20$ points$)$ Consider a neural network, with $N_i=2$ input units, $x=$

$(x_1,x_2),N_h=3$ hidden units with ReLU activations and one output unit for regression,

$z_j^H={\displaystyle \underset{k=1}{\overset{N_i}{}}}{W}_{jk}^H{x}_k+b_j^H,u_j^H=$max$\{0,z_j^H\},j=1,$dots,$N_h$

widehat$(y)={\displaystyle \underset{k=1}{\overset{N_h}{}}}{W}_k^O{u}_k^H+b^O,$

$($a$)(5$ points$)$ Suppose that

$W^H=\left[\begin{array}{cc}1& 0\\ 0& 1\\ -1& -1\end{array}\right],b^H=\left[\begin{array}{c}-0\mathrm{.}5\\ -1\\ 3\end{array}\right]$

Write equations for $z_j^H$ in terms of $x$ for $j=1,2,3.$

$($b$)(5$ points$)$ Draw the region of inputs $(x_1,x_2)$ where $u_j^H>0$ for all $j.$

$($c$)(5$ points$)$ Assuming you were given the following training data set, and the parameters of

the hidden layer are as above. What parameter $W^O$ and $b^O,$ the weight and bias for the

output layer that minimizes the MSE? What is the MSE of the training set with those

parameters?

$($d$)(5$ points$)$ You are given data $x,y$ as well as weights and biases Wh$,$ bh for the hidden layer.

Write a few lines of python code to fit wo$,$ bo for the output layer by minimizing the MSE.

You may assume you have a function

beta $=lstsq(A,b)\frac{?}{?}$# Solves the least squares problem $A.$dot$(beta)=b$