Question $3.$ XOR Problem

In this question, we'll train a neural network using backpropagation by hand. Don't worry, we will only train the network for a single iteration.

Our neural network will have two input features $x_1$ and $x_2,$ and two hidden units $h_1$ and $h_2.$ The hidden units $h_1$ and $h_2$ will use the ReLU activation, and for final prediction $y$ we will use the sigmoid activation to obtain a prediction between $0$ and $1.$

Our training example will look like this:

$-x^{(1)}=[0,0{]}^T,t^{(1)}=0$

$-x^{(2)}=[0,1{]}^T,t^{(2)}=1$

$-x^{(3)}=[1,0{]}^T,t^{(3)}=1$

$-x^{(4)}=[1,1{]}^T,t^{(4)}=0$

Part $($a$)$

Our initial weights and biases will look like this:

$W^{(1)}=\left[\begin{array}{cc}0& 1\\ 1& 1\end{array}\right],b^{(1)}=\left[\begin{array}{c}0\\ 0\end{array}\right],W^{(2)}=\left[\begin{array}{cc}1& 1\end{array}\right],b^{(2)}=[0]$

Compute the forward pass for each of the four examples. That is$,$ for each $x^{(n)},$ compute $h_1,h_2$ and the prediction $y.$ Use the notation $m_1$ and $m_2$ to denote the pre$-$activations values of the hidden states, so that $h_1=$relu$(m_1)$ and $h_2=$relu$(m_2).$ Use $z$ to represent the logit of the prediction, so that $y=(z).$

$2$

You may also use the vector representations $h=\left[\begin{array}{c}{h}_1\\ h_2\end{array}\right],m=\left[\begin{array}{c}{m}_1\\ m_2\end{array}\right],$ but always start with the scalar representations first.

Part $($b$)$

Suppose that we are using a mini$-$batch size of $1,$ and that we are using $L(x^{(4)},t^{(4)})$ to estimate the cost function. Use backpropagation to compute the below terms. For the vector and matrix derivatives, always start with the scalar derivatives of its elements before putting them together in vector or matrix.

Update $($Feb $1)$: Please use the cross$-$entropy loss.

$\frac{?}{\text{b}\text{a}\text{r}}(L)=1\frac{?}{b}$

$ar(y)=\frac{?}{b}$

$ar(z)=\frac{?}{b}$

$ar(w_k^{(2)})=\frac{?}{b}$

$ar(b^{(2)})=\frac{?}{b}$

$ar(h_k)=\frac{?}{b}$

$ar(m_k)=\frac{?}{b}$

$ar(W_{jk}^{(1)})=\frac{?}{b}$

$ar(b_k^{(1)})=$

Part $($c$)$

Assuming a learning rate of $0\mathrm{.}5,$ perform one iteration of weight update based on the values you obtained from part $($b$).$