Given the following neural network with fully connection layer and ReLU activations, including two input units $(11,12),$ four hidden units $($h$1,$ h$2)$ and $($h$3,$ h$4).$ The output units are indicated as $(01,02)$ and their targets are indicated as $($t$1,$ t$2).$ The weights and bias of fully connected layer are called w and b with specific sub$-$descriptors.

bi

b$3$

$21$

W$11$

h$1$

ReLU$-$ h$3$

W$31-$

$01$

w$32$

W$12.$

W$41$

W$21$

W$22$

h$2$

ReLU ha

$$W$42$

$02$

$22$

The values of variables are given in the following table:

Variable i Value

i$2$ W$11$ W$12$ W$21$ W$22$ W$31$ W$32$ W$41$ W$42$ b$1$ ba t$1$ t$22\mathrm{.}0-1\mathrm{.}01\mathrm{.}0-0\mathrm{.}50\mathrm{.}5-1\mathrm{.}00\mathrm{.}5-1\mathrm{.}0-0\mathrm{.}51\mathrm{.}00\mathrm{.}5-0\mathrm{.}5-1\mathrm{.}00\mathrm{.}51\mathrm{.}00\mathrm{.}5$

b$2.$

$($a$)(3$ points$)$ Compute the output $(01,02)$ with the input $(1,2)$ and network paramters as specified above. Write down all calculations, including intermediate layer results.

Solution:

Forward pass:

h$1$ixw$11+$ i$2\backslash$times w$21+$ b$1=2\mathrm{.}0$ x $1\mathrm{.}01\mathrm{.}0$ x $0\mathrm{.}5+0\mathrm{.}5=2\mathrm{.}0$ h$2$i$1$w$12+$ i$2\backslash$times w$22+$ b$2=2\mathrm{.}0$ x $-0\mathrm{.}5-1\mathrm{.}0$x$-1\mathrm{.}0-0\mathrm{.}5=-0\mathrm{.}5$ h$=$ max$(0,$ h$1)=$ h$1=2$

h $=$ max$(0,$ h$2)=0$

$01$h$3$w$31+$ h$4\backslash$times W$41+$b$3=2$ x $0\mathrm{.}5+0$x$-0\mathrm{.}5-1\mathrm{.}0=0$

$02$h$3$ W$32+$ h$4\backslash$times W$42+$ b$1=2$x$-1\mathrm{.}0+0$ x $1\mathrm{.}0+0\mathrm{.}5-1\mathrm{.}5$