Ex $5\mathrm{.}4$: Activation and weight scaling. Consider the two hidden unit network shown in

Figure $5\mathrm{.}62,$ which uses ReLU activation functions and has no additive bias parameters. Your

task is to find a set of weights that will fit the function

$y=|x_1+1\mathrm{.}1x_2|.$

Can you guess a set of weights that will fit this function?

Starting with the weights shown in column $b,$ compute the activations for the hid$-$

den and final units as well as the regression loss for the nine input values $(x_1,x_2)in$

$\{-1,0,1\}\{-1,0,1\}.$

Now compute the gradients of the squared loss with respect to all six weights using the

backpropagation chain rule equations $(5\mathrm{.}65-5\mathrm{.}68)$ and sum them up across the training

samples to get a final gradient.

What step size should you take in the gradient direction, and what would your update

squared loss become?

Repeat this exercise for the initial weights in column $($c$)$ of Figure $5\mathrm{.}62.$

Given this new set of weights, how much worse is your error decrease, and how many

iterations would you expect it to take to achieve a reasonable solution?

Figure $5\mathrm{.}63$ Function optimization: $($a$)$ the contour plot of $f(x,y)=x^2+20y^2$ with

the function being minimized at $(0,0)$; $($b$)$ ideal gradient descent optimization that quickly

converges towards the minimum at $x=0,y=0.$

Would batch normalization help in this case?