Please answer a, b, c, d and e. Thank you.

3. (16%) This question is about forward and backward propagation of a basic neural network. Specifically, the following graph shows the architecture of a neural network with a single hidden layer. The input layer has three neurons x=(x1,x2,x3). The hidden layer has two neurons h=(h1,h2). The output layer has one unit y. a) We would like to use ReLU activation function for the hidden layer and the output layer. Please write down the mathematical formula (z) for ReLU function. ( z just generally denotes a variable). State two advantages of ReLU activation function compared to logistic sigmoid activation. (3%) b) We use ReLU as the activation function in this network. Moreover, denote by W as the weight matrix connecting input and hidden layer, and V as the weight matrix connecting hidden layer and output unit. Write out the symbolic function of the mapping xy using ,W,V. c) We further design the loss function as (y,t)=21(yt)2 where t is the ground-truth value for the output unit y. Assume the network parameters are initialized as follows. W=[110110]andV=[01] Assume that we have one training sample (x,t) with x=[1,2,1] and t=1. Compute the numerical value of hidden neurons h and output unit y. d) Compute the gradient of the loss function with respect to the weights. In particular, compute the following terms symbolically: - The gradient relative to V, i.e., V - The gradient relative to W, i.e., W - Compute the values numerically for the values of W,V,x,y as given above. e) Does it make sense to initialize all weights in a neural network to 0 ? Why