Problem 3.1 (5 pts) Consider a 2-layer fully-connected NN, where we have input x = R1,...

 

Transcribed Image Text:

Problem 3.1 (5 pts) Consider a 2-layer fully-connected NN, where we have input x = R1, hidden feature x2 = Rmx1, output x3 E Rkx1 and weights and bias W = Rmxn, W2 Rkxm, b = Rmx1, b2 E Rkx1 of the two layers. The hidden features and outputs are computed as follows x2 = Sigmoid (Wx1 + b) x3 = W2x2 + b +b2 (1) (2) A MSE loss function L = (t x3)(t - x3) is applied in the end, where t = Rk1 is the target value. Following the chain rule, derive the gradient w' 2' ab' b L L aL in a vectorized format. L Problem 3.2 (5 pts) Replace the Sigmoid function with ReLU function. Given a data x = target value t = [1, 2], weights and bias at this iteration are 3 -1 , b1 21] - 2 - 1 -2 , b -3 1 " == W1 W2 = = = L L L L Following the results in Problem 3.1, calculate the values of L, 1' ' ' b [0, 1, 2], (3) (4) Problem 3.1 (5 pts) Consider a 2-layer fully-connected NN, where we have input x = R1, hidden feature x2 = Rmx1, output x3 E Rkx1 and weights and bias W = Rmxn, W2 Rkxm, b = Rmx1, b2 E Rkx1 of the two layers. The hidden features and outputs are computed as follows x2 = Sigmoid (Wx1 + b) x3 = W2x2 + b +b2 (1) (2) A MSE loss function L = (t x3)(t - x3) is applied in the end, where t = Rk1 is the target value. Following the chain rule, derive the gradient w' 2' ab' b L L aL in a vectorized format. L Problem 3.2 (5 pts) Replace the Sigmoid function with ReLU function. Given a data x = target value t = [1, 2], weights and bias at this iteration are 3 -1 , b1 21] - 2 - 1 -2 , b -3 1 " == W1 W2 = = = L L L L Following the results in Problem 3.1, calculate the values of L, 1' ' ' b [0, 1, 2], (3) (4)