This is MOCK only NOT Assignment NO marks

5. Question 5: (a) The gradient descent algorithm is usually used for training neural networks. The gradient descent algorithm for finding the local minimum of a continuous and differentiable loss function LO) is given below 1. Init current solution ecur, e.g., by random initialization Repeat until converge: al 2. Compute gradient: g = acur 3. Compute a new solution: Anew - ng 4. Assign new solution to Acur: Acur = = Anew in which > 0 in the step 3 is the learning rate. At the step 3 in the above algorithm, to get the new solution Anew, we move the current solution Acur in the direction that is opposite with the gradient direction. Explain (mathe- matically) why we should move Acur in that direction. (15 marks) (b) Sigmoid function is a common activation function used in trainning neural networks. (i) Write the sigmoid function (ii) What is the output range of the sigmoid function. (iii) Compute derivative of the sigmoid function. (5 marks) 5. Question 5: (a) The gradient descent algorithm is usually used for training neural networks. The gradient descent algorithm for finding the local minimum of a continuous and differentiable loss function LO) is given below 1. Init current solution ecur, e.g., by random initialization Repeat until converge: al 2. Compute gradient: g = acur 3. Compute a new solution: Anew - ng 4. Assign new solution to Acur: Acur = = Anew in which > 0 in the step 3 is the learning rate. At the step 3 in the above algorithm, to get the new solution Anew, we move the current solution Acur in the direction that is opposite with the gradient direction. Explain (mathe- matically) why we should move Acur in that direction. (15 marks) (b) Sigmoid function is a common activation function used in trainning neural networks. (i) Write the sigmoid function (ii) What is the output range of the sigmoid function. (iii) Compute derivative of the sigmoid function