Consider the strongly convex function y[x] = (x 2). 1. Analytically obtain the global minimizer...

 

Transcribed Image Text:

Consider the strongly convex function y[x] = (x − 2)². 1. Analytically obtain the global minimizer of the function via taking the gradient of the function and finding the root of the gradient. 2. Assume that you have written a code for gradient descent over this function and xo=0 is chosen as the initial point. Mimic what the code would do and fill out the following steps for three different choices of step size (n = {0.001, 0.1,5}): Iteration 1: x₁ = xo − nVy[xo] ⇒ X₁ =? Iteration 2: x₂ = x₁ − nVy[x₁] ⇒ x₂ =? Iteration 2: x3 = x₂ − nVy[x₂] ⇒ X3 =? Iteration 2: x4 = x3 − nVy[x3] ⇒ X4 =? Iteration 2: *5 = x4 − nVy[x4] ⇒ X5 =? you need to generate 15 numbers (5 iterations for each choice of step size) " 3. What is the difference in terms of the behavior of the solutions (i.e., x₁, · x5) for the three aforementioned choices of step size? Hint: How is the sequence approaching or bouncing back and forth around the global minimizer? Hint: 4. Given the above intuition, what do you think about this statement: "It is always better to choose a smaller step size for gradient descent". Hint: think about the speed of conver- gence vs. accuracy! Sometimes there is no definite Yes/No answer and we have to study the tradeoff. Consider the strongly convex function y[x] = (x − 2)². 1. Analytically obtain the global minimizer of the function via taking the gradient of the function and finding the root of the gradient. 2. Assume that you have written a code for gradient descent over this function and xo=0 is chosen as the initial point. Mimic what the code would do and fill out the following steps for three different choices of step size (n = {0.001, 0.1,5}): Iteration 1: x₁ = xo − nVy[xo] ⇒ X₁ =? Iteration 2: x₂ = x₁ − nVy[x₁] ⇒ x₂ =? Iteration 2: x3 = x₂ − nVy[x₂] ⇒ X3 =? Iteration 2: x4 = x3 − nVy[x3] ⇒ X4 =? Iteration 2: *5 = x4 − nVy[x4] ⇒ X5 =? you need to generate 15 numbers (5 iterations for each choice of step size) " 3. What is the difference in terms of the behavior of the solutions (i.e., x₁, · x5) for the three aforementioned choices of step size? Hint: How is the sequence approaching or bouncing back and forth around the global minimizer? Hint: 4. Given the above intuition, what do you think about this statement: "It is always better to choose a smaller step size for gradient descent". Hint: think about the speed of conver- gence vs. accuracy! Sometimes there is no definite Yes/No answer and we have to study the tradeoff.