(a) Consider minimizing the function f(w)=w - 5w+3 using gradient descent (graph of f shown below)....

Transcribed Image Text:

(a) Consider minimizing the function f(w)=w² - 5w+3 using gradient descent (graph of f shown below). Say the current value of w is equal to 5. What is the derivative of f at this point? (show your work) If our step size a = -0.1, what is the new value of w after this single gradient descent update? (b) Say we are part way through SGD for a linear regression problem with p= 1. The current weights are woand w₁=-. Next we are analyzing (z. y.)= (-1,2). If a = 0.1, what are the new weights after considering this point? Show work for full credit: Wo Explain your result geometrically (include a sketch of the model and a discussion of cost): (c) Assuming n training examples, p features, and T iterations needed for convergence, how long does it take to compute the stochastic gradient descent solution (i.e. w) for linear regression? Answer using big-O notation and briefly explain your reasoning for full credit. In this question we will analyze SGD for a simple (meaning p = 1) linear regression problem. We have n = 2, where (ri, y1)= (2, 1) and (x2, 32)= (1, -1) (plotted below), and wish to fit a linear model to this data. (Note the different scales on z and y below.) . (X2.Y2) = (1,-1) (d) Before we begin SGD, we will set wo - w₁0. At this point, what is the numerical value of the cost function for linear regression? Our cost function for linear regression is: J(w) == • (X₁,Y₁) = (2,1) n (hw (x₁) - y)² (e) For SGD we will use a = 1 (learning rate). Using (21, 31), compute the SGD updates (show your work) to find new values for wo and w₁. Use these new values to draw the current linear model on the plot above and label it M₁ (model 1). (a) Consider minimizing the function f(w)=w² - 5w+3 using gradient descent (graph of f shown below). Say the current value of w is equal to 5. What is the derivative of f at this point? (show your work) If our step size a = -0.1, what is the new value of w after this single gradient descent update? (b) Say we are part way through SGD for a linear regression problem with p= 1. The current weights are woand w₁=-. Next we are analyzing (z. y.)= (-1,2). If a = 0.1, what are the new weights after considering this point? Show work for full credit: Wo Explain your result geometrically (include a sketch of the model and a discussion of cost): (c) Assuming n training examples, p features, and T iterations needed for convergence, how long does it take to compute the stochastic gradient descent solution (i.e. w) for linear regression? Answer using big-O notation and briefly explain your reasoning for full credit. In this question we will analyze SGD for a simple (meaning p = 1) linear regression problem. We have n = 2, where (ri, y1)= (2, 1) and (x2, 32)= (1, -1) (plotted below), and wish to fit a linear model to this data. (Note the different scales on z and y below.) . (X2.Y2) = (1,-1) (d) Before we begin SGD, we will set wo - w₁0. At this point, what is the numerical value of the cost function for linear regression? Our cost function for linear regression is: J(w) == • (X₁,Y₁) = (2,1) n (hw (x₁) - y)² (e) For SGD we will use a = 1 (learning rate). Using (21, 31), compute the SGD updates (show your work) to find new values for wo and w₁. Use these new values to draw the current linear model on the plot above and label it M₁ (model 1).

Answer rating: 100% (QA)

a To find the derivative of the function f w w 25 w 3 at a point where w 5 we can differentiate it with respect to w f w 2 w 5 Now lets calculate the derivative at w 5 f 52551055 So the derivative of ... View the full answer