Stochastic gradient descent $($SGD$)$ is a simple but widely applicable optimization technique. For example, we

can use it to train a Support Vector Machine. The objective function in this case is given by:

$J()=[\frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{}}}Los{s}_h(y^{(i)}*x^{(i)})]+\frac{}{2}|||{|}^2$

where $Los{s}_h(z)=$max$\{0,1-z\}$ is the hinge loss function, $(x^{(i)},y^{(i)})$ with for $i=1,$dotsn are the

training examples, with $y^{(i)}in\{1,-1\}$ being the label for the vector $x^{(i)}.$

For simplicity, we ignore the offset parameter ${}_0$ in all problems on this page.