Assume you are training a deep neural network using stochastic gradient descent (a) Sketch a typical curve which shows the training loss as a function of training steps, for the following cases: too small of a learning rate, to large of a learning rate, optimal learning rate. (b) Explain two reasons the training cost might go up when performing stochastic gradient descent, and what can be done to partly alleviate these problems. (c) Explain the difference between batch, mini-batch and stochastic gradient descent. Why do we then to usually use mini-batch? (d) One of the problems with backpropagation is the vanishing gradient when networks get very deep. Explain how GoogLeNet and Resnet attempt to alleviate this problem.