The gradient descent algorithm is commonly used in back propagation of Artificial Neural Network.

(a) Explain the principle of the gradient descent algorithm. Accompany your explanation with a diagram. Include all the terms and constants that you introduce and comment on the range of values that they can take.

(b)Derive the gradient descent training rule assuming that the target function representation is: o_d = w₀ + w₁x₁ + + w_nx_n. In your derivation, the cost/error function E is defined explicitly, assuming that a set of training examples D is provided, where each training example d D is associated with the target output t_d.

(c) Show that the Least Mean Square (LMS) training rule performs a gradient descent to minimize the cost/error function E defined in (b).