Problem # $1.$ Backpropagation.

$1($a$)$ For this problem, we will follow the tutorial on autograd given at: PyTorch

autograd.

We define neural networks using a l$(.)$ that takes a vector and outputs a

scalar as in traditional optimization methods of studying f $($x$).$ However, neural

networks use function composition to propagate the inputs to the outputs. In

the autograd example, the notation is as follows:

l$(.)$ is the output scalar function $($e$.$g$.,$ mean squared error$).$

y denotes the top output layers of the neural network..

x denotes the input to the neural network.

In order to train Neural$-$networks using the loss function, we need to compute

gradients of l$(.)$ with respect to all of the variables. In the simplest form, we

use functional composition to express l$(.)$ for a specific input: l$($y$($x$)).$ When

designing neural$-$networks, we can assume that the derivatives of each level are

available to us$.$ This means that we assume that the following derivatives are

given to us:

$$l

$$yi

$,$yi

$$xj

for i $=1,2,...,$ m j $=1,2,...,$ n$.$

We then have to determine all other derivatives. To determine the dependence

of the loss$-$function based on the input, we use:

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$l

x$1$

$...$

$$l

xn

$$

$$

$$

$$

$$

$$

$$

$$

$$

$=$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$y$1$

$$x$1$

$...$ym

$$x$1$

$.........$

$$y$1$

$$xn

$...$ym

$$xn

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$

$$l

y$1$

$...$

$$l

ym

$$

$$

$$

$$

$$

$$

$$

$$

$$

$.$

Multiply out the matrix by the vector to derive an expression for $$l$/$xi$.$ How

many terms do you have?

$1($b$)$ Let us introduce one more layer. Suppose that the loss is given by l$($y$($g$($x$)).$

Apply $1($a$)$ recursively to derive an expression for $$l$/$xi for this case. Assume

that g outputs a p dimensional vector that is input to y$.$ Assume that all other

dimensions remain as given in $1($a$).$ You will need to clearly label all matrix

dimensions in your derivation.

$1($c$)$ Repeat $1($b$)$ for l$($y$($g$2($g$1($x$)))).$ Assume that the output of g$2$ is p$2$ dimen$-$

sional and the output for g$1$ is p$1$ dimensional. Make sure to clearly indicate all

the matrix dimensions.