Backpropagation We want to train a simple deep neural network $f_w(x)$ with $w=(w_1,w_2,w_3{)}^{TT}in{R}^3$ and xinR, defined as:

$f_w(x)$:$=w_3{}_2(w_2{}_1(w_{1}x))$

where ${}_1(u)={}_2(u)=\frac{1}{1+\text{e}\text{x}\text{p}(-u)},$ i$.$e$.,$ sigmoid activation. You may denote $x_1$:$=w_{1}x$ and $x_2$:$=w_2{}_1(x_1)$ for notational convenience.

$($a$)$ Illustrate a directed acyclic graph corresponding to the computation of $f_w(x).$

$($b$)$ Compute $\frac{\text{d}\text{e}\text{l}{}_1}{\text{d}\text{e}\text{l}\text{u}}$ and provide the answer in two different forms: $($i$)$ an expression using only $u$ and the exponential functions; and $($ii$)$ an expression using only ${}_1(u).$

$($c$)$ Describe briefly what is meant by a forward pass and a backward pass?

$($d$)2pt$ Compute $\frac{del{f}_w}{del{w}_3}.$ Which result should we retain from the forward pass in order for efficiently computing this derivative?

$($e$)$ Compute $\frac{del{f}_w}{del{w}_2}$ using the second option in Problem $($b$).$ Which results should we retain from the forward pass in order for efficiently computing this derivative?

$($f$)[5$pt$]$ Compute $\frac{del{f}_w}{del{w}_1}$ using the second option in Problem $($b$).$ Which results should we retain from the forward pass in order for efficiently computing this derivative? In what order should we compute the derivatives $\frac{del{f}_w}{del{w}_1},\frac{del{f}_w}{del{w}_2}$ and $\frac{del{f}_w}{del{w}_3}$ in order for maximizing computational efficiency? How is this order related to the forward pass?