## Part $3.$ Model Building: Backwards Pass

We are ready to complete the function that computes the backward pass of

our model!

You should start by reviewing the lecture slides on backpropagation.

One difference between the slides and our implementation here is that the

slides express the required computations for computing the gradients of

the loss for a $*$single data point$*.$

However, our implementation of backpropagation is further vectorized to

compute gradients of the loss for a $*$batch consisting of multiple data points$*.$

We begin with applying the backpropagation algorithm on our forward pass

steps from earlier. Recall that our model's forward pass is as follows:

$\backslash$begin$\{$align$*\}$

$\backslash$bf$\{$x$\_$a$\}$ &$=\backslash$textrm$\{$the one$-$hot vector for word $1\}\backslash \backslash$

$\backslash$bf$\{$x$\_$b$\}$ &$=\backslash$textrm$\{$the one$-$hot vector for word $2\}\backslash \backslash$

$\backslash$bf$\{$x$\_$c$\}$ &$=\backslash$textrm$\{$the one$-$hot vector for word $3\}\backslash \backslash$

$\backslash$bf$\{$v$\_$a$\}$ &$=\backslash$bf$\{$W$\}^\{($word$)\}\backslash$bf$\{$x$\_$a$\}\backslash \backslash$

$\backslash$bf$\{$v$\_$b$\}$ &$=\backslash$bf$\{$W$\}^\{($word$)\}\backslash$bf$\{$x$\_$b$\}\backslash \backslash$

$\backslash$bf$\{$v$\_$c$\}$ &$=\backslash$bf$\{$W$\}^\{($word$)\}\backslash$bf$\{$x$\_$c$\}\backslash \backslash$

$\backslash$bf$\{$v$\}$ &$=\backslash$textrm$\{$concatenate$\}(\backslash$bf$\{$v$\_$a$\},\backslash$bf$\{$v$\_$b$\},\backslash$bf$\{$v$\_$c$\})\backslash \backslash$

$\backslash$bf$\{$m$\}$ &$=\backslash$bf$\{$W$^\{(1)\}\}\backslash$bf$\{$v$\}+\backslash$bf$\{$b$^\{(1)\}\}\backslash \backslash$

$\backslash$bf$\{$h$\}$ &$=\backslash$textrm$\{$ReLU$\}(\backslash$bf$\{$m$\})\backslash \backslash$

$\backslash$bf$\{$z$\}$ &$=\backslash$bf$\{$W$^\{(2)\}\}\backslash$bf$\{$h$\}+\backslash$bf$\{$b$^\{(2)\}\}\backslash \backslash$

$\backslash$bf$\{$y$\}$ &$=\backslash$textrm$\{$softmax$\}(\backslash$bf$\{$z$\})\backslash \backslash$

L &$=\backslash$mathcal$\{$L$\}\_\backslash$textrm$\{$Cross$-$Entropy$\}(\backslash$bf$\{$y$\},\backslash$bf$\{$t$\})\backslash \backslash$

$\backslash$end$\{$align$*\}$

Following the steps discussed in this week's lecture, we should get

the following backward$-$pass computation $($verify this yourself!$)$:

$\backslash$begin$\{$align$*\}$

$\backslash$overline$\{\{\backslash$bf z$\}\}$ &$=\{\backslash$bf y$\}-\{\backslash$bf t$\}\backslash \backslash$

$\backslash$overline$\{$W$^\{(2)\}\}$ &$=\backslash$overline$\{\{\backslash$bf z$\}\}\{\backslash$bf h$\}^$T $\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf b$^\{(2)\}\}\}$ &$=\backslash$overline$\{\{\backslash$bf z$\}\}\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf h$\}\}$ &$=\{$W$^\{(2)\}\}^$T$\backslash$overline$\{$z$\}\backslash \backslash$

$\backslash$overline$\{$W$^\{(1)\}\}$ &$=\backslash$overline$\{\{\backslash$bf m$\}\}\{\backslash$bf v$\}^$T $\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf b$\}^\{(1)\}\}$ &$=\backslash$overline$\{\{\backslash$bf m$\}\}\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf m$\}\}$ &$=\backslash$overline$\{\{\backslash$bf h$\}\}\backslash$circ $\backslash$textrm$\{$ReLU$\}\text{'}(\{\backslash$bf m$\})\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf v$\}\}$ &$=\{$W$^\{(1)\}\}^$T $\backslash$overline$\{\{\backslash$bf m$\}\}\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf v$\_$a$\}\}$ &$=\backslash$dots $\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf v$\_$b$\}\}$ &$=\backslash$dots $\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf v$\_$c$\}\}$ &$=\backslash$dots $\backslash \backslash$

$\backslash$overline$\{\{\backslash$bf W$^\{($word$)\}\}\}$ &$=\backslash$dots $\backslash \backslash$

$\backslash$end$\{$align$*\}$

$**$Task$**$: What is the error signal $$\backslash$overline$\{\{\backslash$bf v$\_$a$\}\}$$$?$

How does this quantity relate to $$\backslash$overline$\{\{\backslash$bf v$\}\}$$$?$

To answer this question, reason about the scalars that make up the elements of

$$\backslash$overline$\{\{\backslash$bf v$\}\}$$$.$ Which of these scalars also appear in $$\backslash$overline$\{\{\backslash$bf v$\_$a$\}\}$$$?$

Express your answer by computing $`$va$\_$bar$`($representing the quantity $$\backslash$overline$\{\{\backslash$bf v$\_$a$\}\}$$$)$

given $`$v$\_$bar$`($representing the quantity $$\backslash$overline$\{\{\backslash$bf v$\}\}$$$).$