LSTM states $2$

$\frac{0}{1}$ point $($graded$)$

Now, we run the same model again with the same parameters and same initial conditions as in the previous

question. The only difference is that our input sequence in now: $1,1,0,1,1.$

Calculate the values $h_t$ at each time$-$step and enter them below as an array $h_0,h_1,h_2,h_3,h_4.$

$($Please round $h_t$ to the closest integer in every time$-$step. If $h_t=+-0\mathrm{.}5,$ then round it to $0.$

For ease of calculation, assume that sigmoid $(x)$~~$1$ and $tanh(x)$~~$1$ for $x1,$ and sigmoid $(x)$~~$0$

and $tanh(x)$~~$-1$ for $x-1.)$LSTM

$$ Bookmark this page

Homework due Jul $17,202417$:$29$ IST Completed

The diagram below shows a single LSTM unit that consists of Input, Output, and Forget gates.

The behavior of such a unit as a recurrent neural network is specified by a set of update equations. These

equations define how the gates, "memory cell" $c_t$ and the "visible state" $h_t$ are updated in response to input

$x_t$ and previous states $c_{t-1},h_{t-1}.$ For the LSTM unit,

$f_1=$sigmoid$(W^{f,h}{h}_{t-1}+W^{f,c}{x}_t+b_t)$

$i_t=$sigmoid$(W^{\frac{i}{4}}{h}_{t-1}+W^{i,x}{x}_t+b_i)$

$o_t=$sigmoid$(W^{0,b}{h}_{t-1}+W^{0,}{x}_t+b_0)$

$c_t=f_{t}o.c_{t-1}+i_{t}o.tanh(W^{c,h}{h}_{t-1}+W^c,x_t+b_c)$

$h_t=a_{t}o.tanh(a_t)$

where symbol C stands for element$-$wise multiplication. The adjustable parameters in this unit are matrices

$W^{f_{1}h},W^{f_{1}x},W^{i,h},W^{i_{1}x},W^{0,h},W^{o,x},W^{c,h},W^{c,},$ as well as the offset parameter vectors $b_f,b_i,b_a,$ and

$b_c.$ By changing these parameters, we change how the unit evolves as a function of inputs $x_t.$

To keep things simple, in this problem we assume that $x_t,c_f,$ and $h_t$ are all scalars. Concretely, suppose that

the parameters are given by

$W^{1,k}=0,W^{f{x}^t}=0,b_f=-100,W^s=-100$

$W^{ts}=0,W^{4,t}=100,b_i=100,W^{\text{'}x}=50$

$W^{as}=0,W^{a,}=100,b_a=0,b_c=0$

We run this unit with initial conditions $h_{-1}=0$ and $c_{-1}=0,$ and in response to the following input sequence:

$0,0,1,1,1,0($For example, $x_0=0,x_1=0,x_2=1,$ and so on$).$