$1$ Problem $1$: Recurrent Neural Network $(20$

points$)$

Recurrent Neural Networks $($RNN$)$ are artificial neural networks that contains

temporal connections among neural nodes in the same layer. Compare to normal

feedforward neural networks, RNNs are good at processing sequential data with

arbitrary length and can learn to model temporal dynamic behaviors. Long

Short$-$Term memory $[1]$ is the most famous RNN variant that are widely used

in deep learning field.

a$.(10$ points$)$ LSTM contains a forget gate to decide how many previous

information need to be kept. There is a Sigmoid activation in the forget gate.

Explain why we use Sigmoid here, instead of Tanh.

b$.(10$ points$)$ In neural language processing and many other fields that

process sequential data, more and more LSTM$-$based models are replaced by

Transformer$-$based models $[2]$ nowadays. Why LSTM is not as popular as before

now? List at least $2$ drawbacks of LSTM$-$compared to Transformer. Explain

each of them with $1$ to $3$ sentences.