4. (a) Consider an Elman-type recurrent neural network (RNN) that receives 2- dimensional input patterns x...

 

Transcribed Image Text:

4. (a) Consider an Elman-type recurrent neural network (RNN) that receives 2- dimensional input patterns x E R² and has one hidden layer. The RNN has five neurons in the hidden layer and two neurons in the output layer. We denote the weight matrices connecting the input to the hidden layer as U, the weight matrices connecting the previous hidden state to the next hidden state as W, and the weight matrices connecting the hidden output to the output layer as V. (i) What is the dimension of U, W and V, respectively? (ii) (iii) (3 marks) If we change this RNN to a Jordan-type RNN, and keep the same number of input dimensions, hidden and output neurons, what is the dimension of the top-down recurrence weight matrix W? (1 mark) Explain the reason of observing exploding gradients in RNN training. Describe a way to address this problem. (5 marks) 4. (a) Consider an Elman-type recurrent neural network (RNN) that receives 2- dimensional input patterns x E R² and has one hidden layer. The RNN has five neurons in the hidden layer and two neurons in the output layer. We denote the weight matrices connecting the input to the hidden layer as U, the weight matrices connecting the previous hidden state to the next hidden state as W, and the weight matrices connecting the hidden output to the output layer as V. (i) What is the dimension of U, W and V, respectively? (ii) (iii) (3 marks) If we change this RNN to a Jordan-type RNN, and keep the same number of input dimensions, hidden and output neurons, what is the dimension of the top-down recurrence weight matrix W? (1 mark) Explain the reason of observing exploding gradients in RNN training. Describe a way to address this problem. (5 marks)

Answer rating: 100% (QA)

The dimensions of U W and V in an Elmantype recurrent neural network RNN are as follows U The input ... View the full answer