a) Suppose you design a multilayer perceptron with a single hidden layer that has a hard...

Transcribed Image Text:

a) Suppose you design a multilayer perceptron with a single hidden layer that has a hard threshold activation function. The output layer uses the softmax activation function. What will go wrong if you try to train this network using gradient descent? b) Consider the following two multilayer perceptrons, where all of the layers use linear activation functions. 100 units 100 units Network 1 100 units 10 units 100 units Network 2 Which one is more advantageous in terms of the following? Expressive power: i) ii) The number of operations for backpropagation: Overfitting: Please first write Network 1 or Network 2, then explain your argument. a) Suppose you design a multilayer perceptron with a single hidden layer that has a hard threshold activation function. The output layer uses the softmax activation function. What will go wrong if you try to train this network using gradient descent? b) Consider the following two multilayer perceptrons, where all of the layers use linear activation functions. 100 units 100 units Network 1 100 units 10 units 100 units Network 2 Which one is more advantageous in terms of the following? Expressive power: i) ii) The number of operations for backpropagation: Overfitting: Please first write Network 1 or Network 2, then explain your argument.