Choose the correct statement

Activation layers must be a piecewise linear function

Deep neural networks can give better approximation capacity $($better approximation capacity $==$ smaller approximation error$)$ compared to linear models

All neural networks $($including convolutional neural networks, recurrent neural networks, etc.$)$ are feedforward neural networks

In multiclass classification problems, we cannot guarantee to find a unique mapping from one$-$hot labels to ordinal labels

Choose the correct statement

In gradient$-$based optimization algorithms, a larger step size or learning rate is always better than a smaller one

Dying ReLU issue is due to the zero$-$value of ReLu function when the input is positive

Softmax function outputs probability$-$format prediction, i$.$e$.,$ a vector where each element represents the predicted probability of the corresponding class.

Cross entropy loss does not give ERM solution, because MLE does not follow the ERM framework.

Choose the correct statement

Back$-$propagation $($BP$)$ is simply another name of chain rule in the context of deep learning

We do not need to compute gradients for hidden layers in BP to learn the neural networks

Chain rule can only hold in the one$-$dimensional cases, rather than high dimensional ones

To update the deep learning model, we need to calculate the stochastic gradients using BP for all layers

Choose the correct statement

Gradient explosion issue does not happen in practice, since it only happens in theory

$BP$ algorithm cannot handle regularization terms, so deep learning does not use regularization technique

Before we can compute stochastic gradients of all layers in deep learning models, we need to first have forward propagation