Implementation $3)$ In another notebook, using the architecture you built in implementation $2$ and the best performing activation function, re$-$train the model by adding $2$ new parameters, learning rate and momentum. The range of the values you should try $($between $5$ and $10$ trials$)$ is:

Learning rate:

Momentum:

Print on the code, for each combination, the corresponding accuracy. What are the optimum learning rate and momentum values?

Example of the output:

Best performing activation function: ReLu

Trial $1$: Learning Rate $=0\mathrm{.}5796,$ Momentum $=0\mathrm{.}9178,$ Accuracy $=0\mathrm{.}6316$

Trial $2$: Learning Rate $=0\mathrm{.}6370,$ Momentum $=0\mathrm{.}1148,$ Accuracy $=0\mathrm{.}6316$

Trial $3$: Learning Rate $=0\mathrm{.}1863,$ Momentum $=0\mathrm{.}5109,$ Accuracy $=0\mathrm{.}6316$

Trial $4$: Learning Rate $=0\mathrm{.}1431,$ Momentum $=0\mathrm{.}7752,$ Accuracy $=0\mathrm{.}9649$

Trial $5$: Learning Rate $=0\mathrm{.}3938,$ Momentum $=0\mathrm{.}4581,$ Accuracy $=0\mathrm{.}6316$

Trial $6$: Learning Rate $=0\mathrm{.}6761,$ Momentum $=0\mathrm{.}2949,$ Accuracy $=0\mathrm{.}6316$

Trial $7$: Learning Rate $=0\mathrm{.}1724,$ Momentum $=0\mathrm{.}0836,$ Accuracy $=0\mathrm{.}6316$

Trial $8$: Learning Rate $=0\mathrm{.}6177,$ Momentum $=0\mathrm{.}8061,$ Accuracy $=0\mathrm{.}6316$

Trial $9$: Learning Rate $=0\mathrm{.}5382,$ Momentum $=0\mathrm{.}3641,$ Accuracy $=0\mathrm{.}6316$

Trial $10$: Learning Rate $=0\mathrm{.}6640,$ Momentum $=0\mathrm{.}4447,$ Accuracy $=0\mathrm{.}6316$