$1$ Most Likely Digits $(20$ Points$)$

In this problem, I want you to compare two networks: the no$-$hidden$-$layer MNIST model, and the best model you found from Problem $4$ in the previous assignment. Specifically, for a model that gives classification probabilities for each digit, I want you to find the images $x_{{?}^{?}}(0),$dots,$x_{{?}^{?}}(9).$ Note, I am not looking for the images in the data sets with largest probabilities: instead, I want you to solve the input over the entire input space that maximizes the probability of being classified in a certain way, for each digit.

Hints:

How can you formulate this as a minimization problem? What would the variables be$,$ and what would the loss function be$?$

Note: Regularization may be useful here.

It may be useful to note that an arbitrary real number $(-$infinity to infinity$)$ can be turned into a value between $0$ and $1$ by applying the sigmoid function.

Formulate how to solve the problem for the optimal digit images. $(5$ points$)$

Find and display the ten optimal images for the no hidden layer network. $(5$ points$)$

Find and display the ten optimal images for the optimal network. $(5$ points$)$

What do the images suggest about what the two networks are looking for, in terms of features? Any similarities or differences? $(5$ points$)$

Bonus $(5$ points$)$: How should you decide when to stop training? What does overfitting$/$over training mean here?