Question: Consider an optimization problem, where you want to train a neural network that is robust to small perturbations of its parameters 0 (weights and biases).

 Consider an optimization problem, where you want to train a neuralnetwork that is robust to small perturbations of its parameters 0 (weights

Consider an optimization problem, where you want to train a neural network that is robust to small perturbations of its parameters 0 (weights and biases). In other words, rather than minimizing your loss function L with respect to parameters 0, you want to minimize some of its "smoothened" version obtained by averaging over parameters from a local neighborhood of 0. There are many different ways of defining the "smoothened" version and one of the most popular approaches is to give smooth as (even though you are not constrained to use that smoothing in this problem !): "smooth = Eg~N(0,1) [C(0 + og)], (2) where o > 0 is a small positive hyperparameter defining the size of the local neighborhood. Propose Monte Carlo based algorithm for optimizing parameters of the neural network with respect to the smoothened metric. Assume that the loss function L is a blackbox, i.e. you can query it but you do not have its explicit form. Your algorithm does not need to be efficient in terms of time complexity.Can you think about scenarios, where such robust neural networks might be particularly useful ? Provide justification why your algorithm works. Hint: Think about most naive way of approximationadients of blackbox differentiable func- tions (see: definition of the gradient and finite difference approach) and improve that method by proving that VLsmooth (0) = Eg~N(0,D) [C(0 + og)g]. IMPORTANT: It is also fine if you propose your own smoothing mechanism and prove the analogue of formula: VCo "smooth (0) = =Eg~N(0,I) [C(0 + og)g] for or accurate approximate version of it that you can use with Monte Carlo techniques

Step by Step Solution

There are 3 Steps involved in it

1 Expert Approved Answer
Step: 1 Unlock blur-text-image
Question Has Been Solved by an Expert!

Get step-by-step solutions from verified subject matter experts

Step: 2 Unlock
Step: 3 Unlock

Students Have Also Explored These Related Mathematics Questions!