Let us consider a single hidden layer MLP with M hidden units. Suppose the input vector xRN1. The hidden activations hRM1 are computed as follows, h=(Wx+b) where weight matrix WRMN, bias vector bRM1, and is the nonlinear activation function. Dropout [1] is a technique to help reduce overfitting for neural networks. In PyTorch, Dropout is implemented as follows. During training, we independently zero out elements of h with probability p and then rescale it with 1p1, i.e., h~m[i]=1pmhBernoulli(1p)i=1,,M, where is the Hadamard product (a.k.a., element-wise product) and Bernoulli (1p) is the Bernoulli distribution where the random variable takes the value 1 with the probability 1p. During testing, we just use h~=h. 1.1 [5pts] Explain why we need to rescale the hidden activations by 1p1. 1.2[10pts] Assume xN(0,I),b=0,WW=IM(IM is an indentity matrix with size MM), and we use rectified linear units (ReLU) as the nonlinear activation function, i.e., (x)=max(x,0), derive the variance of the activations before Dropout (i.e., h) and after Dropout (i.e., h~ ). 1.3[10pts] What is the expected number of hidden units that are kept (i.e., those with m[i]=1 ) by Dropout? Derive the probability distribution (i.e., the probability mass function) of the number of kept hidden units. 1.4 [Bonus 20pts] Assume the number of hidden units M goes to infinity and the probability of keeping units 1p goes to 0 in a way that their product M(1p) stays fixed. Derive the probability distribution of the number of kept hidden units. 1.5 [Bonus 20pts] Suppose the number of hidden units M follows a Poisson distribution with parameter , i.e., the probability mass function is P(M=k)=k!ke. Derive the probability distribution of the number of kept hidden units