Q$1.[5$ points$]$ The optimization problem of training logistic regression for the binary classification problem, i$.$e$.,Y=\{0,1\},$ uses the cross$-$entropy $($CE$)$ loss which is defined as

$w_{CE}^{**}=argmi{n}_w{l}_{CE}(w)=argmi{n}_w-\frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{}}}{y}_{i}loghat(y{)}_i+(1-y_i)log(1-$hat$(y{)}_i),$

where hat$(y)=\frac{1}{1+\text{e}\text{x}\text{p}(-w^{T}x)}.$

One may believe that we could have used mean squared error $($MSE$)$ as the loss function instead of the $CE$ loss. In this case, the optimization problem for the logistic regression can be described as

$w_{\text{M}\text{S}\text{E}}^{**}=argmi{n}_w{l}_{\text{M}\text{S}\text{E}}(w)=argmi{n}_w\frac{1}{n}{\displaystyle \underset{i=1}{\overset{n}{}}}||y_i-$hat$(y{)}_i|{|}_2^2.$

Here, hat$(y)=\frac{1}{1+\text{e}\text{x}\text{p}(-w^{T}x)}.$

The goal of this exercise is to compare the CE loss and the MSE loss when training a logistic regression model and to understand why we should use the CE over the MSE.

$($a$)[2$ points$]$ Compute the first order and the second order derivatives of $l_{CE}(w).$

$($b$)[2$ points$]$ Compute the first order and the second order derivatives of $l_{\text{M}\text{S}\text{E}}(w).$

$($c$)[1$ point$]$ Using the seconder order derivatives, which are computed in $($a$)$ and $($b$),$ explain why we should prefer the cross$-$entropy loss over the MSE loss for training a logistic regression model. $($Hint$.$ Use the second derivative and check the convexity.$)$

$($d$)[1$ point$]$ Given the gradients computed in $($a$),$ derive the GD algorithm for optimization problems in equations $(1).$