Question $4$:

In logistic regression, we assume $Y=(Y_1,$dots,$Y_n{)}^T$ are a collection of $n$ binary observa$-$

tions. For each $Y_i,$ we observe $x_i=(x_{i1},x_{i2},x_{i3}{)}^T$ predictors. We assume

$log\frac{p_i}{1-p_i}=x_i^T,$

where $=({}_1,$dots,${}_3{)}^T$ is the vector of regression coefficients.

a$)$ Formulate the overall loglikelihood of the dataset $l(Y).$

b$)$ Derive the first derivative dell$\frac{Y}{d}el{}_2.$

c$)$Derive the second derivative $de{l}^{2}l\frac{Y}{d}el{}_2$del${}_3.$

d$)$ Suppose hat$({)}^{\text{c}\text{u}\text{r}\text{r}\text{e}\text{n}\text{t}}=(0\mathrm{.}1,0\mathrm{.}2,0\mathrm{.}3{)}^T,$ and we have a new observation $x_{n+1}=(3,2,4{)}^T.$

Predict the probability $p$ of success for this new observation.

e$)$ Suppose $Y_{n+1}=1,$ calculate the first derivative vector dell$\frac{Y_{n+1}}{d}el.$

f$)$ Suppose $Y_{n+1}=1,$ calculate the second order derivative matrix $de{l}^{2}l\frac{Y_{n+1}}{d}eldel{}^T.$

g$)$ If instead of using Newton's method, we decide to use stochastic gradient accent

algorithm, which updates the parameter in the following manner:

hat$({)}^{\text{n}\text{e}\text{w}}=$hat$({)}^{\text{c}\text{u}\text{r}\text{r}\text{e}\text{n}\text{t}}+dell\frac{Y_{n+1}}{d}el$

Please update the estimate for $.$ Use the learning rate $=0\mathrm{.}001.$

$($You may notice that we are maximizing the objective function $($loglikelihood$),$ we are

using stochastic gradient accent algorithm, which is very similar to the gradient descent

algorithm for minimization purpose, which we learned a few weeks ago. "Accent" means

"going up$"$ and "descent" means "going down". The two methods only differ by the sign of

the second term, for maximization, we add the second term, for minimization we minus the

second term.