$3.$ Linear regression $(15$ marks$)$

$($a$)$ Write the explicit formula of a linear regression model, f $=$ f$($x$,\backslash$lambda $),$ in the

case x in R$.$ Specify how you can use f to predict the label of a test object.

$[3$ marks$]$

$($b$)$ Explain how you can train f on a given data set and what is the least$-$square

estimate of $\backslash$lambda $.$ Do you need an iterative algorithm, e$.$g$.$ gradient descent, to

obtain it$?$ Justify your answer. $[4$ marks$]$

$($c$)$ Consider a $2-$dimensional linear regression model, g$($X$,\backslash$lambda $)=\backslash$lambda $0+$

P$2$

i$=1\backslash$lambda iXi

$,$

with parameter $\backslash$lambda $=[1,0,1]$T

$.$ Evaluate the Residual Sum of Squares $($RSS$)$

of g on

D $=\{($xn$,$ yn$)$ in R

$2\backslash$times R$\}$

$4$

n$=1$

$=\{([0\mathrm{.}1,0\mathrm{.}2]$T

$,0\mathrm{.}3),([0\mathrm{.}4,0\mathrm{.}5]$T

$,0\mathrm{.}4),([0\mathrm{.}3,0\mathrm{.}2]$T

$,0\mathrm{.}1),([0\mathrm{.}2,0\mathrm{.}3]$T

$,0\mathrm{.}4)\}$

Hint: In this case, the RSS is defined as

RSS$($D$,\backslash$lambda $)=$ X

$($x$,$y$)$ in D

$($y $$ g$($x$,\backslash$lambda $))2$

$.$

$[4$ marks$]$

$($d$)$ Discuss the problem of regularization in machine learning. In particular,

how can you regularize a linear regression model? Write the expression of

the L$2-$regularized RSS of g on D as a function of $\backslash$lambda and a regularization

parameter $\backslash$rho $.$

Hint: The L$2$ norm of a d$-$dimensional vector is defined as

kvk

$2=$

X

d

i$=1$

v

$2$

i

$.$

$[4$ marks$]$