Problem $2.(30$pt$)$ Given a Markov stationary policy $,$ we studied the minimization of the

projected Bellman error for policy evaluation via function approximation. Alternatively, we

can choose the objective function as

$J()=\frac{1}{2}{\displaystyle \underset{sinS}{\overset{?}{}}}(s)|v^{}(s)-v(s$;$){|}^2,$

where $in(S)$ is the stationary distribution of the Markov chain induced by $$ and $v(*$;$)$ is

the approximation of $v^{}$ with the parameter $in{R}^d.$ Then, the gradient of $J()$ with respect to

$$ is given by

gradJ$()=E_s[(v^{}(s)-v(s$;$))*gradv(s$;$)].$

To find $$ approximating $v^{}(s),$ we can apply the stochastic gradient method according to

${}_{k+1}={}_k+(v^{}(s_k)-v(s_k$;${}_k))*gradv(s_k$;${}_k)$

where $s_k$inS denotes the current state at stage $k.$

$($a$)$ Show that with direct parametrization, i$.$e$.,v=,$ the update ${(}^{**})$ reduces to

$n-$step TD learning algorithm if we use $v^{}(s_t)$~~$G_t^{(n)},$

Monte Carlo method if we use $v^{}(s_t)$~~$G_t,$ where $G_t=\underset{n}{lim}{G}_t^{(n)},$

$-$return update if we use $v^{}(s_t)$~~$G_t^{}.$

Recall the indicator function $I_{\{s_t)}=s$ in these $($non$-$parametric$)$ updates.

$($b$)$ The direct parameterization can be viewed as linear function approximation with the fea$-$

ture matrix $Iin{R}^{|S||S|}.$ What if we have the feature matrix

$=\left[\begin{array}{ccc}{}_1& \text{d}\text{o}\text{t}\text{s}& {}_d\end{array}\right]=\left[\begin{array}{c}\text{h}\text{a}\text{t}({)}^T(s_0)\\ \text{v}\text{d}\text{o}\text{t}\text{s}\\ \text{h}\text{a}\text{t}({)}^T(s_{|S|})\end{array}\right]in{R}^{|S|d},$

where ${}_{i}in{R}^{|S|}$ and hat$()(s)in{R}^d.$ We have $d|S|$ and $$ is full column rank. Formulate the

counterparts of $n-$step TD learning, Monte Carlo method, and $-$return algorithms based on

$(**)$ under linear function approximation according to the feature matrix $.$