Answer Part $2$ Plz: REINFORCE: Monte$-$Carlo Policy$-$Gradient Control $($episodic$)$ for ${}_{*}$

Input: a differentiable policy parameterization $(a|s,)$

Algorithm parameter: step size $>0$

Initialize policy parameter $in{R}^{d^{\text{'}}}($e$.$g$.,$ to $0)$

Loop forever $($for each episode$)$:

Generate an episode $S_0,A_0,R_1,$dots,$S_{T-1},A_{T-1},R_T,$ following $(*|*,)$

Loop for each step of the episode $t=0,1,$dots,$T-1$ :

Glarr${\displaystyle \underset{k=t+1}{\overset{T}{}}}{}^{k-t-1}{R}_k$

$larr+{}^t$Ggradln$(A_t|S_t,)$

Assume the agent uses REINFORCE for learning a policy while navigating in a continuous $2$D square maze, with center at origin. It starts at the state $(0,0).$ The agent's policy is

parameterized by a linear function where the final layer outputs the mean action $Ws==({}_x,{}_y).$ Here, $Win{R}^{2x2}$ is a $22$ matrix initialized as all zeros, and $s$ is the state.

During execution, the agent then samples an action $(a_x,a_y)N(,I),$ a $2-$dimensional Gaussian distribution with mean $$ and identity variance. The first trajectory is:

$s_0=(0,0),a_0=(\mathrm{.}5,-\mathrm{.}2),s_1=(1,-\mathrm{.}2),a_1=(\mathrm{.}2,\mathrm{.}1),s_2=(1\mathrm{.}2,-\mathrm{.}1)dots,s_5=(3\mathrm{.}2,1\mathrm{.}3).$ The trajectory ends in $s_5$ because the agent falls into a trap and receives a

negative reward of $-1(R(s_4,a_4)=-1).$ Otherwise, the agent receives a reward of $0$ for every previous step. Assume $=0\mathrm{.}9.$

What is the return $G$ at state $s_0?$ Please specify to the $4$ th decimal place.

Feedback

Based on answering correctly

$G={\displaystyle \underset{t=0}{\overset{4}{}}}{}^{t}R(s_t,a_t)={}^4*-1=-0\mathrm{.}6561$

$0$ points earned

Assume learning rate $=0\mathrm{.}1,$ what will be the sum of all the elements in the updated $W$ right after we loop over the first state $s_0($so we have just updated $W$ based on the

return from state $s_0$ of this episode and haven't updated the parameters based on $s_1$ yet.

Please refer to the REINFORCE algorithm on page $328$ of the textbook$)?$ Please write the answer to the $4$th decimal place.

Hint: recall the formula of multivariate Gaussian, $pdf=\frac{1}{\sqrt[\phantom{2}]{(2{)}^d||}}$exp$(-\frac{1}{2}(x-{)}^T{}^{-1}(x-)),$ where $d$ is the dimension.