Question $4,[20$ marks$]$

Having the following XO sequence of states a long with their values

O

$-$

$-$

$-$

X

$-$

X

O

$-$

$0\mathrm{.}500$

O

$-$

$-$

$-$

X

X

X

O

$-$

$0\mathrm{.}576$

O

$-$

$-$

$-$

X

X

X

O

O

$0\mathrm{.}545$

O

X

$-$

$-$

X

X

X

O

O

$0\mathrm{.}556$

O

X

$-$

O

X

X

X

O

O

$0\mathrm{.}577$

O

X

X

O

X

X

X

O

O

$1\mathrm{.}000$

a$.$ Assume a learning rate of $0\mathrm{.}73$ what will be updated values adopting gradient$-$based state value update with each move.

b$.$ Assume a learning rate $($$)$ of $0\mathrm{.}86$ and discount factor $($$)$ of $0\mathrm{.}82,$ what will be updated values adopting TD$-$based state value update with each move. Assume all rewards are $-1$ except for the actions leading to the goal state with respect to X$-$player.

c$.$ Compare both algorithms applied in a$.$ and b$..$