Question $6.(6$ marks$)$

Consider the following MDP: the set of states is $S=\{s_0,s_1,s_2,s_3\}$ and the set of actions available at each

state is $A=\{l,r\}.$ Each episode of the MDP starts in $s_1$ and terminates in $s_0.$

You do not know the transition probabilities or the reward function of the MDP$,$ so you are using Sarsa

to find the optimal policy. Suppose the current $Q-$values are:

$Q(s_0,l)=0,Q(s_0,r)=0$

$Q(s_1,l)=3\mathrm{.}4,Q(s_1,r)=-1\mathrm{.}8$

$Q(s_2,l)=-0\mathrm{.}8,Q(s_2,r)=-0\mathrm{.}7$

$Q(s_3,l)=-0\mathrm{.}5,Q(s_3,r)=7\mathrm{.}5$

Suppose the next episode is as follows:

$s_1,l,-1,s_1,r,-1,s_2,l,-1,s_1,l,10,s_0.$

$($a$)(4$ marks$)$ Do all the Sarsa updates to the $Q-$values that would result from this episode, using $=0\mathrm{.}25$

and $=0\mathrm{.}9.$ Show your working.

$($b$)(1$ mark$)$ Based on the updated $Q-$values, give the final policy $$ determined by $Q,$ i$.$e$.,$ give $(s_1),(s_2)$

and $(s_3).$ Show your working.

$($c$)(1$ mark$)$ Give an $lon-$greedy policy based on the $Q-$values obtained in $($a$).$