Question $2$ Model$-$Based RL: Cycle

Consider an MDP with $3$ states, A$,$ B and C; and $2$ actions Clockwise and Counterclockwise. We do not know the transition function or the reward function for the MDP$,$ but instead, we are given samples of what an agent experiences when it interacts with the environment $($although$,$ we do know that we do not remain in the same state after taking an action$).$ In this problem, we will first estimate the model $($the transition function and the reward function$),$ and then use the estimated model to find the optimal actions.

To find the optimal actions, model$-$based RL proceeds by computing the optimal V or Q value function with respect to the estimated T and R $.$ This could be done with any of value iteration, policy iteration, or Q $-$value

iteration. Last week you already solved some exercises that involved value iteration and policy iteration, so we will go with $\backslash ($ Q $\backslash )$ value iteration in this exercise.

Consider the following samples that the agent encountered.

Q$2\mathrm{.}1$

We start by estimating the transition function, T$($s$,$a$,\backslash (\backslash$left$.\backslash$mathrm$\{$s$\}^\{\backslash$prime$\}\backslash$right$)\backslash )$ and reward function $\backslash (\backslash$mathrm$\{$R$\}\backslash$left$(\backslash$mathrm$\{$s$\},\backslash$mathrm$\{$a$\},\backslash$mathrm$\{$s$\}^\{\backslash$prime$\}\backslash$right$)\backslash )$ for this MDP$.$ Fill in the missing values in the following table for $\backslash (\backslash$mathrm$\{$T$\}\backslash$left$(\backslash$mathrm$\{$s$\},\backslash$mathrm$\{$a$\},\backslash$mathrm$\{$s$\}^\{\backslash$prime$\}\backslash$right$)\backslash )$ and $\backslash (\backslash$mathrm$\{$R$\}\backslash$left$(\backslash$mathrm$\{$s$\},\backslash$mathrm$\{$a$\},\backslash$mathrm$\{$s$\}^\{\backslash$prime$\}\backslash$right$)\backslash ).$

Discount Factor, $\backslash (\backslash$gamma$=0\mathrm{.}5\backslash )$

Answer the following:

$\backslash [$

$\backslash$begin$\{$array$\}\{$l$\}$

$\backslash$mathrm$\{$M$\}=1\backslash \backslash$

$\backslash$mathrm$\{$~N$\}=\backslash$mid $\backslash \backslash$

$\backslash$mathrm$\{$O$\}=\backslash$mid $\backslash \backslash$

$\backslash$mathrm$\{$P$\}=\backslash$mid

$\backslash$end$\{$array$\}$

$\backslash ]$