Q$9$ Policy Iteration: Cycle

$14$ Points

Consider the following transition diagram, transition function and reward function for an MDP$.$

Discount Factor, $\backslash (\backslash$gamma$=0\mathrm{.}5\backslash )$;

Suppose we are doing policy evaluation, by following the policy given by the left$-$hand side table below. Our current estimates $($at the end of some iteration of policy evaluation$)$ of the value of states when following the current policy is given in the righthand side table.

We recommend you work out the solutions to the following questions on a sheet of scratch paper, and then enter your results into the answer boxes.

Part $1$

What is $\backslash ($ V$\_\{$k$+1\}^\{\backslash$pi$\}($A$)\backslash )?$

Suppose that policy evaluation converges to the following value function, $\backslash ($ V$\_\{\backslash$infty$\}^\{\backslash$pi$\}\backslash ).$

Now let's execute policy improvement.

Part $2$

What is $\backslash ($ Q$\_\{\backslash$infty$\}^\{\backslash$pi$\}\backslash )($A$,$ clockwise$)?$

Part $3$: What is $\backslash ($ Q$\_\{\backslash$infty$\}^\{\backslash$pi$\}\backslash )($A$,$ counterclockwise$)?$

Part $4$: What is the updated action for state A$?$

Clockwise

Counterclockwise