Q$3$ Va$1$ue Iteration: Cyc$1$e

We recommend you work out the solutions to the following questions on a sheet of scratch paper, and then enter your results into the answer boxes.

Consider the following transition diagram, transition function and reward function for an MDP$.$

Discount Factor, $=0\mathrm{.}5$

$\backslash$table$[[$S$,$a$,$s$^(\text{'}),$T$($s$,$a$,$s$^(\text{'})),$R$($s$,$a$,$s$^(\text{'}))],[$A$,$Clockwise,B$,1\mathrm{.}0,0\mathrm{.}0],[$A$,$Counterclockwise,C$,1\mathrm{.}0,-2\mathrm{.}0],[$B$,$Clockwise,A$,0\mathrm{.}4,-1\mathrm{.}0],[$B$,$Clockwise,C$,0\mathrm{.}6,2\mathrm{.}0],[$B$,$Counterclockwise,A$,0\mathrm{.}6,2\mathrm{.}0],[$B$,$Counterclockwise,C$,0\mathrm{.}4,-1\mathrm{.}0],[$C$,$Clockwise,A$,0\mathrm{.}6,2\mathrm{.}0],[$C$,$Clockwise,B$,0\mathrm{.}4,2\mathrm{.}0],[$C$,$Counterclockwise,A$,0\mathrm{.}4,2\mathrm{.}0],[$C$,$Counterclockwise,B$,0\mathrm{.}6,0\mathrm{.}0]]$

Q$3\mathrm{.}1$

Suppose that after iteration $k$ of value iteration we end up with the following values for $V_k$:

Vk$($A$)=0\mathrm{.}4$

Vk$($B$)=1\mathrm{.}4$

Vk$($C$)=2\mathrm{.}16$

What is Vk$+1($A$)?$

$[\_\_\_\_\_\_]$