Value Iteration Example: $4$ x $4$ Grid World

$k=2$ : Given the state$-$value we obtained at $k=1,$ consider updating state $9,$ there are four actions, and we have

$V_2^W(s=9),=r_9^W+P_{98}^W**V_1(s=8)$ Where $r_9^W=-1,P_{98}^W=1,V_1(s=8)=-1$

$,=-1+1**-1=-2,$

$V_2^E(s=9),=r_9^E+P_{9,10}^E**V_1(s=10)$ Where $r_9^E=-1,P_{9,10}^E=1,V_1(s=10)=-1$

$,=-1+1**-1=-2,$

$V_2^N(s=9),=r_9^N+P_{95}^N**V_1(s=5)$ Where $r_9^N=-1,P_{95}^N=1,V_1(s=5)=-1$

$,=-1+1**-1=-2,$

$V_2^S(s=9),=r_9^S+P_{9,13}^S**V_1(s=13)$ Where $r_9^S=-1,P_{9,13}^W=1,V_1(s=13)=-1$

$,=-1+1**-1=-2,$

$k=1$ :

We then compare the values of each action and choose the one with the maximum value.

Since $V_2^W(s=9)=V_2^E(s=9)=V_2^N(s=9)=V_2^S(s=9)=-2,$ we have

$V_2(s=9)=ma{x}_{\text{a}\text{i}\text{n}\text{A}}\{V_2^W(s=9),V_2^E(s=9),V_2^N(s=9),V_2^S(s=9)\}=-2$

For the grid world example we discussed in the lecture, consider using the value iteration, given the state value at

$k=2$ :

Answer the following questions:

$1\mathrm{.}1$: what is the state value of state $"1"$ when $k=3?$

$1\mathrm{.}2$ : what is the state value of state $"9"$ when $k=3?$