Problem $1.$ Consider a MDP with two states $S=\{0,1\},$ two actions $A=\{1,2\},$ and the follow reward function

$R_s^{(a)}=\{\begin{array}{ccc}1,& (s,a)=(0,& 1)\\ 4,& (s,a)=(0,& 2)\\ 3,& (s,a)=(1,& 1)\\ 2,& (s,a)=(1,& 2)\end{array}$

and the transition probabilities $P_{s{s}^{\text{'}}}^{(a)}$ as follows:

$\left[\begin{array}{cc}{P}_{00}^{(1)}& P_{00}^{(2)}\\ P_{10}^{(1)}& P_{10}^{(2)}\end{array}\right]=\left[\begin{array}{cc}\frac{1}{3}& \frac{1}{2}\\ \frac{1}{4}& \frac{2}{3}\end{array}\right]$

The other probabilities can be deduced, for example:

$P_{01}^{(1)}=1-P_{00}^{(1)}=1-\frac{1}{3}=\frac{2}{3}.$

The discount factor is

$=\frac{3}{4.}$

$($a$)$ For the policy $$ that chooses action $1$ in state $0,$ and action $2$ in state $1,$ find the state value function $v_{}(s),$ by writing out the Bellman's expectation equation, and solve the equation explicitly.

$($b$)$ For the same policy $,$ obtain the state value function using iterative update based on the Bellman's expectation equation. You need to list the first $5$ iteration values of $v(s).$

$($c$)$ For the policy $,$ calculate the $q_{}(s,a)$ function.

$($d$)$ Based on the value function $v_{}(s),$ obtain an improved policy ${}^{\text{'}}$ based on

${}^{\text{'}}(s)=argma{x}_a{q}_{}(s,a).$

$($e$)$ Obtain the optimal value function $v_{**}(s)$ using value iteration based on the Bellman's optimality equation, with all initial values set to $0.$

$($f$)$ Obtain the optimal policy.