Problem $4.(25$ points$)$ Consider the following MDP with two states $S=\{s_1,s_2\}$ and three actions $A=\{0,\frac{1}{2},1\}.$ The expressions on the arrows indicate the probability of corresponding transition on taking an action ainA. That is start with $v^0(s_1)=0,v^0(s_2)=0.$ You may execute these algorithms either by hand or using a computer program.

$($Approximate answers rounded to two decimal places will be accepted$).$

In your solution, copy the code, and provide the value vector $v^k$ for at least $4$ iterations of value iteration, and

policy ${}^k$ for at least $4$ iterations of policy iteration.

You are required to implement value iteration and policy iteration in your code, and not use a built in tool like

MDP toolbox. $Pr(s_1|s_1,a)=\frac{a^2}{2},Pr(s_2|s_2,a)=\frac{a^2}{4}.$ Rewards are given by $r(s_1,a)=-a,r(s_2,a)=-1+\frac{a}{12}.$

Solve this MDP $($i$.$e$.,$ find a stationary policy that maximizes expected discounted reward$)$ for $=0\mathrm{.}5,$ using policy

iteration and value iteration. For policy iteration, start with ${}^0(s_1)=0,{}^0(s_2)=0.$ For value iteration, you may