Problem $3(35\%$ of the total mark$).$ Consider a robot in the environment shown

in Figure $3.$ The robot is initially located as indicated. The numbers inside each cell

represent the payoff function for arriving to the cell. The process is terminated once

the robot arrives to the target location designated by the prize cup. The crosshatched

locations $($cells$)$ are forbidden. Also, once the robot leaves the initial location $($the top

right corner cell$),$ the latter becomes forbidden $($i$.$e$.,$ the robot cannot go back to the

initial location$).$

There are four possible control actions, which are: $u_1$:$=$ "move up$",u_2$:$=$ "move down",

$u_3$:$=$ "move right", $u_4$:$=$ "move left". However, an action is forbidden if it leads the

robot outside of the designated environment. $($For example, if the robot is in the bottom

right corner cell, then there are only two admissible control actions: $u_1$:$=$ "move up$",$

and $u_4$:$=$ "move left". The actions $u_2$:$=$ "move down" and $u_3$:$=$ "move right" are

forbidden.$)$ At each time, the robot knows its position without ambiguity. However,

the robot motion is uncertain. Specifically, there is a chance that the robot executes

another admissible control action instead of the correct one. The probability of exe$-$

cuting an incorrect action is $15\%$ per admissible action. In addition, there is a $10\%$

chance that the chosen control action fails, and the robot does not move at all, i$.$e$.,$ it

stays in the same cell. $($For example, if the robot is located in a cell where there are

three admissible actions $u_2$:$=$ "move down", $u_3$:$=$ "move right", $u_4$:$=$ "move left",

and the correct action is $u_2$:$=$ "move down", then the robot executes $u_2$ with $60\%$

probability, $u_3$ with $15\%$ probability, $u_4$ with $15\%$ probability, and does not move at

all with probability $10\%.)$

Using the Markov Decision Processes framework, do the following:

i$)$ write the value iteration algorithm for the problem $(25\%)$;

ii$)$ calculate the value function for each cell and find an optimal control policy $(10\%).$

For calculations, the use of MATLAB is recommended $($however not required$).$