$1$ Consider a game where a frog repeatedly jumps a random number of steps that is equally likely to be $2,3,$ or $4.$ The frog can either Jump or Stop if the total number of steps is less than $6.$ If the total step is $6$ or higher, the game automatically ends, and the frog receives a reward of $0.$ When the frog Stops, the reward is equal to the total steps $($up to $5),$ and the game ends. There is no reward for the Jump action. Formulate this problem as an MDP with the states $\{0,2,3,4,5,$ Done$\}.$

a$)$ What is the transition function p$($s$$ s$,$ a$)$ for this MDP$?$

b$)$ What is the reward function for this MDP$?$

c$)$ Perform value iteration for $4$ iterations with $=1$ and mention the value function as:

States

$0$

$2$

$3$

$4$

$5$

Done

V$0$

$0$

$0$

$0$

$0$

$0$

$0$

V$1$

$$

$$

$$

$$

$$

$$

V$2$

$$

$$

$$

$$

$$

$$

V$3$

$$

$$

$$

$$

$$

$$

V$4$

$$

$$

$$

$$

$$

$$

d$)$ Based on the above value function after $4$ iterations, what is the current best policy?