Consider a two$-$state Markov decision process $($MDP$)$ with state s$1$ and state s$2.$ In state s$1,$ the decision maker chooses either action a$1$ or action a$2$; In state s$2,$ only action a$3$ is available. The immediate returns and transition probabilities are as follows.

r$($s$1,$ a$1)=4,$ r$($s$1,$ a$2)=10,$ r$($s$2,$ a$3)=2,$ p$($s$1|$s$1,$ a$1)=$ p$($s$2|$s$1,$ a$1)=0\mathrm{.}5,$ p$($s$2|$s$1,$ a$2)=1,$ p$($s$1|$s$2,$ a$3)=0\mathrm{.}2,$ p$($s$2|$s$2,$ a$3)=0\mathrm{.}8.$

$($a$)$ Solve the three$-$periods problem with terminal reward r$4($s$1)=$ r$4($s$2)=0$ to maximize the expected total rewards and find the optimal decision rule in each period.

$($b$)$ Consider the infinite$-$horizon discounted MDP with discounted factor $\backslash$lambda $=0\mathrm{.}5.$ Calculate the expected total discounted reward of a stationary policy $\backslash$delta $\backslash$infty with $\backslash$delta $($s$1)=$ a$1$ and $\backslash$delta $($s$2)=$ a$3.$ Also, use the optimality equations to check if it is the optimal policy.