$(30$ points$)$ Figure $1$ in the last page shows an MDP M $=$S$,$ A$,$ P$,$ R$.$

$($a$)(10$ points$)$ Write down the state space $($S$),$ action space $($A$),$ statetransition matrix $($P$)$ and reward vector $($R$)$ of M$.$ Label the rows and

columns of P and the rows of R$.$

$($b$)(10$ points$)$ How many policies does M have? List all the policies of M in

the format shown below. The first component is the action for state s$1,$

the second is for state s$2,$ and the third is for state s$3.$

$\backslash$pi $1=$

$$

$$

a$1$

a$1$

a$1$

$$

$$

$1$

$($c$)(10$ points$)$ Let $\backslash$pi be the policy that takes action a$2$ in all the states, i$.$e$.,$

$\backslash$pi $=$

$$

$$

a$2$

a$2$

a$2$

$$

$$

Define $\backslash$pi using symbols. Draw $\backslash$pi as an MDP$.$ Write down the statetransition matrix and the reward vector of $\backslash$pi $.$Figure $1$: An MDP $M=($:$S,A,P,R$:$).$ The immediate rewards of all state$-$action

pairs are shown within square brackets. The state$-$transition probability of each

state$-$action pair is shown close to the arrows representing actions. Action $a_2$ is

deterministic, so the probability $1$ is not shown in the figure. $(30$ points$)$ Figure $1$ in the last page shows an MDP $M=($:$S,A,P,R$:$).$

$($a$)(10$ points$)$ Write down the state space $(S),$ action space $(A),$ state$-$

transition matrix $(P)$ and reward vector $(R)$ of $M.$ Label the rows and

columns of $P$ and the rows of $R.$

$($b$)(10$ points$)$ How many policies does $M$ have? List all the policies of $M$ in

the format shown below. The first component is the action for state $s_1,$

the second is for state $s_2,$ and the third is for state $s_3.$

${}_1=\left[\begin{array}{c}{a}_1\\ a_1\\ a_1\end{array}\right]$

$($c$)(10$ points$)$ Let $$ be the policy that takes action $a_2$ in all the states, i$.$e$.,$

$=\left[\begin{array}{c}{a}_2\\ a_2\\ a_2\end{array}\right]$

Define $$ using symbols. Draw $$ as an MDP$.$ Write down the state$-$

transition matrix and the reward vector of $.$