Problem $3.(25$ points$)$ Consider the MDP shown below. There are five states $\{A,B,C,D,T\}$ in MDP$,$ where state $\{T\}$ is the target state, and two actions $\{a_1,a_2\}.$ The numbers on each transition show the probability of moving to the next state and the reward of transition, respectively. For example, if agent takes action $a_1$ at state A$,$ it will end up at state B with probability $0\mathrm{.}8$ and will be rewarded $-10,$ and with probability $0\mathrm{.}2$ will move to state $C$ and will be rewarded $-10.$

a$)$ For a policy $$ which always takes action $a_1$ at every state, write down the Bellman recursive value function for each state, i$.$e$.,v_{}(A),v_{}(B),v_{}(C),v_{}(D),v_{}(T),$ and compute the final state values when $=\frac{1}{2}.$

b$)$ Consider a random policy which uniformly selects actions at each state $($the probability of taking each of two actions under this policy is $\frac{1}{2}).$ Apply one iteration of Value Iteration algorithm $($one$-$step policy evaluation followed by policy greedification$)$ on this MDP with $=1$ and show the new improved policy.

c$)$ Consider the following episode generated by an arbitrary policy $.$ Assume the current values of all the state values are: $v_{}(A)=0,v_{}(B)=5,v_{}(C)=2,v_{}(D)=10,$ and $v_{}(T)=0.$

Please, i$)$ write down the Temporal Difference $($TD$)$ evaluation equation for updating the values of states, ii$)$ compute the final values after processing the episode shown in the figure with $==\frac{1}{2}.$