## $4.$D $-$ Success Rate Plot

Use Matplotlib to create a line plot showing the progress in the success rate for the Monte Carlo agent. The y$-$values for the line plot should come from the success rate list created in the previous cell. The x$-$values should be the corresponding number of episodes. The figure should also have the following characteristics.

$*$ A figsize of $`[4,3]`.$

$*$ The title should read $"$MC Agent Success Rate".

$*$ The x and y axes should be labeled "Number of Episodes" and "Success Rate", respectively.

$*$ Add a grid to your plot.

## $4.$E $-$ Display Policy

Calculate the mean absolute difference between the optimal state$-$value function and the current estimate produced by Monte Carlo control. Print the message shown below with the blank filled in with the appropriate value, rounded to $2$ decimal places.

The mean absolute difference in V is $\_\_\_\_.$

Display the environment from $4.$A$,$ setting $`$fill$`$ to shade the the cells according to their value under the policy found by MC control, and set $`$contents$`$ to display that policy. When calling display$(),$ set $`$size$=2`$ and $`$show$\_$nums$=$False$`.$

## $4.$F $-$ Q$-$Learning

Starter code has been provided in the cell below. Complete this code to repeat the process outline in Step $4.$C$,$ but using Q$-$learning insteasd of MC Control. The process is identical to that described in Step $4,$C$,$ with two exceptions:

$1.$ You will use Q$-$learning instead of MC control.

$2.$ The characters $"$MC$"$ in the output should be replaced with $"$TD$".$

$\_\_\_\_\_\_=$ TDAgent$($env$=\_\_\_\_\_\_,$ gamma$=1,$ random$\_$state$=1)$

s$\_$rates$\_2=[]$

for i in range$(1,11)$:

num$\_$eps $=\_\_\_\_\_\_$

$\_\_\_\_\_\_.$q$\_$learning$($episodes$=$num$\_$eps, epsilon$=10**(-$i$),$ alpha$=0\mathrm{.}01,$ max$\_$steps$=200,$ exploring$\_$starts$=\_\_\_\_\_\_)$

sr $=$ success$\_$rate$($env$=\_\_\_\_\_\_,$ policy$=\_\_\_\_\_\_,$ episodes$=1000,$ max$\_$steps$=200,$ random$\_$state$=$i$)$

s$\_$rates$\_2.$append$($sr$)$

print$($f$"$After $\{$i $*$ num$\_$eps$\}$ episodes, the TD agent's success rate was $\{$sr:$\mathrm{.}3$f$\}.")$

## $4.$G $-$ Success Rate Plot

Repeat the steps outlined in Step $4.$D$,$ but using the list created for Q$-$Learning in $4.$F instead. The title of this figure should be $"$TD Agent Success Rate".

## $4.$H $-$ Display Policy

Repeat the steps outlined in Step $4.$E$,$ but using the policy and state$-$value function estimates found using Q$-$learning rather than those found by Monte Carlo control.