Math: An alternative learning algorithm $[10$ points$]$ Consider a learning algorithm which at$-$

tempts to learn a Q$-$function, but instead of using the usual Q$-$learning target $R+ma{x}_{a}Q(s^{\text{'}},a),$

it uses as target a mixture of

$R+((1-)ma{x}_{a}Q(s^{\text{'}},a)+{\displaystyle \underset{a}{\overset{?}{}}}(s^{\text{'}},a)Q(s^{\text{'}},a))$

where $in(0,1)$ is a hyper$-$parameter.

Assume that $$ is an $lon-$greedy policy derived from $Q,$ and the episodes used for training are collected

using $$ only.

$($a$)[5$ points$]$ Recall that an on$-$policy control algorithm estimates $q_{}(s,a)$ for the current be$-$

haviour policy $$ and for all states $s$ and actions $a.$ Is this algorithm on$-$policy or off$-$policy?

Justify your answer.

$($b$)[5$ points$]$ For different values of $,$ how would you expect this algorithm to perform com$-$

pared to Q$-$learning and SARSA? Include bias, variance, and maximization bias in your

discussion.

$($c$)[5$ points$]$ Bonus question: try this algorithm on the Taxi Problem in Question $1,$ and compare

it to the other algorithms. Are the results consistent with your hypothesis?