$1$ Problem $1($Multi$-$step Q learning$)$

We update the multi$-$step $($with step length $N)$ Q learning in the following

manner

$Q(s_t,a_t)=(1-)Q(s_t,a_t)+(({\displaystyle \underset{k=t}{\overset{t+N-1}{}}}{}^{k-t}{r}_k)+ma{x}_{a_{t+N}}Q(s_{t+N},a_{t+N}))$

Note that when $N=1,$ it is standard $Q-$learning where data is collected from

some policy $.$ State whether the following statements are true or false $($you

need to give justification$).$

Multi$-$step Q learning is an unbiased estimator for $Q^{}$ when $=1,$ and

$N$ is any finite number

Multi$-$step Q learning is an unbiased estimator for $Q^{}$ when $=1,$ and

$N.$

Suppose that the policy $$ is $lon-$greedy, Multi$-$step $Q$ learning is an on$-$policy

estimator if $N$ is finite and $=1.$

As $N$ increases multi$-$step Q learning has a higher variance if $=1.$