Problem $1.(50$pt$)$ Given a Markov stationary policy $\backslash$pi $,$ consider the policy evaluation problem to compute v

$\backslash$pi

$.$ For example, we can apply the temporal difference $($TD$)$ learning algorithm given by v

t$+1$

$$

$($s$)=$v

t

$$

$($s$)+\backslash$alpha $\backslash$delta

t

$$

$($s$)$I

$\{$s

t

$$

$=$s$\}$

$$

$,$ where $\backslash$delta

t

$$

:$=$r

t

$$

$+\backslash$gamma v

t

$$

$($s

t$+1$

$$

$)$v

t

$$

$($s

t

$$

$)$ is known as TD error. Alternatively, we can apply the n$-$step TD learning algorithm given by v

t$+1$

$$

$($s$)=$v

t

$$

$($s$)+\backslash$alpha $($G

t

$($n$)$

$$

$$v

t

$$

$($s$))$I

$\{$s

t

$$

$=$s$\}$

$$

$,$ where G

t

$($n$)$

$$

:$=$r

t

$$

$+\backslash$gamma r

t$+1$

$$

$+...+\backslash$gamma

n$1$

r

t$+$n$1$

$$

$+\backslash$gamma

n

v

t

$\backslash$pi

$$

$($s

t$+$n

$$

$)$ for n$=1,2,...$ Note that $\backslash$delta

t

$$

$=$ G

t

$(1)$

$$

$$v

t

$$

$($s

t

$$

$).$ The n$-$step TD algorithms for n$<\backslash$infty use bootstrapping. Therefore, they use biased estimate of v

$\backslash$pi

$.$ On the other hand, as n$->\backslash$infty $,$ the n$-$step TD algorithm becomes a Monte Carlo method, where we use an unbiased estimate of v

$\backslash$pi

$.$ However, these approaches delay the update for n stages and we update the value function estimate only for the current state. As an intermediate step to address these challenges, we first introduce the $\backslash$lambda $-$return algorithm given by v

t$+1$

$$

$($s$)=$v

t

$$

$($s$)+\backslash$alpha $($G

t

$\backslash$lambda

$$

$$v

t

$$

$($s$))$I

$\{$s

t

$$

$=$s$\}$

$$

$,$ where given $\backslash$lambda in $[0,1],$ we define G

t

$\backslash$lambda

$$

:$=(1\backslash$lambda $)$

n$=1$

$\backslash$infty

$$

$\backslash$lambda

n$1$

G

t

$($n$)$

$$

taking a weighted average of G

t

$($n$),$s$.$

$$

$($a$)$ By the definition of G

t

$($n$)$

$$

$,$ we can show that G

t

$($n$)$

$$

$=$r

t

$$

$+\backslash$gamma G

t$+1$

$($n$1)$

$$

$.$ Derive an analogous recursive relationship for G

t

$\backslash$lambda

$$

and G

t$+1$

$\backslash$lambda

$$

$.($b$)$ Show that the term G

t

$\backslash$lambda

$$

$$v

t

$$

$($s$)$ in the $\backslash$lambda $-$return update can be written as the sum of TD errors. The TD algorithm, Monte Carlo method and $\backslash$lambda $-$return algorithm looks forward to approximate v

$\backslash$pi

$.$ Alternatively, we can look backward via the eligibility trace method. TheTD$(\backslash$lambda $)$