Semi$-$Gradient Update

This problem presents a brief glimpse of the problems that can arise in off$-$policy learning with function approximation, through the concepts that have been introduced so far. If

you would like a more detailed discussion on these issues, you may refer to Chapter $11.$ Let us now apply semi$-$gradient TD learning from Chapter $9$ with batch updates $($Section

$6\mathrm{.}3)$ to the following value$-$function approximation problem, based on a problem known as Baird's Counterexample:

In the above diagram, each circle is a state. The arrows represent some possible transitions between states.

The formulas shown in each state are for their values in terms of some parameters $w_i,i=0,1,$dots,$6$ that comprise the value function approximator that we wish to learn.

Consider that we see each of the transitions shown above exactly once in a batch of data.

The reward for each transition is $0,$ and the discount factor $=0\mathrm{.}95.$

We update the weights in the function approximation using semi$-$gradient TD$(0).$

Before the update, the weights are: $(1,1,1,1,1,1,5),$ with $w_6=5.$

With a learning rate $=0\mathrm{.}1,$ what will be the weights after the update?

Consider the updates happening in a batch, with the individual updates summed up and applied to the weights after computing the update for all the six transitions. Enter your

answers to $3$ decimal places.

$5$

$0$ points earned

$w_0$

$1\mathrm{.}185$

$0$ points earned

$w_1$

$1$

$7$

$0$ points earned

$8$

$0$ points earned

$9$

O points earned

$w_4$

$10$

$0$ points earned