We claimed that its demonstration crucially depended on the conjunction of three factors: off-policy updating, bootstrapping,...

Transcribed Image Text:

We claimed that its demonstration crucially depended on the conjunction of three factors: off-policy updating, bootstrapping, and generalisation. In this question, we consider the effect of removing one of these factors: specifically we replace the off-policy update with an on-policy update. Refer to the MDP in the counterexample on Slide 9 of the lecture. Suppose that episodes always start at state $₁. Since there is a deterministic transition to 82, the number of time steps per episode in 8₁ is exactly T(81) = 1. Similarly, what is T(82), the expected number of time steps per episode in 82? Naturally T(82) must depend on €; assume €€ (0, 1). We use the same linear architecture as described in the lecture. For k0, the new update rule we propose is We claimed that its demonstration crucially depended on the conjunction of three factors: off-policy updating, bootstrapping, and generalisation. In this question, we consider the effect of removing one of these factors: specifically we replace the off-policy update with an on-policy update. Refer to the MDP in the counterexample on Slide 9 of the lecture. Suppose that episodes always start at state $₁. Since there is a deterministic transition to 82, the number of time steps per episode in 8₁ is exactly T(81) = 1. Similarly, what is T(82), the expected number of time steps per episode in 82? Naturally T(82) must depend on €; assume €€ (0, 1). We use the same linear architecture as described in the lecture. For k0, the new update rule we propose is

Answer rating: 100% (QA)

The detailed ... View the full answer