Consider the Bellman equation for deterministic policies and state-only rewards: VT (s) = R(s) + yT(s,...

 

Transcribed Image Text:

Consider the Bellman equation for deterministic policies and state-only rewards: VT (s) = R(s) + yΣT(s, π(s), s')V™ (s') s' We often need to consider stochastic policies as well, which we denote by (als) instead of π(s). T(als) specifies the probability of taking action a in state s. When the policy is deterministic, exactly one action a will have probability 1, so we overload notation and refer to that action as a = π(s). Note: The output types are different; 7(als) outputs a probability, whereas T(s) outputs an action. A more general version of the Bellman equation can be derived for stochastic policies and reward functions depending on (s, a, s'): V" (s) = Σπ(a|s) ΣT(s, a, s') [R(s, a, s') + yV™ (s')] a (a) Explain, in words, what the general version of the Bellman equation means. Additionally, show that it reduces to the simpler version when using deterministic policies 7(s) and state-only re- wards R(s). Consider the Bellman equation for deterministic policies and state-only rewards: VT (s) = R(s) + yΣT(s, π(s), s')V™ (s') s' We often need to consider stochastic policies as well, which we denote by (als) instead of π(s). T(als) specifies the probability of taking action a in state s. When the policy is deterministic, exactly one action a will have probability 1, so we overload notation and refer to that action as a = π(s). Note: The output types are different; 7(als) outputs a probability, whereas T(s) outputs an action. A more general version of the Bellman equation can be derived for stochastic policies and reward functions depending on (s, a, s'): V" (s) = Σπ(a|s) ΣT(s, a, s') [R(s, a, s') + yV™ (s')] a (a) Explain, in words, what the general version of the Bellman equation means. Additionally, show that it reduces to the simpler version when using deterministic policies 7(s) and state-only re- wards R(s).

Answer rating: 100% (QA)

This is boundlessness We can address this with the assistance of the rebate factor previously presen... View the full answer