In this problem, we consider mild modifications of the standard MDP setting. (a) (10 points) Sometimes...

Transcribed Image Text:

In this problem, we consider mild modifications of the standard MDP setting. (a) (10 points) Sometimes MDPs are formulated with a reward function R(s) that depends only on the current state. Write the Bellman equation for this formulation, keeping in mind the definition of the utility of a state. (b) (10 points) Consider an MDP with reward function R(s) and transition model P(s's, a). Instead of a deterministic policy (s) = a, which assigns a single optimal action a for each states, consider allowing probabilistic policies (s) = p(a|s), where p(als) is a probability distribution over possible actions. Write the Bellman equation for this formulation, keeping in mind the definition of the utility of a state. In this problem, we consider mild modifications of the standard MDP setting. (a) (10 points) Sometimes MDPs are formulated with a reward function R(s) that depends only on the current state. Write the Bellman equation for this formulation, keeping in mind the definition of the utility of a state. (b) (10 points) Consider an MDP with reward function R(s) and transition model P(s's, a). Instead of a deterministic policy (s) = a, which assigns a single optimal action a for each states, consider allowing probabilistic policies (s) = p(a|s), where p(als) is a probability distribution over possible actions. Write the Bellman equation for this formulation, keeping in mind the definition of the utility of a state.