Sometimes MDPs are formulated with a reward function R(s, a) that depends on the action taken or a reward function R (s, a, s’) that also depends on the outcome state.

a. Write the Bellman equations for these formulations.

b. Show how an MDP with reward function R (s. a. s’) can be transformed into a different MDP with reward function R(s, a), such that optimal policies in the new MDP correspond exactly to optimal policies in the original MDP.

c. Now do the same to convert MDPs with R (s, a) into MDPs with R (s).