1.1.2 Example (6 pts) Does the following MDP obey the Markov Property? Explain your answer. Let the MDP M be defined as the tuple where S is the state space with 2 discrete states {81, s2), A is the action space with 2 discrete actions {al, a2} . The transition function P defines the distribution over next states Pr(St +1 s, St = 8, At = a) Vs, s, S,Ya E A. R is the reward function R(s, a, s') that defines the reward Rt the agent receives at time t for the transition St s, At-a, St+-s'. The reward function in M always returns 1. The initial state distribution do is defined such that Pr( o = 81) 1. The reward discount factor ? = 1 . Let the transition function be: P(s, a, s' ift5 S1 a1.0 S1 a82 0.0 S2 1 1.0 S2 a21.0 P(s, a, s') if t > 5 S1 a82 0.0 S a1.0 s2 a11.0 s2 a1 1.0 Table 1: Transition Function for a MDP Answer: 1.1.2 Example (6 pts) Does the following MDP obey the Markov Property? Explain your answer. Let the MDP M be defined as the tuple where S is the state space with 2 discrete states {81, s2), A is the action space with 2 discrete actions {al, a2} . The transition function P defines the distribution over next states Pr(St +1 s, St = 8, At = a) Vs, s, S,Ya E A. R is the reward function R(s, a, s') that defines the reward Rt the agent receives at time t for the transition St s, At-a, St+-s'. The reward function in M always returns 1. The initial state distribution do is defined such that Pr( o = 81) 1. The reward discount factor ? = 1 . Let the transition function be: P(s, a, s' ift5 S1 a1.0 S1 a82 0.0 S2 1 1.0 S2 a21.0 P(s, a, s') if t > 5 S1 a82 0.0 S a1.0 s2 a11.0 s2 a1 1.0 Table 1: Transition Function for a MDP