Consider the following Markov decision process (MDP). reward 1 reward=1 reward=1 reward=10 S1 S2 (S3) S4...

 

Transcribed Image Text:

Consider the following Markov decision process (MDP). reward 1 reward=1 reward=1 reward=10 S1 S2 (S3) S4 S5 reward=1 reward 1 reward=1 reward=10 We have five states representing steps along one direction. Call these states S1, S2, S3, S4, and $5. From each state, except the end states, we can move either left or right. The avail- able actions in state S1 is just to move right while the action available in S5 is just to move left. When we move left and right, we go to the preceding and following states respectively with probability 1. The reward for taking any action to the next state is 1 except when moving right from S4 or left from $5 which provides a reward of 10. Assume a discount factor y = 0.5. Note that 0.5 = 1. Suppose we apply value iteration on this MDP. What are the values of state S3 after one iteration, two iterations, and an infinite number of iterations? Show how you arrived at your answers. Consider the following Markov decision process (MDP). reward 1 reward=1 reward=1 reward=10 S1 S2 (S3) S4 S5 reward=1 reward 1 reward=1 reward=10 We have five states representing steps along one direction. Call these states S1, S2, S3, S4, and $5. From each state, except the end states, we can move either left or right. The avail- able actions in state S1 is just to move right while the action available in S5 is just to move left. When we move left and right, we go to the preceding and following states respectively with probability 1. The reward for taking any action to the next state is 1 except when moving right from S4 or left from $5 which provides a reward of 10. Assume a discount factor y = 0.5. Note that 0.5 = 1. Suppose we apply value iteration on this MDP. What are the values of state S3 after one iteration, two iterations, and an infinite number of iterations? Show how you arrived at your answers.