1 Markov Decision Process for Robot Soccer A soccer robot R is on a fast break toward the goal, starting in position 1. From positions 1 through 3, it can either shoot (S) or dribble the ball forward (D). From 4 it can only shoot. If it shoots, it either scores a goal (state G) or misses (state M). If it dribbles, it either advances a square or loses the ball, ending up in state M. goose In this Markov Decision Process (MDP), the states are 1, 2, 3, 4, G, and M, where G and M are terminal states. The transition model depends on the parameter y, which is the probability of dribbling successfully (ate, advancing a square). Assume a discount of qr =1. For k E {1,2,3,4}, we have P(le.5)= and rewards are [l for all other transitions. (a) (3 points) Denote by V\" the value function for the specic policy :rr. What is VH1) for the policy 7r that always shoots? (b) (4 points) Denote by Q*(s,a) the value of a qstate (s,a), which is the expected utility when starting with action a at state a, and thereafter acting optimally. What is Q'(3, D) in terms of y? (c) (5 points) Denote by l4\" (s) the value of a state 3 at iteration t, which is the expected utility when starting in state .9 and acting optimally. Using y = 3-, complete the rst two iterations (t = 1, 2) of value iteration. Iteration [3 corresponds to having value [ll in every state: L311) = 113(2) = 1/313) = 175(4) = D. Hint: Recall that JHLS) = nairZqus, o)(R(s, a, 3') + VITS'D. (d) (3 points) For what range of values of y is Q'(3, 5') 2 Q'(3, D)