9. (15 points) Value Function Approximation. The robot given below is trying to explore the area and find safe routes to resources. The state of the robot is the grid it is in. Robot can move in four cardinal directions. The landmarks, L1 and L2, signify that there is a resource close-by. The locations of these landmarks are known to the robot (L1 = (m: 1) and L2 = (112, U12). 4 Actions: Up Left + Right 3 L1 2 L2 Down State: (x,y) location of the robot, e.g. (2,1) in the figure L1 and L2: Known landmarks Discount: 1.0 1 6 1 2 3 4 The robot wants to use function approximation get the values of each state. It decides to use the following features, given the current state 8 = (x,y). Current x-coordinate: f1() = 1 Current y-coordinate: fa(8) = y Manhattan Distance to Ll: f(s) = 1- | + y - 1 Manhattan Distance to L2: 4(8) = 1 - 112l+ly-nal Furthermore, it was a linear function approximator: (s, t) = wifi(s) +w2f2(s) + ufa(s) + w4f(s) = w' f(s) The robot then observes the following transitions: (2,1), -0.1 +(2,2), -0.1 +(2,3), +1 Answer the questions below: (a) (3 points) Calculate the feature vectors of the observed states Page 14 COMP 341 Final - Page 15 of 15 18/01/2022 (b) (12 points) Use the observed transitions to update the weights, starting from zero weights with the learning rate a = 0.2 and the discount factory = 1.0