Value Function

$1$ point possible $($graded$)$

As above, we are working with the $33$ grid example with $+1$ reward at the top right corner and $-1$ at the

cell below it$.$ The agent also gets a reward of $-10$ for every action that it takes. The action outcomes are

deterministic. The agent continues to act until it reaches the $+1$ cell, when it stops.

The following figures show states $s_1,s_2,s_3,$ in which the letter $"$ A $"$ marks the current location of the agent.

$s_1$

$s_2$

$s_3$

A value function $V(s)$ of a given state $s$ is the expected reward $($i$.$e the expectation of the utility function$)$ if

the agent acts optimally starting at state $s.$ In the given MDP$,$ since the action outcome is deterministic, the

expected reward simply equals the utility function.

Which of the following should hold true for a good value function $V(s)$ under the reward structure in the given

MDP$?$

Note: You may want to watch the video on the next page before submitting this question.

$V(s_3)V(s_3)$

$V(s_3)V(s_1)$

$V(s_3)$

$V(s_3)$