Consider the grid environment with six states $($numbered from l to $6)$ as shown in Figure$4\mathrm{.}1.$ with the thick border indicating walls. Suppose that at each state the agent can moveup $($denoted by ai$),$ right $($a$2),$ dow $($as$),$ or left $($a$4).$ When the agent moves from state sto state s$\text{'},$ it receives a reward of l if s$>$s'and $0$ otherwise. For example, the agentreceives a reward of $1$ when moving from state $4$ to state $3.$ Assume that state $1$ and state$5$ are goal $($i$.$e$.,$ terminal$)$ states. Let the discount rate y be $0\mathrm{.}7.$