What values do we have for Q$($s$1,$ a$1)$ and Q$($s$2,$ a$1)$ now, after these three steps of updates? Write

$112$ down how you obtained them.

$1132.$ Suppose from here we will use the $\backslash$epsi $-$greedy strategy with $\backslash$epsi $=0\mathrm{.}3,$ which means that with $\backslash$epsi probability

$114$ we will use an arbitrary action $($each of the two actions will be chosen equally likely in this case$),$ and

$115$ with $1\backslash$epsi probability we will choose the best action according to the current Q$-$values. Now that we

$116$ are in s$2$ after Step $3,$ what is the probability of seeing the transition $($s$2,$ a$1,$ s$1)$ in the next step? That

$117$ is$,$ calculate the probability of the event $$according to the $\backslash$epsi $-$greedy policy, we obtained the action a$1$

$118$ in the current state s$2,$ and after applying this action, the MDP puts us in s$1$ as the next state.$$

$1193.$ If instead of $\backslash$epsi $-$greedy policy, we take the greedy policy that always takes the action that maximizes

$120$ Q$-$values in each step, then what is the probability of seeing $($s$2,$ a$1,$ s$1)$ in the next step?