Q$-$Learning

Let's simulate the $Q-$learning algorithm! Assume there are states $0,1,2,3$ and actions $(b\text{'},\text{'}$c$),$ and discount factor $=0\mathrm{.}9.$ Furthermore, assume that all the $Q$ values are initialized to $0$ and that the learning rate $=0\mathrm{.}5.$

Each row, $t,$ in the table represents a record of experience at time $t$ : $(s_t,a_t,r_t).$

In each row $t,$ indicate what update $Q(s_t,a_t)larrq$ will be made by the $Q$ learning algorithm based on $(s_t,a_t,r_t,s_{t+1}).$ Note that $s_{t+1}$ is on the next row $($you might need to look ahead to the next part of the problem to see that next state value.$)$ You will want to keep track of the overall table $Q(s_t,a_t)$ as these updates take place, spanning the multiple parts of this question.

As a reminder, the $Q-$learning update formula is the following:

$Q(s,a)=(1-)Q(s,a)+(r+ma{x}_{a^{\text{'}}}Q(s^{\text{'}},a^{\text{'}}))$

You are welcome to do this problem by hand, though writing a small program to solve may be a good idea. To help with that, here is a variable with the history of experience: