Bandit example Consider a k$-$armed bandit problem with k $=4$ actions, denoted $1,2,3,$ and $4.$ Consider applying to this problem a bandit algorithm using $\backslash$epsi $-$greedy action selection, sample$-$average action$-$value estimates, and initial estimates of Q$1($a$)=0,$ for all a$.$ Suppose the initial sequence of actions and rewards is A$1=1,$R$1=1,$A$2=2,$R$2=1,$A$3=2,$R$3=2,$A$4=2,$R$4=2,$ A$5=3,$ R$5=0.$ On some of these time steps the $\backslash$epsi case may have occurred, causing an action to be selected at random. On which time steps did this definitely occur? On which time steps could this possibly have occurred?