$4-$ Assuming that all $Q-$values are initialized to $0,$ what are the $Q-$values for the following state$-$action pairs after running $[$tabular$]$

Q$-$learning for the first episode? $[$skip$/$disregard episodes $2$ and $3].$ Use discount factor $=0\mathrm{.}8$ and learning rate $=0\mathrm{.}6$

Q$($A$,$ Down$)$

$Q(B,Up)$

Hint: Use the following equations and update $Q$ values after each transition until the end of episode $1.$

Consider your new sample estimate

target $=R(s,a,s^{\text{'}})+ma{x}_{a^{\text{'}}}$hat$(Q)(s^{\text{'}},a^{\text{'}})$

Incorporate the new estimate into a running average

hat$(Q)(s,a)larr(1-)hat(Q)(s,a)+()[$ target $]$

$5-$ Repeat part $4$ if you run SARSA $($temporal difference$)$ with the above experience sequence $($again assume that all Q$-$values

are initialized to $0$ and use only episode $1)?$ Use discount factor $=0\mathrm{.}8$ and learning rate $=0\mathrm{.}6$

Hint: Use the following equations and update $Q$ values after each transition until the end of episode $1.$

Sample of hat$(Q{)}^{}(s,a)$:$,$ target $=R(s,a,s^{\text{'}})+hat(Q{)}^{}(s^{\text{'}},a^{\text{'}})$

Update hat$(Q{)}^{}(s,a)$:$,$hat$(Q{)}^{}(s,a)larr(1-)hat(Q{)}^{}(s,a)+$ target