Q$3.$ Consider a reinforcement learning problem with two states and two actions. Compute

the estimate of the action$-$value function obtained after the first $6$ steps assuming that

the learning algorithm is

a$)$ Sarsa;

b$)$ Q$-$learning;

c$)$ Expected Sarsa.

The discount rate is gamma$=1//2.$ The step size alpha is $0\mathrm{.}1.$ The action$-$value estimates are

initialized to $0.$ The sequence of states, actions and rewards is:

Please write with good handwriting, explain all the steps, and inlcude all the formulas used so that it is easy to understand the steps. Thanks