Consider a Markov chain with three states $\{1,2,3\}.$ In each state, we can choose one of the two

possible actions $\{1,2\}.$ The transition probability matrices under the two actions are given below:

$P(1)=([0\mathrm{.}5,0\mathrm{.}3,0\mathrm{.}2],[0\mathrm{.}1,0\mathrm{.}4,0\mathrm{.}5],[0\mathrm{.}3,0\mathrm{.}3,0\mathrm{.}4])$ and $P(2)=([0\mathrm{.}3,0\mathrm{.}3,0\mathrm{.}4],[0\mathrm{.}5,0\mathrm{.}1,0\mathrm{.}4],[0\mathrm{.}2,0\mathrm{.}5,0\mathrm{.}3]).$

The cost for a given $($state$,$ action$)$ pair is a Bernoulli random variable. The mean costs are given

below

$C=([0\mathrm{.}1,0\mathrm{.}9],[0\mathrm{.}8,0\mathrm{.}1],[0,0])$

We are interested in solving the following discounted cost problem

$mi{n}_{}\underset{N}{lim}E[{\displaystyle \underset{k=0}{\overset{N}{}}}{0\mathrm{.}9}^{k}c(x_k,u_k)|x_0=1,u_0=1]$

where $x_k$ is the state at time $k,u_k$ is the action at time $k,$ and $$ denotes a policy.

Assume we do not know the model but are given the following trace $(x_k,u_k,c(x_k,u_k))$ instead:

$(1,1,1)(2,1,0)(3,2,1)(2,2,0).$

Consider the Q$-$learning algorithm with $Q_0=([0,0\mathrm{.}5],[0\mathrm{.}3,0],[0\mathrm{.}2,0\mathrm{.}1])$ and step size $lon=0\mathrm{.}1.$ Please calculate the

sequence of Q$-$values under Q$-$learning with the trace given above.