Consider an undiscounted MDP having three states, $(1,2,3),$ with rewards $1,2,$ and $0,$ respectively. State $3$ is a terminal state. In states $1$ and $2$ there are two possible actions: A and B$.$ The transition model is as follows:

In state $1,$

action A moves the agent to state $2$ with probability $0\mathrm{.}8,$ and$0\mathrm{.}2$ chance do not moveaction B moves the agent to state $3$ with probability $0\mathrm{.}1,$ and$0\mathrm{.}9$ chance do not move

In state $2,$

action A moves the agent to state $1$ with probability $0\mathrm{.}8$ and$0\mathrm{.}2$ chance do not moveaction B moves the agent to state $3$ with probability $0\mathrm{.}1$ and $0\mathrm{.}9$ chance do not move

Let us apply policy iteration. We determine the optimal policy and the values of states $1$ and $2$ each step.

We call utility at state $1,$ u$1,$ utility at state $2,$ u$2,$ and utility at state $3,$ u$3$

The whole process includes several iterations, and each iteration includes three major steps

$1.$ initialization

$2.$ value determination

$3.$ policy update.

Assume that the initial policy choose action b in both states. Let us calculate the first iteration.$$

First, initialization is easy, because we already said "Assume that the initial policy choose action b in both states".$$

After initialization, we do value determination. We have a set of three linear equations with u$1,$ u$2$ and u$3.$

$1.$ find u$1,$ u$2,$ and u$3$

$2.$Which action is preferred for state $1$ at this iteration?

$3.$ Which action is preferred for state $2$ at this iteration?

$4.$ Now we start the second iteration. Based on the preferred action calculated from previous iteration, we initialize it again.Then, in the value determination of this second iteration, the set of equation now been updated. Solve them again, and find u$1,$ u$2,$ and u$3$

$5.$ Which action is preferred for state $1$ at this iteration?

$6.$ Which action is preferred for state $2$ at this iteration?

Now, what will happens to policy iteration if we let the initial policy choose action A in both states?