Let us apply policy iteration. We determine the optimal policy and the values of states $1$ and $2$ each step.

We call utility at state $1,$ u$1,$ utility at state $2,$ u$2,$ and utility at state $3,$ u$3$

The whole process includes several iterations, and each iteration includes three major steps

$1.$ initialization

$2.$ value determination

$3.$ policy update.

Assume that the initial policy choose action b in both states. Let us calculate the first iteration.$$

First, initialization is easy, because we already said "Assume that the initial policy choose action b in both states".$$

After initialization, we do value determination. We have a set of three linear equations with u$1,$ u$2$ and u$3.$ After solving these equations, we have:

u$1=\_\_($with error margin $0)$