• 4.21. (A Simple bandit model) Suppose there are two projects available for selection in each of three periods. Project one yields a reward of 1 unit and always occupies state s and the other, project two, occupies either state t or state u. When project two is selected, and it occupies state u, it yields a reward of 2 and moves to state t and the next decision epoch with probability 0.5. When selected in state t, It yields a reward of 0 and moves to state u at the next decision epoch with probability 1. Assume a terminal reward of 0, and that project two does not change state when it is not selected. Using backward induction determine a strategy that maximizes the expected total reward.