At the start of each day, the condition of a machine is classified as either excellent, acceptable, or poor. The daily behavior of the machine is modeled as a three-state absorbing unichain with the following transition probability matrix:

A machine in excellent condition earns revenue of \($600\) per day. A machine in acceptable condition earns daily revenue of \($300\), and a machine in poor condition earns daily revenue of \($100\). At the start of each day, the engineer responsible for the machine can make one of the following three maintenance decisions: do nothing, repair the machine, or replace it with a new machine. One day is needed to repair the machine at a cost of \($500\), or to replace it at a cost of \($1\),000. A machine which is repaired starts the following day in excellent condition with probability 0.6, in acceptable condition with probability 0.3, or in poor condition with probability 0.1. A machine that is replaced always starts the following day in excellent condition.

(a) Formulate this model as an MDP.

(b) Determine an optimal maintenance policy that will maximize the expected total profi t over the next 3 days.

(c) Use LP to determine an optimal maintenance policy that will maximize the expected average profi t per day over an infi nite planning horizon.

(d) Use PI to determine an optimal maintenance policy over an infi nite planning horizon.

(e) Use LP with a discount factor of α = 0.9 to fi nd an optimal maintenance policy over an infi nite planning horizon.

(f) Use PI with a discount factor of α = 0.9 to fi nd an optimal maintenance policy and the associated expected total discounted reward vector over an infi nite planning horizon.  

Transcribed Image Text:

State Excellent (E) E A P 0.1 0.7 0.2 P = [pi]= Acceptable (A) Poor (P) 0 0.4 0.6 0 0 1