Suppose that the condition of a machine can be described by one of the following three states: 

The daily behavior of the machine when it is left alone for 1 day is modeled as an absorbing unichain with the following transition probability matrix:

The machine is observed at the start of each day. Suppose initially that the machine is in state WP at the beginning of the day. If, with probability 0.3, it works properly throughout the day, it earns \($1\),000 in revenue. If, with probability 0.5, the machine deteriorates during the day, it earns \($500\) in revenue and enters state WI, working intermittently. If, with probability 0.2, the machine fails during the day, it earns zero revenue and enters state NW, not working. Suppose next that the machine starts the day in state WI. If, with probability 0.2, it works intermittently throughout the day, it earns \($500\) in revenue. If, with probability 0.8, the machine fails during the day, it earns zero revenue and enters state NW. Daily revenue is summarized in the table below: 

At the beginning of the day, the engineer responsible for the machine can make one of the following four maintenance decisions: do nothing (DN), perform preventive maintenance (PM), repair the machine (RP), or replace the machine (RL). The costs of the maintenance actions are summarized in the table below: 

Replacement, and a repair which is completely successful, always produces a machine that works properly throughout the day. A partly successful repair when the machine is in state NW enables a machine to work intermittently throughout the day. An unsuccessful repair when the machine is in state NW leaves the machine not working throughout the day. An unsuccessful repair when the machine is in state WI leaves the machine working intermittently throughout the day. Preventive maintenance which is completely successful enables a machine to remain in its current state throughout the day. When the machine begins the day in state NW, a repair is completely successful with probability 0.5, partly successful with probability 0.4, and unsuccessful with probability 0.1. When the machine starts the day in state WI, a repair is successful with probability 0.6, and unsuccessful with probability 0.4. When the machine begins the day in state WP, preventive maintenance is completely successful with probability 0.7, partly successful with probability 0.2, and unsuccessful with probability 0.1. When the machine starts the day in state WI, preventive maintenance is successful with probability 0.8, and unsuccessful with probability 0.2.

(a) Construct the vector of expected immediate rewards associated with each state and decision.

(b) Formulate this model as an MDP.

(c) Determine an optimal maintenance policy that will maximize the expected total profit over the next 3 days.

(d) Use LP to determine an optimal maintenance policy that will maximize the expected average profit per day over an infinite planning horizon.

(e) Use PI to determine an optimal maintenance policy over an infinite planning horizon.

(f) Use LP with a discount factor of α = 0.9 to fi nd an optimal maintenance policy over an infinite planning horizon.

(g) Use PI with a discount factor of α = 0.9 to fi nd an optimal maintenance policy and the associated expected total discounted reward vector over an infinite planning horizon.  

Transcribed Image Text:

State NW WI WP Condition Not Working Working Intermittently Working Properly