Section 2: Markov Decision Processes

Question 1:

What is the special property of Markov chains?

Question 2:

Everyday morning, the working condition of an equipment is inspected, then classified as:

state 0 = as new, state 1 = slight damage, state 2 = important damage, or state 3 = out of order.

Suppose the manager has two operation policies to choose from and applies the first one defined as follows:

Policy 1: do nothing in states 0, 1, 2 and replace the equipment in state 3.

Suppose that the replacement process takes one day to complete with a lost profit of $1000 and a replacement cost of $ 500, and that the costs of defectives are $ 0 for state 0, $ 500 for state 1, and $ 800 for state 2.

If the steady-state probabilities of Policy 1 are:

0=2/9, 1=2/9, 2=3/9, 3=2/9

1. What is the expected average operation cost per day? Explain your answer
2. If the expected average operation cost per day for policy 2 is $ 666.66, what is the optimal policy?
3. Suppose that the manager applies an operation policy with the following transition matrix, What is the steady-state probabilities of the operation policy? Explain your answer

	0	1	2	3
0	0	1/2	1/2	0
1	0	1/2	1/4	1/4
2	0	0	1/2	1/2
3	1	0	0	0

Question 3:

At a jewelry, every month the precision of a laser cutting tool is inspected, then classified as: state 0 = as new, state 1 = low imprecision, state 2 = major imprecision, or state 3 = out of order.

Suppose the jeweler has two operation policies to choose from and applies the first one defined as follows:

Policy 1: do nothing in states 0, 1, 2 and replace the laser cutting tool in state 3.

Suppose that the replacement process takes one month to complete with a lost profit of $16000 and a replacement cost of $32000, and that the costs of defectives are $ 0 for state 0, $8000 for state 1, and $24000 for state 2.

If the steady-state probabilities of Policy 1 are:

0=2/9, 1=3/9, 2=2/9, 3=2/9

What is the expected average operation cost per month?