Section :Markov Decision Processes

Question 1:

Everyday morning, the working condition of an equipment is inspected, then classified as:

state 0 = as new, state 1 = slight damage, state 2 = important damage, or state 3 = out of order.

Suppose the manager has two operation policies to choose from and applies the first one defined as follows:

Policy 1: do nothing in states 0, 1, 2 and replace the equipment in state 3.

Suppose that the replacement process takes one day to complete with a lost profit of $1000 and a replacement cost of $ 500, and that the costs of defectives are $ 0 for state 0, $ 500 for state 1, and $ 800 for state 2.

If the steady-state probabilities of Policy 1 are:

0=2/9, 1=2/9, 2=3/9, 3=2/9

1. What is the expected average operation cost per day? Explain your answer
2. If the expected average operation cost per day for policy 2 is $ 666.66, what is the optimal policy?
3. Suppose that the manager applies an operation policy with the following transition matrix, What is the steady-state probabilities of the operation policy? Explain your answer

	0	1	2	3
0	0	1/2	1/2	0
1	0	1/2	1/4	1/4
2	0	0	1/2	1/2
3	1	0	0	0