At the beginning of every month, a woman executes diagnostic software on her personal computer to determine whether it is virus-free or is infected with a malignant or benign virus. She models the condition of her computer as a Markov chain with the following three states:

If her computer is virus-free at the start of the current month, the probability that it will be virus-free at the start of the next month is 0.45, the probability that it will be infected with a benign virus is 0.40, and the probability that it will be infected 

with a malignant virus is 0.15. If her computer is infected with a virus, she can hire either virus-removal consultant A or virus removal consultant B. Both consultants require a full month in which to attempt to remove a virus, but are not always successful. Consultant A charges higher monthly fees than consultant B because she has higher probabilities of success. To remove a benign virus, consultant A charges \($160\) and consultant B charges \($110\). To remove a malignant virus, consultant A charges \($280\) and consultant B charges \($200\). Consultant A has a 0.64 probability of removing a malignant virus and a 0.28 probability of reducing it to a benign virus. She has a 0.92 probability of removing a benign virus and a 0.08 probability of leaving it unchanged. Consultant B has a 0.45 probability of removing a malignant virus and a 0.38 probability of reducing it to a benign virus. She has a 0.62 probability of removing a benign virus, and a 0.04 probability of transforming it into a malignant virus.

(a) Model this virus-removal problem as a recurrent MDP.

(b) Find an optimal virus-removal policy and its expected total cost over a 3 month planning horizon.

(c) Use LP to find an optimal virus-removal policy and its expected average cost over an infinite planning horizon.

(d) Use PI to find an optimal virus-removal policy and its expected average cost over an infinite planning horizon.

(e) Use LP with a discount factor of α = 0.9 to find an optimal virus-removal policy over an infinite planning horizon.

(f) Use PI with a discount factor of α = 0.9 to find an optimal virus-removal policy and the associated expected total discounted cost vector over an infinite planning horizon.  

Transcribed Image Text:

State Condition F Virus-free RM B Benign virus M Malignant virus