Question 1. Consider the problem from the previous assignment, in which you are designing a control software for a robot that moves packages ariving via automated lines. The number of packages that arrive in a period t = 1,..., N is a random variable and the number of packages arriving in each period is independent of the number of those arriving in other periods. Suppose that pi denotes the probability that i packages arrive in a period (i = 0, 1, ..., m), where i=0 Pi = 1. The expected number of arriving packages in a period is positive. At the end of each period, the robot may move packages to the shipping dock. If the robot takes packages for moving, then it takes all packages that have arrived so far and have not yet been moved. The maximal number of packages that the robot can move in one load is M > m. The cost of moving is K > 0 and the moving time is negligible. The cost incurred in each period is w > 0 for each package that has not been moved. All packages, if any, must be moved to the shipping dock in period N. (Q1.1) Formulate an infinite horizon problem to determine an optimal moving policy that minimizes the combined doscounted expected costs of moving and waiting. The discount factor is 0.9. (Q1.2) Argue that the optimal moving policy for the problem in (Q1.1) is a stationary Markov policy with a threshold decision rule: move if x > x* and wait (do not move) if x x*. Question 2. You are designing a control software for a robot moving packages that arrive via automated lines in a storage room. The number of packages that arrive in one period t = 1,..., N is a random variable and the number of packages arriving in each period is independent of the number of those arriving in other periods. Suppose that p; denotes the probability that i packages arrive in a period (i=0,1,...,m), where o Pi = 1. The expected number of arriving packages in a period is positive. At the end of each period, the robot may move packages to the shipping dock. If the robot takes packages for moving, then it takes all packages that have arrived so far and have not yet been moved. The maximal number of packages that the robot can move in one load is M > 1. The cost of moving is K > 0 and the moving time is negligible. The cost incurred in each period is w > 0 for each package that has not been moved. All packages, if any, must be moved to the shipping dock in period N. Formulate a Markov decision problem to determine an optimal moving policy that minimizes the combined expected costs of moving and waiting over N periods.