Solve the following:

Problem 1 Consider the Markov chain with three states, S = {1, 2,3}, that has the following transition matrix P = O O a. Draw the state transition diagram for this chain. b. If we know P(X1 = 1) = P(X1 = 2) = 1, find P(X1 = 3, X2 = 2, X3 = 1).Problem 3 Consider the Markovr chain of Example 2. Again assume X9 = 3. We would like to find the expected time (number of steps) until the chain gets absorbed in R1 or R2. More specifically, let T be the absorption time, i.e., the rst time the chain visits a state in R1 or R2. We would like to fin-Cl E[T|X|] = 3]. Problem 5 Consider the Markov chain shown in Figure 11.20. Figure 11.2{1 A state transition diagram. a. Is this chain irreducible? b. Is this chain aperiodic? c. Find the stationary distribution for this chain. d. Is the stationary distribution a limiting distribution for the chain? Consider the Markov chain shown in Figure 11.19. Assume Xo = 1, and let R be the first time that the chain returns to state 1, i.e., R = min{n > 1: Xn =1}. Find E[R Xo = 1]. 2 3 Figure 11.19 - A state transition diagram.Problem 6 Consider the Markov chain shown in Figure 11.21. Assume that