Read example 22.10 in the link above. It considers the case of two recurrent classes, where one of the recurrent classes is ergodic, and the other is periodic. Find lim v P. p3 where v 0 = (000100000). Find the answer theoretically (using a mathematical computation) AND confirm it using a simulation. Submit both your 0 code/output as well as mathematical work. 100 It can be used to replace one homework grade (for both parts done entirely and correctly) OR up to 4 points on a midterm. outcome space. By Lemma 22.2, P{X, ET} = v(T) = 0. The measure v may assign zero probability to some of the recurrent classes, but it must assign non- zero probability to at least one recurrent class as v(S) = 1. By the Partition Theorem (see exercises below) we have Vj = P(X) = j) = P[X, = j|X, e Ri] P(X, E Ri] i: P[X, ER:]>0 vr;(j) v(R;). i: v(R)>0 == (22.9) Now v(R) (R.) = 1, i: v(R)>0 i=1 where the first equality holds as the additional terms thrown in are all zero. Letting li = v(Ri) for 1 0 and Xiai li = 1. Using (22.9) and Lemma 22.6 we get that: k Vj = ti:v|r; (j) I like i: >0 i:li>0 i=1 Again, the last equality holds because the addition terms thrown in are all zero. EXAMPLE 22.10. Suppose a Markov chain on states S = {1, 2, ...,9} has PTM 0 0 1/2 0 0 0 0 P = 1 0 0 1/4 0 0 0 0 0 0 1 1/2 0 0 0 0 0 0 0 0 0 0 0 0 1/2 1/4 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1/2 1/2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2/3 1/3 1/2 1/2 0 0 0 0 0 0 a diagram of which is shown in Figure 22.11 below. Figure 22.11. Diagram of P. 2 5 back to 5 $22. ERGODIC DECOMPOSITION 99 - = Clearly R1 {1,2,3}, R2 = {5, 6, 7, 8, 9} and T = {4}. The recurrent class R is aperiodic and the reader will readily verify that UR ( 0 0 0 0 0 0). Also, R2 has period d = 3 and if we take 5 D(0) we have D(0) {5}, D(1) = {6,7} and D(2) {8,9}. Here Q = P3 and the reader should verify that the Q-invariant measures on D(0), D(1), and D(2) are: D(0) = (00 0 0 1 0 0 00), (0 0 0 0 0 00), and D(2) (0 0 0 0 0 0 0 ), producing 4R2 = (0 0 0 0 36 36) as the unique P-invariant measure on R2. The general P-invariant distribution on S is therefore given by ( 0 ), 5 where 11, 12 > 0 and 11 + 12 = 1. MD(1) 0 7X2 36 36 EXERCISES 1. Use the techniques developed in this section to analyze the two-state Markov chains with PTM's and where p + q = 1. Treat the case 0 1 (9 (69) separately. For problems 2 7, the PTM 0 1 0 1 0 0 .250 0.5 0 0.5 .5 0 0 0 0 0 0 0 0 0 P = 0 0 0 0 .5 1 0 0 0 .25 0 0 0 0 0 .5 0 0 0 0 1 defines a Markov chain on state space S = {1,2,3,4,5,6,7}. Read example 22.10 in the link above. It considers the case of two recurrent classes, where one of the recurrent classes is ergodic, and the other is periodic. Find lim v P. p3 where v 0 = (000100000). Find the answer theoretically (using a mathematical computation) AND confirm it using a simulation. Submit both your 0 code/output as well as mathematical work. 100 It can be used to replace one homework grade (for both parts done entirely and correctly) OR up to 4 points on a midterm. outcome space. By Lemma 22.2, P{X, ET} = v(T) = 0. The measure v may assign zero probability to some of the recurrent classes, but it must assign non- zero probability to at least one recurrent class as v(S) = 1. By the Partition Theorem (see exercises below) we have Vj = P(X) = j) = P[X, = j|X, e Ri] P(X, E Ri] i: P[X, ER:]>0 vr;(j) v(R;). i: v(R)>0 == (22.9) Now v(R) (R.) = 1, i: v(R)>0 i=1 where the first equality holds as the additional terms thrown in are all zero. Letting li = v(Ri) for 1 0 and Xiai li = 1. Using (22.9) and Lemma 22.6 we get that: k Vj = ti:v|r; (j) I like i: >0 i:li>0 i=1 Again, the last equality holds because the addition terms thrown in are all zero. EXAMPLE 22.10. Suppose a Markov chain on states S = {1, 2, ...,9} has PTM 0 0 1/2 0 0 0 0 P = 1 0 0 1/4 0 0 0 0 0 0 1 1/2 0 0 0 0 0 0 0 0 0 0 0 0 1/2 1/4 0 0 0 0 0 0 0 1 0 1 0 0 0 0 0 0 0 0 1/2 1/2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2/3 1/3 1/2 1/2 0 0 0 0 0 0 a diagram of which is shown in Figure 22.11 below. Figure 22.11. Diagram of P. 2 5 back to 5 $22. ERGODIC DECOMPOSITION 99 - = Clearly R1 {1,2,3}, R2 = {5, 6, 7, 8, 9} and T = {4}. The recurrent class R is aperiodic and the reader will readily verify that UR ( 0 0 0 0 0 0). Also, R2 has period d = 3 and if we take 5 D(0) we have D(0) {5}, D(1) = {6,7} and D(2) {8,9}. Here Q = P3 and the reader should verify that the Q-invariant measures on D(0), D(1), and D(2) are: D(0) = (00 0 0 1 0 0 00), (0 0 0 0 0 00), and D(2) (0 0 0 0 0 0 0 ), producing 4R2 = (0 0 0 0 36 36) as the unique P-invariant measure on R2. The general P-invariant distribution on S is therefore given by ( 0 ), 5 where 11, 12 > 0 and 11 + 12 = 1. MD(1) 0 7X2 36 36 EXERCISES 1. Use the techniques developed in this section to analyze the two-state Markov chains with PTM's and where p + q = 1. Treat the case 0 1 (9 (69) separately. For problems 2 7, the PTM 0 1 0 1 0 0 .250 0.5 0 0.5 .5 0 0 0 0 0 0 0 0 0 P = 0 0 0 0 .5 1 0 0 0 .25 0 0 0 0 0 .5 0 0 0 0 1 defines a Markov chain on state space S = {1,2,3,4,5,6,7}