Hello, I'm studying stochastic processes but very stuck on this question at the moment. Our professor's explanation of this topic was very unclear, so I'm having difficulty constructing a solution. Any help is greatly appreciated. Thanks in advance!

1. Consider a robot exploring an unknown terrain for new resources. After some time, it has learned that its exploration algorithm takes it through three types of terrains [1: forest, 2: mountains, 3: swamps]. Let us model the terrain exploration problem as a Markov chain {XH}'I'LEU-r with the transition matrix .3 .1 .1 p: .2 .7 .1 . (1} a u 1 Note that once the robot is in state (3: swamps], it cannot get out. a.) {4 points) We assurne that the reward function r : {1, 2, 3} 1- R20 encodes the amount of resources in each terrain. Let T g min{n 23 : X\