Question $3$:

In a Markov Decision Process $($MDP$),$ the state $sin\{1,2,3,$dots,$99\}$ is defined as the total capital

and action ain$\{0,1,$dots,min$(s,100-s)\}$ is the current investment. In each step, an investment

will be successful with $p$ probability that lead to double the invested money. If it fails, the

investment will be lost. The termination condition is defined as reaching the total capital to $100$ or

$0.$ The reward is zero in each transition except the game ends with full capital. In this case, the

reward is $+1.$

a$)$ For $p=0\mathrm{.}25$ and $p=0\mathrm{.}4,$ implement value iteration via MATLAB to obtain optimum

amount of investment with respect to the capital, and illustrate value estimates$-$capital $($i$.$e$,$

$y-$axis is value estimates, $x-$axis is capital$)$ and optimum policy$-$capital graphs.

b$)$ For $p=0\mathrm{.}25$ and $p=0\mathrm{.}4,$ implement policy iteration via MATLAB to obtain optimum

amount of investment with respect to the capital, and illustrate value estimates$-$capital $($i$.$e$,$

$y-$axis is value estimates, $x-$axis is capital$)$ and optimum policy$-$capital graphs.