Objective

Reinforcement Learning

Homework $3$: Model$-$Free Monte Carlo Prediction

In this homework assignment, you will apply the Monte Carlo prediction method to estimate the state values for a four$-$state problem. You will be provided with four episodes. Your task is to calculate the state values using the Monte Carlo method with a specified discount factor $($gamma$)$ and initial values for the states.

Problem Setup

$-$ States $($S$)$: Four states, labeled as S$1,$ S$2,$ S$3,$ and S$4.$

$-$ Rewards $($R$)$: Provided within each episode, including a final reward. $-$ Discount Factor $(\backslash$gamma $)$: $0\mathrm{.}9$

$-$ Initial State Values $($V$)$:

$-$ V$($S$1)=0-$ V$($S$2)=0-$ V$($S$3)=0-$ V$($S$4)=0$

Episodes

$-$ S$1,0,$ S$2,1,$ S$3,0,$ S$4,10$

$-$ S$1,0,$ S$2,0,$ S$2,0,$ S$3,0,$ S$4,-5-$ S$1,0,$ S$1,1,$ S$2,0,$ S$3,0,$ S$4,8-$ S$1,0,$ S$2,0,$ S$2,0,$ S$3,1,$ S$4,12$

Tasks

$1.$ Calculate the returns $($G$)$ for each state in each episode.

$2.$ Use the Every$-$Visit Monte Carlo method to update the state values $($V$)$ based on the returns and the discount factor $(\backslash$gamma $).$

$3.$ Calculate the updated values for each state after processing all four episodes.