Q$\mathrm{.}4.$ Figure $($a$)$ below shows a house with $6$ rooms, where the rooms labeled $0-4$ are internal

rooms and $5$ is the outside "room". Doors lead from each room to some others, as shown.

An Agent can be placed in any of the $6$ rooms, which can be considered his starting State.

The objective is to take Actions to move from the starting room to room $\text{'}5\text{'},$ which can

also be called the Goal State. Transiting from one room $($state$)$ to another through a door

is considered an Action, that leads to a Reward. The Reward associated with each Action

is expressed in the Table $R,$ which is like a Reward Matrix $-$ each row represents a State

and each column an Action, and the value of that Action is the corresponding element.

All infeasible Actions $($i$.$e$.$ no doors exist to execute such actions$)$ are shown as $-1.$

You are to use the Q$-$Learning process to update the Q$-$matrix shown as Table Q$.$ As you

know, the Q$-$Learning process proceeds through Episodes, and in each Episode a

sequence of states is followed which updates the Q$-$matrix at each step, till the Goal

$($Terminal$)$ state is reached signaling the end of that Episode. The Q$-$Learning update

equation is the following:

$Q^{\text{n}\text{i}\text{t}}(s_i,a_t)=(1-)Q^{\text{n}\text{i}\text{t}}(s_t,a_t)+(R(s_t)+ma{x}_{a_{t+4}}Q(s_{i+1},a_{t+1}))$

where all notations and symbols follow from what you have seen in your class.

Fig $($a$)$: the house with six labelled rooms.

Tabie R: Rows are states, columns are

Actions. Values are Rewards.

Tabie Q: Rows are states, columns are

Actions. Elements are Q$-$Values.

You are to take $$ as $1,$ as $0\mathrm{.}8,$ and then use the given Q$-$Matrix and R$-$Matrix to update

the $Q-$Matrix at each state in the following Episode $($Sequence of states$)$: $2=>3=>1=>5.$

Please provide a handwritten solution if possible