Task $2\mathrm{.}2,2\mathrm{.}4(4$ Points $+2$ Points$))$ Test overall q$\_$learning function implementation $(0/6)$

Test Failed: get$\_$next$\_$state$()$ missing $1$ required positional argument: 'grid$\_$size' $```$

# TASKS $2\mathrm{.}2$ and $2\mathrm{.}4$: Q$-$learning algorithm $($following epsilon$-$greedy policy$)$

# Inputs:

# q$\_$table, r$\_$table: initialized by calling the initialize$\_$q$\_$r$\_$tables function inside the main function

# start$\_$pos, goal$\_$pos: given by the get$\_$random$\_$start$\_$goal function based on student$\_$id and grid$\_$size

# num$\_$episodes: taken from the global constant EPISODES $($you need to determine the episodes needed to train the agent

# grid$\_$size: To try different grid sizes, change the GRID$\_$SIZE global constant

# alpha, gamma, epsilon: DO NOT CHANGE

# Outputs:

# q$\_$table: the final q$\_$table after training

def q$\_$learning$($start$\_$pos, goal$\_$pos, q$\_$table$=$q$\_$table$\_$g$,$ r$\_$table$=$r$\_$table$\_$g$,$ num$\_$episodes$=$EPISODES, alpha$=0\mathrm{.}1,$ gamma$=0\mathrm{.}9,$ epsil

for episode in range$($num$\_$episodes$)$:

# Initialize the state index corresponding to the starting position

current$\_$state$\_$index $=$ start$\_$pos$[0]*$ grid$\_$size $+$ start$\_$pos$[1]$ # Ensure this results in an integer

current$\_$state$\_$pos $=$ start$\_$pos # current$\_$state$\_$pos has current row, column position of the agent

done $=$ False

while not done:

# $[$Task $2\mathrm{.}1]$ Get action using epsilon$-$greedy policy

action $=$ get$\_$action$($q$\_$table, epsilon, current$\_$state$\_$index$)$

# $[$Task $2\mathrm{.}2]$ Get next state based on the chosen action

next$\_$state$\_$pos $=$ get$\_$next$\_$state$($current$\_$state$\_$pos, action, grid$\_$size$)$ # Pass grid$\_$size

next$\_$state$\_$index $=$ next$\_$state$\_$pos$[0]*$ grid$\_$size $+$ next$\_$state$\_$pos$[1]$ # Correct calculation of index

# $[$Task $2\mathrm{.}3]$ Update the Q$-$table using Q$-$learning formula

q$\_$table $=$ update$\_$q$\_$table$($q$\_$table, r$\_$table, current$\_$state$\_$index, action, next$\_$state$\_$index, alpha, gamma$)$

# Update the 'state' to the next state index

current$\_$state$\_$pos $=$ next$\_$state$\_$pos

current$\_$state$\_$index $=$ next$\_$state$\_$index

# $[$Task $2\mathrm{.}4]$ End episode if goal is reached

if current$\_$state$\_$pos $==$ goal$\_$pos:

done $=$ True # Episode ends when the goal is reached

q$\_$table$\_$g $=$ q$\_$table # DO NOT CHANGE THIS LINE

$```$ Task $1(8$ Points$))$ Test get$\_$next$\_$state$()$ function implementation $(0/8)$

Test Failed: unsupported operand type$($s$)$ for divmod$()$: 'tuple' and 'int'

Task $2\mathrm{.}1(6$ Points$))$ Test get$\_$action implementation $(6/6)$

Task $2\mathrm{.}2,2\mathrm{.}4(4$ Points $+2$ Points$))$ Test overall q$\_$learning function implementation $(0/6)$

Test Failed: get$\_$next$\_$state$()$ missing $1$ required positional argument: 'grid$\_$size'

Task $2\mathrm{.}3(5$ Points$))$ Test update$\_$q$\_$table function $(0/5)$

Test Failed: unsupported operand type$($s$)$ for divmod$()$: 'tuple' and 'int' $```$

# TASK $2\mathrm{.}3$

# Complete the update$\_$q$\_$table function $($in Task $2\mathrm{.}3)$

# This function will be called from the q$\_$learning$(...)$ function

# Inputs:

# q$\_$table, r$\_$table, current$\_$state$\_$index, action, next$\_$state$\_$index, alpha$=0\mathrm{.}1,$ gamma$=0\mathrm{.}9$

# Outputs:

# q$\_$table: with updated Q values

def update$\_$q$\_$table$($q$\_$table, r$\_$table, current$\_$state$\_$index, action, next$\_$state$\_$index, alpha$=0\mathrm{.}1,$ gamma$=0\mathrm{.}9)$:

best$\_$next$\_$action $=$ np$.$argmax$($q$\_$table$[$next$\_$state$\_$index$])$

td$\_$target $=$ r$\_$table$[$current$\_$state$\_$index$][$action$]+$ gamma $*$ q$\_$table$[$next$\_$state$\_$index$][$best$\_$next$\_$action$]$

td$\_$error $=$ td$\_$target $-$ q$\_$table$[$current$\_$state$\_$index$][$action$]$

q$\_$table$[$current$\_$state$\_$index$][$action$]+=$ alpha $*$ td$\_$error

return q$\_$table

$```$