Run the code that we discussed above and change the probability calculation $($in step $6)$ to remove the smoothing as follows:

$$ Remove the $+1$ to n$-$gram count to make the equation calculate the count of n$-$grams as it is without smoothing.

$$ Remove the total count of unique n$-$gram vocabulary V$.$

The required change is bolded $($step $6)$ in the following code:

#Step $2$:

ngrams$\_$all $=\{1$:$[],2$:$[],3$:$[],4$:$[]\}$

for i in range$(4)$:

for each in tokenized$\_$text:

for j in ngrams$($each$,$ i$+1)$:

ngrams$\_$all$[$i$+1].$append$($j$)$;

#Step $3$:

ngrams$\_$voc $=\{1$:set$([]),2$:set$([]),3$:set$([]),4$:set$([])\}$

for i in range$(4)$:

for gram in ngrams$\_$all$[$i$+1]$:

if gram not in ngrams$\_$voc$[$i$+1]$:

ngrams$\_$voc$[$i$+1].$add$($gram$)$

#Step $4$:

total$\_$ngrams $=\{1$:$-1,2$:$-1,3$:$-1,4$:$-1\}$

total$\_$voc $=\{1$:$-1,2$:$-1,3$:$-1,4$:$-1\}$

for i in range$(4)$:

total$\_$ngrams$[$i$+1]=$ len$($ngrams$\_$all$[$i$+1])$

total$\_$voc$[$i$+1]=$ len$($ngrams$\_$voc$[$i$+1])$

#Step $5$:

ngrams$\_$prob $=\{1$:$[],2$:$[],3$:$[],4$:$[]\}$

for i in range$(4)$:

for ngram in ngrams$\_$voc$[$i$+1]$:

tlist $=[$ngram$]$

tlist.append$($ngrams$\_$all$[$i$+1].$count$($ngram$))$

ngrams$\_$prob$[$i$+1].$append$($tlist$)$

#Step $6$:

for i in range$(4)$:

for ngram in ngrams$\_$prob$[$i$+1]$:

ngram$[-1]=($ngram$[-1])/($total$\_$ngrams$[$i$+1])$

By comparing the probability results of the two n$-$gram models $($with and without smoothing$),$ we will notice that the probability of some the n$-$grams decreased after the smoothing $($marked in red$),$ as they have higher probability than others. On the other hand, the probability of less frequent n$-$grams are increased after smoothing

Exercises

Now, it$$s Your Turn! Build n$-$gram model with Add$-1$ smoothing for the following tokens:

tokenized$\_$text $=[[\text{'}$This$\text{'},\text{'}$is$\text{'},$ 'ngram','model'$],$

$[\text{'}$This$\text{'},\text{'}$is$\text{'},$ 'smoothed', 'model'$],$

$[\text{'}$This$\text{'},\text{'}$is$\text{'},$ 'unsmoothed', 'model'$],$

$[\text{'}$This$\text{'},\text{'}$is$\text{'},$ 'ngram', 'lab'$]]$

Task $1$: build smoothed n$-$gram model based on the above tokens.

Task $2$: what is the next word that is more likely to appear after $$This is$?$

Task $3$: compare the probability of this word $($from task $2)$ after and before smoothing. Is the probability increased or decreased after smoothing?