Input:

• lm: the language model you trained (the object you returned from the train_ngram_lm function)
• data: test data
• vocab: vocab
• order: order of the lm

Output:

• the perplexity of test data

Hint:

• If the history is not in the lm object, back-off to (n-1) order history to check if it is in lm. If no history can be found, just use 1/|V| where |V| is the size of vocabulary.

[ ]

def compute_perplexity(lm, data, vocab, order=3):

# pad according to order

order -= 1

data = [''] * order + data

for i in range(len(data) - order):

h, w = ' '.join(data[i: i+order]), data[i+order]

"""

IMPLEMENT ME!

# if h not in lm, back-off to n-1 gram and look up again

"""

Input:

• lm: the language model you trained (the object you returned from the train_ngram_lm function)
• data: test data
• vocab: vocab
• order: order of the lm

Output:

• the perplexity of test data

Hint:

• If the history is not in the lm object, back-off to (n-1) order history to check if it is in lm. If no history can be found, just use 1/|V| where |V| is the size of vocabulary.

• def compute_perplexity(lm, data, vocab, order=3):

  # pad according to order

  order -= 1

  data = [''] * order + data

  for i in range(len(data) - order):

  h, w = ' '.join(data[i: i+order]), data[i+order]

  """

  IMPLEMENT ME!

  # if h not in lm, back-off to n-1 gram and look up again

  """

  pass

• for o in [1, 2, 3, 4]:

  lm = train_ngram_lm(data['train'], order=o)

  print('order {} ppl {}'.format(o, compute_perplexity(lm, data['test'], vocab, order=o)))