In the Pollard assignment you computed a unigram frequency distribution for the Brown corpus. You will need that for this assignmewnt. This time you will do a bigram distribution: >>> import nltk >>> from nltk.corpus import brown >>> from nltk import bigrams >>> brown_bigrams = bigrams (brown.words()) It is instructive to compare brown.words, which we used in the last assignment, with brown.bigrams: >>> brown.words[:10] ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of'] >>> brown_bigrams[:10] [('The', 'Fulton'), ('Fulton', 'County'), ('County', 'Grand'), ('Grand', 'Jury'), ('Jury', 'said'), ('said', 'Friday'), ('Friday', 'an'), an', 'investigation'), ('investigation', 'of'), ('of', "Atlanta's")] So brown.words() returns a list of the words, while brown.bigrams() returns a list of word pairs. Notice the the second word of the first pair becomes the first word of the second pair, and the the second word of the second pair, the first word of the third, and so on. Since each word in Brown becaome the first word of a bigram except the last, there is exactly one more word token than there are bogram tokens: >>> len (brown_bigrams) 1161191 >>> len (brown.words()) 1161192 Create a new frequency distribution of the Brown bigrams. Plot the cumulative frequency distribution of the top 50 bigrams. Then do add one smoothing on the bigrams. This will require adding one to all the bigram counts, including those that previously had count 0. You will also need to change the ungram counts appropriately. You will com- pute all possible bigrams using the known vocabulary, so use the keys of the unigram Brown distribution you created before to compute the set of possible bigrams. The vocabulary size from that exercise should be 49815. Then having added 1 to all the bigram counts, you must compute at least the following Probabilities Compute following: 1. P(the lin) before and after smoothing (Pmle and Plaplace); 2. P(in the) before and after smoothing; 3. P(said the) before and after smoothing. 4. P(the I said) before and after smoothing. In some cases you will to use the unigram counts to compute these probabilities. Remember that the unigram counts must change too when smoothing. Turn in these values and the Python code you used to compute them. In the Pollard assignment you computed a unigram frequency distribution for the Brown corpus. You will need that for this assignmewnt. This time you will do a bigram distribution: >>> import nltk >>> from nltk.corpus import brown >>> from nltk import bigrams >>> brown_bigrams = bigrams (brown.words()) It is instructive to compare brown.words, which we used in the last assignment, with brown.bigrams: >>> brown.words[:10] ['The', 'Fulton', 'County', 'Grand', 'Jury', 'said', 'Friday', 'an', 'investigation', 'of'] >>> brown_bigrams[:10] [('The', 'Fulton'), ('Fulton', 'County'), ('County', 'Grand'), ('Grand', 'Jury'), ('Jury', 'said'), ('said', 'Friday'), ('Friday', 'an'), an', 'investigation'), ('investigation', 'of'), ('of', "Atlanta's")] So brown.words() returns a list of the words, while brown.bigrams() returns a list of word pairs. Notice the the second word of the first pair becomes the first word of the second pair, and the the second word of the second pair, the first word of the third, and so on. Since each word in Brown becaome the first word of a bigram except the last, there is exactly one more word token than there are bogram tokens: >>> len (brown_bigrams) 1161191 >>> len (brown.words()) 1161192 Create a new frequency distribution of the Brown bigrams. Plot the cumulative frequency distribution of the top 50 bigrams. Then do add one smoothing on the bigrams. This will require adding one to all the bigram counts, including those that previously had count 0. You will also need to change the ungram counts appropriately. You will com- pute all possible bigrams using the known vocabulary, so use the keys of the unigram Brown distribution you created before to compute the set of possible bigrams. The vocabulary size from that exercise should be 49815. Then having added 1 to all the bigram counts, you must compute at least the following Probabilities Compute following: 1. P(the lin) before and after smoothing (Pmle and Plaplace); 2. P(in the) before and after smoothing; 3. P(said the) before and after smoothing. 4. P(the I said) before and after smoothing. In some cases you will to use the unigram counts to compute these probabilities. Remember that the unigram counts must change too when smoothing. Turn in these values and the Python code you used to compute them