Using Python

TF-IDF Application For this assignment, you will build a application that processes data files containing natural language. The purpose of this assignment is to provide a real-world application for the concepts discussed in Chapter 5, process data files and dictionaries. This application builds upon Example 4, Using the Dictionary as a Frequency Table, from Section 5.3. The goal of this assignment is to create a numerical representation of the natural language data contained in the data files:

80% of this assignment will be graded based on correctness, 20% of this assignment will be graded based on the quality of your code (e.g. your use of functions, variables, whitespace, output formatting etc.). Correctness will be determined based on the contents of your output files.

Submit your code as a single Python file called tfidf.py?. Programs will be downloaded and run using Python 3

For the example corpus (i.e. collection of documents) document_key, document_contents dl, a this is a sample d2, this is another example another example example The goal is to produce the following matrix: document_key, a, another, example,is,sample, this dl,0.12,0,0,0,0.06,0 D2, 0,0.09,0.13,0,0,0 The desired output for each of the input data files takes the form Document space t, t t 2 |erm vector space D a12 a13 .. ain 2n 331 32 33. 3rn 2122 23 m mi m2 dm3. am Where D is the document_key, t is a term, and a is the TF-IDF statistic for t in D For the example corpus (i.e. collection of documents) document_key, document_contents dl, a this is a sample d2, this is another example another example example The goal is to produce the following matrix: document_key, a, another, example,is,sample, this dl,0.12,0,0,0,0.06,0 D2, 0,0.09,0.13,0,0,0 The desired output for each of the input data files takes the form Document space t, t t 2 |erm vector space D a12 a13 .. ain 2n 331 32 33. 3rn 2122 23 m mi m2 dm3. am Where D is the document_key, t is a term, and a is the TF-IDF statistic for t in D