Problem 2: Decision Tree, post-pruning and cost complexity parameter using sklearn 0.22

We will use a pre-processed natural language dataset in the CSV file "spamdata.csv" to classify emails as spam or not. Each row contains the word frequency for 54 words plus statistics on the longest "run" of captial letters.

Word frequency is given by: =/

Where is the frequency for word , is the number of times word appears in the email, and is the total number of words in the email. We will use decision trees to classify the emails.

TO DO 1: Complete the function get_spam_dataset to read in values from the dataset and split the data into train and test sets.

def get_spam_dataset(filepath="data/spamdata.csv", test_split=0.1): ''' get_spam_dataset Loads csv file located at "filepath". Shuffles the data and splits it so that the you have (1-test_split)*100% training examples and (test_split)*100% testing examples. Args: filepath: location of the csv file test_split: percentage/100 of the data should be the testing split Returns: X_train, X_test, y_train, y_test, feature_names (in that order) first four are np.ndarray ''' # complete your code here return 0

TO DO 2: Import the data set into five variables: X_train, X_test, y_train, y_test, label_names # Uncomment and edit the line below to complete this task.

test_split = 0.1 # default test_split; change it if you'd like; ensure that this variable is used as an argument to your function # your code here

# X_train, X_test, y_train, y_test, label_names = np.arange(5)

TO DO 3: Build a decision tree classifier using the sklearn toolbox. Then compute metrics for performance like precision and recall. This is a binary classification problem, therefore we can label all points as either positive (SPAM) or negative (NOT SPAM).

def build_dt(data_X, data_y, max_depth = None, max_leaf_nodes =None): ''' This function builds the decision tree classifier and fits it to the provided data. Arguments data_X - a np.ndarray data_y - np.ndarray max_depth - None if unrestricted, otherwise an integer for the maximum depth the tree can reach. Returns: A trained DecisionTreeClassifier ''' # complete your code here