Law enforcement agencies monitor social media sites on a regular basis, as a way to identify and assess potential crimes and terrorism activities. For example, certain keywords on Facebook pages are tracked, and the data are compiled into a data mining model to determine whether or not the Facebook page is a potential threat. Officer Matthew Osorio is assigned to explore data mining techniques that can be used for this purpose. He starts by experimenting with KNN algorithms to monitor and assess social media sites with war-related terms such as “warfare,” “bomb,” and “attack” as well as suspicious keywords such as “extremist,” “radical,” and “conspiracy.” He collects a data set with 300 observations, a portion of which is shown in the accompanying table. Each record in the data set includes the following variables: Threat (1 if yes, 0 otherwise), the number of suspicious words (WarTerms and Keywords), and the number of hyperlinks to or mentioning of suspicious sites. 

a. Perform KNN analysis on the data set. What is the optimal value of k? 

b. What is the misclassification rate for the optimal k? 

c. Report and interpret the accuracy, specificity, sensitivity, and precision rates for the test data set (for Analytic Solver) or validation data set (for R). 

d. Generate the cumulative lift chart. Does the entire lift curve lie above the baseline? 

e. Generate the decile-wise lift chart. What is the lift value of the leftmost bar? 

f. Generate the ROC curve. What is the area under the ROC curve (or the AUC value)? 

g. Comment on the performance of the KNN classification model.

Transcribed Image Text:

Threat 0 0 : 1 WarTerms 6 3 ⠀ 4 Keywords 5 תות 5 1 4 Links 5 80 5 8 ⠀ 2