There is data in a zip file I have to analyze

20.4 Clustering auto posts. Use the data and scenario described in this chapter's example. The task is to cluster the auto posts.

a)Following the example in this chapter, preprocess the documents, without a label vector.

b)Use the lsa package to create 10 concepts.

c)Before doing the clustering, state how many natural clusters you expect to find.

d)Perform hierarchical clustering and inspect the dendrogram.

i.From the dendrogram, how many natural clusters appear?

ii.Examining the dendrogram as it branches beyond the number of main clusters, select a sub-cluster and assess its characteristics.

e)Perform k-means clustering for two clusters and report how distant and separated they are (using between-cluster distance and within cluster dispersion).

Current code stated is below:

install.packages("SnowballC")

install.packages("tm")

install.packages("lsa")

install.packages("NLP")

library(SnowballC)

library(NLP)

library(tm)

library(lsa)

###A

# step 1: import

# read zip file into a corpus

corp <- Corpus(ZipSource("Documents/GSU/Spring 2020/Data Mining/HW5/AutoAndElectronics.zip", recursive = T))

# step 2: text preprocessing

# tokenization

corp <- tm_map(corp, stripWhitespace)

corp <- tm_map(corp, removePunctuation)

corp <- tm_map(corp, removeNumbers)

# stopwords

corp <- tm_map(corp, removeWords, stopwords("english"))

# stemming

corp <- tm_map(corp, stemDocument)

# step 3: TF-IDF and latent semantic analysis

# compute TF-IDF

tdm <- TermDocumentMatrix(corp)

tfidf <- weightTfIdf(tdm)

###B

lsa.tfidf <- lsa(tfidf, dim = 10)

# convert to data frame

words.df <- as.data.frame(as.matrix(lsa.tfidf$dk))