Python:

Your Task: Define get$\_$top$\_$tokens as follows:

Given a cluster, identify its most frequent tokens.

Inputs:

cid: The ID of a cluster to analyse

labels: cluster assignments where document i was assigned to cluster labels $[$i$].$

corpusdf: A corpus dataframe having the columns $\text{'}$id$\text{'},$ 'title' and 'pesudodoc'.

k: the number of tokens to return

Return: top$\_$tokens, a pythonset of the k most frequent tokens

Steps:

$*)$ Use labels to identify the documents

$*)$ select thier corresponding pesudo$-$documents "corpusdf"

$*)$ For each unique token, count the number of pesudo$-$documents in which it appeared

$*)$Return the k most frequently occuring tokens as a python set. In the case of ties, consider tokens in ascending order

Other notes:

$*)$ to match a document ID i to its pseudo$-$document, note that i is the index value of corpusdf $($and$,$ in particular, not the $\text{'}$id$\text{'}$ column!$)$

$*)$if there are fewer than k unique tokens, return as many as available

$*)$ if there are ties, use the token itself sort the tokens by name

Example: for the demo code, a correct implementation should return:

$\{\text{'}$page$\text{'},$ 'reviews', 'academic', 'book'$\}$

Solution##

def get$\_$top$\_$tokens$($cid: int, labels:np$.$ndarray, corpusdf: pd$.$DataFrame, k$=10)->$ set:

## code

with open$(\text{'}$resource$/$asnlib$/$publicdata$/$demo$\_$args$\_$get$\_$top$\_$tokens.dill', $\text{'}$rb$\text{'})$ as fp:

demo$\_$cid, demo$\_$labels, demo$\_$corpusdf, demo$\_$k$=$dill.load$($fp$)$

print$($get$\_$top$\_$tokens$($demo$\_$cid, demo$\_$labels, demo$\_$corpusdf, demo$\_$k$))$