Task $2$b $-$ More understanding of Semantic Matching $(15$ points$).$

Which terms does LSI find similar?$$

To understand why the LSI$-$expanded vectors get the results they do$,$ we're going to look at what the operator

U

does to text. In particular, the term$-$term matrix

U

U

T

tells us the term expansion behavior of this LSI model. Think of the term$-$term matrix like an operator that first maps a term to the latent space

L

k

$($using

U

$),$ then back again from

L

k

to term space $($using

U

transpose$).$ The

$($

i

$,$

j

$)$

entry of

U

U

T

is a kind of association weight between term

i

and term

j

$.$

Write a function to get the most related terms $($according to LSI$)$ for the word "economy". To do this:

Compute the term$-$term matrix from the matrix U $($the reduced$\_$term$\_$matrix variable$).$

Use the term$-$term matrix to get the association weights of all words related to the term "economy"

Sort by descending weight value.

Your function should return the top $5$ words and their weights as a list of $($string$,$ float$)$ tuples.

Do the related terms match your subjective similarity judgment?

TOC

In $[19]$:

Grade cell: cell$-725$c$6$b$70431$f$4779$Score: $0\mathrm{.}0/0\mathrm{.}0($Top$)$

# Please tell me more!

task$\_$id $="2$b$"$

In $[20]$:

Student's answer$($Top$)$

def answer$\_$semantic$\_$similarity$\_$b$()$:

result $=$ None

# YOUR CODE HERE

term $=$ "economy"

term$\_$index $=$ tfidf$\_$vectorizer.vocabulary$\_[$term$]$

#calc term matrix

term$\_$term$\_$matrix $=$ reduced$\_$term$\_$matrix @reduced$\_$term$\_$matrix.T

#get the associated weights of the term

related$\_$terms$\_$weights $=$ term$\_$term$\_$matrix$[$term$\_$index$]$

#get the top $5$ related terms

top$\_$indices $=$ related$\_$terms$\_$weights.argsort$()[$::$-1][1$:$6]$

top$\_$terms $=[($tfidf$\_$feature$\_$names$[$i$],$ related$\_$terms$\_$weights$[$i$])$ for i in top$\_$indices$]$

return top$\_$terms

#raise NotImplementedError$()$

return result

In $[21]$:

# use this cell to explore your solution

# remember to comment the function call before submitting the notebook

# answer$\_$semantic$\_$similarity$\_$b$()$

In $[22]$:

Grade cell: cell$-683419$d$1$db$09$c$762$Score: $0\mathrm{.}0/15\mathrm{.}0($Top$)$

print$($f$"$Task $\{$task$\_$id$\}-$ AG tests"$)$

stu$\_$ans $=$ answer$\_$semantic$\_$similarity$\_$b$()$

print$($f$"$Task $\{$task$\_$id$\}-$ your answer:

$\{$stu$\_$ans$\}")$

assert isinstance$($stu$\_$ans, list$),$ f$"$Task $\{$task$\_$id$\}$: Your function should return a list. $"$

assert len$($stu$\_$ans$)==5,$ f$"$Task $\{$task$\_$id$\}$: Your list should contain five elements $($the term, score tuples$)."$

for i$,$ item in enumerate$($stu$\_$ans$)$:

assert isinstance$($item$,$ tuple$),$ f$"$Task $\{$task$\_$id$\}$: Your answer at index $\{$i$\}$ should be a tuple. $"$

assert isinstance$($

item$[0],$ str

$),$ f$"$Task $\{$task$\_$id$\}$: The first element of your tuple at index $\{$i$\}$ should be a string. $"$

assert isinstance$($

item$[1],($float$,$ np$.$floating$)$

$),$ f$"$Task $\{$task$\_$id$\}$: The second element of your tuple at index $\{$i$\}$ should be a float. $"$

# Some hidden tests

del stu$\_$ans