$22\mathrm{.}1$ Implement Skip Gram and CBOW from scratch

Implement both the Word$2$Vec algorithms from scratch using softmax. For

Skip$-$Gram, additionally implement with negative sampling.

$2\mathrm{.}2$ Dataset

Link to the dataset is here with train, validation, and test splits. Use the

'wikitext$-103-$raw$-$v$1\text{'}$ subset.

$2\mathrm{.}3$ Evaluation

Run time. Report the time taken for training all $3$ variants $($Skip Gram with

softmax, CBOW with softmax, Skip Gram with negative sampling$)$ on your

system including the system's configuration.

Quality of predictions. For at least $3$ window sizes, report the Mean Reciprocal

Rank $(MRR)$ of the predicted words over all windows in the test set. The $MR{R}_i$

defined for a window $i$ of size $2c$ is given by:

$MR{R}_i={\displaystyle \underset{-cjc}{\overset{?}{}}},j0\frac{1}{2c}*\frac{1}{\text{r}\text{a}\text{n}\text{k}{?}_{(}()j)}$

where $ran{k}_j$ is the position of $w_j$ in the list of terms sorted in decreasing order

of similarity to $w_i$ based on the learnt embeddings. The MRR for the test data

$(MR{R}_d)$ is then aggregated as the average of $MR{R}_i$ over all windows.

$MR{R}_d={\displaystyle \underset{i=1}{\overset{|S|}{}}}MR{R}_i$

$2\mathrm{.}4$ Analysis

Include a detailed analysis of the experiments in your report.

$.$

$1$

$$Implement Skip Gram and CBOW from scratch

Implement both the Word

$2$

Vec algorithms from scratch using softmax. For Skip

$-$

Gram, additionally implement with negative sampling.

$2$

$.$

$2$

$$Dataset

Link to the dataset is here with train, validation, and test splits. Use the

$$

wikitext

$-$

$1$

$0$

$3$

$-$

raw

$-$

v

$1$

$$

$$subset$.$

$2$

$.$

$3$

$$Evaluation

Run time. Report the time taken for training all

$3$

$$variants

$($

Skip Gram with softmax, CBOW with softmax, Skip Gram with negative sampling

$)$

$$on your system including the system

$$

s configuration.

Quality of predictions. For at least

$3$

$$window sizes, report the Mean Reciprocal Rank

$($

MRR

$)$

$$of the predicted words over all windows in the test set. The MRR

$\_$

i defined for a window i of size

$2$

c is given by:

MRR

$\_$

i

$=$

$\backslash$

Sigma

$\_$

$($

$$

c

$$

$=$

j

$$

$=$

c

$,$

j

$$

$=$

$0$

$)$

$($

$1$

$/$

$2$

c

$)$

$.$

$($

$1$

$/$

rank

$\_$

j

$)$

where rank

$\_$

j is the position of w

$\_$

j in the list of terms sorted in decreasing order of similarity to wi based on the learnt embeddings. The MRR for the test data

$($

MRR

$\_$

d

$)$

$$is then aggregated as the average of MRR

$\_$

i over all windows.

MRR

$\_$

d

$=$

$\backslash$

Sigma

$^$

$($

$|$

S

$|$

$)$

$\_$

$($

i

$=$

$1$

$)$

$$MRR

$\_$

i

$2$

$.$

$4$

$$Analysis

Include a detailed analysis of the experiments in your report.