Question Four $(10$ Marks$)$

In a sentiment classification, we use the attention mechanism to compute the encoding for the input sequence "I like the pattern class". Assume the initial embedding transposed will be $1=(0\mathrm{.}9,0\mathrm{.}1,0\mathrm{.}1,0\mathrm{.}3),$ like $(0\mathrm{.}4,0\mathrm{.}5,0\mathrm{.}3,0\mathrm{.}2),$ the $=(0\mathrm{.}1,0\mathrm{.}1,0\mathrm{.}1,0\mathrm{.}1),$ pattern$=(0\mathrm{.}5,0\mathrm{.}5,0\mathrm{.}6\mathrm{.}0\mathrm{.}1),$ class$=(0\mathrm{.}4,0\mathrm{.}1,0\mathrm{.}3,0\mathrm{.}1)$

We will use single head attention with

Wq the concatenation of the embedding of I, pattern, pattern, class

Wv the concatenation of the embedding of I, the, pattern, class

Wk the identity matrix.

$1.$ Find the embedding for the token "like" using self$-$attention. Show steps

$2.$ Use hardcoded sinusoid functions to compute the positional embedding for the word "like". Then add the positional embedding to the embedding you obtained in part $1.$

$3.$ Suppose we have another token "hate" with initial embedding $=(0\mathrm{.}2,0\mathrm{.}3,0\mathrm{.}5,0\mathrm{.}4).$ Given the same context of the token "like" how efficiently can you repeat part $1.$