Suppose that the put to a self$-$attertion layer

is a $5-$dimensional vector.

In the self$-$attention layer, each input vector ${\stackrel{}{x}}_i$ is

first linearly projected into query $\frac{?}{\text{b}\text{a}\text{r}}(q{)}_i,$ key $\frac{?}{\text{b}\text{a}\text{r}}(k{)}_i$

and value $\frac{?}{\text{b}\text{a}\text{r}}(v{)}_i,$ where $\frac{?}{\text{b}\text{a}\text{r}}(q{)}_i\frac{,}{b}ar(k{)}_i$ and $\frac{?}{\text{b}\text{a}\text{r}}(v{)}_i$ are each $3-$dimensional.

Which of the followings could ther possibly represent

the attention matrix, given some input sequence?

Note that softmax operation is applied separately

over the columns of the alignment matrix.

o$){\left[\begin{array}{ccc}0& 0& 1\\ 0& \frac{1}{2}& 0\\ \frac{1}{2}& 0& 0\end{array}\right]}_{33}$

b$){\left[\begin{array}{ccc}0& 0& 1\\ 1& 0& 0\\ 0& 1& 0\end{array}\right]}_{33}$

${\left[\begin{array}{cccc}0& \frac{1}{2}& 0& 1\\ 0& 0& \frac{1}{2}& 0\\ 1& 0& 0& 0\\ 0& 0& 0& 0\end{array}\right]}_{44}$

d$){\left[\begin{array}{cccc}0& 0& 1& 0\\ 1& 0& 0& 1\\ 0& 1& 0& 0\\ 0& 0& 0& 0\end{array}\right]}_{44}$

$55$

$f$

e$)\left[\begin{array}{ccccc}0& 0& 0& 0& 1\\ 1& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 0& 0& 0& 0\\ 0& 1& 0& 0& 0\end{array}\right]$

h$){\left[\begin{array}{ccc}1& 0& 0\\ 0& 0& 0\\ 0& 1& 0\\ 0& 0& 0\\ 0& 0& 1\end{array}\right]}_{53}$