$2$ Linear Example $$ w$2$ Linear Example $$ when $\backslash$Phi is trivial

Suppose we are given the following positively labeled data points in $<$

$2$

:

$3$

$1$

$,$

$3$

$1$

$,$

$6$

$1$

$,$

$6$

$1$

and the following negatively labeled data points in $<$

$2$

$($see Figure $1)$:

$1$

$0$

$,$

$0$

$1$

$,$

$0$

$1$

$,$

$1$

$0$

$1$

Figure $1$: Sample data points in $<$

$2$

$.$ Blue diamonds are positive examples and

red squares are negative examples.

We would like to discover a simple SVM that accurately discriminates the

two classes. Since the data is linearly separable, we can use a linear SVM $($that

is$,$ one whose mapping function $\backslash$Phi $()$ is the identity function$).$ By inspection, it

should be obvious that there are three support vectors $($see Figure $2)$:

s$1=$

$1$

$0$

$,$ s$2=$

$3$

$1$

$,$ s$3=$

$3$

$1$

In what follows we will use vectors augmented with a $1$ as a bias input, and

for clarity we will differentiate these with an over$-$tilde. So$,$ if s$1=(10),$ then

s$1=(101).$ Figure $3$ shows the SVM architecture, and our task is to find values

for the $\backslash$alpha i such that

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$1)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$1)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$1)=1$

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$2)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$2)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$2)=+1$

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$3)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$3)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$3)=+1$

Since for now we have let $\backslash$Phi $()=$ I, this reduces to

$\backslash$alpha $1$s$1$ s$1+\backslash$alpha $2$s$2$ s$1+\backslash$alpha $3$s$3$ s$1=1$

$\backslash$alpha $1$s$1$ s$2+\backslash$alpha $2$s$2$ s$2+\backslash$alpha $3$s$3$ s$2=+1$

$\backslash$alpha $1$s$1$ s$3+\backslash$alpha $2$s$2$ s$3+\backslash$alpha $3$s$3$ s$3=+1$

Now, computing the dot products results in

$2$

Figure $2$: The three support vectors are marked as yellow circles.

Figure $3$: The SVM architecture.

$3$

$2\backslash$alpha $1+4\backslash$alpha $2+4\backslash$alpha $3=1$

$4\backslash$alpha $1+11\backslash$alpha $2+9\backslash$alpha $3=+1$

$4\backslash$alpha $1+9\backslash$alpha $2+11\backslash$alpha $3=+1$

A little algebra reveals that the solution to this system of equations is $\backslash$alpha $1=$

$3\mathrm{.}5,\backslash$alpha $2=0\mathrm{.}75$ and $\backslash$alpha $3=0\mathrm{.}75.$

Now, we can look at how these $\backslash$alpha values relate to the discriminating hyperplane; or$,$ in other words, now that we have the $\backslash$alpha i

$,$ how do we find the hyperplane that discriminates the positive from the negative examples? It turns out

that

w$=$

X

i

$\backslash$alpha is$$i

$=3\mathrm{.}5$

$$

$$

$1$

$0$

$1$

$$

$+0\mathrm{.}75$

$$

$$

$3$

$1$

$1$

$$

$+0\mathrm{.}75$

$$

$$

$3$

$1$

$1$

$$

$$

$=$

$$

$$

$1$

$0$

$2$

$$

$$

Finally, remembering that our vectors are augmented with a bias, we can

equate the last entry in $$w as the hyperplane offset b and write the separating

hyperplane equation y $=$ wx$+$b with w $=$

$1$

$0$

and b $=2.$ Plotting the line

gives the expected decision surface $($see Figure $4).$hen $\backslash$Phi is trivial

Suppose we are given the following positively labeled data points in $<$

$2$

:

$3$

$1$

$,$

$3$

$1$

$,$

$6$

$1$

$,$

$6$

$1$

and the following negatively labeled data points in $<$

$2$

$($see Figure $1)$:

$1$

$0$

$,$

$0$

$1$

$,$

$0$

$1$

$,$

$1$

$0$

$1$

Figure $1$: Sample data points in $<$

$2$

$.$ Blue diamonds are positive examples and

red squares are negative examples.

We would like to discover a simple SVM that accurately discriminates the

two classes. Since the data is linearly separable, we can use a linear SVM $($that

is$,$ one whose mapping function $\backslash$Phi $()$ is the identity function$).$ By inspection, it

should be obvious that there are three support vectors $($see Figure $2)$:

s$1=$

$1$

$0$

$,$ s$2=$

$3$

$1$

$,$ s$3=$

$3$

$1$

In what follows we will use vectors augmented with a $1$ as a bias input, and

for clarity we will differentiate these with an over$-$tilde. So$,$ if s$1=(10),$ then

s$1=(101).$ Figure $3$ shows the SVM architecture, and our task is to find values

for the $\backslash$alpha i such that

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$1)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$1)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$1)=1$

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$2)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$2)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$2)=+1$

$\backslash$alpha $1\backslash$Phi $($s$1)\backslash$Phi $($s$3)+\backslash$alpha $2\backslash$Phi $($s$2)\backslash$Phi $($s$3)+\backslash$alpha $3\backslash$Phi $($s$3)\backslash$Phi $($s$3)=+1$

Since for now we have let $\backslash$Phi $()=$ I, this reduces to

$\backslash$alpha $1$s$1$ s$1+\backslash$alpha $2$s$2$ s$1+\backslash$alpha $3$s$3$ s$1=1$

$\backslash$alpha $1$s$1$ s$2+\backslash$alpha $2$s$2$ s$2+\backslash$alpha $3$s$3$ s$2=+1$

$\backslash$alpha $1$s$1$ s$3+\backslash$alpha $2$s$2$ s$3+\backslash$alpha $3$s$3$ s$3=+1$

Now, computing the dot products results in

$2$

Figure $2$: The three support vectors are marked as yellow circles.

Figure $3$: The SVM architecture.

$3$

$2\backslash$alpha $1+4\backslash$alpha $2+4\backslash$alpha $3=1$

$4\backslash$alpha $1+11\backslash$alpha $2+9\backslash$alpha $3=+1$

$4\backslash$alpha $1+9\backslash$alpha $2+11\backslash$alpha $3=+1$

A little algebra reveals that the solution to this system of equations is $\backslash$alpha $1=$

$3\mathrm{.}5,\backslash$alpha $2=0\mathrm{.}75$ and $\backslash$alpha