### Part $3$: Nonlinear SVM$,$ Parameter Tuning, Accuracy, and Cross$-$Validation

$***$

Any support vector machine classifier will have at least one parameter that needs to be tuned based on the training data. The guaranteed parameter is the $C$ associated with the slack variables in the primal objective function, i$.$e$.$

$$

$\backslash$min$\_\{\{\backslash$bf w$\},$ b$,\{\backslash$bf $\backslash$xi$\}\}\backslash$frac$\{1\}\{2\}\backslash |\{\backslash$bf w$\}\backslash |^2+$ C $\backslash$sum$\_\{$i$=1\}^$m $\backslash$xi$\_$i

$$

If you use a kernel fancier than the linear kernel then you will likely have other parameters as well. For instance in the polynomial kernel $K$(\{\backslash$bf x$\},\{\backslash$bf z$\})=(\{\backslash$bf x$\}^$T$\{\backslash$bf z$\}+$ c$)^$d$ you have to select the shift $c$ and the polynomial degree $d$$.$ Similarly the rbf kernel

$$

K$(\{\backslash$bf x$\},\{\backslash$bf z$\})=\backslash$exp$\backslash$left$[-\backslash$gamma$\backslash |\{\backslash$bf x$\}-\{\backslash$bf z$\}\backslash |^2\backslash$right$]$

$$

has one tuning parameter, namely $$\backslash$gamma$$,$ which controls how fast the similarity measure drops off with distance between $$\{\backslash$bf x$\}$$ and $$\{\backslash$bf z$\}$$$.$

For our examples we'll consider the rbf kernel, which gives us two parameters to tune, namely $C$ and $$\backslash$gamma$$.$

Consider the following two dimensional data