Irreducible data example

Not all datasets dimensionality can be successfully reduced using PCA.

(a) Discuss the cases when PCA will fail.

(b) How do we quantify that it fails?

(c) Provide a minimal example of a dataset (specify the points as vectors of numbers) in which PCA will not work well for dimensionality reduction. Explain why. Hint: Think of 2D points and reduction to 1D.

(d) When linear PCA does not work we naturally go to kernels. For your example above, propose a suitable kernel (x) that you expect would do a better job if we do Kernel-PCA. Justify your selection. Compute the kernel matrix K(xi, xj ) = (xi)(xj ) for the first 3 points of your example data above.