Kernel method

Kernel machines are a class of algorithms for pattern analysis, whose best known member is the Support vector machine (SVM).

Kernel Trick

The Kernel Trick maps data into a higher-dimensional Feature Space where a linear boundary (like a hyperplane in SVMs) can effectively separate the classes.

Takes low-dimensional, nonlinear data and projects it into a higher-dimensional space where it becomes linear.

Uses a Kernel Function $K(x,y)$ that computes the inner product of the vectors in the high-dimensional space directly:
$K(x,y)=⟨\varphi (x),\varphi (y)⟩$

Why This Matters

• Computational Efficiency: Calculating ϕ(x) explicitly can be computationally expensive or even impossible if the feature space is infinite-dimensional (as is the case with the RBF kernel).
• Memory: You only need to store the results of the kernel function (the Gram matrix) rather than the coordinates of the high-dimensional vectors.
• Linearity: It allows algorithms designed for linear problems (SVM, PCA, Ridge Regression) to solve complex, nonlinear problems seamlessly.

Common Kernel Functions

• Linear: $K(x,y)=xTy$
• Polynomial: $K(x,y)=(xTy+c)d$
• Radial Basis Function (RBF/Gaussian): $K(x,y)=\exp (-\gamma \Vert x-y{\Vert }^2)$. This effectively maps data into an infinite-dimensional space.