SVM Using CVXPY

Support Vector Machines or (SVM) is a Supervised model used for classification and regression which can be done by finding the hyperplane in a N-dimensional space where N is the number of features which is used to classify data.

Terminologies

Hyperplane – Decision boundaries that helps to classify the data points. Depending upon data points’ side (either side of the plane), the class of the data is decided.

Support Vectors – Data points closer to the hyperplane and those points influence the position and the orientation of the hyperplane.

Weights – Used to draw the boundaries. Minimizing the weights will maximize the distance between the support vectors and the hyperplane.

Hard Margin

If the training data is linearly separable then we can select two parallel hyperplanes that separate the two classes of data, so that the distance between them is as large as possible. The region bounded by these two hyperplanes is called the “margin”, and the maximum-margin hyperplane is the hyperplane that lies halfway between them. 

Consider the below example. Here the data points are linearly separable since there are no overlap between the data points.

The condition is to have the hyperplane at least 1unit away from both the sides. We can write the problem formulations as mentioned below.

We will be using cvxpy in python to solve this constrained problem. The code is given below.

Soft Margin

If the training data is not linearly separable then we cannot simply select two parallel hyperplanes because it might lead to wrong predictions in some case. So Soft Margin SVM is used to classify these type of datas.

Since there is an overlap between the data points, we introduce a slack variable into the formulation to tally the problem.

The formulation is given below.

Here the psi is the slack variable used to tally the overlapping problem and c is the control parameter used to prevent the misclassification of data to one side. It can be solved using python as shown below.

Check out the other blogs on Machine Learning Algorithms under Python For ML Category.