Why feature scaling in SVM?

machine-learning svm scaling
Kevin picture Kevin · Oct 6, 2014 · Viewed 45.5k times · Source

I found that scaling in SVM (Support Vector Machine) problems really improve its performance... I have read this explanation:

"The main advantage of scaling is to avoid attributes in greater numeric ranges dominating those in smaller numeric ranges."

Unfortunately this didn't help me ... Can somebody provide me a better explanation? Thank you in advance!

Answer

greeness picture greeness · Oct 7, 2014

Feature scaling is a general trick applied to optimization problems (not just SVM). The underline algorithm to solve the optimization problem of SVM is gradient descend. Andrew Ng has a great explanation in his coursera videos here.

I will illustrate the core ideas here (I borrow Andrew's slides). Suppose you have only two parameters and one of the parameters can take a relatively large range of values. Then the contour of the cost function can look like very tall and skinny ovals (see blue ovals below). Your gradients (the path of gradient is drawn in red) could take a long time and go back and forth to find the optimal solution.
enter image description here

Instead if your scaled your feature, the contour of the cost function might look like circles; then the gradient can take a much more straight path and achieve the optimal point much faster. enter image description here

Related questions

What are advantages of Artificial Neural Networks over Support Vector Machines?

ANN (Artificial Neural Networks) and SVM (Support Vector Machines) are two popular strategies for supervised machine learning and classification. It's not often clear which method is better for a particular project, and I'm certain the answer is always "it depends." …

Read More
How to do multi class classification using Support Vector Machines (SVM)

In every book and example always they show only binary classification (two classes) and new vector can belong to any one class. Here the problem is I have 4 classes(c1, c2, c3, c4). I've training data for 4 classes. For new …

Read More
What is the relation between the number of Support Vectors and training data and classifiers performance?

I am using LibSVM to classify some documents. The documents seem to be a bit difficult to classify as the final results show. However, I have noticed something while training my models. and that is: If my training set is …

Read More