Simple, Fast and Accurate Hyper-parameter Tuning in Gaussian-kernel SVM

We consider the parameter tuning problem for Gaussian-kernel support vector machines, i.e., how to set its two hyperparameters - sigma (bandwidth) and C (tradeoff). Among the many methods in the literature, the majority handle this task by maximizing the cross validation accuracy over the first quadrant of the (sigma, C) plane. However, they are all computationally expensive because the objective function has no explicit formula so that one has to resort to numerical methods (which require training and testing the classifier many times). Additionally, these methods ignore the intrinsic geometry of training data and always operate in a large set, thus being computationally inefficient. In this paper we propose a two-step procedure for efficient parameter selection: First, we use a nearest neighbor method to directly set the value of sigma based on the data geometry; afterwards, for the tradeoff parameter C we employ an elbow method that finds the smallest C leading to "nearly" the highest validation accuracy. By slightly sacrificing the validation accuracy our method gains additional attractive properties such as (1) faster training (i.e., much less candidate points to be examined) and (2) better generalizability (due to larger class margins). We conduct extensive experiments to show that such a combination of simple techniques achieves excellent performance - the classification accuracy of our method is comparable to its competitors in most cases, but it is much faster.