Optimally regularised kernel Fisher discriminant analysis

Mika et al. [3] introduce a non-linear formulation of Fisher's linear discriminant, based the now familiar "kernel trick", demonstrating state-of-the-art performance on a wide range of real-world benchmark datasets. In this paper we show that the usual regularisation parameter can be adjusted so as to minimise the leave-one-out cross-validation error with a computational complexity of only O(l(4)) operations, where e is the number of training patterns, rather than the O(l(4)) operations required for a naive implementation of the leave-one-out procedure. This procedure is then used to form a component of an efficient heirarchical model selection strategy where the regularisation parameter is optimised within the inner loop while the kernel parameters are optimised in the outer loop.