Randomized Nystrom Features for Fast Regression: An Error Analysis

We consider the problem of fast approximate kernel regression. Since kernels can map input features into the infinite dimensional space, kernel trick is used to make the algorithms tractable. However on large data set time complexity of O(n(2)) is prohibitive. Therefore, various approximation methods are employed, such as randomization. A Nystrom method (based on a random selection of columns) is usually employed. Main advantage of this algorithm is its time complexity which is reduced to O(nm(2) + m(3)). Space complexity is also reduced to O(nm) because it does not require the computation of the entire matrix. An arbitrary number m << n represents both the size of a random subset of an input set and the dimension of random feature vectors. A Nystrom method can be extended with the randomized SVD so that l (where l > m) randomly selected columns of a kernel matrix without replacement are used for a construction of m-dimensional random feature vectors while keeping time complexity linear in n. Approximated matrix computed in this way is a better approximation than the matrix computed via the Nystrom method. We will prove here that the expected error of the approximated kernel predictor derived via this method is approximately the same in expectation as the error of the error of kernel predictor. Furthermore, we will empirically show that using the l randomly selected columns of a kernel matrix for a construction of m-dimensional random feature vectors produces smaller error on a regression problem, than using m randomly selected columns.