Tight Dimensionality Reduction for Sketching Low Degree Polynomial Kernels

We revisit the classic randomized sketch of a tensor product of q vectors x(i) is an element of R-n. The i-th coordinate (Sx)(i) of the sketch is equal to Pi(q)(j=1) (u(i,j), x(j))/ root m, where u(i,j) are independent random sign vectors. Kar and Karnick (JMLR, 2012) show that if the sketching dimension m = Omega(epsilon(-2) C(Omega)(2)log(1/delta)), where C-Omega is a certain property of the point set Omega one wants to sketch, then with probability 1 - delta, parallel to Sx parallel to(2) = (1 +/- epsilon)parallel to x parallel to(2) for all x epsilon Omega. However, in their analysis C-Omega(2) can be as large as Theta(n(2q)), even for a set Omega of O(1) vectors x. We give a new analysis of this sketch, providing nearly optimal bounds. Namely, we show an upper bound of m = Theta(epsilon(-2) log(1/delta) + epsilon(-1) logq(n/delta)), which by composing with CountSketch, can be improved to Theta(epsilon(-2) log(1/delta epsilon) + epsilon(-1) log(q) (1/(delta epsilon)). For the important case of q = 2 and delta = 1/poly(n), this shows that m = Omega(epsilon(-2) log(n) + epsilon(-1) log(2)(n)), demonstrating that the epsilon(-2) and log(2)(n) terms do not multiply each other. We also show a nearly matching lower bound of m = Omega(epsilon(-2) log(1/delta)) + epsilon(-1) log(q) (1/(delta))). In a number of applications, one has vertical bar Omega vertical bar = poly(n) and in this case our bounds are optimal up to a constant factor. This is the first high probability sketch for tensor products that has optimal sketch size and can be implemented in m . Sigma(q)(i=1) nnz(x(i)) time, where nnz(x(i)) is the number of non -zero entries of x(i). Lastly, we empirically compare our sketch to other sketches for tensor products, and give a novel application to compressing neural networks.