Learning mixtures of structured distributions over discrete domains

Let C be a class of probability distributions over the discrete domain [n] = {1, ..., n}: We show that if C satisfies a rather general condition - essentially, that each distribution in C can be well-approximated by a variable-width histogram with few bins - then there is a highly efficient (both in terms of running time and sample complexity) algorithm that can learn any mixture of k unknown distributions from C. We analyze several natural types of distributions over [n], including log-concave, monotone hazard rate and unimodal distributions, and show that they have the required structural property of being well-approximated by a histogram with few bins. Applying our general algorithm, we obtain near-optimally efficient algorithms for all these mixture learning problems as described below. More precisely, Log-concave distributions: We learn any mixture of k log-concave distributions over [n] using k.(O) over tilde (1/epsilon(4)) samples (independent of n) and running in time (O) over tilde (k log(n)/epsilon(4)) bit-operations (note that reading a single sample from [n] takes Theta (log n) bit operations). For the special case k = 1 we give an efficient algorithm using (O) over tilde (1/epsilon(3)) samples; this generalizes the main result of [DDS12b] from the class of Poisson Binomial distributions to the much broader class of all log-concave distributions. Our upper bounds are not far from optimal since any algorithm for this learning problem requires Omega (k/epsilon(5/2)) samples. Monotone hazard rate (MHR) distributions: We learn any mixture of k MHR distributions over [n] using O (k log(n/epsilon)/epsilon(4)) samples and running in time (O) over tilde (k log(2) (n)/epsilon(4)) bit-operations. Any algorithm for this learning problem must use Omega (k log(n)/epsilon(3)) samples. Unimodal distributions: We give an algorithm that learns any mixture of k unimodal distributions over [n] using O (k log(n)/epsilon(4)) samples and running in time (O) over tilde (k log(2) (n)/epsilon(4)) bit-operations. Any algorithm for this problem must use Omega (k log(n)/epsilon(3)) samples.