Stochastic Convergences in Divergences

The paper presents a new unifying look at the statistical inference. We define score functions and minimum score estimators. We show that the stochastic convergence for estimators, i.e. the consistency of an estimators in score functions, leads to various types of consistency in the well-known statistical distances or disparity measures between probability distributions. We formulate conditions under which a score function is phi-divergence of theoretical and empirical distribution. Conversely, each phi-divergence is a score function. We prove that minimization of arbitrary divergence score function leads to the classical histogram density estimator and that a special score function leads in a similar sense to the minimum Kolmogorov distance estimator.