Kullback-Leibler Distance in Linear Parametric Modeling

This paper addresses the estimation of the Kulback-Leibler (KL) distance in data-driven modeling of parametric probability distributions. Given a finite observation of a parametric probability density function (pdf), the goal is to provide the best representative of the true parameter, which is known to belong to a given parametric model set. The first step in this problem setting is to estimate the true parameter in available nested model sets of different orders. The proposed method calculates the KL distance between these estimates and the unknown true parameter. By using only the observed data, we provide probabilistic worst case bounds on these KL distances. The best candidate among the available estimates is the solution of a resulting probabilistic min-max problem. A comparison of this approach with existing methods that estimate the KL distance is provided.