Kullback-Leibler Divergence Rate Between Probability Distributions on Sets of Different Cardinalities

In this paper we generalize the familiar notion of the Kullback-Leibler divergence between two probability distribitions on a finite set to the case where the two probability distributions are defined on sets of different cardinalities. This is achieved by 'dilating' the distribution on the smaller set to one on the larger set. This idea follows [6]. Then, using the log sum inequality, we give a formula for this divergence that permits all computations to be carried out on the set of smaller cardinality. However, computing the divergence is still in general an intractable problem. Hence we give a greedy algorithm for quickly obtaining an upper bound for this divergence. The ideas given here can be used to study the concept of an optimally aggregated model of a Markov or hidden Markov process. This is done in a companion paper.