Clustgrams: an extension to histogram densities based on the minimum description length principle

Density estimation is one of the most important problems in statistical inference and machine learning. A common approach to the problem is to use histograms, i.e., piecewise constant densities. Histograms are flexible and can adapt to any density given enough bins. However, due to the simplicity of histograms, a large number of parameters and a large sample size might be needed for learning an accurate density, especially in more complex problem instances. In this paper, we extend the histogram density estimation framework by introducing a model called clustgram, which uses arbitrary density functions as components of the density rather than just uniform components. The new model is based on finding a clustering of the sample points and determining the type of the density function for each cluster. We regard the problem of learning clustgrams as a model selection problem and use the theoretically appealing minimum description length principle for solving the task.