Hierarchical models, data augmentation, and Markov chain Monte Carlo

The ever increasing power and sophistication of today's high energy astronomical instruments is opening a new realm of high quality data that is quickly pushing beyond the capabilities of the "classical" data-analysis methods in common use. In this chapter we discuss the use of highly structured models that not only incorporate the scientific model (e.g., for a source spectrum) but also account for stochastic components of data collection and the instrument (e.g., background contamination and pile up). Such hierarchical models when used in conjunction with Bayesian or likelihood statistical methods offer a systematic solution to many challenging data analytic problems (e.g., low count rates and pile up). Hierarchical models are becoming increasingly popular in physical and other scientific disciplines largely because of the recent development of sophisticated methods for statistical computation. Thus, we discuss such methods as the EM algorithm, data augmentation, and Markov chain Monte Carlo in the context of high energy high resolution low count data. This paper is followed by a commentary by astronomer Michael Strauss.