Computational AstroStatistics: Fast algorithms and efficient statistics for density estimation in large astronomical datasets

被引:0
|
作者
Nichol, RC [1 ]
Connolly, AJ [1 ]
Moore, AW [1 ]
Schneider, J [1 ]
Genovese, C [1 ]
Wasserman, L [1 ]
机构
[1] Carnegie Mellon Univ, Dept Phys, Pittsburgh, PA 15213 USA
关键词
D O I
暂无
中图分类号
P1 [天文学];
学科分类号
0704 ;
摘要
We present initial results on the use of Mixture Models for density estimation in large astronomical databases. We provide herein both the theoretical and experimental background for using a mixture model of Gaussians based on the Expectation Maximization (EM) Algorithm. Applying these analyses to simulated data sets we show that the EM algorithm - using the both the AIC & BIC penalized likelihood to score the fit - can out-perform the best kernel density estimate of the distribution while requiring no "fine-tuning" of the input algorithm parameters. We find that EM can accurately recover the underlying density distribution from point processes thus providing an efficient adaptive smoothing method for astronomical source catalogs. To demonstrate the general application of this statistic to astrophysical problems we consider two cases of density estimation; the clustering of galaxies in redshift space and the clustering of stars in color space. From these data we show that EM provides an adaptive smoothing of the distribution of galaxies in redshift space (describing accurately both the small and large-scale features within the data) and a means of identifying outliers in multi-dimensional color-color space (e.g., for the identification of high redshift QSOs). Automated tools such as those based on the EM algorithm will be needed in the analysis of the next generation of astronomical catalogs (2MASS, FIRST, PLANCK, SDSS) and ultimately in in the development of the National Virtual Observatory.
引用
收藏
页码:265 / 271
页数:7
相关论文
共 50 条
  • [1] Computational AstroStatistics: Fast and efficient tools for analysing huge astronomical data sources
    Nichol, RC
    Chong, S
    Connolly, AJ
    Davies, S
    Genovese, C
    Hopkins, AM
    Miller, CJ
    Moore, AW
    Pelleg, D
    Richards, GT
    Schneider, J
    Szapudi, I
    Wasserman, L
    [J]. STATISTICAL CHALLENGES IN ASTRONOMY, 2003, : 265 - 278
  • [2] Comparison of density estimation methods for astronomical datasets
    Ferdosi, B. J.
    Buddelmeijer, H.
    Trager, S. C.
    Wilkinson, M. H. F.
    Roerdink, J. B. T. M.
    [J]. ASTRONOMY & ASTROPHYSICS, 2011, 531
  • [3] Data Mining Projects, Discoveries and Statistics in Large Astronomical Archives: The Astrostatistics Group of the Spanish Virtual Observatory
    Sarro, L. M.
    Garcia Torres, M.
    Lopez, M.
    Berihuete, A.
    Marquez, M. J.
    Garcia Sedano, F.
    [J]. HIGHLIGHTS OF SPANISH ASTROPHYSICS V, 2010, : 541 - 541
  • [4] Comparison of fast regression algorithms in large datasets
    Cangur, Sengul
    Ankarali, Handan
    [J]. KUWAIT JOURNAL OF SCIENCE, 2023, 50 (02)
  • [5] Cached sufficient statistics for efficient machine learning with large datasets
    Moore, A
    Lee, MS
    [J]. JOURNAL OF ARTIFICIAL INTELLIGENCE RESEARCH, 1998, 8 : 67 - 91
  • [6] Space efficient fast isosurface extraction for large datasets
    Bordoloi, UD
    Shen, HW
    [J]. IEEE VISUALIZATION 2003, PROCEEDINGS, 2003, : 201 - 208
  • [7] Efficient computational algorithms for fast electrostatics and molecular docking
    Sandak, B
    [J]. COMPUTATIONAL METHODS FOR MACROMOLECULES: CHALLENGES AND APPLICATIONS, 2002, 24 : 411 - 441
  • [8] Fast and Efficient Algorithms for Computational Electromagnetics on GPU Architecture
    Maceina, Tautvydas
    Bettini, Paolo
    Manduchi, Gabriele
    Passarotto, Mauro
    [J]. IEEE TRANSACTIONS ON NUCLEAR SCIENCE, 2017, 64 (07) : 1983 - 1987
  • [9] Fast statistically efficient algorithms for single frequency estimation
    Klein, JD
    [J]. 2005 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH, AND SIGNAL PROCESSING, VOLS 1-5: SPEECH PROCESSING, 2005, : 389 - 392
  • [10] Fast algorithms and efficient statistics: N-point correlation functions
    Moore, AW
    Connolly, AJ
    Genovese, C
    Gray, A
    Grone, L
    Kanidoris, N
    Nichol, RC
    Schneider, J
    Szalay, AS
    Szapudi, I
    Wasserman, L
    [J]. MINING THE SKY, 2001, : 71 - 82