What will the future of cloud-based astronomical data processing look like?

被引:0
|
作者
Green, Andrew W. [1 ]
Mannering, Elizabeth [1 ]
Harischandra, Lloyd [1 ]
Minh Vuong [1 ]
O'Toole, Simon [1 ]
Sealey, Katrina [1 ]
Hopkins, Andrew M. [1 ]
机构
[1] Australian Astron Observ, POB 915, N Ryde, NSW 1670, Australia
来源
ASTROINFORMATICS | 2017年 / 12卷 / S325期
关键词
astronomical data bases: miscellaneous; methods: data analysis;
D O I
10.1017/S1743921317001363
中图分类号
P1 [天文学];
学科分类号
0704 ;
摘要
Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud-astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example.
引用
收藏
页码:27 / 31
页数:5
相关论文
共 50 条
  • [21] Impact-Minimizing Runtime Adaptation in Cloud-Based Data Stream Processing
    Qin, Cui
    ADVANCES IN SERVICE-ORIENTED AND CLOUD COMPUTING (ESOCC 2016), 2018, 707 : 274 - 285
  • [22] Stability of Cloud-Based UAV Systems Supporting Big Data Acquisition and Processing
    Luo, Feng
    Jiang, Chunxiao
    Yu, Shui
    Wang, Jingjing
    Li, Yipeng
    Ren, Yong
    IEEE TRANSACTIONS ON CLOUD COMPUTING, 2019, 7 (03) : 866 - 877
  • [23] Secure Phrase Search for Intelligent Processing of Encrypted Data in Cloud-Based IoT
    Shen, Meng
    Ma, Baoli
    Zhu, Liehuang
    Du, Xiaojiang
    Xu, Ke
    IEEE INTERNET OF THINGS JOURNAL, 2019, 6 (02) : 1998 - 2008
  • [24] Cloud-based configurable data stream processing architecture in rural economic development
    Chen, Haohao
    Al-Turjman, Fadi
    PEERJ COMPUTER SCIENCE, 2024, 10 : 1 - 19
  • [25] Catalyst: A Cloud-based Media Processing Framework
    Katsak, William
    Nguyen, Hai
    Nagaraja, Kiran
    Halen, Joacim
    Radia, Nimish
    Nguyen, Thu D.
    2019 39TH IEEE INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS (ICDCS 2019), 2019, : 2079 - 2089
  • [26] Information Masking Theory for Data Protection in Future Cloud-Based Energy Management
    Xin, Shujun
    Guo, Qinglai
    Wang, Jianhui
    Chen, Chen
    Sun, Hongbin
    Zhang, Boming
    IEEE TRANSACTIONS ON SMART GRID, 2018, 9 (06) : 5664 - 5676
  • [27] Cloud-Based Ground System for Telemetry Processing
    Mirchandani, Chandru
    COMPLEX ADAPTIVE SYSTEMS, 2015, 2015, 61 : 183 - 190
  • [28] Monitoring a Cloud-based Speech Processing System
    Suciu, George
    Vulpe, Alexandru
    Arseni, Stefan Ciprian
    Stancu, Alexandru
    Butca, Cristina
    Suciu, Victor
    PROCEEDINGS OF THE 2015 7TH INTERNATIONAL CONFERENCE ON ELECTRONICS, COMPUTERS AND ARTIFICIAL INTELLIGENCE (ECAI), 2015,
  • [29] Cloud-based NoSQL Data Migration
    Bansel, Aryan
    Gonzalez-Velez, Horacio
    Chis, Adriana E.
    2016 24TH EUROMICRO INTERNATIONAL CONFERENCE ON PARALLEL, DISTRIBUTED, AND NETWORK-BASED PROCESSING (PDP), 2016, : 224 - 231
  • [30] Cloud-Based Data Architecture Security
    N. A. Semenov
    A. A. Poltavtsev
    Automatic Control and Computer Sciences, 2019, 53 : 1056 - 1064