What will the future of cloud-based astronomical data processing look like?

被引:0
|
作者
Green, Andrew W. [1 ]
Mannering, Elizabeth [1 ]
Harischandra, Lloyd [1 ]
Minh Vuong [1 ]
O'Toole, Simon [1 ]
Sealey, Katrina [1 ]
Hopkins, Andrew M. [1 ]
机构
[1] Australian Astron Observ, POB 915, N Ryde, NSW 1670, Australia
来源
ASTROINFORMATICS | 2017年 / 12卷 / S325期
关键词
astronomical data bases: miscellaneous; methods: data analysis;
D O I
10.1017/S1743921317001363
中图分类号
P1 [天文学];
学科分类号
0704 ;
摘要
Astronomy is rapidly approaching an impasse: very large datasets require remote or cloud-based parallel processing, yet many astronomers still try to download the data and develop serial code locally. Astronomers understand the need for change, but the hurdles remain high. We are developing a data archive designed from the ground up to simplify and encourage cloud-based parallel processing. While the volume of data we host remains modest by some standards, it is still large enough that download and processing times are measured in days and even weeks. We plan to implement a python based, notebook-like interface that automatically parallelises execution. Our goal is to provide an interface sufficiently familiar and user-friendly that it encourages the astronomer to run their analysis on our system in the cloud-astroinformatics as a service. We describe how our system addresses the approaching impasse in astronomy using the SAMI Galaxy Survey as an example.
引用
收藏
页码:27 / 31
页数:5
相关论文
共 50 条
  • [1] Cloud-Based Processing on Data Science for Visualization
    Ashari, Ahmad
    Tjoa, A. Min
    Riasetiawan, Mardhani
    INTERNATIONAL JOURNAL OF ADVANCED COMPUTER SCIENCE AND APPLICATIONS, 2016, 7 (02) : 449 - 454
  • [2] Reliability of Cloud-Based Processing for Satellite Data
    Wood, Paul
    Rossiter, Dan
    Rose, Debra
    2021 IEEE AEROSPACE CONFERENCE (AEROCONF 2021), 2021,
  • [3] A Cloud-based IoT Data Gathering and Processing Platform
    Emeakaroha, Vincent C.
    Cafferkey, Neil
    Healy, Philip
    Morrison, John P.
    2015 3RD INTERNATIONAL CONFERENCE ON FUTURE INTERNET OF THINGS AND CLOUD (FICLOUD) AND INTERNATIONAL CONFERENCE ON OPEN AND BIG (OBD), 2015, : 50 - 57
  • [4] The Future of Cloud-Based Entertainment
    Hughes, Kilroy
    PROCEEDINGS OF THE IEEE, 2012, 100 : 1391 - 1394
  • [5] Cloud-based processing of multi-spectral imaging data
    Bernat, Amir S.
    Bolton, Frank J.
    Weisser, Reuven
    Levitz, David
    OPTICS AND BIOPHOTONICS IN LOW-RESOURCE SETTINGS III, 2017, 10055
  • [6] Elastic Data Partitioning for Cloud-based SQL Processing Systems
    Lim, Lipyeow
    2013 IEEE INTERNATIONAL CONFERENCE ON BIG DATA, 2013,
  • [7] Toward a Cloud-based security intelligence with big data processing
    Benzidane, Karim
    El Alloussi, Hassan
    El Warrak, Othman
    Fetjah, Leila
    Andaloussi, Said Jai
    Sekkaki, Abderrahim
    NOMS 2016 - 2016 IEEE/IFIP NETWORK OPERATIONS AND MANAGEMENT SYMPOSIUM, 2016, : 1089 - 1092
  • [8] SMASH: A Cloud-based Architecture for Big Data Processing and Visualization of Traffic Data
    Wu, Siqi
    Morandini, Luca
    Sinnott, Richard O.
    2015 IEEE INTERNATIONAL CONFERENCE ON DATA SCIENCE AND DATA INTENSIVE SYSTEMS, 2015, : 53 - 60
  • [9] The future of MCMs: What is it? And what will they look like?
    Vardaman, EJ
    1998 INTERNATIONAL CONFERENCE ON MULTICHIP MODULES AND HIGH DENSITY PACKAGING, PROCEEDINGS, 1998, : 512 - 515
  • [10] A Cloud-Based Data Gathering and Processing System for Intelligent Demand Forecasting
    Tay, Colin K. L.
    Shim, Kyong Jin
    2018 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2018, : 5451 - 5453