Cookery: A framework for creating data processing pipeline using online services

被引:0
|
作者
Baranowski M. [1 ]
Belloum A. [1 ,2 ]
Cushing R. [1 ]
Valkering O. [1 ]
机构
[1] Multiscale Networked Systems (MNS), Institute of Informatics University of Amsterdam, Amsterdam
[2] Netherlands eScience Center Science, Park 140, Amsterdam
来源
| 1600年 / Slovak Academy of Sciences卷 / 39期
基金
欧盟地平线“2020”;
关键词
AWS Lambda; Domain specific languages (DSL); Function-as-a-service (FaaS); Serverless computing;
D O I
10.31577/CAI_2020_4_678
中图分类号
学科分类号
摘要
With the increasing amount of data the importance of data analysis in various scientific domains has grown. A large amount of the scientific data has shifted to cloud based storage. The cloud offers storage and computation power. The Cookery framework is a tool developed to build scientific applications using cloud services. In this paper we present the Cookery systems and how they can be used to authenticate and use standard online third party services to easily create data analytic pipelines. Cookery framework is not limited to work with standard web services; it can also integrate and work with the emerging AWS Lambda which is part of a new computing paradigm, collectively, known as serverless computing. The combination of AWS Lambda and Cookery, which makes it possible for users in many scientific domains, who do not have any program experience, to create data processing pipelines using cloud services in a short time. © 2020 Slovak Academy of Sciences. All rights reserved.
引用
收藏
页码:678 / 694
页数:16
相关论文
共 50 条
  • [41] The QUEST Data Processing Software Pipeline
    Andrews, Peter
    Baltay, Charles
    Bauer, Anne
    Ellman, Nancy
    Jerke, Jonathan
    Lauer, Rochelle
    Rabinowitz, David
    Silge, Julia
    PUBLICATIONS OF THE ASTRONOMICAL SOCIETY OF THE PACIFIC, 2008, 120 (868) : 703 - 714
  • [42] Data processing pipeline for Herschel HIFI
    Shipman, R. F.
    Beaulieu, S. F.
    Teyssier, D.
    Morris, P.
    Rengel, M.
    McCoey, C.
    Edwards, K.
    Kester, D.
    Lorenzani, A.
    Coeur-Joly, O.
    Melchior, M.
    Xie, J.
    Sanchez, E.
    Zaal, P.
    Avruch, I.
    Borys, C.
    Braine, J.
    Comito, C.
    Delforge, B.
    Herpin, F.
    Hoac, A.
    Kwon, W.
    Lord, S. D.
    Marston, A.
    Mueller, M.
    Olberg, M.
    Ossenkopf, V.
    Puga, E.
    Akyilmaz-Yabaci, M.
    ASTRONOMY & ASTROPHYSICS, 2017, 608
  • [43] Pipeline processing of data in the VLT era
    Grosbol, P
    Banse, K
    Ballester, P
    ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS VIII, 1999, 172 : 151 - 154
  • [44] Pipeline synchronization in data parallel processing
    Zhang, GS
    Li, XM
    CHINESE SCIENCE BULLETIN, 1996, 41 (02): : 163 - 168
  • [45] Neutron imaging data processing using the Mantid framework
    Pouzols, Federico M.
    Draper, Nicholas
    Nagella, Sri
    Yang, Erica
    Sajid, Ahmed
    Ross, Derek
    Ritchie, Brian
    Hill, John
    Burca, Genoveva
    Minniti, Triestino
    Moreton-Smith, Christopher
    Kockelmann, Winfried
    VI EUROPEAN CONFERENCE ON NEUTRON SCATTERING (ECNS2015), 2016, 746
  • [46] The data processing pipeline for the MUSE instrument
    Weilbacher, Peter M.
    Palsa, Ralf
    Streicher, Ole
    Bacon, Roland
    Urrutia, Tanya
    Wisotzki, Lutz
    Conseil, Simon
    Husemann, Bernd
    Jarno, Aurelien
    Kelz, Andreas
    Pecontal-Rousset, Arlette
    Richard, Johan
    Roth, Martin M.
    Selman, Fernando
    Vernet, Joel
    ASTRONOMY & ASTROPHYSICS, 2020, 641
  • [47] Creating Analytic Online Homework for Digital Signal Processing
    Trussell, H. Joel
    Baron, Dror
    IEEE SIGNAL PROCESSING MAGAZINE, 2015, 32 (05) : 112 - 118
  • [48] A Framework of Co-creating Business Values for IT Services
    Hosono, Shigeru
    Kuno, Ayako
    Hasegawa, Masahiro
    Hara, Tatsunori
    Shimomura, Yoshiki
    Arai, Tamio
    CLOUD: 2009 IEEE INTERNATIONAL CONFERENCE ON CLOUD COMPUTING, 2009, : 167 - 174
  • [49] Automated pipeline framework for processing of large-scale building energy time series data
    Khalilnejad, Arash
    Karimi, Ahmad M.
    Kamath, Shreyas
    Haddadian, Rojiar
    French, Roger H.
    Abramson, Alexis R.
    PLOS ONE, 2020, 15 (12):
  • [50] An Evolutionary Online Framework for MOOC Performance using EEG Data
    Tahmassebi, Amirhessam
    Gandomi, Amir H.
    Meyer-Baese, Anke
    2018 IEEE CONGRESS ON EVOLUTIONARY COMPUTATION (CEC), 2018, : 895 - 902