Cookery: A framework for creating data processing pipeline using online services

被引:0
|
作者
Baranowski M. [1 ]
Belloum A. [1 ,2 ]
Cushing R. [1 ]
Valkering O. [1 ]
机构
[1] Multiscale Networked Systems (MNS), Institute of Informatics University of Amsterdam, Amsterdam
[2] Netherlands eScience Center Science, Park 140, Amsterdam
来源
| 1600年 / Slovak Academy of Sciences卷 / 39期
基金
欧盟地平线“2020”;
关键词
AWS Lambda; Domain specific languages (DSL); Function-as-a-service (FaaS); Serverless computing;
D O I
10.31577/CAI_2020_4_678
中图分类号
学科分类号
摘要
With the increasing amount of data the importance of data analysis in various scientific domains has grown. A large amount of the scientific data has shifted to cloud based storage. The cloud offers storage and computation power. The Cookery framework is a tool developed to build scientific applications using cloud services. In this paper we present the Cookery systems and how they can be used to authenticate and use standard online third party services to easily create data analytic pipelines. Cookery framework is not limited to work with standard web services; it can also integrate and work with the emerging AWS Lambda which is part of a new computing paradigm, collectively, known as serverless computing. The combination of AWS Lambda and Cookery, which makes it possible for users in many scientific domains, who do not have any program experience, to create data processing pipelines using cloud services in a short time. © 2020 Slovak Academy of Sciences. All rights reserved.
引用
收藏
页码:678 / 694
页数:16
相关论文
共 50 条
  • [31] Library web sites: Creating online collections and services
    Parry, F
    ELECTRONIC LIBRARY, 2004, 22 (05): : 454 - 454
  • [32] Library web sites: Creating online collections and services
    Large, A
    JOURNAL OF LIBRARIANSHIP AND INFORMATION SCIENCE, 2004, 36 (03) : 139 - 140
  • [33] Data processing pipeline for cardiogenic shock prediction using machine learning
    Jajcay, Nikola
    Bezak, Branislav
    Segev, Amitai
    Matetzky, Shlomi
    Jankova, Jana
    Spartalis, Michael
    El Tahlawi, Mohammad
    Guerra, Federico
    Friebel, Julian
    Thevathasan, Tharusan
    Berta, Imrich
    Poelzl, Leo
    Naegele, Felix
    Pogran, Edita
    Cader, F. Aaysha
    Jarakovic, Milana
    Gollmann-Tepekoeylue, Can
    Kollarova, Marta
    Petrikova, Katarina
    Tica, Otilia
    Krychtiuk, Konstantin A.
    Tavazzi, Guido
    Skurk, Carsten
    Huber, Kurt
    Boehm, Allan
    FRONTIERS IN CARDIOVASCULAR MEDICINE, 2023, 10
  • [34] Tracing and using data lineage for pipeline processing in Astro-WISE
    Mwebaze, Johnson
    Boxhoorn, Danny
    Valentijn, Edwin A.
    EXPERIMENTAL ASTRONOMY, 2013, 35 (1-2) : 131 - 155
  • [35] Tracing and using data lineage for pipeline processing in Astro-WISE
    Johnson Mwebaze
    Danny Boxhoorn
    Edwin A. Valentijn
    Experimental Astronomy, 2013, 35 : 131 - 155
  • [36] LOCUST: An online analytical processing framework for high dimensional classification of data streams
    Aggarwal, Charu C.
    Yu, Philip S.
    2008 IEEE 24TH INTERNATIONAL CONFERENCE ON DATA ENGINEERING, VOLS 1-3, 2008, : 426 - +
  • [37] Data processing pipeline for Tianlai experiment
    Zuo, S.
    Li, J.
    Li, Y.
    Santanu, D.
    Stebbins, A.
    Masui, K. W.
    Shaw, R.
    Zhang, J.
    Wu, F.
    Chen, X.
    ASTRONOMY AND COMPUTING, 2021, 34
  • [38] A Distributed Pipeline for DIDSON Data Processing
    Li, Liling
    Danner, Tyler
    Eickholt, Jesse
    McCann, Erin
    Pangle, Kevin
    Johnson, Nicholas
    2017 IEEE INTERNATIONAL CONFERENCE ON BIG DATA (BIG DATA), 2017, : 4301 - 4306
  • [39] The data processing pipeline for the MUSE instrument
    Weilbacher, Peter M.
    Palsa, Ralf
    Streicher, Ole
    Bacon, Roland
    Urrutia, Tanya
    Wisotzki, Lutz
    Conseil, Simon
    Husemann, Bernd
    Jarno, Aurélien
    Kelz, Andreas
    Pécontal-Rousset, Arlette
    Richard, Johan
    Roth, Martin M.
    Selman, Fernando
    Vernet, Joël
    Astronomy and Astrophysics, 2020, 641
  • [40] Pipeline synchronization in data parallel processing
    张冠松
    李晓明
    Chinese Science Bulletin, 1996, (02) : 163 - 168