Cookery: A framework for creating data processing pipeline using online services

被引:0
|
作者
Baranowski M. [1 ]
Belloum A. [1 ,2 ]
Cushing R. [1 ]
Valkering O. [1 ]
机构
[1] Multiscale Networked Systems (MNS), Institute of Informatics University of Amsterdam, Amsterdam
[2] Netherlands eScience Center Science, Park 140, Amsterdam
来源
| 1600年 / Slovak Academy of Sciences卷 / 39期
基金
欧盟地平线“2020”;
关键词
AWS Lambda; Domain specific languages (DSL); Function-as-a-service (FaaS); Serverless computing;
D O I
10.31577/CAI_2020_4_678
中图分类号
学科分类号
摘要
With the increasing amount of data the importance of data analysis in various scientific domains has grown. A large amount of the scientific data has shifted to cloud based storage. The cloud offers storage and computation power. The Cookery framework is a tool developed to build scientific applications using cloud services. In this paper we present the Cookery systems and how they can be used to authenticate and use standard online third party services to easily create data analytic pipelines. Cookery framework is not limited to work with standard web services; it can also integrate and work with the emerging AWS Lambda which is part of a new computing paradigm, collectively, known as serverless computing. The combination of AWS Lambda and Cookery, which makes it possible for users in many scientific domains, who do not have any program experience, to create data processing pipelines using cloud services in a short time. © 2020 Slovak Academy of Sciences. All rights reserved.
引用
收藏
页码:678 / 694
页数:16
相关论文
共 50 条
  • [21] The MAGIC data processing pipeline
    Firpo Curcoll, R.
    Delfino, M.
    Neissner, C.
    Reichardt, I.
    Rico, J.
    Tallada, P.
    Tonello, N.
    INTERNATIONAL CONFERENCE ON COMPUTING IN HIGH ENERGY AND NUCLEAR PHYSICS (CHEP 2010): EVENT PROCESSING, 2011, 331
  • [22] GALFACTS Data Processing Pipeline
    Guram, S. S.
    Andrecut, M.
    George, S. J.
    Taylor, A. R.
    ASTRONOMICAL DATA ANALYSIS SOFTWARE AND SYSTEMS XX, 2011, 442 : 317 - 320
  • [23] The ISOPHOT pipeline data processing
    Richards, PJ
    Klaas, U
    Laureijs, RJ
    Abraham, P
    Schulz, B
    Morris, H
    Wilke, K
    Heinrichsen, I
    PROCEEDINGS OF THE CONFERENCE ON THE CALIBRATION LEGACY OF THE ISO MISSION, 2003, 481 : 279 - 284
  • [24] Asynchronous dual-pipeline deep learning framework for online data stream classification
    Lara-Benitez, Pedro
    Carranza-Garcia, Manuel
    Garcia-Gutierrez, Jorge
    Riquelme, Jose C.
    INTEGRATED COMPUTER-AIDED ENGINEERING, 2020, 27 (02) : 101 - 119
  • [25] A Plugin-Based Software Framework for Data Acquisition and Online Processing
    Fan, Shaoshuai
    Gu, Minhao
    Zhang, Hangchang
    IEEE TRANSACTIONS ON NUCLEAR SCIENCE, 2025, 72 (03) : 518 - 525
  • [26] Knowledge Graph Generation and Application for Unstructured Data Using Data Processing Pipeline
    Thushara Sukumar, Sushmi
    Lung, Chung-Horng
    Zaman, Marzia
    Panday, Ritesh
    IEEE ACCESS, 2024, 12 : 136759 - 136770
  • [27] Online data processing
    Bouchachia, Abdelhamid
    NEUROCOMPUTING, 2014, 126 : 116 - 117
  • [28] Library web sites: Creating online collections and services
    Donovan, C
    JOURNAL OF ACADEMIC LIBRARIANSHIP, 2004, 30 (05): : 423 - 423
  • [29] LIBRARY WEB SITES: CREATING ONLINE COLLECTIONS AND SERVICES
    Nisonger, Thomas E.
    TECHNICAL SERVICES QUARTERLY, 2005, 23 (01) : 105 - 107
  • [30] Library web sites: Creating online collections and services
    Hamburger, S
    LIBRARY COLLECTIONS ACQUISITIONS & TECHNICAL SERVICES, 2004, 28 (04): : 494 - 494