NMSTREAM: A SCALABLE EVENT-DRIVEN ETL FRAMEWORK FOR PROCESSING HETEROGENEOUS STREAMING DATA

被引：0
|
作者：
Xiao, Fei [1 ]
Li, Chengming [1 ]
Wu, Zheng [1 ]
Wu, Yinghao [1 ]
机构：
[1] Chinese Acad Surveying & Mapping, Beijing, Peoples R China
来源：
ISPRS TC IV MID-TERM SYMPOSIUM 3D SPATIAL INFORMATION SCIENCE - THE ENGINE OF CHANGE | 2018年 / 4-4卷
关键词：
Streaming data; Extract-Transform-Load; Apache Flume; Apache Cassandra;
D O I：
10.5194/isprs-annals-IV-4-243-2018
中图分类号：
P9 [自然地理学];
学科分类号：
0705 ; 070501 ;
摘要：
ETL (Extraction-Transform-Load) tools, traditionally developed to operate offline on historical data for feeding Data-warehouses need to be enhanced to deal with continuously increased streaming data and be executed at network level during data streams acquisition. In this paper, a scalable and web-based ETL system called NMStream was presented. NMStream is based on event-driven architecture and designed for integrating distributed and heterogeneous streaming data by integrating the Apache Flume and Cassandra DB system, and the ETL processes were conducted through the Flume agent object. NMStream can be used for feeding traditional/ real-time data-warehouses or data analytic tools in a stable and effective manner.
引用
收藏
页码：243 / 246
页数：4
相关论文
共 50 条
  • [1] EPypes: a framework for building event-driven data processing pipelines
    Semeniuta, Oleksandr
    Falkman, Petter
    PEERJ COMPUTER SCIENCE, 2019, 2019 (02)
  • [2] An Event-Driven Serverless ETL Pipeline on AWS
    Pogiatzis, Antreas
    Samakovitis, Georgios
    APPLIED SCIENCES-BASEL, 2021, 11 (01): : 1 - 13
  • [3] Scalable Event-Driven Native Parallel Processing: The SpiNNaker Neuromimetic System
    Rast, Alexander D.
    Jin, Xin
    Galluppi, Francesco
    Plana, Luis A.
    Patterson, Cameron
    Furber, Steve
    PROCEEDINGS OF THE 2010 COMPUTING FRONTIERS CONFERENCE (CF 2010), 2010, : 21 - 30
  • [4] JetStream: Graph Analytics on Streaming Data with Event-Driven Hardware Accelerator
    Rahman, Shafiur
    Afarin, Mahbod
    Abu-Ghazaleh, Nael
    Gupta, Rajiv
    PROCEEDINGS OF 54TH ANNUAL IEEE/ACM INTERNATIONAL SYMPOSIUM ON MICROARCHITECTURE, MICRO 2021, 2021, : 1091 - 1105
  • [5] Event-Driven Packet Processing
    Ibanez, Stephen
    Antichi, Gianni
    Brebner, Gordon
    McKeown, Nick
    PROCEEDINGS OF THE EIGHTEENTH ACM WORKSHOP ON HOT TOPICS IN NETWORKS (HOTNETS '19), 2019, : 133 - 140
  • [6] Scalable Diagnosability Checking of Event-Driven Systems
    Schumann, Anika
    Pencole, Yannick
    20TH INTERNATIONAL JOINT CONFERENCE ON ARTIFICIAL INTELLIGENCE, 2007, : 575 - 580
  • [7] Scalable Proactive Event-Driven Decision Making
    Artikis, Alexander
    Baber, Chris
    Bizarro, Pedro
    Canudas-De-Wit, Carlos
    Etzion, Opher
    Fournier, Fabiana
    Goulart, Paul
    Howes, Andrew
    Lygeros, John
    Paliouras, Georgios
    Schuster, Assaf
    Sharfman, Izchak
    IEEE TECHNOLOGY AND SOCIETY MAGAZINE, 2014, 33 (03) : 35 - 41
  • [8] Neuromorphic Data Processing for Event-Driven Imagery for Acoustic Measurements
    Zheng, Kevin
    Sorensen, Jack
    DeVilliers, Celeste
    Cattaneo, Alessandro
    Moreu, Fernando
    Taylor, Gregory
    Mascarenas, David
    ROTATING MACHINERY, OPTICAL METHODS & SCANNING LDV METHODS, VOL 6, 2023, : 37 - 41
  • [9] Research on Data Processing in RFID Middleware Based on Event-driven
    Fei, Yulian
    Jin, Gonglian
    Wu, Ruqi
    PROCEEDINGS OF THE ICEBE 2008: IEEE INTERNATIONAL CONFERENCE ON E-BUSINESS ENGINEERING, 2008, : 578 - 581
  • [10] Hybrid processing system for sensor networks based on an event-driven framework
    Miyaho, N.
    Iwaki, Y.
    Yamazaki, T.
    Kuji, N.
    IET COMMUNICATIONS, 2010, 4 (07) : 776 - 785
12345