Real Time Delta Extraction Based on Triggers to Support Data Warehousing

被引:7
|
作者
Valencio, Carlos Roberto [1 ]
Marioto, Matheus Henrique [1 ]
Donega Zafalon, Geraldo Francisco [1 ]
Machado, Jose Marcio [1 ]
Momente, Julio Cesar [1 ]
机构
[1] Sao Paulo State Univ, Dept Ciencias Computacao & Estat, Sao Paulo, Brazil
关键词
data warehouse; ETL (Extract Transform and Load); CDC (Change Data Capture); trigger; automatic code generation;
D O I
10.1109/PDCAT.2013.52
中图分类号
TP3 [计算技术、计算机技术];
学科分类号
0812 ;
摘要
Nowadays large corporations require integrated data from diverse sources, leading to the use of data warehouse architectures for this purpose. To bypass problems related to the use of computational resources to process large volumes of data, an ETL (Extract, Transform and Load) technique with zero latency can be used, that works by constantly processing small data loads. Among the extraction techniques of the zero latency ETL are the use of logs, triggers, materialized views and timestamps. This paper proposes a structure capable of performing this task by means of triggers and a tool developed for the automatic generation of the SQL (Structured Query Language) code to create these trigger, besides showing its performance and comparing it to other techniques. Said method is relevant for the extraction of portions of selected information as it permits to combine conventional and real time ETL techniques.
引用
收藏
页码:293 / 297
页数:5
相关论文
共 50 条
  • [1] Towards Near Real-Time Data Warehousing
    Chen, Li
    Rahayu, Wenny
    Taniar, David
    [J]. 2010 24TH IEEE INTERNATIONAL CONFERENCE ON ADVANCED INFORMATION NETWORKING AND APPLICATIONS (AINA), 2010, : 1150 - 1157
  • [2] Bioterrorism surveillance with real-time data warehousing
    Berndt, DJ
    Hevner, AR
    Studnicki, J
    [J]. INTELLIGENCE AND SECURITY INFORMATICS, PROCEEDINGS, 2003, 2665 : 322 - 335
  • [3] HYBRIDJOIN for Near-Real-Time Data Warehousing
    Naeem, M. Asif
    Dobbie, Gillian
    Weber, Gerald
    [J]. INTERNATIONAL JOURNAL OF DATA WAREHOUSING AND MINING, 2011, 7 (04) : 21 - 42
  • [4] An architecture for real-time warehousing of scientific data
    Lawrence, R
    Kruger, A
    [J]. CSC '05: PROCEEDINGS OF THE 2005 INTERNATIONAL CONFERENCE ON SCIENTIFIC COMPUTING, 2005, : 151 - 156
  • [5] Real-time Data Superaddition Method Based on Triggers
    Cen, Ming
    Jiang, Jianchun
    Hu, Ruifei
    [J]. 2008 7TH WORLD CONGRESS ON INTELLIGENT CONTROL AND AUTOMATION, VOLS 1-23, 2008, : 6232 - +
  • [6] A continuous data integration methodology for supporting real-time data warehousing
    Santos, Ricardo Jorge
    Bernardino, Jorge
    [J]. ICEIS 2007: PROCEEDINGS OF THE NINTH INTERNATIONAL CONFERENCE ON ENTERPRISE INFORMATION SYSTEMS: DATABASES AND INFORMATION SYSTEMS INTEGRATION, 2007, : 589 - 595
  • [7] A non-intrusive and reactive architecture to support real-time ETL processes in data warehousing environments
    de Assis Vilela, Flavio
    Times, Valeria Cesario
    de Campos Bernardi, Alberto Carlos
    de Paula Freitas, Augusto
    Ciferri, Ricardo Rodrigues
    [J]. HELIYON, 2023, 9 (05)
  • [8] X-HYBRIDJOIN for Near-Real-Time Data Warehousing
    Naeem, Muhammad Asif
    Dobbie, Gillian
    Weber, Gerald
    [J]. ADVANCES IN DATABASES, 2011, 7051 : 33 - 47
  • [9] A Robust Join Operator to Process Streaming Data in Real-Time Data Warehousing
    Naeem, M. Asif
    [J]. 2013 EIGHTH INTERNATIONAL CONFERENCE ON DIGITAL INFORMATION MANAGEMENT (ICDIM), 2013, : 119 - 124
  • [10] Metadata-Driven SOA-Based Application for Facilitation of Real-Time Data Warehousing
    Pintar, Damir
    Vranic, Mihaela
    Skocir, Zoran
    [J]. E-COMMERCE AND WEB TECHNOLOGIES, PROCEEDINGS, 2009, 5692 : 108 - 119