Data Synchronization Tool for Distributed Heterogeneous Database

被引：0
|
作者：
Xu Z.-J. [1 ,2 ]
Ye S. [1 ,2 ]
Zhang X. [1 ,2 ]
机构：
[1] Key Laboratory of Data Engineering and Knowledge Engineering of the Ministry of Education, Renmin University of China, Beijing
[2] School of Information, Renmin University of China, Beijing
来源：
Ruan Jian Xue Bao/Journal of Software | 2019年 / 30卷 / 03期
基金：
中国国家自然科学基金; 国家重点研发计划;
关键词：
Data synchronization; Read/write separation; SQL reduction;
D O I：
10.13328/j.cnki.jos.005694
中图分类号：
学科分类号：
摘要：
In general, the read-write separation technology can solve some of the problems on mismatch between read and write in the current big data environment, but most of the current read-write separation technology are prepared for homogeneous database. Due to the inconsistent storage structure, heterogeneous distributed database systems composed of a row storage database and a columnar storage database will face many difficulties like format conversion and mismatch of synchronization speed in data synchronization compared to a homogeneous distributed database system. This study proposes the use of MySQL binary log to perform the TD-Reduction of SQL. It designs and implements Binlog parser BinParser and Binlog restorer BinReducer, which based on the mixed format. Different events perform log parsing and restoring according to the corresponding rules to generate executable SQL statements. Based on the above techniques, this study has implemented Cynomys, a distributed database data synchronization tool. In the experimental environment, Cynomys has shown sound performance. The method is suitable for data synchronization between all other heterogeneous databases with a similar mechanism like Binlog. © Copyright 2019, Institute of Software, the Chinese Academy of Sciences. All rights reserved.
引用
收藏
页码：684 / 699
页数：15
相关论文
共 19 条
  • [1] Stonebraker M., Aoki P.M., Litwin W., Pfeffer A., Sah A., Sidell J., Et al., Mariposa: A wide-area distributed database system, VLDB Journal, 5, 1, pp. 48-63, (1996)
  • [2] Chen K., Zhou Y., Cao Y., Online data partitioning in distributed database systems, Proc. of the Int'l Conf. on Extending Database Technology (EDBT 2015), pp. 1-12, (2015)
  • [3] Corbett J.C., Dean J., Epstein M., Et al., Spanner: Google's globally-distributed database, Proc. of the Usenix Conf. on Operating Systems Design and Implementation, 31, pp. 251-264, (2012)
  • [4] Wang J., Zhang D.S., Research and design of distributed database synchronization system based on middleware, Proc. of the Modern Electronics Technique, pp. 685-688, (2016)
  • [5] Lahiri T., Chavan S., Colgan M., Et al., Oracle database in-memory: A dual format in-memory database, Proc. of the IEEE, Int'l Conf. on Data Engineering, pp. 1253-1258, (2016)
  • [6] Mukherjee N., Chavan S., Colgan M., Et al., Distributed architecture of oracle database in-memory, Proc. of the VLDB Endowment, 8, 12, pp. 1630-1641, (2015)
  • [7] Mukherjee N., Kulkarni K., Jin H., Et al., How does oracle database in-memory scale out?, Proc. of the Int'l Joint Conf. on Software Technologies, 1, pp. 1-6, (2015)
  • [8] Farber F., May N., Lehner W., Et al., The sap hana database-An architecture overview, Bulletin of the Technical Committee on Data Engineering, 35, 1, pp. 28-33, (2012)
  • [9] Wang Z., Research and implementation of load balancing algorithm for offline data migration, (2015)
  • [10] Li G.X., Liu S., Liu J.C., Et al., Research and application of data synchronization service platform based on achived logs, Electric Power Information and Communication Technology, 8, 2, pp. 31-35, (2010)
12