Efficient Disk-Based Directed Graph Processing: A Strongly Connected Component Approach

被引:15
|
作者
Zhang, Yu [1 ]
Liao, Xiaofei [1 ]
Shi, Xiang [1 ]
Jin, Hai [1 ]
He, Bingsheng [2 ]
机构
[1] Huazhong Univ Sci & Technol, Serv Comp Technol & Syst Lab, Cluster & Grid Comp, Sch Comp Sci & Technol, Wuhan 430074, Hubei, Peoples R China
[2] Natl Univ Singapore, Dept Comp Sci, Singapore 117418, Singapore
基金
中国国家自然科学基金; 中国博士后科学基金;
关键词
Iterative direct graph processing; strongly connected component; convergence; I/O cost; COMPUTATION; FRAMEWORK;
D O I
10.1109/TPDS.2017.2776115
中图分类号
TP301 [理论、方法];
学科分类号
081202 ;
摘要
Recently, there have been many disk-based systems proposed for iterative graph processing. In the popular vertex/edge-centric systems, an iterative directed graph algorithm needs to reprocess many partitions so as to update their vertices' states according to other non-convergent vertices for the unawareness of their dependencies. As a result, it induces high data access cost and a long time to converge. To tackle this problem, we propose a novel system for iterative directed graph processing with taking advantage of the strongly connected component (SCC) structure. It stores a directed graph into a directed acyclic graph (DAG) sketch, with each node representing a SCC in the original data graph. During execution, the SCCs are loaded into memory for processing in a parallel way according to the topological order of the DAG sketch, and the vertices in each SCC are tried to be handled along the directed paths. In this way, each SCC is able to reach convergence in order and needs to be loaded into the main memory for exactly once, getting much lower data access cost and faster convergence. Besides, the vertices of each SCC need fewer updates for convergence. We further develop a lightweight approach to maintain the DAG sketch and handle SCCs in an incremental way for evolving graphs. Compared with the state-of-the-art methods, experimental results show that our approach achieves a performance improvements of 1.46-8.37 times for static graphs, and can reduce the execution time by 61.4-72.7 percent for evolving graphs.
引用
收藏
页码:830 / 842
页数:13
相关论文
共 50 条
  • [1] Strongly connected dominating and absorbing set in directed disk graph
    Zhang, Zhao
    Wu, Weili
    Wu, Lidong
    Li, Yanjie
    Chen, Zongqing
    [J]. INTERNATIONAL JOURNAL OF SENSOR NETWORKS, 2015, 19 (02) : 69 - 77
  • [2] LUMOS: Dependency-Driven Disk-based Graph Processing
    Vora, Keval
    [J]. PROCEEDINGS OF THE 2019 USENIX ANNUAL TECHNICAL CONFERENCE, 2019, : 429 - 442
  • [3] Redio: Accelerating Disk-Based Graph Processing by Reducing Disk I/Os
    Wu, Chengwen
    Zhang, Guangyan
    Wang, Yang
    Jiang, Xinyang
    Zheng, Weimin
    [J]. IEEE TRANSACTIONS ON COMPUTERS, 2019, 68 (03) : 414 - 425
  • [4] ScaleG: A Distributed Disk-Based System for Vertex-Centric Graph Processing
    Wang, Xubo
    Wen, Dong
    Qin, Lu
    Chang, Lijun
    Zhang, Ying
    Zhang, Wenjie
    [J]. IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, 2023, 35 (02) : 2019 - 2033
  • [5] ON FINDING THE STRONGLY CONNECTED COMPONENTS IN A DIRECTED GRAPH
    NUUTILA, E
    SOISALONSOININEN, E
    [J]. INFORMATION PROCESSING LETTERS, 1994, 49 (01) : 9 - 14
  • [6] Giant strongly connected component of directed networks
    Dorogovtsev, SN
    Mendes, JFF
    Samukhin, AN
    [J]. PHYSICAL REVIEW E, 2001, 64 (02): : 4
  • [7] An Efficient Segment Grouping Approach for Active Disk-based Storage Systems
    Xu, Quanqing
    Xi, Weiya
    Yong, Khai Leong
    Jin, Chao
    [J]. 2018 IEEE 24TH INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED SYSTEMS (ICPADS 2018), 2018, : 178 - 185
  • [8] ScaleG: A Distributed Disk-based System for Vertex-centric Graph Processing (Extended Abstract)
    Wang, Xubo
    Wen, Dong
    Qin, Lu
    Chang, Lijun
    Zhang, Wenjie
    [J]. 2022 IEEE 38TH INTERNATIONAL CONFERENCE ON DATA ENGINEERING (ICDE 2022), 2022, : 1511 - 1512
  • [9] Load the Edges You Need: A Generic I/O Optimization for Disk-based Graph Processing
    Vora, Keval
    Xu, Guoqing
    Gupta, Rajiv
    [J]. PROCEEDINGS OF USENIX ATC '16: 2016 USENIX ANNUAL TECHNICAL CONFERENCE, 2016, : 507 - 522
  • [10] Efficient layout transformation for disk-based multidimensional arrays
    Krishnamoorthy, S
    Baumgartner, G
    Lam, CC
    Nieplocha, J
    Sadayappan, P
    [J]. HIGH PERFORMANCE COMPUTING - HIPC 2004, 2004, 3296 : 386 - 398