Software Development Framework for a Distributed Storage and GPGPU Data Processing Infrastructure

被引：0
|
作者：
Kamenskikh, I. S. [1 ]
Sinelnikov, D. M. [1 ]
Kalintsev, D. S. [1 ]
Kozlov, A. A. [1 ]
Rovnyagin, M. M. [1 ]
Shulga, D. A. [1 ]
机构：
[1] Natl Res Nucl Univ MEPhI, Moscow Engn Phys Inst, Moscow, Russia
来源：
PROCEEDINGS OF THE 2016 IEEE NORTH WEST RUSSIA SECTION YOUNG RESEARCHERS IN ELECTRICAL AND ELECTRONIC ENGINEERING CONFERENCE (ELCONRUSNW) | 2016年
关键词：
Hadoop; YARN; Map reduce; Dynamic load balance; CUDA; Casandra;
D O I：
暂无
中图分类号：
TM [电工技术]; TN [电子技术、通信技术];
学科分类号：
0808 ; 0809 ;
摘要：
The problem of choosing the cluster or a cluster node for task execution is important for the overall performance of a distributed system. This paper presents a complex approach to the planning of computations on heterogeneous distributed systems - a set of clusters and NoSQL storage systems. Dynamic scheduling algorithm depends on: the inter-cluster network parameters, characteristics of cluster interconnect, compute nodes utilization, co-processors computing capabilities, etc. In this work Hadoop YARN, CUDA technology and NoSQL-system Apache Cassandra has been used as the experimental platform.
引用
收藏
页码：216 / 219
页数：4
相关论文
共 50 条
  • [1] Development and evaluation of ParaRuby: A distributed GPGPU framework
    Yoshimi, M., 1600, Japan Society for Software Science and Technology (29):
  • [2] Storage resource broker; Generic software infrastructure for managing globally distributed data
    Moore, RW
    Wan, M
    Rajasekar, A
    Local to Global Data Interoperability - Challenges and Technologies: BEYOND MASS STORAGE TO GLOBALLY DISTRIBUTED DATA, 2005, : 65 - 69
  • [3] A SOFTWARE INFRASTRUCTURE FOR DISTRIBUTED DATA MODELS IN CLOUD
    Jelacic, Bojan
    Vukmirovic, Srdan
    Stoja, Sebastijan
    2013 21ST TELECOMMUNICATIONS FORUM (TELFOR), 2013, : 845 - +
  • [4] A Framework for Distributed Data Processing
    Kadir Febrer-Hernandez, Jose
    Herrera Semenets, Vitali
    PROGRESS IN PATTERN RECOGNITION, IMAGE ANALYSIS, COMPUTER VISION, AND APPLICATIONS (CIARP 2019), 2019, 11896 : 566 - 574
  • [5] MataNui - A Distributed Storage Infrastructure for Scientific Data
    Kloss, Guy K.
    2013 INTERNATIONAL CONFERENCE ON COMPUTATIONAL SCIENCE, 2013, 18 : 2607 - 2610
  • [6] A distributed storage framework of FlowTable in software defined network
    Ma, Huan
    Yang, Yang
    Mi, Zhenqiang
    COMPUTERS & ELECTRICAL ENGINEERING, 2015, 43 : 155 - 168
  • [7] The Distributed Control Framework: A software infrastructure for agent-based distributed control and robotics
    Kulis, Zachary
    Manikonda, Vikram
    Azimi-Sadjadi, Babak
    Ranjan, Priya
    2008 AMERICAN CONTROL CONFERENCE, VOLS 1-12, 2008, : 1329 - 1336
  • [8] A software framework for heterogeneous, distributed data fusion
    Walters, Joshua J.
    Julier, Simon J.
    2006 9TH INTERNATIONAL CONFERENCE ON INFORMATION FUSION, VOLS 1-4, 2006, : 1415 - 1422
  • [9] Computational infrastructure for parallel processing spatially distributed data
    Bychkov, I. V.
    Kitov, A. D.
    Cherkashin, E. A.
    COMPUTATIONAL SCIENCE AND HIGH PERFORMANCE COMPUTING II, 2006, 91 : 233 - +
  • [10] A distributed data storage and processing framework for next-generation residential distribution systems
    Zhang, Ni
    Yan, Yu
    Xu, Shengyao
    Su, Wencong
    ELECTRIC POWER SYSTEMS RESEARCH, 2014, 116 : 174 - 181
12345