Sharing lattice QCD data over a widely distributed file system

被引:19
|
作者
Amagasa, T. [1 ]
Aoki, S. [2 ]
Aoki, Y. [3 ]
Aoyama, T. [3 ]
Doi, T. [4 ]
Fukumura, K. [2 ]
Ishii, N. [5 ]
Ishikawa, K. -I [6 ]
Jitsumoto, H. [7 ]
Kamano, H. [5 ]
Konno, Y. [11 ]
Matsufuru, H. [8 ]
Mikami, Y. [11 ]
Miura, K. [3 ]
Sato, M. [1 ]
Takeda, S. [9 ]
Tatebe, O. [1 ]
Togawa, H. [5 ]
Ukawa, A. [10 ]
Ukita, N. [1 ]
Watanabe, Y. [4 ]
Yamazaki, T. [1 ]
Yoshie, T. [1 ]
机构
[1] Univ Tsukuba, Ctr Computat Sci, Tsukuba, Ibaraki 3058577, Japan
[2] Kyoto Univ, Yukawa Inst Theoret Phys, Kyoto, Kyoto 6068502, Japan
[3] Nagoya Univ, Kobayashi Maskawa Inst Origin Particles & Univers, Nagoya, Aichi 4648602, Japan
[4] RIKEN Nishina Ctr, Wako, Saitama 3510198, Japan
[5] Osaka Univ, Nucl Phys Res Ctr, Ibaraki, Osaka 5670047, Japan
[6] Hiroshima Univ, Grad Sch Sci, Hiroshima 7398526, Japan
[7] Univ Tokyo, Ctr Informat Technol, Bunkyo Ku, Tokyo 1138658, Japan
[8] KEK, Ctr Res Comp, Tsukuba, Ibaraki 3050801, Japan
[9] Kanazawa Univ, Inst Phys, Kanazawa, Ishikawa 9201192, Japan
[10] RIKEN Adv Inst Computat Sci, Kobe, Hyogo 6500047, Japan
[11] Hitachi Solut East Japan Ltd, Kawasaki, Kanagawa 2100007, Japan
关键词
D O I
10.1088/1742-6596/664/4/042058
中图分类号
O57 [原子核物理学、高能物理学];
学科分类号
070202 ;
摘要
JLDG is a data-grid for the lattice QCD (LQCD) community in Japan. Several large research groups in Japan have been working on lattice QCD simulations using supercomputers distributed over distant sites. The JLDG provides such collaborations with an efficient method of data management and sharing. File servers installed on 9 sites are connected to the NH SINET VPN and are bound into a single file system with the GFarm. The file system looks the same from any sites, so that users can do analyses on a supercomputer on a site, using data generated and stored in the JLDG at a different site. We present a brief description of hardware and software of the JLDG, including a recently developed subsystem for cooperating with the HPCI shared storage, and report performance and statistics of the JLDG. As of April 2015, 15 research groups (61 users) store their daily research data of 4.7PB including replica and 68 million files in total. Number of publications for works which used the JLDG is 98. The large number of publications and recent rapid increase of disk usage convince us that the JLDG has grown up into a useful infrastructure for LQCD community in Japan.
引用
收藏
页数:8
相关论文
共 50 条
  • [21] A Distributed File System over Heterogeneous SaaS Storage Platforms
    Scavuzzo, Marco
    16TH INTERNATIONAL SYMPOSIUM ON SYMBOLIC AND NUMERIC ALGORITHMS FOR SCIENTIFIC COMPUTING (SYNASC 2014), 2014, : 417 - 421
  • [22] Distributed File Sharing Using Web Services
    Chang, Chin-Chih
    Wu, Wen-Xiang
    2014 FOURTH INTERNATIONAL CONFERENCE ON DIGITAL INFORMATION AND COMMUNICATION TECHNOLOGY AND IT'S APPLICATIONS (DICTAP), 2014, : 264 - 268
  • [23] Content Espresso: A Distributed Large File Sharing System for Digital Content Productions
    Ando, Daisuke
    Teraoka, Fumio
    Kaneko, Kunitake
    IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, 2017, E100D (09): : 2100 - 2117
  • [24] DISTRIBUTED FILES SHARING MANAGEMENT: A FILE SHARING APPLICATION USING DISTRIBUTED COMPUTING CONCEPTS
    Malgaonkar, Saurabh
    Surve, Sakshi
    Hirave, Tejas
    2012 IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND COMPUTING RESEARCH (ICCIC), 2012, : 106 - 109
  • [25] An Efficient Data Duplication System based on Hadoop Distributed File System
    Veeraiah, D.
    Rao, J. Nageswara
    PROCEEDINGS OF THE 5TH INTERNATIONAL CONFERENCE ON INVENTIVE COMPUTATION TECHNOLOGIES (ICICT-2020), 2020, : 197 - 200
  • [26] Distributed PACS using Distributed File System with Hierarchical Meta Data Servers
    Hiroyasu, Tomoyuki
    Minamitani, Yoshiyuki
    Miki, Mitsunori
    Yokouchi, Hisatake
    Yoshimi, Masato
    2012 ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY (EMBC), 2012, : 5891 - 5894
  • [27] Data Adaptively Storing Approach for Hadoop Distributed File System
    Fu, Yingxun
    Wen, Shilin
    Ma, Li
    2017 2ND IEEE INTERNATIONAL CONFERENCE ON COMPUTATIONAL INTELLIGENCE AND APPLICATIONS (ICCIA), 2017, : 20 - 24
  • [28] Distributed File System: Efficiency Experiments for Data Access and Communication
    Upadhyaya, Bipin
    Azimov, Fahriddin
    Doan, Thanh Tran
    Choi, Eunmi
    Kim, SangBum
    Kim, Pilsung
    NCM 2008: 4TH INTERNATIONAL CONFERENCE ON NETWORKED COMPUTING AND ADVANCED INFORMATION MANAGEMENT, VOL 2, PROCEEDINGS, 2008, : 400 - +
  • [29] A CKAN Plugin for Data Harvesting to the Hadoop Distributed File System
    Scholz, Robert
    Tcholtchev, Nikolay
    Laemmel, Philipp
    Schieferdecker, Ina
    CLOSER: PROCEEDINGS OF THE 7TH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING AND SERVICES SCIENCE, 2017, : 19 - 28
  • [30] Complete Data Deletion Based on Hadoop Distributed File System
    Wang, Fulin
    Wu, Shunxiang
    Cai, Jianhuai
    Zhao, Longze
    Liao, Zhendong
    Ming, Daodong
    PROCEEDINGS OF THE THIRD INTERNATIONAL CONFERENCE ON COMPUTER SCIENCE AND APPLICATION ENGINEERING (CSAE2019), 2019,