Sharing lattice QCD data over a widely distributed file system

被引:19
|
作者
Amagasa, T. [1 ]
Aoki, S. [2 ]
Aoki, Y. [3 ]
Aoyama, T. [3 ]
Doi, T. [4 ]
Fukumura, K. [2 ]
Ishii, N. [5 ]
Ishikawa, K. -I [6 ]
Jitsumoto, H. [7 ]
Kamano, H. [5 ]
Konno, Y. [11 ]
Matsufuru, H. [8 ]
Mikami, Y. [11 ]
Miura, K. [3 ]
Sato, M. [1 ]
Takeda, S. [9 ]
Tatebe, O. [1 ]
Togawa, H. [5 ]
Ukawa, A. [10 ]
Ukita, N. [1 ]
Watanabe, Y. [4 ]
Yamazaki, T. [1 ]
Yoshie, T. [1 ]
机构
[1] Univ Tsukuba, Ctr Computat Sci, Tsukuba, Ibaraki 3058577, Japan
[2] Kyoto Univ, Yukawa Inst Theoret Phys, Kyoto, Kyoto 6068502, Japan
[3] Nagoya Univ, Kobayashi Maskawa Inst Origin Particles & Univers, Nagoya, Aichi 4648602, Japan
[4] RIKEN Nishina Ctr, Wako, Saitama 3510198, Japan
[5] Osaka Univ, Nucl Phys Res Ctr, Ibaraki, Osaka 5670047, Japan
[6] Hiroshima Univ, Grad Sch Sci, Hiroshima 7398526, Japan
[7] Univ Tokyo, Ctr Informat Technol, Bunkyo Ku, Tokyo 1138658, Japan
[8] KEK, Ctr Res Comp, Tsukuba, Ibaraki 3050801, Japan
[9] Kanazawa Univ, Inst Phys, Kanazawa, Ishikawa 9201192, Japan
[10] RIKEN Adv Inst Computat Sci, Kobe, Hyogo 6500047, Japan
[11] Hitachi Solut East Japan Ltd, Kawasaki, Kanagawa 2100007, Japan
关键词
D O I
10.1088/1742-6596/664/4/042058
中图分类号
O57 [原子核物理学、高能物理学];
学科分类号
070202 ;
摘要
JLDG is a data-grid for the lattice QCD (LQCD) community in Japan. Several large research groups in Japan have been working on lattice QCD simulations using supercomputers distributed over distant sites. The JLDG provides such collaborations with an efficient method of data management and sharing. File servers installed on 9 sites are connected to the NH SINET VPN and are bound into a single file system with the GFarm. The file system looks the same from any sites, so that users can do analyses on a supercomputer on a site, using data generated and stored in the JLDG at a different site. We present a brief description of hardware and software of the JLDG, including a recently developed subsystem for cooperating with the HPCI shared storage, and report performance and statistics of the JLDG. As of April 2015, 15 research groups (61 users) store their daily research data of 4.7PB including replica and 68 million files in total. Number of publications for works which used the JLDG is 98. The large number of publications and recent rapid increase of disk usage convince us that the JLDG has grown up into a useful infrastructure for LQCD community in Japan.
引用
收藏
页数:8
相关论文
共 50 条
  • [1] A Peer-to-Peer File Sharing System over Named Data Networking
    Shi, Jian
    Piao, Xuewei
    Li, Lihua
    Xun, Yunbo
    Lei, Kai
    2016 IEEE 22ND INTERNATIONAL CONFERENCE ON PARALLEL AND DISTRIBUTED SYSTEMS (ICPADS), 2016, : 159 - 167
  • [2] Wharf: Sharing Docker Images in a Distributed File System
    Zheng, Chao
    Rupprecht, Lukas
    Tarasov, Vasily
    Thain, Douglas
    Mohamed, Mohamed
    Skourtis, Dimitrios
    Warke, Amit S.
    Hildebrand, Dean
    PROCEEDINGS OF THE 2018 ACM SYMPOSIUM ON CLOUD COMPUTING (SOCC '18), 2018, : 174 - 185
  • [3] Thoughts on distributed file sharing
    Kropf, M
    COMPUTER, 2001, 34 (01) : 10 - 10
  • [4] Sharing Linked Open Data over Peer-to-Peer Distributed File Systems: The Case of IPFS
    Sicilia, Miguel-Angel
    Sanchez-Alonso, Salvador
    Garcia-Barriocanal, Elena
    METADATA AND SEMANTICS RESEARCH, MTSR 2016, 2016, 672 : 3 - 14
  • [5] Data Structures for Storing File Namespace in Distributed File System
    Long, Luu Hoang
    Choi, Eunmi
    Kim, SangBum
    Kim, Pilsung
    NCM 2008 : 4TH INTERNATIONAL CONFERENCE ON NETWORKED COMPUTING AND ADVANCED INFORMATION MANAGEMENT, VOL 1, PROCEEDINGS, 2008, : 250 - 255
  • [6] Private Search Over Big Data Leveraging Distributed File System and Parallel Processing
    Selcuk, Ayse
    Orencik, Cengiz
    Savas, Erkay
    CLOUD COMPUTING 2015: THE SIXTH INTERNATIONAL CONFERENCE ON CLOUD COMPUTING, GRIDS, AND VIRTUALIZATION, 2015, : 116 - 121
  • [7] Data Security in Hadoop Distributed File System
    Shetty, Madhvaraj M.
    Manjaiah, D. H.
    IEEE INTERNATIONAL CONFERENCE ON EMERGING TECHNOLOGICAL TRENDS IN COMPUTING, COMMUNICATIONS AND ELECTRICAL ENGINEERING (ICETT), 2016,
  • [8] A Distributed File System over Unreliable Network Storages
    Oh, Kyunghee
    Choi, Dooho
    2015 INTERNATIONAL CONFERENCE ON ICT CONVERGENCE (ICTC), 2015, : 653 - 657
  • [9] Distributed file sharing: Barbarians at the gates?
    Macedonia, M
    COMPUTER, 2000, 33 (08) : 99 - 101
  • [10] OPTIMAL FILE SHARING IN DISTRIBUTED NETWORKS
    NAOR, M
    ROTH, RM
    SIAM JOURNAL ON COMPUTING, 1995, 24 (01) : 158 - 183