Efficient structured data access in parallel file systems

被引：0

作者：

Ching, A ^{[1
]}

Choudhary, A ^{[1
]}

Liao, WK ^{[1
]}

Ross, R ^{[1
]}

Gropp, W ^{[1
]}

机构：

[1] Northwestern Univ, Dept Elect & Comp Engn, Evanston, IL 60208 USA

来源：

IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING, PROCEEDINGS | 2003年

关键词：

D O I：

暂无

中图分类号：

TP301 [理论、方法];

学科分类号：

081202 ;

摘要：

Parallel scientific applications store and retrieve very large, structured datasets. Directly supporting these structured accesses is an important step in providing high-performance I/O solutions for these applications. High-level interfaces such as HDF5 and Parallel netCDF provide convenient APIs for accessing structured datasets, and the MPI-IO interface also supports efficient access to structured data. However, parallel file systems do not traditionally support such access. In this work, we present an implementation of structured data access support in the context of the Parallel Virtual File System (PVFS). We call this support "datatype I/O" because of its similarity to MPI datatypes. This support is built by using a reusable datatype-processing component from the MPICH2 MPI implementation. We describe how this component is leveraged to efficiently process structured data representations resulting from MPI-IO operations. We quantitatively assess the solution using three test applications. We also point to further optimizations in the processing path that could be leveraged for even more efficient operation.

引用

页码：326 / 335

页数：10

共 50 条

[31] SPECIAL SECTION ON DISK FILE ACCESS SYSTEMS FOR DATA-STORAGE DEVICES
LOW, TS
[J]. IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS, 1995, 42 (03) : 221 - 221
[32] Optimization of Reading Data via Classified Block Access Patterns in File Systems
Liao, Jianwei
Chen, Shanxiong
[J]. IEEE ACCESS, 2016, 4 : 9421 - 9427
[33] Efficient Data-parallel Computations on Distributed Systems
曾志勇
[J]. High Technology Letters, 2002, (03) : 92 - 96
[34] Efficient Data-Parallel Primitives on Heterogeneous Systems
Lai, Zhuohang
Luo, Qiong
Xie, Xiaolong
[J]. PROCEEDINGS OF THE 48TH INTERNATIONAL CONFERENCE ON PARALLEL PROCESSING (ICPP 2019), 2019,
[35] Parallel file access for implementing dynamic load balancing on a massively parallel computer
Shimizu, M
Oue, Y
Ohnishi, K
Kitamura, T
[J]. IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, 1997, E80D (04) : 466 - 472
[36] English Access to Structured Data
Richardson, Kyle D.
Bobrow, Daniel G.
Condoravdi, Cleo
Waldinger, Richard
Das, Amar
[J]. FIFTH IEEE INTERNATIONAL CONFERENCE ON SEMANTIC COMPUTING (ICSC 2011), 2011, : 13 - 20
[37] Modeling and scheduling parallel data flow systems using structured systems of recurrence equations
Charot, F
Nyamsi, M
Quinton, P
Wagner, C
[J]. 15TH IEEE INTERNATIONAL CONFERENCE ON APPLICATION-SPECIFIC SYSTEMS, ARCHITECTURES AND PROCESSORS, PROCEEDINGS, 2004, : 6 - 16
[38] Dynamic file prefetching scheme based on file access patterns in VIA-based parallel file system
Lee, YY
Kim, CY
Seo, DW
[J]. IEICE TRANSACTIONS ON INFORMATION AND SYSTEMS, 2002, E85D (04) : 714 - 721
[39] Quantifying the Effects of Contention on Parallel File Systems
Wright, Steven A.
Jarvis, Stephen A.
[J]. 2015 IEEE 29TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM WORKSHOPS, 2015, : 932 - 940
[40] The Design and Implementation of an Efficient Data Consistency Mechanism for In-Memory File Systems
Chen, Xianzhang
Sha, Edwin H. -M.
Sun, Zhilong
Zhuge, Qingfeng
Jiang, Weiwen
[J]. 2016 13TH INTERNATIONAL CONFERENCE ON EMBEDDED SOFTWARE AND SYSTEMS (ICESS) - PROCEEDINGS, 2016, : 170 - 175

← 1 2 3 4 5 →