I/O characterization and performance evaluation of large-scale storage architectures for heterogeneous workloads

被引：0

作者：

Kogiou, Olga ^{[1
]}

Devarajan, Hariharan ^{[2
]}

Wang, Chen ^{[2
]}

Yu, Weikuan ^{[1
]}

Mohror, Kathryn ^{[2
]}

机构：

[1] Florida State Univ, Tallahassee, FL 32306 USA

[2] Lawrence Livermore Natl Lab, Livermore, CA 94550 USA

来源：

2023 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING WORKSHOPS, CLUSTER WORKSHOPS | 2023年

关键词：

VAST; IOR benchmark; HPC applications;

D O I：

10.1109/CLUSTERWorkshops61457.2023.00017

中图分类号：

TP3 [计算技术、计算机技术];

学科分类号：

0812 ;

摘要：

HPC systems traditionally supported compute-centric workloads. However, the increasing reliance on data has led to a shift towards data-dependent workloads. This transition has necessitated storage technologies that enable fast data sharing among workflow parts, but diverse I/O requirements demand tailored solutions. New HPC architectures incorporate specialized software layers like Datawarp, IME, and VAST. However, user-driven storage system selection may lead to improper choices. Our investigation compares VAST with GPFS and Lustre filesystems across multiple machines, measuring performance, scalability, and identifying suitable I/O behaviors. This work provides a guide for selecting the appropriate storage system to optimize data access based on user requirements.

引用

页码：44 / 45

页数：2

共 50 条

[31] Design and Performance Evaluation of Read/Write Metadata Service Strategy for Large-Scale Storage System
Zhou, Gang
Cao, Qiang
Wu, Chentao
EIGHTH INTERNATIONAL SYMPOSIUM ON OPTICAL STORAGE AND 2008 INTERNATIONAL WORKSHOP ON INFORMATION DATA STORAGE, 2009, 7125
[32] UDP performance characterization in a large-scale SMP system
Lu, YP
Ko, KT
Du, DHC
Proceedings of the Eighth IASTED International Conference on Internet and Multimedia Systems and Applications, 2004, : 190 - 195
[33] A Performance Evaluation of Scientific I/O Workloads on Flash-Based SSDs
Park, Stan
Shen, Kai
2009 IEEE INTERNATIONAL CONFERENCE ON CLUSTER COMPUTING AND WORKSHOPS, 2009, : 501 - 505
[34] SIMULATION OF HPC JOB SCHEDULING AND LARGE-SCALE PARALLEL WORKLOADS
Abu Obaida, Mohammad
Liu, Jason
2017 WINTER SIMULATION CONFERENCE (WSC), 2017, : 920 - 931
[35] π-Hub: Large-scale video learning, storage, and retrieval on heterogeneous hardware platforms
Tang, Jie
Liu, Shaoshan
Cao, Jie
Sun, Dawei
Ding, Bolin
Gaudiot, Jean-Luc
Shi, Weisong
FUTURE GENERATION COMPUTER SYSTEMS-THE INTERNATIONAL JOURNAL OF ESCIENCE, 2020, 102 : 514 - 523
[36] CNN ARCHITECTURES FOR LARGE-SCALE AUDIO CLASSIFICATION
Hershey, Shawn
Chaudhuri, Sourish
Ellis, Daniel P. W.
Gemmeke, Jort F.
Jansen, Aren
Moore, R. Channing
Plakal, Manoj
Platt, Devin
Saurous, Rif A.
Seybold, Bryan
Slaney, Malcolm
Weiss, Ron J.
Wilson, Kevin
2017 IEEE INTERNATIONAL CONFERENCE ON ACOUSTICS, SPEECH AND SIGNAL PROCESSING (ICASSP), 2017, : 131 - 135
[37] Performance and Cost Evaluation of Large-Scale Optical Switch Node Architectures for Space-Division Multiplexed Networks
Szczerban, Mijail
Kelly, Colin
Zamy, Thierry
Neilson, David
Ryf, Roland
Simsarian, Jesse
2023 INTERNATIONAL CONFERENCE ON PHOTONICS IN SWITCHING AND COMPUTING, PSC, 2023,
[38] Compiler-optimized simulation of large-scale applications on high performance architectures
Adve, VS
Bagrodia, R
Deelman, E
Sakellariou, R
JOURNAL OF PARALLEL AND DISTRIBUTED COMPUTING, 2002, 62 (03) : 393 - 426
[39] Practice and Thoughts of Large-scale Instrument Performance Evaluation
Liu Yueqi
Liu Fang
Cui Yuanshun
PROCEEDINGS OF 2013 6TH INTERNATIONAL CONFERENCE ON INFORMATION MANAGEMENT, INNOVATION MANAGEMENT AND INDUSTRIAL ENGINEERING (ICIII 2013) VOL 1, 2013, : 523 - +
[40] Distributed Training Large-Scale Deep Architectures
Zou, Shang-Xuan
Chen, Chun-Yen
Wu, Jui-Lin
Chou, Chun-Nan
Tsao, Chia-Chin
Tung, Kuan-Chieh
Lin, Ting-Wei
Sung, Cheng-Lung
Chang, Edward Y.
ADVANCED DATA MINING AND APPLICATIONS, ADMA 2017, 2017, 10604 : 18 - 32

← 1 2 3 4 5 →