Reducing I/O variability using dynamic I/O path characterization in petascale storage systems

Son, Seung Woo; Sehrish, Saba; Liao, Wei-keng; Oldfield, Ron; Choudhary, Alok

doi:10.1007/s11227-016-1904-7

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems

Journal Article · Tue Nov 01 00:00:00 EDT 2016 · Journal of Supercomputing

DOI:https://doi.org/10.1007/s11227-016-1904-7· OSTI ID:1356839

Son, Seung Woo ^[1]; Sehrish, Saba ^[2]; Liao, Wei-keng ^[3]; Oldfield, Ron ^[4]; Choudhary, Alok ^[3]

Univ. of Massachusetts, Lowell, MA (United States)
Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States)
Northwestern Univ., Evanston, IL (United States)
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

In petascale systems with a million CPU cores, scalable and consistent I/O performance is becoming increasingly difficult to sustain mainly because of I/O variability. Furthermore, the I/O variability is caused by concurrently running processes/jobs competing for I/O or a RAID rebuild when a disk drive fails. We present a mechanism that stripes across a selected subset of I/O nodes with the lightest workload at runtime to achieve the highest I/O bandwidth available in the system. In this paper, we propose a probing mechanism to enable application-level dynamic file striping to mitigate I/O variability. We also implement the proposed mechanism in the high-level I/O library that enables memory-to-file data layout transformation and allows transparent file partitioning using subfiling. Subfiling is a technique that partitions data into a set of files of smaller size and manages file access to them, making data to be treated as a single, normal file to users. Here, we demonstrate that our bandwidth probing mechanism can successfully identify temporally slower I/O nodes without noticeable runtime overhead. Experimental results on NERSC’s systems also show that our approach isolates I/O variability effectively on shared systems and improves overall collective I/O performance with less variation.

View Accepted Manuscript (DOE)

Research Organization:: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States); Sandia National Laboratories (SNL-CA), Livermore, CA (United States); Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)

Grant/Contract Number:: AC02-05CH11231; AC02-07CH11359; AC04-94AL85000; FG02-08ER25848; SC0001283; SC0005309; SC0005340; SC0007456

OSTI ID:: 1356839

Alternate ID(s):: OSTI ID: 1469016

Report Number(s):: FERMILAB-PUB--17-292-CD; SAND--2017-3907J; PII: 1904

Journal Information:: Journal of Supercomputing, Journal Name: Journal of Supercomputing Journal Issue: 5 Vol. 73; ISSN 0920-8542

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (49)

I/O-aware bandwidth allocation for petascale computing systems Zhou, Zhou; Yang, Xu; Zhao, Dongfang Parallel Computing, Vol. 58 https://doi.org/10.1016/j.parco.2016.05.005	journal	October 2016
FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes Fryxell, B.; Olson, K.; Ricker, P. The Astrophysical Journal Supplement Series, Vol. 131, Issue 1 https://doi.org/10.1086/317361	journal	November 2000
Direct numerical simulations of turbulent lean premixed combustion Sankaran, Ramanan; Hawkes, Evatt R.; Chen, Jacqueline H. Journal of Physics: Conference Series, Vol. 46 https://doi.org/10.1088/1742-6596/46/1/004	journal	September 2006
IO strategies and data services for petascale data sets from a global cloud resolving model Schuchardt, K. L.; Palmer, B. J.; Daily, J. A. Journal of Physics: Conference Series, Vol. 78 https://doi.org/10.1088/1742-6596/78/1/012089	journal	July 2007
A case study for scientific I/O: improving the FLASH astrophysics code Latham, Rob; Daley, Chris; Liao, Wei-keng Computational Science & Discovery, Vol. 5, Issue 1 https://doi.org/10.1088/1749-4699/5/1/015001	journal	January 2012
BurstMem: A high-performance burst buffer system for scientific applications Wang, Teng; Oral, Sarp; Wang, Yandong 2014 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2014.7004215	conference	October 2014
Exploiting Lustre File Joining for Effective Collective IO Yu, Weikuan; Vetter, Jeffrey; Canon, R. Shane Seventh IEEE International Symposium on Cluster Computing and the Grid (CCGrid '07) https://doi.org/10.1109/CCGRID.2007.51	conference	May 2007
A User-Level InfiniBand-Based File System and Checkpoint Strategy for Burst Buffers Sato, Kento; Mohror, Kathryn; Moody, Adam 2014 14th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2014.24	conference	May 2014
PIDX: Efficient Parallel I/O for Multi-resolution Multi-dimensional Scientific Datasets Kumar, Sidharth; Vishwanath, Venkatram; Carns, Philip 2011 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2011.19	conference	September 2011
Parallel I/O Performance for Application-Level Checkpointing on the Blue Gene/P System Fu, Jing; Min, Misun; Latham, Robert 2011 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2011.81	conference	September 2011
Dynamic file striping and data layout transformation on parallel system with fluctuating I/O workload Son, Seung Woo; Sehrish, Saba; Liao, Wei-keng 2013 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2013.6702689	conference	September 2013
TRIO: Burst Buffer Based I/O Orchestration Wang, Teng; Oral, Sarp; Pritchard, Michael 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.38	conference	September 2015
24/7 Characterization of petascale I/O workloads Carns, Philip; Latham, Robert; Ross, Robert 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289150	conference	August 2009
Data sieving and collective I/O in ROMIO Thakur, R.; Gropp, W.; Lusk, E. Proceedings. Frontiers '99. Seventh Symposium on the Frontiers of Massively Parallel Computation https://doi.org/10.1109/FMPC.1999.750599	conference	January 1999
ParColl: Partitioned Collective I/O on the Cray XT Yu, Weikuan; Vetter, Jeffrey 2008 37th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2008.76	conference	September 2008
Using Subfiling to Improve Programming Flexibility and Performance of Parallel Shared-file I/O Gao, Kui; Liao, Wei-keng; Nisar, Arifa 2009 International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2009.68	conference	September 2009
Managing I/O Interference in a Shared Burst Buffer System Thapaliya, Sagar; Bangalore, Purushotham; Lofstead, Jat 2016 45th International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2016.54	conference	August 2016
IO-Cop: Managing Concurrent Accesses to Shared Parallel File System Thapaliya, Sagar; Bangalore, Purushotham; Lofstead, Jay 2014 43nd International Conference on Parallel Processing Workshops (ICCPW), 2014 43rd International Conference on Parallel Processing Workshops https://doi.org/10.1109/ICPPW.2014.20	conference	September 2014
Log-Assisted Straggler-Aware I/O Scheduler for High-End Computing Tavakoli, Neda; Dai, Dong; Chen, Yong 2016 45th International Conference on Parallel Processing Workshops (ICPPW) https://doi.org/10.1109/ICPPW.2016.38	conference	August 2016
CALCioM: Mitigating I/O Interference in HPC Systems through Cross-Application Coordination Dorier, Matthieu; Antoniu, Gabriel; Ross, Rob 2014 IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2014 IEEE 28th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2014.27	conference	May 2014
On the Root Causes of Cross-Application I/O Interference in HPC Storage Systems Yildiz, Orcun; Dorier, Matthieu; Ibrahim, Shadi 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.50	conference	May 2016
Toward a General I/O Layer for Parallel-Visualization Applications Kendall, W.; Peterka, T. IEEE Computer Graphics and Applications, Vol. 31, Issue 6 https://doi.org/10.1109/MCG.2011.102	journal	November 2011
On the role of burst buffers in leadership-class storage systems Liu, Ning; Cope, Jason; Carns, Philip 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2012.6232369	conference	April 2012
Jitter-free co-processing on a prototype exascale storage stack Bent, John; Faibish, Sorin; Ahrens, Jim 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2012.6232382	conference	April 2012
ASCAR: Automating contention management for high-performance storage systems Li, Yan; Lu, Xiaoyuan; Miller, Ethan L. 2015 31st Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2015.7208287	conference	May 2015
Dynamically adapting file domain partitioning methods for collective I/O based on underlying parallel file system locking protocols Liao, Wei-keng; Choudhary, Alok 2008 SC - International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2008.5222722	conference	November 2008
Managing Variability in the IO Performance of Petascale Storage Systems Lofstead, Jay; Zheng, Fang; Liu, Qing 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.32	conference	November 2010
Characterizing output bottlenecks in a supercomputer Xie, Bing; Chase, Jeffrey; Dillow, David 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.28	conference	November 2012
Efficient data restructuring and aggregation for I/O acceleration in PIDX Kumar, Sidharth; Vishwanath, Venkatram; Carns, Philip 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.54	conference	November 2012
Two-Choice Randomized Dynamic I/O Scheduler for Object Storage Systems Dai, Dong; Chen, Yong; Kimpe, Dries SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.57	conference	November 2014
Scalable Design and Implementations for MPI Parallel Overlapping I/O No authors listed IEEE Transactions on Parallel and Distributed Systems, Vol. 17, Issue 11 https://doi.org/10.1109/TPDS.2006.163	journal	November 2006
Efficient data restructuring and aggregation for I/O acceleration in PIDX Kumar, Sidharth; Vishwanath, Venkatram; Carns, Philip 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/sc.2012.54	conference	November 2012
Massively Parallel i/o for Partitioned Solver Systems Liu, Ning; Fu, Jing; Carothers, Christopher D. Parallel Processing Letters, Vol. 20, Issue 04 https://doi.org/10.1142/S0129626410000302	journal	December 2010
Massively Parallel i/o for Partitioned Solver Systems Liu, Ning; Fu, Jing; Carothers, Christopher D. Parallel Processing Letters, Vol. 20, Issue 04 https://doi.org/10.1142/s0129626410000302	journal	December 2010
Parallel netCDF: A High-Performance Scientific I/O Interface Li, Jianwei; Zingale, Michael; Liao, Wei-keng Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050189	conference	January 2003
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
Y-lib: a user level library to increase the performance of MPI-IO in a lustre file system environment Dickens, Phillip M.; Logan, Jeremy Proceedings of the 18th ACM international symposium on High performance distributed computing - HPDC '09 https://doi.org/10.1145/1551609.1551617	conference	January 2009
PLFS: a checkpoint filesystem for parallel applications Bent, John; Gibson, Garth; Grider, Gary https://doi.org/10.1145/1654059.1654081	conference	January 2009
I/O performance challenges at leadership scale Lang, Samuel; Carns, Philip; Latham, Robert Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09 https://doi.org/10.1145/1654059.1654100	conference	January 2009
Improved parallel I/O via a two-phase run-time access strategy del Rosario, Juan Miguel; Bordawekar, Rajesh; Choudhary, Alok ACM SIGARCH Computer Architecture News, Vol. 21, Issue 5 https://doi.org/10.1145/165660.165667	journal	December 1993
Server-side I/O coordination for parallel file systems Song, Huaiming; Yin, Yanlong; Sun, Xian-He Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063407	conference	January 2011
QoS support for end users of I/O-intensive applications using shared storage systems Zhang, Xuechen; Davis, Kei; Jiang, Song Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063408	conference	January 2011
Server-directed collective I/O in Panda Seamons, K. E.; Chen, Y.; Jones, P. Proceedings of the 1995 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '95 https://doi.org/10.1145/224170.224371	conference	January 1995
Disk-directed I/O for MIMD multiprocessors Kotz, David ACM Transactions on Computer Systems, Vol. 15, Issue 1 https://doi.org/10.1145/244764.244766	journal	February 1997
Insights for exascale IO APIs from building a petascale IO API Lofstead, Jay; Ross, Robert Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503238	conference	January 2013
How Much SSD Is Useful for Resilience in Supercomputers Fang, Aiman; Chien, Andrew A. Proceedings of the 5th Workshop on Fault Tolerance for HPC at eXtreme Scale - FTXS '15 https://doi.org/10.1145/2751504.2751509	conference	January 2015
Breaking the Cloud Parameterization Deadlock Randall, David; Khairoutdinov, Marat; Arakawa, Akio Bulletin of the American Meteorological Society, Vol. 84, Issue 11 https://doi.org/10.1175/BAMS-84-11-1547	journal	November 2003
The Tau Parallel Performance System Shende, Sameer S.; Malony, Allen D. The International Journal of High Performance Computing Applications, Vol. 20, Issue 2 https://doi.org/10.1177/1094342006064482	journal	May 2006
PLFS: A Checkpoint Filesystem for Parallel Applications Bent, John; Gibson, Garth; Grider, Gary Carnegie Mellon University https://doi.org/10.1184/r1/6608453	text	January 2005

Cited By (1)

Approaches of enhancing interoperations among high performance computing and big data analytics via augmentation Pathak, Ajeet Ram; Pandey, Manjusha; Rautaray, Siddharth S. Cluster Computing, Vol. 23, Issue 2 https://doi.org/10.1007/s10586-019-02960-y	journal	August 2019

Similar Records

Dynamic File Striping and Data Layout Transformation on Parallel System with Fluctuating I/O Workload

Conference · Sat Dec 31 23:00:00 EST 2016 · OSTI ID:1468998

Application-specific I/O Optimizations on Petascale Supercomputers

Journal Article · Thu Jun 05 20:00:00 EDT 2014 · Procedia Computer Science · OSTI ID:1567357

Related Subjects

97 MATHEMATICS AND COMPUTING
I/O variability
PnetCDF
parallel I/O
subfile

Reducing I/O variability using dynamic I/O path characterization in petascale storage systems

Citation Formats

References (49)

Cited By (1)

Similar Records

Related Subjects