Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks.

Liu, Qing Gary; Logan, Jeremy S.; Tian, Yuan; Abbasi, Hasan; Podhorszki, Norbert; Choi, Jong Youl; Klasky, Scott A.; Tchoua, Roselyne B.; Lofstead, J.; Oldfield, Ron A.; Parashar, Manish; Samatova, Nagiza; Schwan, Karsten; Shoshani, A.; Wolf, Matthew D.; Wu, K.; Yu, Weikuan

doi:10.1002/cpe.3125

Title: Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks.

Journal Article · Thu May 01 00:00:00 EDT 2014 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.3125· OSTI ID:1150872

Liu, Qing Gary ^[1]; Logan, Jeremy S. ^[1]; Tian, Yuan ^[1]; Abbasi, Hasan ^[1];

^[1];

^[1]; Tchoua, Roselyne B. ^[1]; Lofstead, J. ^[2]; Oldfield, Ron A. ^[3]; Parashar, Manish ^[1]; Samatova, Nagiza ^[1]; Schwan, Karsten ^[2]; Shoshani, A. ^[4];

^[1]; Wu, K. ^[4]; Yu, Weikuan ^[1]

ORNL
Georgia Institute of Technology, Atlanta
Sandia National Laboratories (SNL)
Lawrence Berkeley National Laboratory (LBNL)

Cite

Export

Save

Research Organization:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Organization:: USDOE Office of Science (SC)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1150872

Journal Information:: Concurrency and Computation. Practice and Experience, Vol. 26, Issue 7; ISSN 1532-0626

Country of Publication:: United States

Language:: English

References (36)

A non-staggered, conservative, , finite-volume scheme for 3D implicit extended magnetohydrodynamics in curvilinear geometries Chacón, L. Computer Physics Communications, Vol. 163, Issue 3 https://doi.org/10.1016/j.cpc.2004.08.005	journal	November 2004
Ueber die stetige Abbildung einer Line auf ein Fl�chenst�ck Hilbert, David Mathematische Annalen, Vol. 38, Issue 3 https://doi.org/10.1007/BF01199431	journal	September 1891
Terascale direct numerical simulations of turbulent combustion using S3D Chen, J. H.; Choudhary, A.; de Supinski, B. Computational Science & Discovery, Vol. 2, Issue 1 https://doi.org/10.1088/1749-4699/2/1/015001	journal	January 2009
Gyro-kinetic simulation of global turbulent transport properties in tokamak experiments Wang, W. X.; Lin, Z.; Tang, W. M. Physics of Plasmas, Vol. 13, Issue 9 https://doi.org/10.1063/1.2338775	journal	September 2006
Analysis of the clustering properties of the Hilbert space-filling curve Moon, B.; Jagadish, H. V.; Faloutsos, C. IEEE Transactions on Knowledge and Data Engineering, Vol. 13, Issue 1 https://doi.org/10.1109/69.908985	journal	January 2001
Grid -Based Parallel Data Streaming implemented for the Gyrokinetic Toroidal Code Klasky, S.; Ethier, S.; Lin, Z. Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050175	conference	January 2003
Scalable Earthquake Simulation on Petascale Supercomputers Cui, Yifeng; Olsen, Kim B.; Jordan, Thomas H. 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.45	conference	November 2010
Direct numerical simulation of flame stabilization downstream of a transverse fuel jet in cross-flow Grout, R. W.; Gruber, A.; Yoo, C. S. Proceedings of the Combustion Institute, Vol. 33, Issue 1 https://doi.org/10.1016/j.proci.2010.06.013	journal	January 2011
Disk-directed I/O for MIMD multiprocessors Kotz, David ACM Transactions on Computer Systems, Vol. 15, Issue 1 https://doi.org/10.1145/244764.244766	journal	February 1997
Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS) Lofstead, Jay F.; Klasky, Scott; Schwan, Karsten Proceedings of the 6th international workshop on Challenges of large applications in distributed environments - CLADE '08 https://doi.org/10.1145/1383529.1383533	conference	January 2008
On implementing MPI-IO portably and with high performance Thakur, Rajeev; Gropp, William; Lusk, Ewing Proceedings of the sixth workshop on I/O in parallel and distributed systems - IOPADS '99 https://doi.org/10.1145/301816.301826	conference	January 1999
Parallel netCDF: A High-Performance Scientific I/O Interface Li, Jianwei; Zingale, Michael; Liao, Wei-keng Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03 https://doi.org/10.1145/1048935.1050189	conference	January 2003
PLFS: a checkpoint filesystem for parallel applications Bent, John; Gibson, Garth; Grider, Gary https://doi.org/10.1145/1654059.1654081	conference	January 2009
A Plugin for HDF5 Using PLFS for Improved I/O Performance and Semantic Analysis Mehta, Kshitij; Bent, John; Torres, Aaron https://doi.org/10.1109/SC.Companion.2012.102	conference	November 2012
Lightweight I/O for Scientific Applications Oldfield, Ron; Ward, Lee; Riesen, Rolf 2006 IEEE International Conference on Cluster Computing https://doi.org/10.1109/CLUSTR.2006.311853	conference	September 2006
Collaborative monitoring and visualization of HPC data Tchoua, Roselyne; Abbasi, Hasan; Klasky, Scott 2012 International Conference on Collaboration Technologies and Systems (CTS) https://doi.org/10.1109/CTS.2012.6261083	conference	May 2012
Six degrees of scientific data: reading patterns for extreme scale science IO Lofstead, Jay; Polte, Milo; Gibson, Garth Proceedings of the 20th international symposium on High performance distributed computing - HPDC '11 https://doi.org/10.1145/1996130.1996139	conference	January 2011
Extending scalability of collective IO through nessie and staging Lofstead, Jay; Oldfield, Ron; Kordenbrock, Todd Proceedings of the sixth workshop on Parallel Data Storage - PDSW '11 https://doi.org/10.1145/2159352.2159355	conference	January 2011
The Gemini System Interconnect Alverson, Robert; Roweth, Duncan; Kaplan, Larry 2010 IEEE 18th Annual Symposium on High-Performance Interconnects (HOTI), 2010 18th IEEE Symposium on High Performance Interconnects https://doi.org/10.1109/HOTI.2010.23	conference	August 2010
Adaptable, metadata rich IO methods for portable high performance IO Lofstead, Jay; Zheng, Fang; Klasky, Scott Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing https://doi.org/10.1109/IPDPS.2009.5161052	conference	May 2009
Data organization and I/O in a parallel ocean circulation model Ding, Chris H. Q.; He, Yun Proceedings of the 1999 ACM/IEEE conference on Supercomputing (CDROM) - Supercomputing '99 https://doi.org/10.1145/331532.331565	conference	January 1999
Investigation of leading HPC I/O performance using a scientific-application derived benchmark Borrill, Julian; Oliker, Leonid; Shalf, John Proceedings of the 2007 ACM/IEEE conference on Supercomputing - SC '07 https://doi.org/10.1145/1362622.1362636	conference	January 2007
DataSpaces: an interaction and coordination framework for coupled simulation workflows Docan, Ciprian; Parashar, Manish; Klasky, Scott Proceedings of the 19th ACM International Symposium on High Performance Distributed Computing - HPDC '10 https://doi.org/10.1145/1851476.1851481	conference	January 2010
DataStager: scalable data staging services for petascale applications Abbasi, Hasan; Wolf, Matthew; Eisenhauer, Greg Proceedings of the 18th ACM international symposium on High performance distributed computing - HPDC '09 https://doi.org/10.1145/1551609.1551618	conference	January 2009
Extending I/O through high performance data services Abbasi, Hasan; Lofstead, Jay; Zheng, Fang 2009 IEEE International Conference on Cluster Computing and Workshops https://doi.org/10.1109/CLUSTR.2009.5289167	conference	August 2009
Toward simulation-time data analysis and I/O acceleration on leadership-class systems Vishwanath, Venkatram; Hereld, Mark; Papka, Michael E. 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092178	conference	October 2011
Enhancing Checkpoint Performance with Staging IO and SSD Ouyang, Xiangyong; Marcarelli, Sonya; Panda, Dhabaleswar K. 2010 International Workshop on Storage Network Architecture and Parallel I/Os (SNAPI) https://doi.org/10.1109/SNAPI.2010.10	conference	May 2010
Using active NVRAM for I/O staging Kannan, Sudarsun; Gavrilovska, Ada; Schwan, Karsten Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11 https://doi.org/10.1145/2110205.2110209	conference	January 2011
On the role of burst buffers in leadership-class storage systems Liu, Ning; Cope, Jason; Carns, Philip 2012 IEEE 28th Symposium on Mass Storage Systems and Technologies (MSST) https://doi.org/10.1109/MSST.2012.6232369	conference	April 2012
The NAS parallel benchmarks---summary and preliminary results Bailey, D. H.; Schreiber, R. S.; Simon, H. D. Proceedings of the 1991 ACM/IEEE conference on Supercomputing - Supercomputing '91 https://doi.org/10.1145/125826.125925	conference	January 1991
Skel: Generative Software for Producing Skeletal I/O Applications Logan, Jeremy; Klasky, Scott; Lofstead, Jay 2011 IEEE Seventh International Conference on e-Science Workshops (eScienceW) https://doi.org/10.1109/eScienceW.2011.26	conference	December 2011
ISOBAR hybrid compression-I/O interleaving for large-scale parallel I/O optimization Schendel, Eric R.; Klasky, Scott; Ross, Robert Proceedings of the 21st international symposium on High-Performance Parallel and Distributed Computing - HPDC '12 https://doi.org/10.1145/2287076.2287086	conference	January 2012
ISOBAR Preconditioner for Effective and High-throughput Lossless Data Compression Schendel, Eric R.; Jin, Ye; Shah, Neil 2012 IEEE International Conference on Data Engineering (ICDE 2012), 2012 IEEE 28th International Conference on Data Engineering https://doi.org/10.1109/ICDE.2012.114	conference	April 2012
Just in time: adding value to the IO pipelines of high performance applications with JITStaging Abbasi, Hasan; Eisenhauer, Greg; Wolf, Matthew Proceedings of the 20th international symposium on High performance distributed computing - HPDC '11 https://doi.org/10.1145/1996130.1996137	conference	January 2011
FastBit: interactively searching massive data Wu, K.; Ahern, S.; Bethel, E. W. Journal of Physics: Conference Series, Vol. 180 https://doi.org/10.1088/1742-6596/180/1/012053	journal	July 2009
Parallel in situ indexing for data-intensive computing Kim, Jinoh; Abbasi, Hasan; Chacon, Luis 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092319	conference	October 2011

Similar Records

Hello ADIOS: The Challenges and Lessons of Developing Leadership Class I/O Frameworks.

Journal Article · Mon Oct 01 00:00:00 EDT 2012 · Proposed for publication in Concurrency and Computation: Practice and Experience. · OSTI ID:1150872

Lofstead, Gerald Fredrick; Liu, Qing; Logan, Jeremy; +14 more

Comparative I/O Workload Characterization of Two Leadership Class Storage Clusters

Conference · Thu Jan 01 00:00:00 EST 2015 · OSTI ID:1150872

Gunasekaran, Raghul; Oral, H Sarp; Hill, Jason J; +3 more

Enabling Event Tracing at Leadership-Class Scale through I/O Forwarding Middleware

Conference · Sun Jan 01 00:00:00 EST 2012 · OSTI ID:1150872

Ilsche, Thomas; Schuchart, Joseph; Cope, Joseph; +7 more

Title: Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks.

Citation Formats

References (36)

Similar Records

Related Subjects