High-Performance I/O: HDF5 for Lattice QCD

Journal Article · 09 May 2017 · PoS Proceedings of Science

OSTI ID:1398478

Kurth, Thorsten ^[1]; Pochinsky, Andrew ^[2]; Sarje, Abhinav ^[3]; Syritsyn, Sergey ^[4]; Walker-Loud, Andre ^[5]

---

1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Nuclear Science Division
2. Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States). Center for Theoretical Physics
3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States). Computational Research Division
4. Brookhaven National Lab. (BNL), Upton, NY (United States). RIKEN Research Center
5. College of William and Mary, Williamsburg, VA (United States). Dept. of Physics; Thomas Jefferson National Accelerator Facility (TJNAF), Newport News, VA (United States)

Practitioners of lattice QCD/QFT have been some of the primary pioneer users of the state-of-the-art high-performance-computing systems, and contribute towards the stress tests of such new machines as soon as they become available. As with all aspects of high-performance-computing, I/O is becoming an increasingly specialized component of these systems. In order to take advantage of the latest available high-performance I/O infrastructure, to ensure reliability and backwards compatibility of data files, and to help unify the data structures used in lattice codes, we have incorporated parallel HDF5 I/O into the SciDAC supported USQCD software stack. Here we present the design and implementation of this I/O framework. Our HDF5 implementation outperforms optimized QIO at the 10-20% level and leaves room for further improvement by utilizing appropriate dataset chunking.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• Grant/Contract Number: AC02-05CH11231; FG02-94ER40818
• OSTI ID: 1398478
• Journal Information: PoS Proceedings of Science, Journal Name: PoS Proceedings of Science; ISSN 1824-8039
• Publisher: SISSACopyright Statement
• Country of Publication: United States
• Language: English