A Methodology for the Rapid Development of Scalable HPC Data Services
There is growing evidence in the scientifc computing community that parallel file systems are not sufficient for all HPC storage workloads. This realization has motivated extensive research in new storage system designs. The question of which alternative we should turn towards implies that there could be a single answer satisfying a wide range of very diverse applications. We argue that such a generic solution does not exist. Instead, custom data services should be designed and tailored to the needs of specific applications on specific hardware. Furthermore, custom data services should be designed in close collaboration with users. In this paper, we present a methodology for the rapid development of such data services. This methodology promotes the design of reusable building blocks that can be composed together efficiently through a runtime based on high-performance threading, tasking, and remote procedure calls. We illustrate the success of our methodology by showcasing three examples of data services designed from the same building blocks, yet targeting entirely different applications.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States); Fermi National Accelerator Lab. (FNAL), Batavia, IL (United States); Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP)
- DOE Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1490839
- Report Number(s):
- FERMILAB-CONF-18-575-CD; 1712343
- Resource Relation:
- Conference: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Dallas, TX, USA, 11/11-11/16/2018
- Country of Publication:
- United States
- Language:
- English
Similar Records
Challenges and Opportunities of User-Level File Systemsfor HPC
Pufferscale: Rescaling HPC Data Services for High Energy Physics Applications