Delta: Data Reduction for Integrated Application Workflows
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States); Florida International Univ. (FIU), Miami, FL (United States)
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Integrated Application Workflows (IAWs) run multiple simulation workflow components concurrently on an HPC resource connecting these components using compute area resources and compensating for any performance or data processing rate mismatches. These IAWs require high frequency and high volume data transfers between compute nodes and staging area nodes during the lifetime of a large parallel computation. The available network band-width between the two areas may not be enough to efficiently support the data movement. As the processing power available to compute resources increases, the requirements for this data transfer will become more difficult to satisfy and perhaps will not be satisfiable at all since network capabilities are not expanding at a comparable rate. Furthermore, energy consumption in HPC environments is expected to grow by an order of magnitude as exascale systems become a reality. The energy cost of moving large amounts of data frequently will contribute to this issue. It is necessary to reduce the volume of data without reducing the quality of data when it is being processed and analyzed. Delta resolves the issue by addressing the lifetime data transfer operations. Delta removes subsequent identical copies of already transmitted data during transfers and restores those copies once the data has reached the destination. Delta is able to identify duplicated information and determine the most space efficient way to represent it. Initial tests show about 50% reduction in data movement while maintaining the same data quality and transmission frequency.
- Research Organization:
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sponsoring Organization:
- USDOE National Nuclear Security Administration (NNSA), Office of Defense Nuclear Security
- DOE Contract Number:
- AC04-94AL85000
- OSTI ID:
- 1193147
- Report Number(s):
- SAND-2015-5029; 594733
- Country of Publication:
- United States
- Language:
- English
Similar Records
SDN-NGenIA, a Software Defined Next Generation Integrated Architecture for HEP and Data Intensive Science
Autonomic Management of Application Workflows on Hybrid Computing Infrastructure