Data Management in the Continuum: Cross-facility Object-based Data Transfers

Conference · Fri Oct 31 00:00:00 EDT 2025

DOI:https://doi.org/10.1109/sbac-pad66369.2025.00015· OSTI ID:3024529

Bez, Jean Luca; Tang, Houjun; Wang, Chen; Byna, Suren

Scientific workflows are evolving from relying on a monolithic storage subsystem at a single High-Performance Computing (HPC) facility to using geographically distributed file systems, repositories, and cloud storage. As a result, storing, accessing, transferring, and managing scientific data have become highly complex and prone to performance inefficiencies. This paper delves into these challenges by exploring an optimized end-to-end interface designed to seamlessly connect various local and remote storage systems, enabling efficient data movement of objects across HPC–Cloud and HPC–HPC environments. We showcase this capability through an object-focused data management runtime system, discuss the effects of relaxed consistency semantics in distributed object scenarios, and illustrate its application in an earthquake simulation workflow. Besides reducing the amount of data by selectively transferring regions of interest, our facility-local results achieved a speedup of 45 × over an optimized HDF5 usage and 15 × over the HDF5 with caching by using the new interface in PDC-XF.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: US Department of Energy; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-ASCR)
• DOE Contract Number: AC02-05CH11231
• OSTI ID: 3024529
• Resource Type: Conference paper
• Conference Information: 2025 IEEE/SBC 37th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)
• Country of Publication: United States
• Language: English