Data transfer for STAR grid jobs
- Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
- Brookhaven National Laboratory (BNL), Upton, NY (United States)
The Solenoidal Tracker at RHIC (STAR) is a multipurpose experiment at the Relativistic Heavy Ion Collider (RHIC) with the primary goal to study the formation and properties of the quark-gluon plasma. STAR is an international collaboration of member institutions and laboratories from around the world. Yearly data-taking period produces PBytes of raw data collected by the experiment. STAR primarily uses its dedicated facility at BNL to process this data, but has routinely leveraged distributed systems, both high throughput (HTC) and high performance (HPC) computing clusters, to significantly augment the processing capacity available to the experiment. The ability to automate the efficient transfer of large data sets on reliable, scalable, and secure infrastructure is critical for any large-scale distributed processing campaign. For more than a decade, STAR computing has relied upon GridFTP with its x509-based authentication to build such data transfer systems and integrate them into its larger production workflow. The end of support by the community for both GridFTP and the x509 standard requires STAR to investigate other approaches to meet its distributed processing needs. In this study we investigate two multi-purpose data distribution systems, Globus.org and XRootD, as alternatives to GridFTP. We compare both their performance and the ease by which each service is integrated into the type of secure and automated data transfer systems STAR has previously built using GridFTP. The presented approach and study may be applicable to other distributed data processing use cases beyond STAR.
- Research Organization:
- Brookhaven National Laboratory (BNL), Upton, NY (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Nuclear Physics (NP)
- Grant/Contract Number:
- SC0012704
- OSTI ID:
- 1996622
- Report Number(s):
- BNL-224736-2023-JAAM
- Journal Information:
- Journal of Physics. Conference Series, Journal Name: Journal of Physics. Conference Series Journal Issue: 1 Vol. 2438; ISSN 1742-6588
- Publisher:
- IOP PublishingCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Physics Data Production on HPC: Experience to be efficiently running at scale
|
journal | January 2020 |
STAR Data Production Workflow on HPC: Lessons Learned & Best Practices
|
journal | April 2020 |
STAR Data Reconstruction at NERSC/Cori, an adaptable Docker container approach for HPC
|
journal | October 2017 |
Similar Records
XRootD Third Party Copy for the WLCG and HL- LHC
Security in the CernVM File System and the Frontier Distributed Database Caching System