Enabling distributed petascale science.
Abstract
Petascale science is an end-to-end endeavour, involving not only the creation of massive datasets at supercomputers or experimental facilities, but the subsequent analysis of that data by a user community that may be distributed across many laboratories and universities. The new SciDAC Center for Enabling Distributed Petascale Science (CEDPS) is developing tools to support this end-to-end process. These tools include data placement services for the reliable, high-performance, secure, and policy-driven placement of data within a distributed science environment; tools and techniques for the construction, operation, and provisioning of scalable science services; and tools for the detection and diagnosis of failures in end-to-end data placement and distributed application hosting configurations. In each area, we build on a strong base of existing technology and have made useful progress in the first year of the project. For example, we have recently achieved order-of-magnitude improvements in transfer times (for lots of small files) and implemented asynchronous data staging capabilities; demonstrated dynamic deployment of complex application stacks for the STAR experiment; and designed and deployed end-to-end troubleshooting services. We look forward to working with SciDAC application and technology projects to realize the promise of petascale science.
- Authors:
- more »
- Publication Date:
- Research Org.:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC)
- OSTI Identifier:
- 973002
- Report Number(s):
- ANL/MCS/CP-59797
TRN: US201005%%399
- DOE Contract Number:
- DE-AC02-06CH11357
- Resource Type:
- Conference
- Journal Name:
- J. Phys.: Conf. Ser.
- Additional Journal Information:
- Journal Volume: 78; Journal Issue: 2007; Conference: Scientific Discovery through Advanced Computing (SciDAC 2007); Jun. 24, 2007 - Jun. 26, 2007; Boston, MA
- Country of Publication:
- United States
- Language:
- ENGLISH
- Subject:
- 97 MATHEMATICAL METHODS AND COMPUTING; 99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; SUPERCOMPUTERS; SIZE; DATA ANALYSIS; PARALLEL PROCESSING; DATA TRANSMISSION; PERFORMANCE
Citation Formats
Baranovski, A, Bharathi, S, Bresnahan, J, chervenak, A, Foster, I, Fraser, D, Freeman, T, Gunter, D, Jackson, K, Keahey, K, Kesselman, C, Konerding, D E, Leroy, N, Link, M, Livny, M, Miller, N, Miller, R, Oleynik, G, Pearlman, L, Schopf, J M, Schuler, R, Tierney, B, Mathematics and Computer Science, FNL,, Univ. of Southern California, Univ. of Chicago, LBNL,, and Univ. of Wisconsin. Enabling distributed petascale science.. United States: N. p., 2007.
Web. doi:10.1088/1742-6596/78/1/012020.
Baranovski, A, Bharathi, S, Bresnahan, J, chervenak, A, Foster, I, Fraser, D, Freeman, T, Gunter, D, Jackson, K, Keahey, K, Kesselman, C, Konerding, D E, Leroy, N, Link, M, Livny, M, Miller, N, Miller, R, Oleynik, G, Pearlman, L, Schopf, J M, Schuler, R, Tierney, B, Mathematics and Computer Science, FNL,, Univ. of Southern California, Univ. of Chicago, LBNL,, & Univ. of Wisconsin. Enabling distributed petascale science.. United States. https://doi.org/10.1088/1742-6596/78/1/012020
Baranovski, A, Bharathi, S, Bresnahan, J, chervenak, A, Foster, I, Fraser, D, Freeman, T, Gunter, D, Jackson, K, Keahey, K, Kesselman, C, Konerding, D E, Leroy, N, Link, M, Livny, M, Miller, N, Miller, R, Oleynik, G, Pearlman, L, Schopf, J M, Schuler, R, Tierney, B, Mathematics and Computer Science, FNL,, Univ. of Southern California, Univ. of Chicago, LBNL,, and Univ. of Wisconsin. 2007.
"Enabling distributed petascale science.". United States. https://doi.org/10.1088/1742-6596/78/1/012020.
@article{osti_973002,
title = {Enabling distributed petascale science.},
author = {Baranovski, A and Bharathi, S and Bresnahan, J and chervenak, A and Foster, I and Fraser, D and Freeman, T and Gunter, D and Jackson, K and Keahey, K and Kesselman, C and Konerding, D E and Leroy, N and Link, M and Livny, M and Miller, N and Miller, R and Oleynik, G and Pearlman, L and Schopf, J M and Schuler, R and Tierney, B and Mathematics and Computer Science and FNL, and Univ. of Southern California and Univ. of Chicago and LBNL, and Univ. of Wisconsin},
abstractNote = {Petascale science is an end-to-end endeavour, involving not only the creation of massive datasets at supercomputers or experimental facilities, but the subsequent analysis of that data by a user community that may be distributed across many laboratories and universities. The new SciDAC Center for Enabling Distributed Petascale Science (CEDPS) is developing tools to support this end-to-end process. These tools include data placement services for the reliable, high-performance, secure, and policy-driven placement of data within a distributed science environment; tools and techniques for the construction, operation, and provisioning of scalable science services; and tools for the detection and diagnosis of failures in end-to-end data placement and distributed application hosting configurations. In each area, we build on a strong base of existing technology and have made useful progress in the first year of the project. For example, we have recently achieved order-of-magnitude improvements in transfer times (for lots of small files) and implemented asynchronous data staging capabilities; demonstrated dynamic deployment of complex application stacks for the STAR experiment; and designed and deployed end-to-end troubleshooting services. We look forward to working with SciDAC application and technology projects to realize the promise of petascale science.},
doi = {10.1088/1742-6596/78/1/012020},
url = {https://www.osti.gov/biblio/973002},
journal = {J. Phys.: Conf. Ser.},
number = 2007,
volume = 78,
place = {United States},
year = {Mon Jan 01 00:00:00 EST 2007},
month = {Mon Jan 01 00:00:00 EST 2007}
}