Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
The primary challenge motivating this team’s work is the widening gap between the ability to compute information and to store it for subsequent analysis. This gap adversely impacts science code teams, who are able to perform analysis only on a small fraction of the data they compute, resulting in the very real likelihood of lost or missed science, when results are computed but not analyzed. Our approach is to perform as much analysis or visualization processing on data while it is still resident in memory, an approach that is known as in situ processing. The idea in situ processing was not new at the time of the start of this effort in 2014, but efforts in that space were largely ad hoc, and there was no concerted effort within the research community that aimed to foster production-quality software tools suitable for use by DOE science projects. In large, our objective was produce and enable use of production-quality in situ methods and infrastructure, at scale, on DOE HPC facilities, though we expected to have impact beyond DOE due to the widespread nature of the challenges, which affect virtually all large-scale computational science efforts. To achieve that objective, we assembled a unique team of researchers consisting of representatives from DOE national laboratories, academia, and industry, and engaged in software technology R&D, as well as engaged in close partnerships with DOE science code teams, to produce software technologies that were shown to run effectively at scale on DOE HPC platforms.
- Research Organization:
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- DOE Contract Number:
- SC0012561; AC02-05CH11231
- OSTI ID:
- 1421430
- Report Number(s):
- DE-SC0012561
- Country of Publication:
- United States
- Language:
- English
Similar Records
Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures
Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures