Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery

Technical Report · Wed Sep 13 04:00:00 EDT 2017

DOI:https://doi.org/10.2172/1389857· OSTI ID:1389857

O'Leary, Patrick ^[1]

---

1. Kitware, Inc., Clifton Park, NY (United States); Kitware, Inc.

The primary challenge motivating this project is the widening gap between the ability to compute information and to store it for subsequent analysis. This gap adversely impacts science code teams, who can perform analysis only on a small fraction of the data they calculate, resulting in the substantial likelihood of lost or missed science, when results are computed but not analyzed. Our approach is to perform as much analysis or visualization processing on data while it is still resident in memory, which is known as in situ processing. The idea in situ processing was not new at the time of the start of this effort in 2014, but efforts in that space were largely ad hoc, and there was no concerted effort within the research community that aimed to foster production-quality software tools suitable for use by Department of Energy (DOE) science projects. Our objective was to produce and enable the use of production-quality in situ methods and infrastructure, at scale, on DOE high-performance computing (HPC) facilities, though we expected to have an impact beyond DOE due to the widespread nature of the challenges, which affect virtually all large-scale computational science efforts. To achieve this objective, we engaged in software technology research and development (R&D), in close partnerships with DOE science code teams, to produce software technologies that were shown to run efficiently at scale on DOE HPC platforms.

• Research Organization: Kitware, Inc., Clifton Park, NY (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• DOE Contract Number: SC0012387
• OSTI ID: 1389857
• Report Number(s): DOE-KITWARE--12387
• Country of Publication: United States
• Language: English

Related Subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION