skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery

Abstract

The primary challenge motivating this project is the widening gap between the ability to compute information and to store it for subsequent analysis. This gap adversely impacts science code teams, who can perform analysis only on a small fraction of the data they calculate, resulting in the substantial likelihood of lost or missed science, when results are computed but not analyzed. Our approach is to perform as much analysis or visualization processing on data while it is still resident in memory, which is known as in situ processing. The idea in situ processing was not new at the time of the start of this effort in 2014, but efforts in that space were largely ad hoc, and there was no concerted effort within the research community that aimed to foster production-quality software tools suitable for use by Department of Energy (DOE) science projects. Our objective was to produce and enable the use of production-quality in situ methods and infrastructure, at scale, on DOE high-performance computing (HPC) facilities, though we expected to have an impact beyond DOE due to the widespread nature of the challenges, which affect virtually all large-scale computational science efforts. To achieve this objective, we engaged in softwaremore » technology research and development (R&D), in close partnerships with DOE science code teams, to produce software technologies that were shown to run efficiently at scale on DOE HPC platforms.« less

Authors:
 [1]
  1. Kitware, Inc., Clifton Park, NY (United States)
Publication Date:
Research Org.:
Kitware, Inc., Clifton Park, NY (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1389857
Report Number(s):
DOE-KITWARE-12387
DOE Contract Number:
SC0012387
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
96 KNOWLEDGE MANAGEMENT AND PRESERVATION

Citation Formats

O'Leary, Patrick. Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery. United States: N. p., 2017. Web. doi:10.2172/1389857.
O'Leary, Patrick. Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery. United States. doi:10.2172/1389857.
O'Leary, Patrick. Wed . "Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery". United States. doi:10.2172/1389857. https://www.osti.gov/servlets/purl/1389857.
@article{osti_1389857,
title = {Final Report Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery},
author = {O'Leary, Patrick},
abstractNote = {The primary challenge motivating this project is the widening gap between the ability to compute information and to store it for subsequent analysis. This gap adversely impacts science code teams, who can perform analysis only on a small fraction of the data they calculate, resulting in the substantial likelihood of lost or missed science, when results are computed but not analyzed. Our approach is to perform as much analysis or visualization processing on data while it is still resident in memory, which is known as in situ processing. The idea in situ processing was not new at the time of the start of this effort in 2014, but efforts in that space were largely ad hoc, and there was no concerted effort within the research community that aimed to foster production-quality software tools suitable for use by Department of Energy (DOE) science projects. Our objective was to produce and enable the use of production-quality in situ methods and infrastructure, at scale, on DOE high-performance computing (HPC) facilities, though we expected to have an impact beyond DOE due to the widespread nature of the challenges, which affect virtually all large-scale computational science efforts. To achieve this objective, we engaged in software technology research and development (R&D), in close partnerships with DOE science code teams, to produce software technologies that were shown to run efficiently at scale on DOE HPC platforms.},
doi = {10.2172/1389857},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Wed Sep 13 00:00:00 EDT 2017},
month = {Wed Sep 13 00:00:00 EDT 2017}
}

Technical Report:

Save / Share: