skip to main content

DOE PAGESDOE PAGES

Title: Performance Analysis, Design Considerations, and Applications of Extreme-Scale In Situ Infrastructures

A key trend facing extreme-scale computational science is the widening gap between computational and I/O rates, and the challenge that follows is how to best gain insight from simulation data when it is increasingly impractical to save it to persistent storage for subsequent visual exploration and analysis. One approach to this challenge is centered around the idea of in situ processing, where visualization and analysis processing is performed while data is still resident in memory. Our paper examines several key design and performance issues related to the idea of in situ processing at extreme scale on modern platforms: Scalability, overhead, performance measurement and analysis, comparison and contrast with a traditional post hoc approach, and interfacing with simulation codes. We illustrate these principles in practice with studies, conducted on large-scale HPC platforms, that include a miniapplication and multiple science application codes, one of which demonstrates in situ methods in use at greater than 1M-way concurrency.
Authors:
 [1] ;  [1] ;  [2] ;  [3] ;  [4] ;  [5] ;  [6] ;  [5] ;  [5] ;  [6] ;  [5] ;  [1] ;  [6] ;  [6] ;  [7] ;  [4] ;  [5] ;  [2] ;  [3] ;  [5] more »;  [5] « less
  1. Kitware Inc., Clifton Park, NY (United States)
  2. Intelligent Light, Rutherford, NJ (United States)
  3. Georgia Inst. of Technology, Atlanta, GA (United States)
  4. Argonne National Lab. (ANL), Argonne, IL (United States)
  5. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  6. Univ. of Colorado, Boulder, CO (United States)
  7. Computational Science and Engineering, LLC, Chicago, IL (United States)
Publication Date:
Grant/Contract Number:
AC02-05CH11231
Type:
Accepted Manuscript
Journal Name:
International Conference for High Performance Computing, Networking, Storage and Analysis (Online)
Additional Journal Information:
Journal Name: International Conference for High Performance Computing, Networking, Storage and Analysis (Online); Journal Volume: 2017; Conference: International Conference for High Performance Computing, Networking, Storage and Analysis, Salt Lake City, UT (United States), 13-18 Nov 2016; Journal ID: ISSN 2167-4337
Publisher:
IEEE
Research Org:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING
OSTI Identifier:
1379776