Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers

Kress, James; Churchill, Randy Michael; Klasky, Scott; Kim, Mark; Childs, Hank; Pugmire, David

doi:10.14529/jsfi160404

Title: Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers

Full Record
Other Related Research

Abstract

High performance computing applications are producing increasingly large amounts of data and placing enormous stress on current capabilities for traditional post-hoc visualization techniques. Because of the growing compute and I/O imbalance, data reductions, including in situ visualization, are required. These reduced data are used for analysis and visualization in a variety of different ways. Many of he visualization and analysis requirements are known a priori, but when they are not, scientists are dependent on the reduced data to accurately represent the simulation in post hoc analysis. The contributions of this paper is a description of the directions we are pursuing to assist a large scale fusion simulation code succeed on the next generation of supercomputers. Finally, these directions include the role of in situ processing for performing data reductions, as well as the tradeoffs between data size and data integrity within the context of complex operations in a typical scientific workflow.

Authors:

Kress, James ^[1]; Churchill, Randy Michael ^[2]; Klasky, Scott ^[3]; Kim, Mark ^[3]; Childs, Hank ^[4]; Pugmire, David ^[3]

Univ. of Oregon, Eugene, OR (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Univ. of Oregon, Eugene, OR (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Publication Date:: Thu Dec 01 00:00:00 EST 2016

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Contributing Org.:: Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States); Univ. of Oregon, Eugene, OR (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

OSTI Identifier:: 1338570

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Supercomputing frontiers and innovations

Additional Journal Information:: Journal Volume: 3; Journal Issue: 4; Related Information: Oct - Dec 2016 issue; Journal ID: ISSN 2313-8734

Publisher:: South Ural State University

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; scientific visualization; in situ methods; data staging methods; data reductions; high performance computing

Citation Formats


                    Kress, James, Churchill, Randy Michael, Klasky, Scott, Kim, Mark, Childs, Hank, and Pugmire, David. Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers.  United States: N. p., 2016. 
Web.  doi:10.14529/jsfi160404.

Copy to clipboard


                    Kress, James, Churchill, Randy Michael, Klasky, Scott, Kim, Mark, Childs, Hank, & Pugmire, David. Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers.  United States.  https://doi.org/10.14529/jsfi160404

Copy to clipboard


                    Kress, James, Churchill, Randy Michael, Klasky, Scott, Kim, Mark, Childs, Hank, and Pugmire, David. Thu .  
"Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers".  United States.  https://doi.org/10.14529/jsfi160404.  https://www.osti.gov/servlets/purl/1338570.

Copy to clipboard


                    
@article{osti_1338570,

  title        = {Preparing for In Situ Processing on Upcoming Leading-edge Supercomputers},

  author       = {Kress, James and Churchill, Randy Michael and Klasky, Scott and Kim, Mark and Childs, Hank and Pugmire, David},

  abstractNote = {High performance computing applications are producing increasingly large amounts of data and placing enormous stress on current capabilities for traditional post-hoc visualization techniques. Because of the growing compute and I/O imbalance, data reductions, including in situ visualization, are required. These reduced data are used for analysis and visualization in a variety of different ways. Many of he visualization and analysis requirements are known a priori, but when they are not, scientists are dependent on the reduced data to accurately represent the simulation in post hoc analysis. The contributions of this paper is a description of the directions we are pursuing to assist a large scale fusion simulation code succeed on the next generation of supercomputers. Finally, these directions include the role of in situ processing for performing data reductions, as well as the tradeoffs between data size and data integrity within the context of complex operations in a typical scientific workflow.},

  doi          = {10.14529/jsfi160404},

  journal      = {Supercomputing frontiers and innovations},

  number       = 4,

  volume       = 3,

  place        = {United States},

  year         = {Thu Dec 01 00:00:00 EST 2016},

  month        = {Thu Dec 01 00:00:00 EST 2016}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.14529/jsfi160404

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Demonstrating the viability of Lagrangian in situ reduction on supercomputers

Journal Article Sane, Sudhanshu ; Johnson, Chris R. ; Childs, Hank - Journal of Computational Science

Performing exploratory analysis and visualization of large-scale time-varying computational science applications is challenging due to inaccuracies that arise from under-resolved data. In recent years, Lagrangian representations of the vector field computed using in situ processing are being increasingly researched and have emerged as a potential solution to enable exploration. However, prior works have offered limited estimates of the encumbrance on the simulation code as they consider “theoretical” in situ environments. Further, the effectiveness of this approach varies based on the nature of the vector field, benefitting from an in-depth investigation for each application area. With this study, an extended versionmore »« less
https://doi.org/10.1016/j.jocs.2022.101615

Full Text Available
In Situ Visualization and Production of Extract Databases

Journal Article Whitlock, Brad J. ; Duque, Earl P.N. - Supercomputing frontiers and innovations

Simulations running at high concurrency on HPC systems generate large volumes of data that are impractical to write to disk due to time and storage constraints. Applications often adapt by saving data infrequently, resulting in datasets with poor temporal resolution. This can make datasets difficult to interpret during post hoc visualization and analysis, or worse, it can lead to lost science. In Situ visualization and analysis can enable efficient production of small data products such as rendered images or surface extracts that consist of polygonal geometry plus fields. These data products are far smaller than their source data and canmore »« less
https://doi.org/10.14529/jsfi160402

Full Text Available
Sim-Situ: A Framework for the Faithful Simulation of in situ Processing

Conference Honoré, Valentin ; Mai anh do, Tu ; Pottier, Loic ; ...

The amount of data generated by numerical simulations in various scientific domains led to a fundamental redesign of how the analysis and visualization of simulation outputs are performed. The throughput and capacity of storage subsystems have not evolved as fast as the computing power in extreme-scale supercomputers, making the classical post-hoc approach highly inefficient. In situ processing has then emerged as a solution in which simulation and data analysis/visualization are intertwined for better performance and greater interactivity.Determining the best allocation, i.e., how many resources to allocate to simulation and analysis respectively, mapping, i.e., where and at which frequency to runmore »« less
https://doi.org/10.1109/eScience55777.2022.00032

Full Text Available
Towards a Scalable and Adaptive Application Support Platform for Large-Scale Distributed E-Sciences in High-Performance Network Environments

Technical Report Wu, Chase Qishi ; Zhu, Michelle Mengxia

The advent of large-scale collaborative scientific applications has demonstrated the potential for broad scientific communities to pool globally distributed resources to produce unprecedented data acquisition, movement, and analysis. System resources including supercomputers, data repositories, computing facilities, network infrastructures, storage systems, and display devices have been increasingly deployed at national laboratories and academic institutes. These resources are typically shared by large communities of users over Internet or dedicated networks and hence exhibit an inherent dynamic nature in their availability, accessibility, capacity, and stability. Scientific applications using either experimental facilities or computation-based simulations with various physical, chemical, climatic, and biological models featuremore »« less
https://doi.org/10.2172/1255487

Full Text Available
Visualization on supercomputing platform level II ASC milestone (3537-1B) results from Sandia.

Technical Report Geveci, Berk ; Fabian, Nathan ; Marion, Patrick ; ...

This report provides documentation for the completion of the Sandia portion of the ASC Level II Visualization on the platform milestone. This ASC Level II milestone is a joint milestone between Sandia National Laboratories and Los Alamos National Laboratories. This milestone contains functionality required for performing visualization directly on a supercomputing platform, which is necessary for peta-scale visualization. Sandia's contribution concerns in-situ visualization, running a visualization in tandem with a solver. Visualization and analysis of petascale data is limited by several factors which must be addressed as ACES delivers the Cielo platform. Two primary difficulties are: (1) Performance of interactivemore »« less
https://doi.org/10.2172/992321

Full Text Available

Similar Records