Skip to main content
U.S. Department of Energy
Office of Scientific and Technical Information

In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms, a State-of-the-art (STAR) Report

Journal Article · · Computer Graphics Forum
DOI:https://doi.org/10.1111/cgf.12930· OSTI ID:1439995
The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed /visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU’s and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.
Research Organization:
Kitware, Inc., Clifton Park, NY (United States); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Organization:
USDOE Office of Nuclear Energy (NE); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
Grant/Contract Number:
AC02-05CH11231; SC0012037
OSTI ID:
1439995
Alternate ID(s):
OSTI ID: 1595010
Journal Information:
Computer Graphics Forum, Journal Name: Computer Graphics Forum Journal Issue: 3 Vol. 35; ISSN 0167-7055
Publisher:
WileyCopyright Statement
Country of Publication:
United States
Language:
English

References (59)

Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS journal August 2013
ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing journal October 2014
DataSpaces: an interaction and coordination framework for coupled simulation workflows journal February 2011
Consuming Network Bandwidth with Visapult book January 2005
Runtime volume visualization for parallel CFD book January 1996
PARAMESH: A parallel adaptive mesh refinement community toolkit journal April 2000
A survey of computational steering environments journal February 1999
A multi-resolution approach to global ocean modeling journal September 2013
Cinema image-based in situ analysis and visualization of MPAS-ocean simulations journal July 2016
ProteinShop: A tool for interactive protein manipulation and steering journal April 2004
FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes journal November 2000
yt: A MULTI-CODE ANALYSIS TOOLKIT FOR ASTROPHYSICAL SIMULATION DATA journal December 2010
Computational steering. Software systems and strategies journal January 1997
Experiences Applying Data Staging Technology in Unconventional Ways
  • Lofstead, J.; Oldfield, R.; Kordenbrock, T.
  • 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing https://doi.org/10.1109/CCGrid.2013.27
conference May 2013
HPCCloud: A Cloud/Web-Based Simulation Environment conference November 2015
Adaptable, metadata rich IO methods for portable high performance IO conference May 2009
PreDatA – preparatory data analytics on peta-scale machines conference April 2010
I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes conference May 2013
Efficient parallel volume rendering of large-scale adaptive mesh refinement data conference October 2013
Damaris/Viz: A nonintrusive, adaptable and user-friendly in situ visualization framework conference October 2013
ADR visualization: A generalized framework for ranking large-scale scientific data using Analysis-Driven Refinement conference November 2014
In-situ multi-resolution and temporal data compression for visual exploration of large-scale scientific simulations conference November 2014
Space-time volumetric depth images for in-situ visualization conference November 2014
Improved post hoc flow analysis via Lagrangian representations conference November 2014
In situ depth maps based feature extraction and tracking conference October 2015
Evaluating the efficacy of wavelet configurations on turbulent-flow data conference October 2015
Research Challenges for Visualization Software journal May 2013
In Situ Visualization at Extreme Scale: Challenges and Opportunities journal November 2009
In Situ Visualization for Large-Scale Combustion Simulations journal May 2010
Visualization at Supercomputing Centers: The Tale of Little Big Iron and the Three Skinny Guys journal January 2011
Increasing Scientific Data Insights about Exascale Class Simulations under Power and Storage Constraints journal March 2015
The Tensions of In Situ Visualization journal March 2016
WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations journal May 2016
Scalable Implicit Flow Solver for Realistic Wing Simulations with Flow Control journal November 2014
From Mesh Generation to Scientific Visualization: An End-to-End Approach to Parallel Supercomputing conference November 2006
An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis
  • Ahrens, James; Jourdain, Sebastien; OLeary, Patrick
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.40
conference November 2014
Oh, $#*@! Exascale! The Effect of Emerging Architectures on Scientific Discovery
  • Moreland, Kenneth
  • 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis https://doi.org/10.1109/SC.Companion.2012.38
conference November 2012
Concurrent Visualization in a Production Supercomputing Environment journal September 2006
Interactive, Internet Delivery of Visualization via Structured Prerendered Multiresolution Imagery journal March 2008
Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data journal November 2010
Interactive Exploration and Analysis of Large-Scale Simulations Using Topology-Based Data Segmentation journal September 2011
Parallel Computational Steering for HPC Applications Using HDF5 Files in Distributed Shared Memory journal June 2012
Site remediation in a virtual environment
  • Bethel, E. W.; Jacobsen, Janet; Holland, Preston
  • IS&T/SPIE 1994 International Symposium on Electronic Imaging: Science and Technology, SPIE Proceedings https://doi.org/10.1117/12.172079
conference April 1994
Parallel netCDF: A High-Performance Scientific I/O Interface conference January 2003
Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems
  • Vishwanath, Venkatram; Hereld, Mark; Morozov, Vitali
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063409
conference January 2011
High end scientific codes with computational I/O pipelines: improving their end-to-end performance conference January 2011
Extending scalability of collective IO through nessie and staging conference January 2011
Distributed merge trees journal August 2013
Evaluation of methods to integrate analysis into a large-scale shock shock physics code conference January 2014
Optimal scheduling of in-situ analysis for large-scale scientific simulations
  • Malakar, Preeti; Vishwanath, Venkatram; Munson, Todd
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15 https://doi.org/10.1145/2807591.2807656
conference January 2015
An Approach to Lowering the In Situ Visualization Barrier
  • Fogal, Thomas; Krüger, Jens
  • Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015 https://doi.org/10.1145/2828612.2828618
conference January 2015
Strawman: A Batch In Situ Visualization and Analysis Infrastructure for Multi-Physics Simulation Codes
  • Larsen, Matthew; Brugger, Eric; Childs, Hank
  • Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015 https://doi.org/10.1145/2828612.2828625
conference January 2015
Computer-made perspective movies as a scientific and communication tool journal March 1964
Petroleum Reservoir Simulation in a Virtual Environment conference April 2013
In Situ Pathtube Visualization with Explorable Images null January 2013
pV3 - A distributed system for large-scale unsteady CFD visualization conference February 2013
EPIC - An Extract Plug-In Components Toolkit for In-Situ Data Extracts Architecture conference June 2015
In Situ Infrastructure Enhancements for Data Extract Generation conference January 2016
A Multi-Code Analysis Toolkit for Astrophysical Simulation Data text January 2010

Cited By (9)

Coupling the Uintah Framework and the VisIt Toolkit for Parallel In Situ Data Analysis and Visualization and Computational Steering book January 2018
Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets book January 2018
A novel in situ compression method for CFD data based on generative adversarial network journal October 2018
Scientific post hoc and in situ visualisation of high-order polynomial solutions from massively parallel simulations journal April 2019
Data Reduction Techniques for Simulation, Visualization and Data Analysis: Survey on Scientific Data Reduction Techniques journal March 2018
A decade of particle-based scientific visualization journal March 2019
Scalable In situ Analysis of Molecular Dynamics Simulations
  • Malakar, Preeti; Knight, Christopher; Munson, Todd
  • Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization - ISAV'17 https://doi.org/10.1145/3144769.3144772
conference January 2017
In situ data-driven adaptive sampling for large-scale simulation data summarization
  • Biswas, Ayan; Dutta, Soumya; Pulido, Jesus
  • Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV '18 https://doi.org/10.1145/3281464.3281467
conference January 2018
A Violently Tornadic Supercell Thunderstorm Simulation Spanning a Quarter-Trillion Grid Volumes: Computational Challenges, I/O Framework, and Visualizations of Tornadogenesis journal September 2019

Similar Records

Related Subjects