Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms

Abstract

The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed /visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU’s and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.

Authors:
 [1];  [2];  [3];  [4];  [1];  [2];  [5];  [1];  [6];  [7];  [8]
+ Show Author Affiliations
  1. Kitware, Inc., Clifton Park, NY (United States)
  2. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  4. Univ. of Oregon, Eugene, OR (United States)
  5. Sandia National Lab. (SNL-CA), Livermore, CA (United States)
  6. Argonne National Lab. (ANL), Argonne, IL (United States)
  7. Intelligent Light, USA, Rutherford, NJ (United States)
  8. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Publication Date:
Research Org.:
Kitware, Inc., Clifton Park, NY (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Nuclear Energy (NE); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1595010
Alternate Identifier(s):
OSTI ID: 1439995
Grant/Contract Number:  
SC0012037; AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Computer Graphics Forum
Additional Journal Information:
Journal Volume: 35; Journal Issue: 3; Journal ID: ISSN 0167-7055
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Bauer, A. C., Abbasi, H., Ahrens, J., Childs, H., Geveci, B., Klasky, S., Moreland, K., O'Leary, P., Vishwanath, V., Whitlock, B., and Bethel, E. W. In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms. United States: N. p., 2016. Web. doi:10.1111/cgf.12930.
Copy to clipboard
Bauer, A. C., Abbasi, H., Ahrens, J., Childs, H., Geveci, B., Klasky, S., Moreland, K., O'Leary, P., Vishwanath, V., Whitlock, B., & Bethel, E. W. In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms. United States. https://doi.org/10.1111/cgf.12930
Copy to clipboard
Bauer, A. C., Abbasi, H., Ahrens, J., Childs, H., Geveci, B., Klasky, S., Moreland, K., O'Leary, P., Vishwanath, V., Whitlock, B., and Bethel, E. W. Wed . "In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms". United States. https://doi.org/10.1111/cgf.12930. https://www.osti.gov/servlets/purl/1595010.
Copy to clipboard
@article{osti_1595010,
title = {In Situ Methods, Infrastructures, and Applications on High Performance Computing Platforms},
author = {Bauer, A. C. and Abbasi, H. and Ahrens, J. and Childs, H. and Geveci, B. and Klasky, S. and Moreland, K. and O'Leary, P. and Vishwanath, V. and Whitlock, B. and Bethel, E. W.},
abstractNote = {The considerable interest in the high performance computing (HPC) community regarding analyzing and visualization data without first writing to disk, i.e., in situ processing, is due to several factors. First is an I/O cost savings, where data is analyzed /visualized while being generated, without first storing to a filesystem. Second is the potential for increased accuracy, where fine temporal sampling of transient analysis might expose some complex behavior missed in coarse temporal sampling. Third is the ability to use all available resources, CPU’s and accelerators, in the computation of analysis products. This STAR paper brings together researchers, developers and practitioners using in situ methods in extreme-scale HPC with the goal to present existing methods, infrastructures, and a range of computational science and engineering applications using in situ analysis and visualization.},
doi = {10.1111/cgf.12930},
journal = {Computer Graphics Forum},
number = 3,
volume = 35,
place = {United States},
year = {Wed Jun 01 00:00:00 EDT 2016},
month = {Wed Jun 01 00:00:00 EDT 2016}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1111/cgf.12930
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 83 works
Citation information provided by
Web of Science
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

FLASH: An Adaptive Mesh Hydrodynamics Code for Modeling Astrophysical Thermonuclear Flashes
journal, November 2000

  • Fryxell, B.; Olson, K.; Ricker, P.
  • The Astrophysical Journal Supplement Series, Vol. 131, Issue 1
  • DOI: 10.1086/317361

High end scientific codes with computational I/O pipelines: improving their end-to-end performance
conference, January 2011

  • Zheng, Fang; Cao, Jianting; Dayal, Jai
  • Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11
  • DOI: 10.1145/2110205.2110210

Increasing Scientific Data Insights about Exascale Class Simulations under Power and Storage Constraints
journal, March 2015

  • Ahrens, James
  • IEEE Computer Graphics and Applications, Vol. 35, Issue 2
  • DOI: 10.1109/MCG.2015.35

In-situ multi-resolution and temporal data compression for visual exploration of large-scale scientific simulations
conference, November 2014

  • Lehmann, Henry; Jung, Bernhard
  • 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2014.7013204

Improved post hoc flow analysis via Lagrangian representations
conference, November 2014

  • Agranovsky, Alexy; Camp, David; Garth, Christoph
  • 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2014.7013206

Space-time volumetric depth images for in-situ visualization
conference, November 2014

  • Fernandes, Oliver; Frey, Steffen; Sadlo, Filip
  • 2014 IEEE 4th Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2014.7013205

Distributed merge trees
journal, August 2013

Topology-aware data movement and staging for I/O acceleration on Blue Gene/P supercomputing systems
conference, January 2011

  • Vishwanath, Venkatram; Hereld, Mark; Morozov, Vitali
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11
  • DOI: 10.1145/2063384.2063409

In situ depth maps based feature extraction and tracking
conference, October 2015

  • Ye, Yucong Chris; Wang, Yang; Miller, Robert
  • 2015 IEEE 5th Symposium on Large Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2015.7348065

Runtime volume visualization for parallel CFD
book, January 1996

Consuming Network Bandwidth with Visapult
book, January 2005

Adaptable, metadata rich IO methods for portable high performance IO
conference, May 2009

  • Lofstead, Jay; Zheng, Fang; Klasky, Scott
  • Distributed Processing (IPDPS), 2009 IEEE International Symposium on Parallel & Distributed Processing
  • DOI: 10.1109/IPDPS.2009.5161052

Extending scalability of collective IO through nessie and staging
conference, January 2011

  • Lofstead, Jay; Oldfield, Ron; Kordenbrock, Todd
  • Proceedings of the sixth workshop on Parallel Data Storage - PDSW '11
  • DOI: 10.1145/2159352.2159355

ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing: ActiveSpaces: Exploring dynamic code deployment for extreme scale data processing
journal, October 2014

  • Docan, Ciprian; Zhang, Fan; Jin, Tong
  • Concurrency and Computation: Practice and Experience, Vol. 27, Issue 14
  • DOI: 10.1002/cpe.3407

Visualization by Proxy: A Novel Framework for Deferred Interaction with Volume Data
journal, November 2010

  • Tikhonova, A.; Correa, C. D.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 16, Issue 6
  • DOI: 10.1109/TVCG.2010.215

Visualization at Supercomputing Centers: The Tale of Little Big Iron and the Three Skinny Guys
journal, January 2011

  • Bethel, E. W.; van Rosendale, J.; Southard, D.
  • IEEE Computer Graphics and Applications, Vol. 31, Issue 1
  • DOI: 10.1109/MCG.2011.13

Site remediation in a virtual environment
conference, April 1994

  • Bethel, E. W.; Jacobsen, Janet; Holland, Preston
  • IS&T/SPIE 1994 International Symposium on Electronic Imaging: Science and Technology, SPIE Proceedings
  • DOI: 10.1117/12.172079

Hello ADIOS: the challenges and lessons of developing leadership class I/O frameworks: HELLO ADIOS
journal, August 2013

  • Liu, Qing; Logan, Jeremy; Tian, Yuan
  • Concurrency and Computation: Practice and Experience, Vol. 26, Issue 7
  • DOI: 10.1002/cpe.3125

EPIC - An Extract Plug-In Components Toolkit for In-Situ Data Extracts Architecture
conference, June 2015

  • Duque, Earl P.; Hiepler, `Daniel E.; Haimes, Robert
  • 22nd AIAA Computational Fluid Dynamics Conference
  • DOI: 10.2514/6.2015-3410

Petroleum Reservoir Simulation in a Virtual Environment
conference, April 2013

  • Jacobsen, J. S.; Bethel, E. W.; Datta-Gupta, Akhil
  • SPE Reservoir Simulation Symposium
  • DOI: 10.2118/29118-MS

Research Challenges for Visualization Software
journal, May 2013

  • Childs, Hank; Geveci, Berk; Schroeder, Will
  • Computer, Vol. 46, Issue 5
  • DOI: 10.1109/MC.2013.179

I/O Containers: Managing the Data Analytics and Visualization Pipelines of High End Codes
conference, May 2013

  • Dayal, Jai; Cao, Jianting; Eisenhauer, Greg
  • 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW)
  • DOI: 10.1109/IPDPSW.2013.198

The Tensions of In Situ Visualization
journal, March 2016

  • Moreland, Kenneth
  • IEEE Computer Graphics and Applications, Vol. 36, Issue 2
  • DOI: 10.1109/MCG.2016.35

Computational steering. Software systems and strategies
journal, January 1997

  • Parker, S. G.; Johnson, C. R.; Beazley, D.
  • IEEE Computational Science and Engineering, Vol. 4, Issue 4
  • DOI: 10.1109/99.641609

A multi-resolution approach to global ocean modeling
journal, September 2013

Interactive, Internet Delivery of Visualization via Structured Prerendered Multiresolution Imagery
journal, March 2008

  • Jerry Chen, ; Bethel, E. W.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 14, Issue 2
  • DOI: 10.1109/TVCG.2007.70428

Cinema image-based in situ analysis and visualization of MPAS-ocean simulations
journal, July 2016

In Situ Visualization at Extreme Scale: Challenges and Opportunities
journal, November 2009

In Situ Visualization for Large-Scale Combustion Simulations
journal, May 2010

  • Hongfeng Yu, ; Grout, Ray W.
  • IEEE Computer Graphics and Applications, Vol. 30, Issue 3
  • DOI: 10.1109/MCG.2010.55

Parallel netCDF: A High-Performance Scientific I/O Interface
conference, January 2003

  • Li, Jianwei; Zingale, Michael; Liao, Wei-keng
  • Proceedings of the 2003 ACM/IEEE conference on Supercomputing - SC '03
  • DOI: 10.1145/1048935.1050189

Computer-made perspective movies as a scientific and communication tool
journal, March 1964

HPCCloud: A Cloud/Web-Based Simulation Environment
conference, November 2015

  • OLeary, Patrick; Christon, Mark; Jourdain, Sebastien
  • 2015 IEEE 7th International Conference on Cloud Computing Technology and Science (CloudCom)
  • DOI: 10.1109/CloudCom.2015.33

In Situ Pathtube Visualization with Explorable Images
null, January 2013

Efficient parallel volume rendering of large-scale adaptive mesh refinement data
conference, October 2013

  • Leaf, Nick; Vishwanath, Venkatram; Insley, Joseph
  • 2013 IEEE Symposium on Large-Scale Data Analysis and Visualization (LDAV)
  • DOI: 10.1109/LDAV.2013.6675156

Evaluation of methods to integrate analysis into a large-scale shock shock physics code
conference, January 2014

  • Oldfield, Ron A.; Moreland, Kenneth; Fabian, Nathan
  • Proceedings of the 28th ACM international conference on Supercomputing - ICS '14
  • DOI: 10.1145/2597652.2597668

A survey of computational steering environments
journal, February 1999

  • Mulder, Jurriaan D.; van Wijk, Jarke J.; van Liere, Robert
  • Future Generation Computer Systems, Vol. 15, Issue 1
  • DOI: 10.1016/S0167-739X(98)00047-8

yt: A MULTI-CODE ANALYSIS TOOLKIT FOR ASTROPHYSICAL SIMULATION DATA
journal, December 2010

  • Turk, Matthew J.; Smith, Britton D.; Oishi, Jeffrey S.
  • The Astrophysical Journal Supplement Series, Vol. 192, Issue 1
  • DOI: 10.1088/0067-0049/192/1/9

Scalable Implicit Flow Solver for Realistic Wing Simulations with Flow Control
journal, November 2014

  • Rasquin, Michel; Smith, Cameron; Chitale, Kedar
  • Computing in Science & Engineering, Vol. 16, Issue 6
  • DOI: 10.1109/MCSE.2014.75

WarpIV: In Situ Visualization and Analysis of Ion Accelerator Simulations
journal, May 2016

  • Rubel, Oliver; Loring, Burlen; Vay, Jean-Luc
  • IEEE Computer Graphics and Applications, Vol. 36, Issue 3
  • DOI: 10.1109/MCG.2016.62

Interactive Exploration and Analysis of Large-Scale Simulations Using Topology-Based Data Segmentation
journal, September 2011

  • Bremer, P-T; Weber, G.; Tierny, J.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 17, Issue 9
  • DOI: 10.1109/TVCG.2010.253

Optimal scheduling of in-situ analysis for large-scale scientific simulations
conference, January 2015

  • Malakar, Preeti; Vishwanath, Venkatram; Munson, Todd
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '15
  • DOI: 10.1145/2807591.2807656

pV3 - A distributed system for large-scale unsteady CFD visualization
conference, February 2013

Experiences Applying Data Staging Technology in Unconventional Ways
conference, May 2013

  • Lofstead, J.; Oldfield, R.; Kordenbrock, T.
  • 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing
  • DOI: 10.1109/CCGrid.2013.27

DataSpaces: an interaction and coordination framework for coupled simulation workflows
journal, February 2011

From Mesh Generation to Scientific Visualization: An End-to-End Approach to Parallel Supercomputing
conference, November 2006

  • Tu, Tiankai; Yu, Hongfeng; Ramirez-Guzman, Leonardo
  • ACM/IEEE SC 2006 Conference (SC'06)
  • DOI: 10.1109/SC.2006.32

PreDatA – preparatory data analytics on peta-scale machines
conference, April 2010

  • Zheng, Fang; Abbasi, Hasan; Docan, Ciprian
  • 2010 IEEE International Symposium on Parallel & Distributed Processing (IPDPS)
  • DOI: 10.1109/IPDPS.2010.5470454

An Approach to Lowering the In Situ Visualization Barrier
conference, January 2015

  • Fogal, Thomas; Krüger, Jens
  • Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015
  • DOI: 10.1145/2828612.2828618

PARAMESH: A parallel adaptive mesh refinement community toolkit
journal, April 2000

Strawman: A Batch In Situ Visualization and Analysis Infrastructure for Multi-Physics Simulation Codes
conference, January 2015

  • Larsen, Matthew; Brugger, Eric; Childs, Hank
  • Proceedings of the First Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV2015
  • DOI: 10.1145/2828612.2828625

Oh, $#*@! Exascale! The Effect of Emerging Architectures on Scientific Discovery
conference, November 2012

  • Moreland, Kenneth
  • 2012 SC Companion: High Performance Computing, Networking, Storage and Analysis (SCC), 2012 SC Companion: High Performance Computing, Networking Storage and Analysis
  • DOI: 10.1109/SC.Companion.2012.38

Concurrent Visualization in a Production Supercomputing Environment
journal, September 2006

  • Ellsworth, D.; Green, B.; Henze, C.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 12, Issue 5
  • DOI: 10.1109/TVCG.2006.128

An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis
conference, November 2014

  • Ahrens, James; Jourdain, Sebastien; OLeary, Patrick
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2014.40

In Situ Infrastructure Enhancements for Data Extract Generation
conference, January 2016

  • Whitlock, Brad J.; Legensky, Steve M.; Forsythe, Jim
  • 54th AIAA Aerospace Sciences Meeting
  • DOI: 10.2514/6.2016-1928

Parallel Computational Steering for HPC Applications Using HDF5 Files in Distributed Shared Memory
journal, June 2012

  • Biddiscombe, J.; Soumagne, J.; Oger, G.
  • IEEE Transactions on Visualization and Computer Graphics, Vol. 18, Issue 6
  • DOI: 10.1109/TVCG.2012.63

A Multi-Code Analysis Toolkit for Astrophysical Simulation Data
text, January 2010

    Sort by:
    [ × clear filter / sort ]

    Works referencing / citing this record:

    Design of a Flexible In Situ Framework with a Temporal Buffer for Data Processing and Visualization of Time-Varying Datasets
    book, January 2018

    • Ono, Kenji; Nonaka, Jorji; Yoshikawa, Hiroyuki
    • High Performance Computing. Lecture Notes in Computer Science
    • DOI: 10.1007/978-3-030-02465-9_17

    A novel in situ compression method for CFD data based on generative adversarial network
    journal, October 2018

    Scientific post hoc and in situ visualisation of high-order polynomial solutions from massively parallel simulations
    journal, April 2019

    • Rasquin, M.; Bauer, A.; Hillewaert, K.
    • International Journal of Computational Fluid Dynamics, Vol. 33, Issue 4
    • DOI: 10.1080/10618562.2019.1618453

    Data Reduction Techniques for Simulation, Visualization and Data Analysis: Survey on Scientific Data Reduction Techniques
    journal, March 2018

    • Li, S.; Marsaglia, N.; Garth, C.
    • Computer Graphics Forum, Vol. 37, Issue 6
    • DOI: 10.1111/cgf.13336

    A decade of particle-based scientific visualization
    journal, March 2019

    • Reina, Guido; Gralka, Patrick; Ertl, Thomas
    • The European Physical Journal Special Topics, Vol. 227, Issue 14
    • DOI: 10.1140/epjst/e2019-800172-4

    Scalable In situ Analysis of Molecular Dynamics Simulations
    conference, January 2017

    • Malakar, Preeti; Knight, Christopher; Munson, Todd
    • Proceedings of the In Situ Infrastructures on Enabling Extreme-Scale Analysis and Visualization - ISAV'17
    • DOI: 10.1145/3144769.3144772

    In situ data-driven adaptive sampling for large-scale simulation data summarization
    conference, January 2018

    • Biswas, Ayan; Dutta, Soumya; Pulido, Jesus
    • Proceedings of the Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization - ISAV '18
    • DOI: 10.1145/3281464.3281467
      Sort by:
      [ × clear filter / sort ]

      Similar Records in DOE PAGES and OSTI.GOV collections: