skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Large-scale compute-intensive analysis via a combined in-situ and co-scheduling workflow approach, In: SC '15 Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Conference · · PROCEEDINGS OF SC15: THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS
 [1];  [2];  [3];  [1];  [4];  [5];  [6];  [7];  [1];  [4];  [8];  [1]
  1. Los Alamos National Lab, Los Alamos, NM
  2. OLCF, Oak Ridge National Lab, Oak Ridge, TN
  3. High Energy Physics, Argonne National Lab, Lemont, IL
  4. Argonne National Lab, Lemont, IL
  5. ALCF, Argonne National Lab, Lemont, IL
  6. Lawrence Livermore Nat. Lab, Livermore, CA
  7. OLCF Oak Ridge National Lab, Oak Ridge, TN
  8. University of Chicago and Argonne National Laboratory
+ Show Author Affiliations

Large-scale simulations can produce hundreds of terabytes to petabytes of data, complicating and limiting the efficiency of workflows. Traditionally, outputs are stored on the file system and analyzed in post-processing. With the rapidly increasing size and complexity of simulations, this approach faces an uncertain future. Trending techniques consist of performing the analysis in-situ, utilizing the same resources as the simulation, and/or off-loading subsets of the data to a compute-intensive analysis system. We introduce an analysis framework developed for HACC, a cosmological N-body code, that uses both in-situ and co-scheduling approaches for handling petabyte-scale outputs. We compare different analysis set-ups ranging from purely off-line, to purely in-situ to in-situ/co-scheduling. The analysis routines are implemented using the PISTON/VTK-m framework, allowing a single implementation of an algorithm that simultaneously targets a variety of GPU, multi-core, and many-core architectures.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Organization:
USDOE Office of Science (SC)
OSTI ID:
1567400
Journal Information:
PROCEEDINGS OF SC15: THE INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS, Conference: International Conference for High Performance Computing, Networking, Storage and Analysis, Austin, Texas, November 15-20, 2015
Country of Publication:
United States
Language:
English

References (31)

Extending I/O through high performance data services conference August 2009
Improved post hoc flow analysis via Lagrangian representations conference November 2014
An Image-Based Approach to Extreme Scale in Situ Visualization and Analysis
  • Ahrens, James; Jourdain, Sebastien; OLeary, Patrick
  • SC14: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2014.40
conference November 2014
Legion: Expressing locality and independence with logical regions
  • Bauer, Michael; Treichler, Sean; Slaughter, Elliott
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.71
conference November 2012
Jitter-free co-processing on a prototype exascale storage stack conference April 2012
Mechanisms of Core-Collapse Supernovae & Simulation Results from the CHIMERA Code
  • Bruenn, S. W.; Mezzacappa, A.; Hix, W. R.
  • PROBING STELLAR POPULATIONS OUT TO THE DISTANT UNIVERSE: CEFALU 2008, Proceedings of the International Conference, AIP Conference Proceedings https://doi.org/10.1063/1.3141615
conference January 2009
The ParaView Coprocessing Library: A scalable, general purpose in situ visualization library conference October 2011
The q Continuum Simulation: Harnessing the Power of gpu Accelerated Supercomputers journal August 2015
Large-Scale Simulations of Sky Surveys journal September 2014
Verification of the time evolution of cosmological simulations via hypothesis-driven comparative and quantitative visualization conference March 2010
Groups of galaxies. I - Nearby groups journal June 1982
A Multitier System for the Verification, Visualization and Management of CHIMERA journal January 2011
Near Real-time Data Analysis of Core-collapse Supernova Simulations with Bellerophon journal January 2014
Extending scalability of collective IO through nessie and staging conference January 2011
Experiences Applying Data Staging Technology in Unconventional Ways
  • Lofstead, J.; Oldfield, R.; Kordenbrock, T.
  • 2013 13th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid), 2013 13th IEEE/ACM International Symposium on Cluster, Cloud, and Grid Computing https://doi.org/10.1109/CCGrid.2013.27
conference May 2013
In Situ Visualization at Extreme Scale: Challenges and Opportunities journal November 2009
Phase-space structures - II. Hierarchical Structure Finder journal July 2009
Petascale supernova simulation with CHIMERA journal July 2007
Examples of in transit visualization
  • Moreland, Kenneth; Hereld, Mark; Papka, Michael E.
  • Proceedings of the 2nd international workshop on Petascal data analytics: challenges and opportunities - PDAC '11 https://doi.org/10.1145/2110205.2110207
conference January 2011
A new scalable parallel DBSCAN algorithm using the disjoint-set data structure
  • Patwary, Md. Mostofa Ali; Palsetia, Diana; Agrawal, Ankit
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.9
conference November 2012
Data co-processing for extreme scale analysis level II ASC milestone (4745). report March 2013
Visualization and parallel I/O at extreme scale journal July 2008
Portable data-parallel visualization and analysis in distributed memory environments conference October 2013
Utilizing many-core accelerators for halo and center finding within a cosmology simulation conference October 2015
GADGET: a code for collisionless and gasdynamical cosmological simulations journal April 2001
Toward a Halo Mass Function for Precision Cosmology: The Limits of Universality journal December 2008
Toward simulation-time data analysis and I/O acceleration on leadership-class systems conference October 2011
Data-parallel halo finding with variable linking lengths conference November 2014
ANALYZING AND VISUALIZING COSMOLOGICAL SIMULATIONS WITH ParaView journal June 2011
On-demand unstructured mesh translation for reducing memory pressure during in situ analysis conference January 2013
In Situ Visualization for Large-Scale Combustion Simulations journal May 2010

Similar Records

Large-Scale Compute-Intensive Analysis via a Combined In-situ and Co-scheduling Workflow Approach
Conference · Thu Jan 01 00:00:00 EST 2015 · OSTI ID:1567400
+6 more
PipeSight: A High-Performance Computing Platform for Pipeline Integrity Management
Technical Report · Wed Mar 01 00:00:00 EST 2023 · OSTI ID:1567400
+2 more
Physics-based seismic hazard analysis on petascale heterogeneous supercomputers, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Conference · Tue Jan 01 00:00:00 EST 2013 · 2013 INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS (SC) · OSTI ID:1567400
+11 more

Related Subjects

Computer Science
Engineering