skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel Tasks

Abstract

The University of Utah’s Carbon Capture Multidisciplinary Simulation Center (CCMSC) is using the Uintah Computational Framework to predict performance of a 1000 MWe ultra-supercritical clean coal boiler. The center aims to utilize the Intel Xeon Phi-based DOE systems, Theta and Aurora, through the Aurora Early Science Program by using the Kokkos C++ library to enable node-level performance portability. This paper describes infrastructure advancements and portability improvements made possible by the integration of Kokkos within Uintah. This integration marks a step towards consolidating Uintah’s MPI+PThreads and MPI+CUDA hybrid parallelism approaches into a single MPI+Kokkos approach. Scalability results are presented that compare serial and data parallel task execution models for a challenging radiative heat transfer calculation, central to the center’s predictive boiler simulations. Here, these results demonstrate both good strong-scaling characteristics to 256 Knights Landing (KNL) processors on the NSF Stampede system, and show the KNL-based calculation to compete with prior GPU-based results for the same calculation.

Authors:
 [1];  [1];  [2];  [1]
  1. Univ. of Utah, Salt Lake City, UT (United States)
  2. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Publication Date:
Research Org.:
Univ. of Utah, Salt Lake City, UT (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1582430
DOE Contract Number:  
NA0002375
Resource Type:
Conference
Resource Relation:
Conference: PEARC17: Practice and Experience in Advanced Research Computing 2017 on Sustainability, Success and Impact, New Orleans, LA (United States), Jul 2017
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Hybrid Parallelism; Kokkos; Knights Landing; Many-Core; MIC Parallel; Portability; Radiation Modeling; Reverse Monte-Carlo Ray Tracing Scalability; Stampede; Uintah; Xeon Phi

Citation Formats

Holmen, John K., Humphrey, Alan, Sunderland, Daniel, and Berzins, Martin. Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel Tasks. United States: N. p., 2017. Web. doi:10.1145/3093338.3093388.
Holmen, John K., Humphrey, Alan, Sunderland, Daniel, & Berzins, Martin. Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel Tasks. United States. doi:10.1145/3093338.3093388.
Holmen, John K., Humphrey, Alan, Sunderland, Daniel, and Berzins, Martin. Sat . "Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel Tasks". United States. doi:10.1145/3093338.3093388. https://www.osti.gov/servlets/purl/1582430.
@article{osti_1582430,
title = {Improving Uintah's Scalability Through the Use of Portable Kokkos-Based Data Parallel Tasks},
author = {Holmen, John K. and Humphrey, Alan and Sunderland, Daniel and Berzins, Martin},
abstractNote = {The University of Utah’s Carbon Capture Multidisciplinary Simulation Center (CCMSC) is using the Uintah Computational Framework to predict performance of a 1000 MWe ultra-supercritical clean coal boiler. The center aims to utilize the Intel Xeon Phi-based DOE systems, Theta and Aurora, through the Aurora Early Science Program by using the Kokkos C++ library to enable node-level performance portability. This paper describes infrastructure advancements and portability improvements made possible by the integration of Kokkos within Uintah. This integration marks a step towards consolidating Uintah’s MPI+PThreads and MPI+CUDA hybrid parallelism approaches into a single MPI+Kokkos approach. Scalability results are presented that compare serial and data parallel task execution models for a challenging radiative heat transfer calculation, central to the center’s predictive boiler simulations. Here, these results demonstrate both good strong-scaling characteristics to 256 Knights Landing (KNL) processors on the NSF Stampede system, and show the KNL-based calculation to compete with prior GPU-based results for the same calculation.},
doi = {10.1145/3093338.3093388},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2017},
month = {7}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Works referenced in this record:

Dynamic task scheduling for the Uintah framework
conference, November 2010

  • Meng, Qingyu; Luitjens, Justin; Berzins, Martin
  • 2010 3rd Workshop on Many-Task Computing on Grids and Supercomputers (MTAGS)
  • DOI: 10.1109/MTAGS.2010.5699431

An Evaluation of Emerging Many-Core Parallel Programming Models
conference, January 2016

  • Martineau, Matt; McIntosh-Smith, Simon; Boulton, Mike
  • Proceedings of the 7th International Workshop on Programming Models and Applications for Multicores and Manycores - PMAM'16
  • DOI: 10.1145/2883404.2883420

Investigating applications portability with the Uintah DAG-based runtime system on PetaScale supercomputers
conference, January 2013

  • Meng, Qingyu; Humphrey, Alan; Schmidt, John
  • Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13
  • DOI: 10.1145/2503210.2503250

Extending the Uintah Framework through the Petascale Modeling of Detonation in Arrays of High Explosive Devices
journal, January 2016

  • Berzins, Martin; Beckvermit, Jacqueline; Harman, Todd
  • SIAM Journal on Scientific Computing, Vol. 38, Issue 5
  • DOI: 10.1137/15M1023270

Spatial Domain-Based Parallelism in Large-Scale, Participating-Media, Radiative Transport Applications
journal, June 1997

  • Burns, Shawn P.; Christen, Mark A.
  • Numerical Heat Transfer, Part B: Fundamentals, Vol. 31, Issue 4
  • DOI: 10.1080/10407799708915117

Kokkos: Enabling manycore performance portability through polymorphic memory access patterns
journal, December 2014

  • Carter Edwards, H.; Trott, Christian R.; Sunderland, Daniel
  • Journal of Parallel and Distributed Computing, Vol. 74, Issue 12
  • DOI: 10.1016/j.jpdc.2014.07.003

Parallelization of the P-1 Radiation Model
journal, January 2006

  • Krishnamoorthy, Gautham; Rawat, Rajesh; Smith, Philip J.
  • Numerical Heat Transfer, Part B: Fundamentals, Vol. 49, Issue 1
  • DOI: 10.1080/10407790500344068

A survey of high level frameworks in block-structured adaptive mesh refinement packages
journal, December 2014

  • Dubey, Anshu; Almgren, Ann; Bell, John
  • Journal of Parallel and Distributed Computing, Vol. 74, Issue 12
  • DOI: 10.1016/j.jpdc.2014.07.001

Using hybrid parallelism to improve memory use in the Uintah framework
conference, January 2011

  • Meng, Qingyu; Berzins, Martin; Schmidt, John
  • Proceedings of the 2011 TeraGrid Conference on Extreme Digital Discovery - TG '11
  • DOI: 10.1145/2016741.2016767

Large Eddy Simulations of Accidental Fires Using Massively Parallel Computers
conference, June 2012

  • Smith, Philip; Rawat, Rajesh; Spinti, Jennifer
  • 16th AIAA Computational Fluid Dynamics Conference
  • DOI: 10.2514/6.2003-3697

Legion: Expressing locality and independence with logical regions
conference, November 2012

  • Bauer, Michael; Treichler, Sean; Slaughter, Elliott
  • 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2012.71

Efficient Parallelization of RMCRT for Large Scale LES Combustion Simulations
conference, June 2012


Fast Parallel Algorithms for Short-Range Molecular Dynamics
journal, March 1995


CHARM++: a portable concurrent object oriented system based on C++
conference, January 1993

  • Kale, Laxmikant V.; Krishnan, Sanjeev
  • Proceedings of the eighth annual conference on Object-oriented programming systems, languages, and applications - OOPSLA '93
  • DOI: 10.1145/165854.165874

Parallel Computations of Radiative heat Transfer Using the Discrete Ordinates Method
journal, December 2004

  • Krishnamoorthy, Gautham; Rawat, Rajesh; Smith, Philip J.
  • Numerical Heat Transfer, Part B: Fundamentals, Vol. 47, Issue 1
  • DOI: 10.1080/10407790490487451

Uintah: a scalable framework for hazard analysis
conference, January 2010

  • Berzins, Martin; Luitjens, Justin; Meng, Qingyu
  • Proceedings of the 2010 TeraGrid Conference on - TG '10
  • DOI: 10.1145/1838574.1838577

Preliminary experiences with the uintah framework on Intel Xeon Phi and stampede
conference, January 2013

  • Meng, Qingyu; Humphrey, Alan; Schmidt, John
  • Proceedings of the Conference on Extreme Science and Engineering Discovery Environment Gateway to Discovery - XSEDE '13
  • DOI: 10.1145/2484762.2484779

Radiation modeling using the Uintah heterogeneous CPU/GPU runtime system
conference, January 2012

  • Humphrey, Alan; Meng, Qingyu; Berzins, Martin
  • Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment on Bridging from the eXtreme to the campus and beyond - XSEDE '12
  • DOI: 10.1145/2335755.2335791

An overview of the Trilinos project
journal, September 2005

  • Heroux, Michael A.; Phipps, Eric T.; Salinger, Andrew G.
  • ACM Transactions on Mathematical Software, Vol. 31, Issue 3
  • DOI: 10.1145/1089014.1089021

Radiative Heat Transfer Calculation on 16384 GPUs Using a Reverse Monte Carlo Ray Tracing Approach with Adaptive Mesh Refinement
conference, May 2016

  • Humphrey, Alan; Sunderland, Daniel; Harman, Todd
  • 2016 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)
  • DOI: 10.1109/IPDPSW.2016.93

An Overview of Performance Portability in the Uintah Runtime System through the Use of Kokkos
conference, November 2016

  • Sunderland, Daniel; Peterson, Brad; Schmidt, John
  • 2016 Second International Workshop on Extreme Scale Programming Models and Middleware (ESPM2), 2016 Second International Workshop on Extreme Scale Programming Models and Middlewar (ESPM2)
  • DOI: 10.1109/ESPM2.2016.012