Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation Laws

Bremer, Maximilian; Bachan, John; Chan, Cy; Dawson, Clint

doi:10.1145/3545996

Title: Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation Laws

Journal Article · Sat Nov 05 00:00:00 EDT 2022 · ACM Transactions on Modeling and Computer Simulation

DOI:https://doi.org/10.1145/3545996· OSTI ID:1994322

^[1];

^[2]

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Univ. of Texas, Austin, TX (United States)

Stable simulation of conservation laws, such as those used to model fluid dynamics and plasma physics applications, requires the satisfaction of the so-called Courant-Friedrichs-Lewy condition. By allowing regions of the mesh to advance with different timesteps that locally satisfy this stability constraint, significant work reduction can be attained when compared to a time integration scheme using a single timestep size. However, parallelizing this algorithm presents considerable difficulty. Since the stability condition depends on the state of the system, dependencies become dynamic and potentially non-local. In this article, we present an adaptive local timestepping algorithm using an optimistic (Timewarp-based) parallel discrete event simulation. We introduce waiting heuristics to limit misspeculation and a semi-static load balancing scheme to eliminate load imbalance as parts of the mesh require finer or coarser timesteps. Last, we outline an interface for separating the physics of the specific conservation law from the temporal integration allowing for productive adoption of our proposed algorithm. We present a misspeculation study for three conservation laws, demonstrating both the productivity of the local timestepping API, for which 74% of the lines of code are reused across different conservation laws, and the robustness of the waiting heuristics—at most 1.5% of element updates are rolled back. Our performance studies demonstrate up to a 2.8× speedup versus a baseline unoptimized local timestepping approach, a 4x improvement in per-node throughput compared to an MPI parallelization of synchronous timestepping, and scalability up to 3,072 cores on NERSC’s Cori Haswell partition.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC); National Science Foundation (NSF)

Grant/Contract Number:: AC02-05CH11231; 1854986

OSTI ID:: 1994322

Journal Information:: ACM Transactions on Modeling and Computer Simulation, Vol. 32, Issue 4; ISSN 1049-3301

Publisher:: Association for Computing MachineryCopyright Statement

Country of Publication:: United States

Language:: English

References (56)

Conservative explicit local time-stepping schemes for the shallow water equations Hoang, Thi-Thao-Phuong; Leng, Wei; Ju, Lili Journal of Computational Physics, Vol. 382 https://doi.org/10.1016/j.jcp.2019.01.006	journal	April 2019
Efficient Inter-Process Synchronization for Parallel Discrete Event Simulation on Multicores Bauer, Pavol; Lindén, Jonatan; Engblom, Stefan Proceedings of the 3rd ACM SIGSIM Conference on Principles of Advanced Discrete Simulation https://doi.org/10.1145/2769458.2769476	conference	June 2015
High resolution schemes for hyperbolic conservation laws Harten, Ami Journal of Computational Physics, Vol. 49, Issue 3 https://doi.org/10.1016/0021-9991(83)90136-5	journal	March 1983
HYPERS: A unidimensional asynchronous framework for multiscale hybrid simulations Omelchenko, Y. A.; Karimabadi, H. Journal of Computational Physics, Vol. 231, Issue 4 https://doi.org/10.1016/j.jcp.2011.11.004	journal	February 2012
An arbitrary high-order Discontinuous Galerkin method for elastic waves on unstructured meshes - V. Local time stepping and p -adaptivity Dumbser, Michael; Käser, Martin; Toro, Eleuterio F. Geophysical Journal International, Vol. 171, Issue 2 https://doi.org/10.1111/j.1365-246X.2007.03427.x	journal	November 2007
Event-driven, hybrid particle-in-cell simulation: A new paradigm for multi-scale plasma modeling Omelchenko, Y. A.; Karimabadi, H. Journal of Computational Physics, Vol. 216, Issue 1 https://doi.org/10.1016/j.jcp.2005.11.029	journal	July 2006
High-order discontinuous Galerkin methods for coastal hydrodynamics applications Brus, S. R.; Wirasaet, D.; Kubatko, E. J. Computer Methods in Applied Mechanics and Engineering, Vol. 355 https://doi.org/10.1016/j.cma.2019.07.003	journal	October 2019
GASNet-EX: A High-Performance, Portable Communication Library for Exascale Bonachea, Dan; Hargrove, Paul H.; Hall, Mary Languages and Compilers for Parallel Computing: 31st International Workshop, LCPC 2018, Salt Lake City, UT, USA, October 9–11, 2018, Revised Selected Papers, p. 138-158 https://doi.org/10.1007/978-3-030-34627-0_11	book	November 2019
High Resolution Schemes Using Flux Limiters for Hyperbolic Conservation Laws Sweby, P. K. SIAM Journal on Numerical Analysis, Vol. 21, Issue 5 https://doi.org/10.1137/0721062	journal	October 1984
Automated Simulation of Modelica Models with QSS Methods : The Discontinuous Case Floros, Xenofon; Bergero, Federico; Cellier, François E. Linköping Electronic Conference Proceedings https://doi.org/10.3384/ecp11063657	conference	June 2011
Local time stepping for the shallow water equations in MPAS Capodaglio, Giacomo; Petersen, Mark Journal of Computational Physics, Vol. 449 https://doi.org/10.1016/j.jcp.2021.110818	journal	January 2022
CHARM++: a portable concurrent object oriented system based on C++ Kale, Laxmikant V.; Krishnan, Sanjeev Proceedings of the eighth annual conference on Object-oriented programming systems, languages, and applications - OOPSLA '93 https://doi.org/10.1145/165854.165874	conference	January 1993
Adaptive memory management and optimism control in time warp Das, Samir R.; Fujimoto, Richard M. ACM Transactions on Modeling and Computer Simulation, Vol. 7, Issue 2 https://doi.org/10.1145/249204.249207	journal	April 1997
Petascale Local Time Stepping for the ADER-DG Finite Element Method Breuer, Alexander; Heinecke, Alexander; Bader, Michael 2016 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS.2016.109	conference	May 2016
An efficient local time-stepping scheme for solution of nonlinear conservation laws Krivodonova, Lilia Journal of Computational Physics, Vol. 229, Issue 22 https://doi.org/10.1016/j.jcp.2010.07.037	journal	November 2010
Split Runge-Kutta method for simultaneous equations Rice, John R. Journal of Research of the National Bureau of Standards Section B Mathematics and Mathematical Physics, Vol. 64B, Issue 3 https://doi.org/10.6028/jres.064B.018	journal	July 1960
The numerical simulation of two-dimensional fluid flow with strong shocks Woodward, Paul; Colella, Phillip Journal of Computational Physics, Vol. 54, Issue 1 https://doi.org/10.1016/0021-9991(84)90142-6	journal	April 1984
Runge-Kutta Discontinuous Galerkin Methods for Convection-Dominated Problems Cockburn, Bernardo; Shu, Chi-Wang Journal of Scientific Computing, Vol. 16, Issue 3, p. 173-261 https://doi.org/10.1023/A:1012873910884	journal	September 2001
Numerical Methods for Conservation Laws LeVeque, Randall J. Lectures in Mathematics. ETH Zürich https://doi.org/10.1007/978-3-0348-8629-1	book	January 1992
Strong Stability-Preserving High-Order Time Discretization Methods Gottlieb, Sigal; Shu, Chi-Wang; Tadmor, Eitan SIAM Review, Vol. 43, Issue 1 https://doi.org/10.1137/S003614450036757X	journal	January 2001
Multirate Timestepping Methods for Hyperbolic Conservation Laws Constantinescu, Emil M.; Sandu, Adrian Journal of Scientific Computing, Vol. 33, Issue 3 https://doi.org/10.1007/s10915-007-9151-y	journal	September 2007
On process migration and load balancing in Time Warp Glazer, D. W.; Tropper, C. IEEE Transactions on Parallel and Distributed Systems, Vol. 4, Issue 3 https://doi.org/10.1109/71.210814	journal	March 1993
A survey of several finite difference methods for systems of nonlinear hyperbolic conservation laws Sod, Gary A. Journal of Computational Physics, Vol. 27, Issue 1 https://doi.org/10.1016/0021-9991(78)90023-2	journal	April 1978
Semi-Static and Dynamic Load Balancing for Asynchronous Hurricane Storm Surge Simulations Bremer, Maximilian H.; Bachan, John D.; Chan, Cy P. 2018 IEEE/ACM Parallel Applications Workshop, Alternatives To MPI (PAW-ATM) https://doi.org/10.1109/PAW-ATM.2018.00010	conference	November 2018
Discontinuous Galerkin methods for modeling Hurricane storm surge Dawson, Clint; Kubatko, Ethan J.; Westerink, Joannes J. Advances in Water Resources, Vol. 34, Issue 9 https://doi.org/10.1016/j.advwatres.2010.11.004	journal	September 2011
A Fast and Stable Well-Balanced Scheme with Hydrostatic Reconstruction for Shallow Water Flows Audusse, Emmanuel; Bouchut, François; Bristeau, Marie-Odile SIAM Journal on Scientific Computing, Vol. 25, Issue 6 https://doi.org/10.1137/S1064827503431090	journal	January 2004
SUNDIALS: Suite of nonlinear and differential/algebraic equation solvers Hindmarsh, Alan C.; Brown, Peter N.; Grant, Keith E. ACM Transactions on Mathematical Software, Vol. 31, Issue 3 https://doi.org/10.1145/1089014.1089020	journal	September 2005
A Fast and High Quality Multilevel Scheme for Partitioning Irregular Graphs Karypis, George; Kumar, Vipin SIAM Journal on Scientific Computing, Vol. 20, Issue 1 https://doi.org/10.1137/S1064827595287997	journal	January 1998
Adaptive Methods for Irregular Parallel Discrete Event Simulation Workloads Mikida, Eric; Kale, Laxmikant Proceedings of the 2018 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation https://doi.org/10.1145/3200921.3200936	conference	May 2018
Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance Slaughter, Elliott; Wu, Wei; Fu, Yuankun SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00066	conference	November 2020
Virtual time Jefferson, David R. ACM Transactions on Programming Languages and Systems, Vol. 7, Issue 3 https://doi.org/10.1145/3916.3988	journal	July 1985
Mobiliti: Scalable Transportation Simulation Using High-Performance Parallel Computing Chan, Cy; Wang, Bin; Bachan, John 2018 21st International Conference on Intelligent Transportation Systems (ITSC) https://doi.org/10.1109/ITSC.2018.8569397	conference	November 2018
Performance Comparison of HPX Versus Traditional Parallelization Strategies for the Discontinuous Galerkin Method Bremer, Maximilian; Kazhyken, Kazbek; Kaiser, Hartmut Journal of Scientific Computing, Vol. 80, Issue 2 https://doi.org/10.1007/s10915-019-00960-z	journal	May 2019
Parallel discrete event simulation Fujimoto, Richard M. Communications of the ACM, Vol. 33, Issue 10 https://doi.org/10.1145/84537.84545	journal	October 1990
Third order asynchronous time integration for gas dynamics Unfer, Thomas Journal of Computational Physics, Vol. 440 https://doi.org/10.1016/j.jcp.2021.110434	journal	September 2021
Numerical approximations to nonlinear conservation laws with locally varying time and space grids Osher, Stanley; Sanders, Richard Mathematics of Computation, Vol. 41, Issue 164 https://doi.org/10.1090/S0025-5718-1983-0717689-8	journal	January 1983
A time-accurate explicit multi-scale technique for gas dynamics Omelchenko, Y. A.; Karimabadi, H. Journal of Computational Physics, Vol. 226, Issue 1 https://doi.org/10.1016/j.jcp.2007.04.010	journal	September 2007
Exposing Inter-Process Information for Efficient Parallel Discrete Event Simulation of Spatial Stochastic Systems Lindén, Jonatan; Bauer, Pavol; Engblom, Stefan Proceedings of the 2017 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation https://doi.org/10.1145/3064911.3064916	conference	May 2017
The Performance Implication of Task Size for Applications on the HPX Runtime System Grubel, Patricia; Kaiser, Hartmut; Cook, Jeanine 2015 IEEE International Conference on Cluster Computing (CLUSTER) https://doi.org/10.1109/CLUSTER.2015.119	conference	September 2015
Speculative Parallel Execution for Local Timestepping Bremer, Maximilian; Bachan, John; Chan, Cy Proceedings of the 2021 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation https://doi.org/10.1145/3437959.3459257	conference	May 2021
Speculative parallel simulation with an adaptive throttle scheme Tay, Seng Chuan; Teo, Yong Meng; Kong, Siew Theng ACM SIGSIM Simulation Digest, Vol. 27, Issue 1 https://doi.org/10.1145/268823.268909	journal	July 1997
Weighted Essentially Non-oscillatory Schemes Liu, Xu-Dong; Osher, Stanley; Chan, Tony Journal of Computational Physics, Vol. 115, Issue 1 https://doi.org/10.1006/jcph.1994.1187	journal	November 1994
Fine-Grained Local Dynamic Load Balancing in PDES Lindén, Jonatan; Bauer, Pavol; Engblom, Stefan Proceedings of the 2018 ACM SIGSIM Conference on Principles of Advanced Discrete Simulation https://doi.org/10.1145/3200921.3200928	conference	May 2018
Urgent Computing of Storm Surge for North Carolina's Coast Blanton, Brian; McGee, John; Fleming, Jason Procedia Computer Science, Vol. 9 https://doi.org/10.1016/j.procs.2012.04.185	journal	January 2012
Flooding and Drying in Discontinuous Galerkin Finite-Element Discretizations of Shallow-Water Equations. Part 1: One Dimension Bokhove, Onno Journal of Scientific Computing, Vol. 22-23, Issue 1-3 https://doi.org/10.1007/s10915-004-4136-6	journal	June 2005
Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake Uphoff, Carsten; Rettenberger, Sebastian; Bader, Michael Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3126908.3126948	conference	November 2017
Asynchronous Variational Integrators Lew, A.; Marsden, J. E.; Ortiz, M. Archive for Rational Mechanics and Analysis, Vol. 167, Issue 2 https://doi.org/10.1007/s00205-002-0212-y	journal	April 2003
Multirate Runge–Kutta schemes for advection equations Schlegel, Martin; Knoth, Oswald; Arnold, Martin Journal of Computational and Applied Mathematics, Vol. 226, Issue 2 https://doi.org/10.1016/j.cam.2008.08.009	journal	April 2009
Adaptive total variation stable local timestepping for conservation laws Bremer, Maximilian; Bachan, John; Chan, Cy Journal of Computational Physics, Vol. 463 https://doi.org/10.1016/j.jcp.2022.111176	journal	August 2022
Feature Article—Parallel Discrete Event Simulation: Will the Field Survive? Fujimoto, Richard M. ORSA Journal on Computing, Vol. 5, Issue 3 https://doi.org/10.1287/ijoc.5.3.213	journal	August 1993
Water waves of finite amplitude on a sloping beach Carrier, G. F.; Greenspan, H. P. Journal of Fluid Mechanics, Vol. 4, Issue 1 https://doi.org/10.1017/S0022112058000331	journal	May 1958
A parallel local timestepping Runge–Kutta discontinuous Galerkin method with applications to coastal ocean modeling Dawson, Clint; Trahan, Corey Jason; Kubatko, Ethan J. Computer Methods in Applied Mechanics and Engineering, Vol. 259 https://doi.org/10.1016/j.cma.2013.03.015	journal	June 2013
ExaHyPE: An engine for parallel dynamically adaptive simulations of wave problems Reinarz, Anne; Charrier, Dominic E.; Bader, Michael Computer Physics Communications, Vol. 254 https://doi.org/10.1016/j.cpc.2020.107251	journal	September 2020
High Resolution Schemes for Conservation Laws with Locally Varying Time Steps Dawson, Clint; Kirby, Robert SIAM Journal on Scientific Computing, Vol. 22, Issue 6 https://doi.org/10.1137/S1064827500367737	journal	January 2001
Performance and Scalability Improvements for Discontinuous Galerkin Solutions to Conservation Laws on Unstructured Grids Brus, S. R.; Wirasaet, D.; Westerink, J. J. Journal of Scientific Computing, Vol. 70, Issue 1 https://doi.org/10.1007/s10915-016-0249-y	journal	August 2016
Adaptive protocols for parallel discrete event simulation Das, S. R. Journal of the Operational Research Society, Vol. 51, Issue 4 https://doi.org/10.1057/palgrave.jors.2600895	journal	April 2000

Similar Records

Deploy Nalu/Kokkos algorithmic infrastructure with performance benchmarking.

Technical Report · Fri Sep 29 00:00:00 EDT 2017 · OSTI ID:1994322

Domino, Stefan P.; Ananthan, Shreyas; Knaus, Robert C.; +1 more

Roofline Analysis in the Intel® Advisor to Deliver Optimized Performance for applications on Intel® Xeon Phi™ Processor

Conference · Tue May 23 00:00:00 EDT 2017 · OSTI ID:1994322

Koskela, Tuomas S.; Lobet, Mathieu; Deslippe, Jack; +1 more

Parallel Implementation and Scaling of an Adaptive Mesh Discrete Ordinates Algorithm for Transport

Conference · Mon Nov 29 00:00:00 EST 2004 · OSTI ID:1994322

Howell, L H

Related Subjects

97 MATHEMATICS AND COMPUTING
local timestepping
parallel discrete event simulation
timewarp
shallow water equations
conservation laws
mathematics of computing
partial differential equations
applied computing
earth sciences
atmospheric sciences

Title: Performance Analysis of Speculative Parallel Adaptive Local Timestepping for Conservation Laws

Citation Formats

References (56)

Similar Records

Related Subjects