Skip to main content
U.S. Department of Energy
Office of Scientific and Technical Information

Performance-portable Binary Neutron Star Mergers with AthenaK

Journal Article · · The Astrophysical Journal. Supplement Series

We introduce an extension to the AthenaK code for general-relativistic magnetohydrodynamics (GRMHD) in dynamical spacetimes using a 3+1 conservative Eulerian formulation. Like the fixed-spacetime GRMHD solver, we use standard finite-volume methods to evolve the fluid and a constrained-transport scheme to preserve the divergence-free constraint for the magnetic field. We also utilize a first-order flux correction (FOFC) scheme to reduce the need for an artificial atmosphere and optionally enforce a maximum principle to improve robustness. We demonstrate the accuracy of AthenaK using a set of standard tests in flat and curved spacetimes. Using a SANE accretion disk around a Kerr black hole, we compare the new solver to the existing solver for stationary spacetimes using the so-called "HARM-like" formulation. We find that both formulations converge to similar results. We also include the first published binary neutron star (BNS) mergers performed on graphical processing units (GPUs). Thanks to the FOFC scheme, our BNS mergers maintain a relative error of $$\mathcal{O}$$(10–11) or better in baryon mass conservation up to collapse. Finally, we perform scaling tests of AthenaK on OLCF Frontier, where we show excellent weak scaling of ≥80% efficiency up to 32,768 GPUs and 74% up to 65,536 GPUs for a GRMHD problem in dynamical spacetimes with six levels of mesh refinement. AthenaK achieves an order-of-magnitude speedup using GPUs compared to CPUs, demonstrating that it is suitable for performing numerical relativity problems on modern exascale resources.

Research Organization:
Pennsylvania State Univ., University Park, PA (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Nuclear Physics (NP); National Aeronautics and Space Administration (NASA); National Science Foundation (NSF)
Grant/Contract Number:
SC0021177; SC0024388; AC05-00OR22725; 80NSSC21K1720; PHY-2011725; PHY-2020275; PHY-2116686; AST-2108467
OSTI ID:
2500239
Journal Information:
The Astrophysical Journal. Supplement Series, Vol. 276, Issue 2; ISSN 0067-0049
Publisher:
IOP PublishingCopyright Statement
Country of Publication:
United States
Language:
English

References (97)

Observation of Gravitational Waves from a Binary Black Hole Merger journal February 2016
GW170817: Observation of Gravitational Waves from a Binary Neutron Star Inspiral journal October 2017
Exploring the sensitivity of next generation gravitational wave detectors journal January 2017
Cosmos++ : Relativistic Magnetohydrodynamics on Unstructured Grids with Local Adaptive Refinement journal December 2005
Numerical 3+1 General Relativistic Magnetohydrodynamics: A Local Characteristic Approach journal January 2006
Total Variation Diminishing Scheme for Relativistic Magnetohydrodynamics journal January 2001
Numerical {3 + 1} General Relativistic Hydrodynamics: A Local Characteristic Approach journal February 1997
A Second-Order Godunov Method for Multi-Dimensional Relativistic Magnetohydrodynamics journal January 2011
Adaptive mesh refinement for hyperbolic partial differential equations journal March 1984
Constraint violation in free evolution schemes: Comparing the BSSNOK formulation with a conformal decomposition of the Z4 formulation journal April 2010
Tidal effects in binary neutron star coalescence journal August 2012
Accuracy of numerical relativity waveforms from binary neutron star mergers and their comparison with post-Newtonian waveforms journal May 2012
A finite element method for angular discretization of the radiation transport equation on spherical geodesic grids journal October 2023
An improved weighted essentially non-oscillatory scheme for hyperbolic conservation laws journal March 2008
High-accuracy numerical simulation of black-hole binaries: Computation of the gravitational-wave energy flux and comparisons with post-Newtonian approximants journal November 2008
Solving 3D relativistic hydrodynamical problems with weighted essentially nonoscillatory discontinuous Galerkin methods journal October 2016
A high-order finite volume method for systems of conservation laws—Multi-dimensional Optimal Order Detection (MOOD) journal May 2011
GRChombo : Numerical relativity with adaptive mesh refinement journal December 2015
A limiter for PPM that preserves accuracy at smooth extrema journal July 2008
The Piecewise Parabolic Method (PPM) for gas-dynamical simulations journal April 1984
Energy Versus Angular Momentum in Black Hole Binaries journal March 2012
GR-Athena++: Puncture Evolutions on Vertex-centered Oct-tree Adaptive Mesh Refinement journal November 2021
ECHO: a Eulerian conservative high-order scheme for general relativistic magnetohydrodynamics and magnetodynamics journal July 2007
A positivity-preserving adaptive-order finite-difference scheme for GRMHD journal November 2023
Flux-corrected transport techniques for multidimensional compressible magnetohydrodynamics journal January 1991
On Godunov-type methods near low densities journal February 1991
Quantifying uncertainties in general relativistic magnetohydrodynamic codes journal May 2023
IllinoisGRMHD: an open-source, user-friendly GRMHD code for dynamical spacetimes journal August 2015
Massively Parallel Simulations of Binary Black Hole Intermediate-Mass-Ratio Inspirals journal January 2019
Massively parallel simulations of binary black holes with adaptive wavelet multiresolution journal March 2023
Relativistic fluid disks in orbit around Kerr black holes journal July 1976
Implementation of Monte Carlo Transport in the General Relativistic SpEC Code journal October 2021
Numerical Simulations of Optically Thick Accretion onto a Black Hole. i. Spherical case journal June 2012
Numerical Simulations of Optically Thick Accretion onto a Black Hole. ii. Rotating flow journal October 2014
Waveform systematics in the gravitational-wave inference of tidal parameters and equation of state from binary neutron-star signals journal June 2021
HARM: A Numerical Scheme for General Relativistic Magnetohydrodynamics journal May 2003
An unsplit Godunov method for ideal MHD via constrained transport journal May 2005
An unsplit Godunov method for ideal MHD via constrained transport in three dimensions journal April 2008
The exact solution of the Riemann problem in relativistic magnetohydrodynamics journal August 2006
WhiskyMHD: a new numerical code for general relativistic magnetohydrodynamics journal June 2007
Can magnetic fields be detected during the inspiral of binary neutron stars? journal October 2009
High Order Strong Stability Preserving Time Discretizations journal September 2008
Parthenon—a performance portable block-structured adaptive mesh refinement framework journal December 2022
On Upstream Differencing and Godunov-Type Schemes for Hyperbolic Conservation Laws journal January 1983
Compact binary evolutions with the Z4c formulation journal October 2013
Pseudospectral method for gravitational wave collapse journal March 2016
Distinguishing the nature of comparable-mass neutron star binary systems with multimessenger observations: GW170817 case study journal September 2019
Gravitational Waves from Inspiraling Compact Binaries with Magnetic Dipole Moments journal July 2000
Global high-order numerical schemes for the time evolution of the general relativistic radiation magneto-hydrodynamics equations journal June 2023
AsterX: a new open-source GPU-accelerated GRMHD code for dynamical spacetimes journal December 2024
Robust recovery of primitive variables in relativistic ideal magnetohydrodynamics journal January 2021
SpECTRE: A task-based discontinuous Galerkin code for relativistic astrophysics journal April 2017
Implementation of advanced Riemann solvers in a neutrino-radiation magnetohydrodynamics code in numerical relativity and its application to a binary neutron star merger journal December 2022
A splitting method for numerical relativistic magnetohydrodynamics journal November 2024
A Godunov-type scheme for relativistic magnetohydrodynamics journal February 1999
First Order Quasilinear Equations in Several Independent Variables journal February 1970
Relativistic MHD simulations of extragalactic jets journal May 2005
Dissipation and Heating in Supersonic Hydrodynamic and mhd Turbulence journal February 2009
The Einstein Toolkit: a community computational infrastructure for relativistic astrophysics journal May 2012
Science case for the Einstein telescope journal March 2020
A Measurement of the Electromagnetic Luminosity of a Kerr Black Hole journal August 2004
An HLLC Riemann solver for relativistic flows – II. Magnetohydrodynamics journal April 2006
Beyond second-order convergence in simulations of magnetized binary neutron stars with realistic microphysics journal October 2019
Type II critical phenomena of neutron star collapse journal September 2008
Primitive Variable Solvers for Conservative General Relativistic Magnetohydrodynamics journal April 2006
Direct Calculation of the Radiative Efficiency of an Accretion disk Around a Black hole journal February 2009
Velocity-induced collapses of stable neutron stars journal September 2001
Large eddy simulations of magnetized mergers of neutron stars with neutrinos journal May 2022
Effects of the microphysical equation of state in the mergers of magnetized neutron stars with neutrino cooling journal August 2015
An operator-splitting numerical scheme for relativistic magnetohydrodynamics journal April 2024
High accuracy binary black hole simulations with an extended wave zone journal February 2011
The Event Horizon General Relativistic Magnetohydrodynamic Code Comparison Project journal August 2019
The black hole accretion code journal May 2017
Increasing the accuracy of binary neutron star simulations with an improved vacuum treatment journal November 2020
Gravitational waveform accuracy requirements for future ground-based detectors journal May 2020
Lessons for adaptive mesh refinement in numerical relativity journal June 2022
A new moment-based general-relativistic neutrino-radiation transport code: Methods and first applications to neutron star mergers journal March 2022
GW170817: Joint Constraint on the Neutron Star Equation of State from Multimessenger Observations journal January 2018
THC: a new high-order finite-difference high-resolution shock-capturing code for special-relativistic hydrodynamics journal October 2012
High-order fully general-relativistic hydrodynamics: new approaches and tests journal March 2014
Beyond second-order convergence in simulations of binary neutron stars in full general relativity journal October 2013
High-Order Numerical-Relativity Simulations of Binary Neutron Stars preprint January 2015
Adaptive mesh refinement in binary black holes simulations journal April 2024
Formation and Coalescence of Cosmological Supermassive-Black-Hole Binaries in Supermassive-Star Collapse journal October 2013
GW170817, general relativistic magnetohydrodynamic simulations, and the neutron star maximum mass journal January 2018
Numerical simulations of super-critical black hole accretion flows in general relativity journal January 2014
Semi-implicit scheme for treating radiation under M1 closure in general relativistic conservative fluid dynamics codes journal January 2013
GRaM-X: a new GPU-accelerated dynamical spacetime GRMHD code for Exascale computing with the Einstein Toolkit journal September 2023
Modeling GW170817 based on numerical relativity and its implications journal December 2017
Non-linear pulsations in differentially rotating neutron stars: mass-shedding-induced damping and splitting of the fundamental mode journal August 2004
The Athena++ Adaptive Mesh Refinement Framework: Design and Magnetohydrodynamic Solvers journal June 2020
The new discontinuous Galerkin methods based numerical relativity program Nmesh journal December 2022
Kokkos 3: Programming Model Extensions for the Exascale Era journal January 2021
Towards the ultimate conservative difference scheme. II. Monotonicity and conservation combined in a second-order scheme journal March 1974
Quasi-radial modes of rotating stars in general relativity journal April 2001
Solving the relativistic magnetohydrodynamics equations with ADER discontinuous Galerkin methods, a posteriori subcell limiting and adaptive mesh refinement journal July 2015
Binary neutron star merger simulations with neutrino transport and turbulent viscosity: impact of different schemes and grid resolution journal January 2023