skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code

Abstract

We present a new code for astrophysical magnetohydrodynamics specifically designed and optimized for high performance and scaling on modern and future supercomputers. We describe a novel hybrid OpenMP/MPI programming model that emerged from a collaboration between Cray, Inc. and the University of Minnesota. This design utilizes MPI-RMA optimized for thread scaling, which allows the code to run extremely efficiently at very high thread counts ideal for the latest generation of multi-core and many-core architectures. Such performance characteristics are needed in the era of “exascale” computing. We describe and demonstrate our high-performance design in detail with the intent that it may be used as a model for other, future astrophysical codes intended for applications demanding exceptional performance.

Authors:
; ;  [1];  [2]; ; ; ;  [3];  [4]
  1. Cray Inc., St. Paul, MN 55101 (United States)
  2. Minnesota Supercomputing Institute for Advanced Computational Research, Minneapolis, MN USA (United States)
  3. School of Physics and Astronomy, University of Minnesota, Minneapolis, MN 55455 (United States)
  4. Institute for Theory and Computation, Center for Astrophysics, Harvard University, Cambridge, MA 02138 (United States)
Publication Date:
OSTI Identifier:
22661346
Resource Type:
Journal Article
Resource Relation:
Journal Name: Astrophysical Journal, Supplement Series; Journal Volume: 228; Journal Issue: 2; Other Information: Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICAL METHODS AND COMPUTING; 79 ASTROPHYSICS, COSMOLOGY AND ASTRONOMY; ASTROPHYSICS; COMPUTER ARCHITECTURE; DEMAND; DESIGN; MAGNETOHYDRODYNAMICS; MINNESOTA; PERFORMANCE; PROGRAMMING; SUPERCOMPUTERS

Citation Formats

Mendygral, P. J., Radcliffe, N., Kandalla, K., Porter, D., O’Neill, B. J., Nolting, C., Donnert, J. M. F., Jones, T. W., and Edmon, P., E-mail: pjm@cray.com, E-mail: nradclif@cray.com, E-mail: kkandalla@cray.com, E-mail: oneill@astro.umn.edu, E-mail: nolt0040@umn.edu, E-mail: donnert@ira.inaf.it, E-mail: twj@umn.edu, E-mail: dhp@umn.edu, E-mail: pedmon@cfa.harvard.edu. WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code. United States: N. p., 2017. Web. doi:10.3847/1538-4365/AA5B9C.
Mendygral, P. J., Radcliffe, N., Kandalla, K., Porter, D., O’Neill, B. J., Nolting, C., Donnert, J. M. F., Jones, T. W., & Edmon, P., E-mail: pjm@cray.com, E-mail: nradclif@cray.com, E-mail: kkandalla@cray.com, E-mail: oneill@astro.umn.edu, E-mail: nolt0040@umn.edu, E-mail: donnert@ira.inaf.it, E-mail: twj@umn.edu, E-mail: dhp@umn.edu, E-mail: pedmon@cfa.harvard.edu. WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code. United States. doi:10.3847/1538-4365/AA5B9C.
Mendygral, P. J., Radcliffe, N., Kandalla, K., Porter, D., O’Neill, B. J., Nolting, C., Donnert, J. M. F., Jones, T. W., and Edmon, P., E-mail: pjm@cray.com, E-mail: nradclif@cray.com, E-mail: kkandalla@cray.com, E-mail: oneill@astro.umn.edu, E-mail: nolt0040@umn.edu, E-mail: donnert@ira.inaf.it, E-mail: twj@umn.edu, E-mail: dhp@umn.edu, E-mail: pedmon@cfa.harvard.edu. Wed . "WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code". United States. doi:10.3847/1538-4365/AA5B9C.
@article{osti_22661346,
title = {WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code},
author = {Mendygral, P. J. and Radcliffe, N. and Kandalla, K. and Porter, D. and O’Neill, B. J. and Nolting, C. and Donnert, J. M. F. and Jones, T. W. and Edmon, P., E-mail: pjm@cray.com, E-mail: nradclif@cray.com, E-mail: kkandalla@cray.com, E-mail: oneill@astro.umn.edu, E-mail: nolt0040@umn.edu, E-mail: donnert@ira.inaf.it, E-mail: twj@umn.edu, E-mail: dhp@umn.edu, E-mail: pedmon@cfa.harvard.edu},
abstractNote = {We present a new code for astrophysical magnetohydrodynamics specifically designed and optimized for high performance and scaling on modern and future supercomputers. We describe a novel hybrid OpenMP/MPI programming model that emerged from a collaboration between Cray, Inc. and the University of Minnesota. This design utilizes MPI-RMA optimized for thread scaling, which allows the code to run extremely efficiently at very high thread counts ideal for the latest generation of multi-core and many-core architectures. Such performance characteristics are needed in the era of “exascale” computing. We describe and demonstrate our high-performance design in detail with the intent that it may be used as a model for other, future astrophysical codes intended for applications demanding exceptional performance.},
doi = {10.3847/1538-4365/AA5B9C},
journal = {Astrophysical Journal, Supplement Series},
number = 2,
volume = 228,
place = {United States},
year = {Wed Feb 01 00:00:00 EST 2017},
month = {Wed Feb 01 00:00:00 EST 2017}
}
  • A method for implementing cylindrical coordinates in the Athena magnetohydrodynamics (MHD) code is described. The extension follows the approach of Athena's original developers and has been designed to alter the existing Cartesian-coordinates code as minimally and transparently as possible. The numerical equations in cylindrical coordinates are formulated to maintain consistency with constrained transport (CT), a central feature of the Athena algorithm, while making use of previously implemented code modules such as the Riemann solvers. Angular momentum transport, which is critical in astrophysical disk systems dominated by rotation, is treated carefully. We describe modifications for cylindrical coordinates of the higher-order spatialmore » reconstruction and characteristic evolution steps as well as the finite-volume and CT updates. Finally, we present a test suite of standard and novel problems in one, two, and three dimensions designed to validate our algorithms and implementation and to be of use to other code developers. The code is suitable for use in a wide variety of astrophysical applications and is freely available for download on the Web.« less
  • FISH is a fast and simple ideal magnetohydrodynamics code that scales to {approx}10,000 processes for a Cartesian computational domain of {approx}1000{sup 3} cells. The simplicity of FISH has been achieved by the rigorous application of the operator splitting technique, while second-order accuracy is maintained by the symmetric ordering of the operators. Between directional sweeps, the three-dimensional data are rotated in memory so that the sweep is always performed in a cache-efficient way along the direction of contiguous memory. Hence, the code only requires a one-dimensional description of the conservation equations to be solved. This approach also enables an elegant novelmore » parallelization of the code that is based on persistent communications with MPI for cubic domain decomposition on machines with distributed memory. This scheme is then combined with an additional OpenMP parallelization of different sweeps that can take advantage of clusters of shared memory. We document the detailed implementation of a second-order total variation diminishing advection scheme based on flux reconstruction. The magnetic fields are evolved by a constrained transport scheme. We show that the subtraction of a simple estimate of the hydrostatic gradient from the total gradients can significantly reduce the dissipation of the advection scheme in simulations of gravitationally bound hydrostatic objects. Through its simplicity and efficiency, FISH is as well suited for hydrodynamics classes as for large-scale astrophysical simulations on high-performance computer clusters. In preparation for the release of a public version, we demonstrate the performance of FISH in a suite of astrophysically orientated test cases.« less
  • This work presents the publicly available moving-mesh magnetohydrodynamics (MHD) code DISCO. DISCO is efficient and accurate at evolving orbital fluid motion in two and three dimensions, especially at high Mach numbers. DISCO employs a moving-mesh approach utilizing a dynamic cylindrical mesh that can shear azimuthally to follow the orbital motion of the gas. The moving mesh removes diffusive advection errors and allows for longer time-steps than a static grid. MHD is implemented in DISCO using an HLLD Riemann solver and a novel constrained transport (CT) scheme that is compatible with the mesh motion. DISCO is tested against a wide varietymore » of problems, which are designed to test its stability, accuracy, and scalability. In addition, several MHD tests are performed which demonstrate the accuracy and stability of the new CT approach, including two tests of the magneto-rotational instability, one testing the linear growth rate and the other following the instability into the fully turbulent regime.« less
  • We continue our presentation of VINE. In this paper, we begin with a description of relevant architectural properties of the serial and shared memory parallel computers on which VINE is intended to run, and describe their influences on the design of the code itself. We continue with a detailed description of a number of optimizations made to the layout of the particle data in memory and to our implementation of a binary tree used to access that data for use in gravitational force calculations and searches for smoothed particle hydrodynamics (SPH) neighbor particles. We describe the modifications to the codemore » necessary to obtain forces efficiently from special purpose 'GRAPE' hardware, the interfaces required to allow transparent substitution of those forces in the code instead of those obtained from the tree, and the modifications necessary to use both tree and GRAPE together as a fused GRAPE/tree combination. We conclude with an extensive series of performance tests, which demonstrate that the code can be run efficiently and without modification in serial on small workstations or in parallel using the OpenMP compiler directives on large-scale, shared memory parallel machines. We analyze the effects of the code optimizations and estimate that they improve its overall performance by more than an order of magnitude over that obtained by many other tree codes. Scaled parallel performance of the gravity and SPH calculations, together the most costly components of most simulations, is nearly linear up to at least 120 processors on moderate sized test problems using the Origin 3000 architecture, and to the maximum machine sizes available to us on several other architectures. At similar accuracy, performance of VINE, used in GRAPE-tree mode, is approximately a factor 2 slower than that of VINE, used in host-only mode. Further optimizations of the GRAPE/host communications could improve the speed by as much as a factor of 3, but have not yet been implemented in VINE. Finally, we find that although parallel performance on small problems may reach a plateau beyond which more processors bring no additional speedup, performance never decreases, a factor important for running large simulations on many processors with individual time steps, where only a small fraction of the total particles require updates at any given moment.« less
  • Long-length, high-temperature superconducting (HTS) wires capable of carrying high critical current, Ic, are required for a wide range of applications. Here, we report extremely high performance HTS wires based on 5 m thick SmBa2Cu3O7- (SmBCO) single layer films on textured metallic templates. SmBCO layer wires over 20 meters long were deposited by a cost-effective, scalable co-evaporation process using a batch-type drum in a dual chamber. All deposition parameters influencing the composition, phase, and texture of the films were optimized via a unique combinatorial method that is broadly applicable for co-evaporation of other promising complex materials containing several cations. Thick SmBCOmore » layers deposited under optimized conditions exhibit excellent cube-on-cube epitaxy. Such excellent structural epitaxy over the entire thickness results in exceptionally high Ic performance, with average Ic over 1000 A/cm for the entire 22 meter long wire and maximum Ic over 1,500 A/cm for a short 12 cm long tape. The Ic values reported in this work are the highest values ever reported from any lengths of cuprate-based HTS wire or conductor.« less