GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials

Eisenbach, Markus; Larkin, Jeff; Lutjens, Justin; Rennich, Steven; Rogers, James H.

doi:10.1016/j.cpc.2016.07.013

Title: GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials

Abstract

The Locally Self-consistent Multiple Scattering (LSMS) code solves the first principles Density Functional theory Kohn–Sham equation for a wide range of materials with a special focus on metals, alloys and metallic nano-structures. It has traditionally exhibited near perfect scalability on massively parallel high performance computer architectures. In this paper, we present our efforts to exploit GPUs to accelerate the LSMS code to enable first principles calculations of O(100,000) atoms and statistical physics sampling of finite temperature properties. We reimplement the scattering matrix calculation for GPUs with a block matrix inversion algorithm that only uses accelerator memory. Finally, using the Cray XK7 system Titan at the Oak Ridge Leadership Computing Facility we achieve a sustained performance of 14.5PFlop/s and a speedup of 8.6 compared to the CPU only code.

Authors:

Eisenbach, Markus ^[1]; Larkin, Jeff ^[2]; Lutjens, Justin ^[2]; Rennich, Steven ^[2]; Rogers, James H. ^[1]

Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
NVIDIA Corporation, Santa Clara, CA (United States)

Publication Date:: Tue Jul 12 00:00:00 EDT 2016

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Contributing Org.:: NVIDIA Corporation, Santa Clara, CA (United States)

OSTI Identifier:: 1335344

Alternate Identifier(s):: OSTI ID: 1396465

Grant/Contract Number:: AC05-00OR22725

Resource Type:: Accepted Manuscript

Journal Name:: Computer Physics Communications

Additional Journal Information:: Journal Volume: 211; Journal ID: ISSN 0010-4655

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 36 MATERIALS SCIENCE; First-principles; Monte-Carlo; Phase transitions

Citation Formats


                    Eisenbach, Markus, Larkin, Jeff, Lutjens, Justin, Rennich, Steven, and Rogers, James H. GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials.  United States: N. p., 2016. 
Web.  doi:10.1016/j.cpc.2016.07.013.

Copy to clipboard


                    Eisenbach, Markus, Larkin, Jeff, Lutjens, Justin, Rennich, Steven, & Rogers, James H. GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials.  United States.  https://doi.org/10.1016/j.cpc.2016.07.013

Copy to clipboard


                    Eisenbach, Markus, Larkin, Jeff, Lutjens, Justin, Rennich, Steven, and Rogers, James H. Tue .  
"GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials".  United States.  https://doi.org/10.1016/j.cpc.2016.07.013.  https://www.osti.gov/servlets/purl/1335344.

Copy to clipboard


                    
@article{osti_1335344,

  title        = {GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials},

  author       = {Eisenbach, Markus and Larkin, Jeff and Lutjens, Justin and Rennich, Steven and Rogers, James H.},

  abstractNote = {The Locally Self-consistent Multiple Scattering (LSMS) code solves the first principles Density Functional theory Kohn–Sham equation for a wide range of materials with a special focus on metals, alloys and metallic nano-structures. It has traditionally exhibited near perfect scalability on massively parallel high performance computer architectures. In this paper, we present our efforts to exploit GPUs to accelerate the LSMS code to enable first principles calculations of O(100,000) atoms and statistical physics sampling of finite temperature properties. We reimplement the scattering matrix calculation for GPUs with a block matrix inversion algorithm that only uses accelerator memory. Finally, using the Cray XK7 system Titan at the Oak Ridge Leadership Computing Facility we achieve a sustained performance of 14.5PFlop/s and a speedup of 8.6 compared to the CPU only code.},

  doi          = {10.1016/j.cpc.2016.07.013},

  journal      = {Computer Physics Communications},

  number       = ,

  volume       = 211,

  place        = {United States},

  year         = {Tue Jul 12 00:00:00 EDT 2016},

  month        = {Tue Jul 12 00:00:00 EDT 2016}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.cpc.2016.07.013

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 12 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Inhomogeneous Electron Gas
journal, November 1964

Hohenberg, P.; Kohn, W.
Physical Review, Vol. 136, Issue 3B, p. B864-B871
DOI: 10.1103/PhysRev.136.B864

Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965

Kohn, W.; Sham, L. J.
Physical Review, Vol. 140, Issue 4A, p. A1133-A1138
DOI: 10.1103/PhysRev.140.A1133

Calculating condensed matter properties using the KKR-Green's function method—recent developments and applications
journal, August 2011

Ebert, H.; Ködderitzsch, D.; Minár, J.
Reports on Progress in Physics, Vol. 74, Issue 9
DOI: 10.1088/0034-4885/74/9/096501

Order- N Multiple Scattering Approach to Electronic Structure Calculations
journal, October 1995

Wang, Yang; Stocks, G. M.; Shelton, W. A.
Physical Review Letters, Vol. 75, Issue 15
DOI: 10.1103/PhysRevLett.75.2867

Magnetic anisotropy of monoatomic iron chains embedded in copper
journal, March 2002

Eisenbach, Markus; Györffy, Balazs L.; Stocks, G. Malcolm
Physical Review B, Vol. 65, Issue 14
DOI: 10.1103/PhysRevB.65.144424

On calculating the magnetic state of nanostructures
journal, February 2007

Stocks, G.; Eisenbach, M.; Ujfalussy, B.
Progress in Materials Science, Vol. 52, Issue 2-3
DOI: 10.1016/j.pmatsci.2006.10.015

Equation of State Calculations by Fast Computing Machines
journal, June 1953

Metropolis, Nicholas; Rosenbluth, Arianna W.; Rosenbluth, Marshall N.
The Journal of Chemical Physics, Vol. 21, Issue 6
DOI: 10.1063/1.1699114

First principles calculation of finite temperature magnetism in Fe and Fe ₃ C
journal, April 2011

Eisenbach, M.; Nicholson, D. M.; Rusanu, A.
Journal of Applied Physics, Vol. 109, Issue 7
DOI: 10.1063/1.3562218

First principles approach to the magneto caloric effect: Application toNi ₂ MnGa
journal, April 2011

Nicholson, D. M.; Odbadrakh, Kh.; Rusanu, A.
Journal of Applied Physics, Vol. 109, Issue 7
DOI: 10.1063/1.3562199

Density-functional Monte-Carlo simulation of CuZn order-disorder transition
journal, January 2016

Khan, S. N.; Eisenbach, Markus
Physical Review B, Vol. 93, Issue 2
DOI: 10.1103/PhysRevB.93.024203

Onsager cavity fields in itinerant-electron paramagnets
journal, July 1992

Staunton, J. B.; Gyorffy, B. L.
Physical Review Letters, Vol. 69, Issue 2
DOI: 10.1103/PhysRevLett.69.371

Massively parallel Wang–Landau sampling on multiple GPUs
journal, August 2012

Yin, Junqi; Landau, D. P.
Computer Physics Communications, Vol. 183, Issue 8
DOI: 10.1016/j.cpc.2012.02.023

Similar Records in DOE PAGES and OSTI.GOV collections:

GPU Acceleration of the Locally Selfconsistent Multiple Scattering Code for First Principles Calculation of the Ground State and Statistical Physics of Materials

Book Eisenbach, Markus ; Larkin, Jeff ; Lutjens, Justin ; ...

The Locally Self-consistent Multiple Scattering (LSMS) code solves the first principles Density Functional theory Kohn-Sham equation for a wide range of materials with a special focus on metals, alloys and metallic nano-structures. It has traditionally exhibited near perfect scalability on massively parallel high performance computer architectures. We present our efforts to exploit GPUs to accelerate the LSMS code to enable first principles calculations of O(100,000) atoms and statistical physics sampling of finite temperature properties. Using the Cray XK7 system Titan at the Oak Ridge Leadership Computing Facility we achieve a sustained performance of 14.5PFlop/s and a speedup of 8.6 comparedmore » to the CPU only code.« less
https://doi.org/10.1007/978-981-10-0457-5_24
GPU acceleration of a petascale application for turbulent mixing at high Schmidt number using OpenMP 4.5

Journal Article Clay, M. P. ; Buaria, D. ; Yeung, P. K. ; ... - Computer Physics Communications

This paper reports on the successful implementation of a massively parallel GPU-accelerated algorithm for the direct numerical simulation of turbulent mixing at high Schmidt number. The work stems from a recent development (Comput. Phys. Commun., vol. 219, 2017, 313–328), in which a low-communication algorithm was shown to attain high degrees of scalability on the Cray XE6 architecture when overlapping communication and computation via dedicated communication threads. An even higher level of performance has now been achieved using OpenMP 4.5 on the Cray XK7 architecture, where on each node the 16 integer cores of an AMD Interlagos processor share a singlemore »« less
https://doi.org/10.1016/j.cpc.2018.02.020
Nekbone performance on GPUs with OpenACC and CUDA Fortran implementations

Journal Article Gong, Jing ; Markidis, Stefano ; Laure, Erwin ; ... - Journal of Supercomputing

We present a hybrid GPU implementation and performance analysis of Nekbone, which represents one of the core kernels of the incompressible Navier–Stokes solver Nek5000. The implementation is based on OpenACC and CUDA Fortran for local parallelization of the compute-intensive matrix–matrix multiplication part, which significantly minimizes the modification of the existing CPU code while extending the simulation capability of the code to GPU architectures. Our discussion includes the GPU results of OpenACC interoperating with CUDA Fortran and the gather–scatter operations with GPUDirect communication. We demonstrate performance of up to 552 Tflops on 16, 384 GPUs of the OLCF Cray XK7 Titan.
https://doi.org/10.1007/s11227-016-1744-5
GPU acceleration of local and semilocal density functional calculations in the SPARC electronic structure code

Journal Article Sharma, Abhiraj ; Metere, Alfredo ; Suryanarayana, Phanish ; ... - Journal of Chemical Physics

We present a Graphics Processing Unit (GPU)-accelerated version of the real-space SPARC electronic structure code for performing Kohn–Sham density functional theory calculations within the local density and generalized gradient approximations. In particular, we develop a modular math-kernel based implementation for NVIDIA architectures wherein the computationally expensive operations are carried out on the GPUs, with the remainder of the workload retained on the central processing units (CPUs). Here, using representative bulk and slab examples, we show that relative to CPU-only execution, GPUs enable speedups of up to 6× and 60× in node and core hours, respectively, bringing time to solution downmore »« less
https://doi.org/10.1063/5.0147249
Porting the WAVEWATCH III (v6.07) wave action source terms to GPU

Journal Article Ikuyajolu, Olawale James ; Van Roekel, Luke ; Brus, Steven R. ; ... - Geoscientific Model Development (Online)

Surface gravity waves play a critical role in several processes, including mixing, coastal inundation, and surface fluxes. Despite the growing literature on the importance of ocean surface waves, wind–wave processes have traditionally been excluded from Earth system models (ESMs) due to the high computational costs of running spectral wave models. The development of the Next Generation Ocean Model for the DOE’s (Department of Energy) E3SM (Energy Exascale Earth System Model) Project partly focuses on the inclusion of a wave model, WAVEWATCH III (WW3), into E3SM. WW3, which was originally developed for operational wave forecasting, needs to be computationally less expensivemore »« less
https://doi.org/10.5194/gmd-16-1445-2023

Similar Records

Title: GPU acceleration of the Locally Selfconsistent Multiple Scattering code for first principles calculation of the ground state and statistical physics of materials

Abstract

Citation Formats

Inhomogeneous Electron Gas journal, November 1964

Self-Consistent Equations Including Exchange and Correlation Effects journal, November 1965

Calculating condensed matter properties using the KKR-Green's function method—recent developments and applications journal, August 2011

Order- N Multiple Scattering Approach to Electronic Structure Calculations journal, October 1995

Magnetic anisotropy of monoatomic iron chains embedded in copper journal, March 2002

On calculating the magnetic state of nanostructures journal, February 2007

Equation of State Calculations by Fast Computing Machines journal, June 1953

First principles calculation of finite temperature magnetism in Fe and Fe 3 C journal, April 2011

First principles approach to the magneto caloric effect: Application toNi 2 MnGa journal, April 2011

Density-functional Monte-Carlo simulation of CuZn order-disorder transition journal, January 2016

Onsager cavity fields in itinerant-electron paramagnets journal, July 1992

Massively parallel Wang–Landau sampling on multiple GPUs journal, August 2012

Inhomogeneous Electron Gas
journal, November 1964

Self-Consistent Equations Including Exchange and Correlation Effects
journal, November 1965

Calculating condensed matter properties using the KKR-Green's function method—recent developments and applications
journal, August 2011

Order- N Multiple Scattering Approach to Electronic Structure Calculations
journal, October 1995

Magnetic anisotropy of monoatomic iron chains embedded in copper
journal, March 2002

On calculating the magnetic state of nanostructures
journal, February 2007

Equation of State Calculations by Fast Computing Machines
journal, June 1953

First principles calculation of finite temperature magnetism in Fe and Fe ₃ C
journal, April 2011

First principles approach to the magneto caloric effect: Application toNi ₂ MnGa
journal, April 2011

Density-functional Monte-Carlo simulation of CuZn order-disorder transition
journal, January 2016

Onsager cavity fields in itinerant-electron paramagnets
journal, July 1992

Massively parallel Wang–Landau sampling on multiple GPUs
journal, August 2012