DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers

Abstract

We have developed a highly scalable 3D Finite Difference GPU code for use in earthquake engineering and disaster management through regional petascale earthquake simulations. This MPI-CUDA code is based on a widely-used wave propagation code called AWP-ODC and restructured for high throughput and efficiency on a heterogeneous computing architecture. We present an effective communication reduction technique for leveraging GPUs with minimal PCI-e overhead, and a novel overlapping method to fully hide data communication latency between GPUs. The optimization concept used in this work can be extended to general stencil computing on a structured grid. The benchmarks demonstrated sustained 100 TFlops in single precision for 49 billion mesh points using 952 GPUs on the NCCS Titan Phase 5 system, which is a 77-fold speedup compared to the CPU version of the code. This multi-GPU implementation has been validated and used for a large-scale verification wave propagation simulation of Mw5.4 Chino Hills earthquake using 128 GPUs.

Authors:
 [1];  [2];  [1];  [2];  [3]
  1. Univ. of California, San Diego, CA (United States). San Diego Supercomputer Center; Univ. of California, San Diego, CA (United States). Dept. of Electrionic and Computer Engineering
  2. Univ. of California, San Diego, CA (United States). San Diego Supercomputer Center
  3. Univ. of California, San Diego, CA (United States). Dept. of Electrionic and Computer Engineering
Publication Date:
Research Org.:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1567324
Grant/Contract Number:  
AC05-00OR22725
Resource Type:
Accepted Manuscript
Journal Name:
Procedia Computer Science
Additional Journal Information:
Journal Volume: 18; Journal Issue: C; Journal ID: ISSN 1877-0509
Publisher:
Elsevier
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; 58 GEOSCIENCES; Computer Science; Earthquake Simulations; CUDA-MPI; Enhanced Overlapping Design; Heterogeneous Supercomputers

Citation Formats

Zhou, Jun, Cui, Yifeng, Poyraz, Efecan, Choi, Dong Ju, and Guest, Clark C. Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers. United States: N. p., 2013. Web. doi:10.1016/j.procs.2013.05.292.
Zhou, Jun, Cui, Yifeng, Poyraz, Efecan, Choi, Dong Ju, & Guest, Clark C. Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers. United States. https://doi.org/10.1016/j.procs.2013.05.292
Zhou, Jun, Cui, Yifeng, Poyraz, Efecan, Choi, Dong Ju, and Guest, Clark C. Sat . "Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers". United States. https://doi.org/10.1016/j.procs.2013.05.292. https://www.osti.gov/servlets/purl/1567324.
@article{osti_1567324,
title = {Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers},
author = {Zhou, Jun and Cui, Yifeng and Poyraz, Efecan and Choi, Dong Ju and Guest, Clark C.},
abstractNote = {We have developed a highly scalable 3D Finite Difference GPU code for use in earthquake engineering and disaster management through regional petascale earthquake simulations. This MPI-CUDA code is based on a widely-used wave propagation code called AWP-ODC and restructured for high throughput and efficiency on a heterogeneous computing architecture. We present an effective communication reduction technique for leveraging GPUs with minimal PCI-e overhead, and a novel overlapping method to fully hide data communication latency between GPUs. The optimization concept used in this work can be extended to general stencil computing on a structured grid. The benchmarks demonstrated sustained 100 TFlops in single precision for 49 billion mesh points using 952 GPUs on the NCCS Titan Phase 5 system, which is a 77-fold speedup compared to the CPU version of the code. This multi-GPU implementation has been validated and used for a large-scale verification wave propagation simulation of Mw5.4 Chino Hills earthquake using 128 GPUs.},
doi = {10.1016/j.procs.2013.05.292},
journal = {Procedia Computer Science},
number = C,
volume = 18,
place = {United States},
year = {2013},
month = {6}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 14 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Hands-on Performance Tuning of 3D Finite Difference Earthquake Simulation on GPU Fermi Chipset
journal, January 2012


Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA
journal, May 2009

  • Komatitsch, Dimitri; Michéa, David; Erlebacher, Gordon
  • Journal of Parallel and Distributed Computing, Vol. 69, Issue 5
  • DOI: 10.1016/j.jpdc.2009.01.006

Scalable Earthquake Simulation on Petascale Supercomputers
conference, November 2010

  • Cui, Yifeng; Olsen, Kim B.; Jordan, Thomas H.
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2010.45

CUDA Implementation of a Navier-Stokes Solver on Multi-GPU Desktop Platforms for Incompressible Flows
conference, June 2012

  • Thibault, Julien; Senocak, Inanc
  • 47th AIAA Aerospace Sciences Meeting including The New Horizons Forum and Aerospace Exposition
  • DOI: 10.2514/6.2009-758

42 TFlops hierarchical N -body simulations on GPUs with applications in both astrophysics and turbulence
conference, January 2009

  • Hamada, Tsuyoshi; Narumi, Tetsu; Yokota, Rio
  • Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09
  • DOI: 10.1145/1654059.1654123

An 80-Fold Speedup, 15.0 TFlops Full GPU Acceleration of Non-Hydrostatic Weather Model ASUCA Production Code
conference, November 2010

  • Shimokawabe, Takashi; Aoki, Takayuki; Muroi, Chiashi
  • 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis
  • DOI: 10.1109/SC.2010.9

Peta-scale phase-field simulation for dendritic solidification on the TSUBAME 2.0 supercomputer
conference, January 2011

  • Shimokawabe, Takashi; Aoki, Takayuki; Takaki, Tomohiro
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11
  • DOI: 10.1145/2063384.2063388

3D finite difference computation on GPUs using CUDA
conference, January 2009

  • Micikevicius, Paulius
  • Proceedings of 2nd Workshop on General Purpose Processing on Graphics Processing Units - GPGPU-2
  • DOI: 10.1145/1513895.1513905

Fast seismic modeling and Reverse Time Migration on a GPU cluster
conference, June 2009

  • Abdelkhalek, Rached; Calandra, Henri; Coulaud, Olivier
  • Simulation (HPCS), 2009 International Conference on High Performance Computing & Simulation
  • DOI: 10.1109/HPCSIM.2009.5192786

Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA
journal, May 2009

  • Komatitsch, Dimitri; Michéa, David; Erlebacher, Gordon
  • Journal of Parallel and Distributed Computing, Vol. 69, Issue 5
  • DOI: 10.1016/j.jpdc.2009.01.006

High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster
journal, October 2010

  • Komatitsch, Dimitri; Erlebacher, Gordon; Göddeke, Dominik
  • Journal of Computational Physics, Vol. 229, Issue 20
  • DOI: 10.1016/j.jcp.2010.06.024

Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs
journal, April 2010

  • Komatitsch, Dimitri; Göddeke, Dominik; Erlebacher, Gordon
  • Computer Science - Research and Development, Vol. 25, Issue 1-2
  • DOI: 10.1007/s00450-010-0109-1

Accelerating a three-dimensional finite-difference wave propagation code using GPU graphics cards
journal, May 2010


Accelerating large-scale simulation of seismic wave propagation by multi-GPUs and three-dimensional domain decomposition
journal, December 2010

  • Okamoto, Taro; Takenaka, Hiroshi; Nakamura, Takeshi
  • Earth, Planets and Space, Vol. 62, Issue 12
  • DOI: 10.5047/eps.2010.11.009

Hands-on Performance Tuning of 3D Finite Difference Earthquake Simulation on GPU Fermi Chipset
journal, January 2012