Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Multi-GPU Implementation of a 3D Finite Difference Time Domain Earthquake Code on Heterogeneous Supercomputers

Journal Article · · Procedia Computer Science
 [1];  [2];  [1];  [2];  [3]
  1. Univ. of California, San Diego, CA (United States). San Diego Supercomputer Center; Univ. of California, San Diego, CA (United States). Dept. of Electrionic and Computer Engineering
  2. Univ. of California, San Diego, CA (United States). San Diego Supercomputer Center
  3. Univ. of California, San Diego, CA (United States). Dept. of Electrionic and Computer Engineering
+ Show Author Affiliations

We have developed a highly scalable 3D Finite Difference GPU code for use in earthquake engineering and disaster management through regional petascale earthquake simulations. This MPI-CUDA code is based on a widely-used wave propagation code called AWP-ODC and restructured for high throughput and efficiency on a heterogeneous computing architecture. We present an effective communication reduction technique for leveraging GPUs with minimal PCI-e overhead, and a novel overlapping method to fully hide data communication latency between GPUs. The optimization concept used in this work can be extended to general stencil computing on a structured grid. The benchmarks demonstrated sustained 100 TFlops in single precision for 49 billion mesh points using 952 GPUs on the NCCS Titan Phase 5 system, which is a 77-fold speedup compared to the CPU version of the code. This multi-GPU implementation has been validated and used for a large-scale verification wave propagation simulation of Mw5.4 Chino Hills earthquake using 128 GPUs.

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
Sponsoring Organization:
USDOE Office of Science (SC)
Grant/Contract Number:
AC05-00OR22725
OSTI ID:
1567324
Journal Information:
Procedia Computer Science, Journal Name: Procedia Computer Science Journal Issue: C Vol. 18; ISSN 1877-0509
Publisher:
ElsevierCopyright Statement
Country of Publication:
United States
Language:
English

References (15)

Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA journal May 2009
Hands-on Performance Tuning of 3D Finite Difference Earthquake Simulation on GPU Fermi Chipset journal January 2012
Modeling the propagation of elastic waves using spectral elements on a cluster of 192 GPUs journal April 2010
High-order finite-element seismic wave propagation modeling with MPI on a large GPU cluster journal October 2010
Porting a high-order finite-element earthquake modeling application to NVIDIA graphics cards using CUDA journal May 2009
Hands-on Performance Tuning of 3D Finite Difference Earthquake Simulation on GPU Fermi Chipset journal January 2012
Fast seismic modeling and Reverse Time Migration on a GPU cluster conference June 2009
Scalable Earthquake Simulation on Petascale Supercomputers
  • Cui, Yifeng; Olsen, Kim B.; Jordan, Thomas H.
  • 2010 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.45
conference November 2010
An 80-Fold Speedup, 15.0 TFlops Full GPU Acceleration of Non-Hydrostatic Weather Model ASUCA Production Code
  • Shimokawabe, Takashi; Aoki, Takayuki; Muroi, Chiashi
  • 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2010.9
conference November 2010
Accelerating a three-dimensional finite-difference wave propagation code using GPU graphics cards journal May 2010
3D finite difference computation on GPUs using CUDA conference January 2009
42 TFlops hierarchical N -body simulations on GPUs with applications in both astrophysics and turbulence conference January 2009
Peta-scale phase-field simulation for dendritic solidification on the TSUBAME 2.0 supercomputer
  • Shimokawabe, Takashi; Aoki, Takayuki; Takaki, Tomohiro
  • Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063388
conference January 2011
CUDA Implementation of a Navier-Stokes Solver on Multi-GPU Desktop Platforms for Incompressible Flows conference June 2012
Accelerating large-scale simulation of seismic wave propagation by multi-GPUs and three-dimensional domain decomposition journal December 2010

Cited By (1)

Alternating direction implicit time integrations for finite difference acoustic wave propagation: Parallelization and convergence text January 2020

Similar Records

Hands-on Performance Tuning of 3D Finite Difference Earthquake Simulation on GPU Fermi Chipset
Journal Article · 2012 · Procedia Computer Science · OSTI ID:1567289
+2 more
Scalable Earthquake Simulation on Petascale Supercomputers
Conference · 2010 · SC '10: Proceedings of the 2010 ACM/IEEE International Conference for High Performance Computing, Networking, Storage and Analysis; 13-19 Nov. 2010; New Orleans, LA, USA · OSTI ID:1567599
+10 more
Physics-based seismic hazard analysis on petascale heterogeneous supercomputers, In: SC '13 Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis
Conference · 2012 · 2013 INTERNATIONAL CONFERENCE FOR HIGH PERFORMANCE COMPUTING, NETWORKING, STORAGE AND ANALYSIS (SC) · OSTI ID:1567348
+11 more

Related Subjects

58 GEOSCIENCES
97 MATHEMATICS AND COMPUTING
CUDA-MPI
Computer Science
Earthquake Simulations
Enhanced Overlapping Design
Heterogeneous Supercomputers