Hierarchical Roofline analysis for GPUs: Accelerating performance optimization for the NERSC‐9 Perlmutter system

Journal Article · 12 November 2019 · Concurrency and Computation. Practice and Experience

DOI: https://doi.org/10.1002/cpe.5547 · OSTI ID:1574050

Yang, Charlene  ^[1]; Kurth, Thorsten  ^[1]; Williams, Samuel ^[2]

---

1. National Energy Research Scientific Computing Center (NERSC) Lawrence Berkeley National Laboratory Berkeley California
2. Computational Research Division (CRD) Lawrence Berkeley National Laboratory Berkeley California

Summary The Roofline performance model provides an intuitive and insightful approach to identifying performance bottlenecks and guiding performance optimization. In preparation for the next‐generation supercomputer Perlmutter at NERSC, this paper presents a methodology to construct a hierarchical Roofline on NVIDIA GPUs and extends it to support reduced precision and Tensor Cores. The hierarchical Roofline incorporates L1, L2, device memory, and system memory bandwidths into one single figure, and it offers more profound insights into performance analysis than the traditional DRAM‐only Roofline. We use our Roofline methodology to analyze three proxy applications: GPP from BerkeleyGW, HPGMG from AMReX, and conv2d from TensorFlow. In doing so, we demonstrate the ability of our methodology to readily understand various aspects of performance and performance bottlenecks on NVIDIA GPUs and motivate code optimizations.

• Sponsoring Organization: USDOE
• Grant/Contract Number: AC02-05CH11231
• OSTI ID: 1574050
• Journal Information: Concurrency and Computation. Practice and Experience, Journal Name: Concurrency and Computation. Practice and Experience Vol. 32 Journal Issue: 20; ISSN 1532-0626
• Publisher: Wiley Blackwell (John Wiley & Sons)Copyright Statement
• Country of Publication: United Kingdom
• Language: English