Parallel performance optimizations on unstructured mesh-based simulations

Journal Article · Mon Jun 01 04:00:00 UTC 2015 · Procedia Computer Science

DOI: https://doi.org/10.1016/j.procs.2015.05.466 · OSTI ID:1202396

Sarje, Abhinav ^[1]; Song, Sukhyun ^[2]; Jacobsen, Douglas ^[3]; Huck, Kevin ^[4]; Hollingsworth, Jeffrey ^[2]; Malony, Allen ^[4]; Williams, Samuel ^[1]; Oliker, Leonid ^[1]

---

1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
2. Univ. of Maryland, College Park (United States)
3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
4. Univ. of Oregon, Eugene, OR (United States)

This paper addresses two key parallelization challenges the unstructured mesh-based ocean modeling code, MPAS-Ocean, which uses a mesh based on Voronoi tessellations: (1) load imbalance across processes, and (2) unstructured data access patterns, that inhibit intra- and inter-node performance. Our work analyzes the load imbalance due to naive partitioning of the mesh, and develops methods to generate mesh partitioning with better load balance and reduced communication. Furthermore, we present methods that minimize both inter- and intranode data movement and maximize data reuse. Our techniques include predictive ordering of data elements for higher cache efficiency, as well as communication reduction approaches. We present detailed performance data when running on thousands of cores using the Cray XC30 supercomputer and show that our optimization strategies can exceed the original performance by over 2×. Additionally, many of these solutions can be broadly applied to a wide variety of unstructured grid-based computations.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)
• Grant/Contract Number: AC02-05CH11231; SC0006723
• OSTI ID: 1202396
• Journal Information: Procedia Computer Science, Journal Name: Procedia Computer Science Journal Issue: C Vol. 51; ISSN 1877-0509
• Publisher: ElsevierCopyright Statement
• Country of Publication: United States
• Language: English