Hierarchical Krylov and nested Krylov methods for extreme-scale computing

Journal Article · Thu Oct 17 00:00:00 EDT 2013 · Parallel Computing

DOI:https://doi.org/10.1016/j.parco.2013.10.001· OSTI ID:1565143

McInnes, Lois Curfman ^[1]; Smith, Barry ^[1]; Zhang, Hong ^[2]; Mills, Richard Tran ^[3]

---

1. Argonne National Lab. (ANL), Argonne, IL (United States)
2. Argonne National Lab. (ANL), Argonne, IL (United States); Illinois Inst. of Technology, Chicago, IL (United States)
3. Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States); Univ. of Tennessee, Knoxville, TN (United States)

The solution of large, sparse linear systems is typically a dominant phase of computation for simulations based on partial differential equations, which are ubiquitous in scientific and engineering applications. While preconditioned Krylov methods are widely used and provide many advantages for solving sparse linear systems that do not have highly convergent, geometric multigrid solvers or specialized fast solvers, Krylov methods encounter well-known scaling difficulties for over 10,000 processor cores because each iteration requires at least one vector inner product, which in turn requires a global synchronization that scales poorly because of internode latency. To aid in overcoming these difficulties, we have developed hierarchical Krylov methods and nested Krylov methods in the PETSc library that reduce the number of global inner products required across the entire system (where they are expensive), though freely allow vector inner products across smaller subsets of the entire system (where they are inexpensive) or use inner iterations that do not invoke vector inner products at all. Nested Krylov methods are a generalization of inner-outer iterative methods with two or more layers. Hierarchical Krylov methods are a generalization of block Jacobi and overlapping additive Schwarz methods, where each block itself is solved by Krylov methods on smaller blocks. Conceptually, the hierarchy can continue recursively to an arbitrary number of levels of smaller and smaller blocks. As a specific case, we introduce the hierarchical FGMRES method, or h-FGMRES, and we demonstrate the impact of two-level h-FGMRES with a variable preconditioner on the PFLOTRAN subsurface flow application. We also demonstrate the impact of nested FGMRES, BiCGStab and Chebyshev methods. These hierarchical Krylov methods and nested Krylov methods significantly reduced overall PFLOTRAN simulation time on the Cray XK6 when using 10,000 through 224,000 cores through the combined effects of reduced global synchronization due to fewer global inner products and stronger inner hierarchical or nested preconditioners.

• Research Organization: Argonne National Laboratory (ANL), Argonne, IL (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); Office of Science (SC), Biological and Environmental Research (BER). Earth and Environmental Systems Science Division
• Grant/Contract Number: AC02-05CH11231; AC02-06CH11357; AC05-00OR22725
• OSTI ID: 1565143
• Journal Information: Parallel Computing, Vol. 40, Issue 1; ISSN 0167-8191
• Publisher: Elsevier
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Hierarchical
Nested
Krylov methods
Variable preconditioner