skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Terascale Optimal PDE Simulations (TOPS) Center

Abstract

This report covers the period from Oct. 2002 to Sep. 2004 when Old Dominion University (ODU) was the lead institution for the TOPS ISIC, until in Oct. 2004 Columbia University replaced ODU as the lead institution. The TOPS members from ODU focused on various aspects of the linear and nonlinear solver infrastructure required by the partial differential equations simulation codes, working directly with SciDAC teams from the Fusion Energy Sciences program: the Center for Extended agnetohydrodynamic Modeling (CEMM) at Princeton, and with the Center for Magnetic Reconnection Studies (CMRS) at University of New Hampshire. With CEMM we worked with their MHD simulation code, called M3D, which is semi-implicit, requiring linear solves but no onlinear solves. We contributed several improvements to their current semi-implicit code. Among these was the use of multilevel reconditioning, which provides optimal scaling. This was done through the multigrid preconditioner available in Hypre, another major solver package available in TOPS. We also provided them direct solver functionality for their linear solves since they may be required for more accurate solutions in some regimes. With the CMRS group, we implemented a fully implicit parallel magnetic reconnection simulation code, built on top of PETSc. Our first attempt was amore » Krylov linear iteration (GMRES because of the lack of symmetry), within each nonlinear (Newton) iteration, with optimal multilevel preconditioning, using the geometric multigrid preconditioner from PETSc. However, for reasons that we have not yet fully understood, the multigrid preconditioner fails early in the simulation, breaking the outer Newton iteration. Much better results were obtained after switching from optimal multilevel preconditioning to suboptimal one level preconditioning. Our current code, based on the additive Schwartz preconditioner from in PETSc, with ILU on subdomains, scales reasonably well, while matching the output of the original explicit code. The new Newton-Krylov-Schwarz implicitcode can take time-steps that are hundreds or thousands of times larger than the explicit code. During the three year period of this grant, we published thirteen papers and gave several invited talks at international conferences. Work on these TOPS projects continues with Columbia University as lead until Sep. 2006.« less

Authors:
Publication Date:
Research Org.:
Old Dominion University Research Foundation
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
890547
Report Number(s):
DOE/ER/25476-3 Final Report
ODURF313401
DOE Contract Number:
FC02-01ER25476
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; 01 COAL, LIGNITE, AND PEAT

Citation Formats

Pothen, Alex. Terascale Optimal PDE Simulations (TOPS) Center. United States: N. p., 2006. Web. doi:10.2172/890547.
Copy to clipboard
Pothen, Alex. Terascale Optimal PDE Simulations (TOPS) Center. United States. doi:10.2172/890547.
Copy to clipboard
Pothen, Alex. Wed . "Terascale Optimal PDE Simulations (TOPS) Center". United States. doi:10.2172/890547. https://www.osti.gov/servlets/purl/890547.
Copy to clipboard
@article{osti_890547,
title = {Terascale Optimal PDE Simulations (TOPS) Center},
author = {Pothen, Alex},
abstractNote = {This report covers the period from Oct. 2002 to Sep. 2004 when Old Dominion University (ODU) was the lead institution for the TOPS ISIC, until in Oct. 2004 Columbia University replaced ODU as the lead institution. The TOPS members from ODU focused on various aspects of the linear and nonlinear solver infrastructure required by the partial differential equations simulation codes, working directly with SciDAC teams from the Fusion Energy Sciences program: the Center for Extended agnetohydrodynamic Modeling (CEMM) at Princeton, and with the Center for Magnetic Reconnection Studies (CMRS) at University of New Hampshire. With CEMM we worked with their MHD simulation code, called M3D, which is semi-implicit, requiring linear solves but no onlinear solves. We contributed several improvements to their current semi-implicit code. Among these was the use of multilevel reconditioning, which provides optimal scaling. This was done through the multigrid preconditioner available in Hypre, another major solver package available in TOPS. We also provided them direct solver functionality for their linear solves since they may be required for more accurate solutions in some regimes. With the CMRS group, we implemented a fully implicit parallel magnetic reconnection simulation code, built on top of PETSc. Our first attempt was a Krylov linear iteration (GMRES because of the lack of symmetry), within each nonlinear (Newton) iteration, with optimal multilevel preconditioning, using the geometric multigrid preconditioner from PETSc. However, for reasons that we have not yet fully understood, the multigrid preconditioner fails early in the simulation, breaking the outer Newton iteration. Much better results were obtained after switching from optimal multilevel preconditioning to suboptimal one level preconditioning. Our current code, based on the additive Schwartz preconditioner from in PETSc, with ILU on subdomains, scales reasonably well, while matching the output of the original explicit code. The new Newton-Krylov-Schwarz implicitcode can take time-steps that are hundreds or thousands of times larger than the explicit code. During the three year period of this grant, we published thirteen papers and gave several invited talks at international conferences. Work on these TOPS projects continues with Columbia University as lead until Sep. 2006.},
doi = {10.2172/890547},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Wed Aug 23 00:00:00 EDT 2006},
month = {Wed Aug 23 00:00:00 EDT 2006}
}
Copy to clipboard
Technical Report:
View Technical Report (0.02 MB)
DOI:  10.2172/890547
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

  • Our work has focused on the development and analysis of domain decomposition algorithms for a variety of problems arising in continuum mechanics modeling. In particular, we have extended and analyzed FETI-DP and BDDC algorithms; these iterative solvers were first introduced and studied by Charbel Farhat and his collaborators, see [11, 45, 12], and by Clark Dohrmann of SANDIA, Albuquerque, see [43, 2, 1], respectively. These two closely related families of methods are of particular interest since they are used more extensively than other iterative substructuring methods to solve very large and difficult problems. Thus, the FETI algorithms are part ofmore » the SALINAS system developed by the SANDIA National Laboratories for very large scale computations, and as already noted, BDDC was first developed by a SANDIA scientist, Dr. Clark Dohrmann. The FETI algorithms are also making inroads in commercial engineering software systems. We also note that the analysis of these algorithms poses very real mathematical challenges. The success in developing this theory has, in several instances, led to significant improvements in the performance of these algorithms. A very desirable feature of these iterative substructuring and other domain decomposition algorithms is that they respect the memory hierarchy of modern parallel and distributed computing systems, which is essential for approaching peak floating point performance. The development of improved methods, together with more powerful computer systems, is making it possible to carry out simulations in three dimensions, with quite high resolution, relatively easily. This work is supported by high quality software systems, such as Argonne's PETSc library, which facilitates code development as well as the access to a variety of parallel and distributed computer systems. The success in finding scalable and robust domain decomposition algorithms for very large number of processors and very large finite element problems is, e.g., illustrated in [24, 25, 26]. This work is based on [29, 31]. Our work over these five and half years has, in our opinion, helped advance the knowledge of domain decomposition methods significantly. We see these methods as providing valuable alternatives to other iterative methods, in particular, those based on multi-grid. In our opinion, our accomplishments also match the goals of the TOPS project quite closely.« less

  • In many areas of science, physical experimentation may be too dangerous, too expensive or even impossible. Instead, large-scale simulations, validated by comparison with related experiments in well-understood laboratory contexts, are used by scientists to gain insight and confirmation of existing theories in such areas, without benefit of full experimental verification. The goal of the TOPS ISIC was to develop and implement algorithms and support scientific investigations performed by DOE-sponsored researchers. A major component of this effort is to provide software for large scale parallel computers capable of efficiently solving the enormous systems of equations arising from the nonlinear PDEs underlyingmore » these simulations. Several TOPS supported packages where designed in part (ScaLAPACK) or in whole (SuperLU) at Berkeley, and are widely used beyond SciDAC and DOE. Beyond continuing to develop these codes, our main effort focused on automatic performance tuning of the sparse matrix kernels (eg sparse-matrix-vector-multiply, or SpMV) at the core of many TOPS iterative solvers. Based on the observation that the fastest implementation of SpMV (and other kernels) can depend dramatically both on the computer and the matrix (the latter of which is not known until run-time), we developed and released a system called OSKI (Optimized Sparse Kernel Interface) that will automatically produce optimized version of SpMV (and other kernels), hiding complicated implementation details from the user. OSKI led to a 2x speedup in SpMV in a DOE accelerator design code, a 2x speedup in a commercial lithography simulation, and has been downloaded over 500 times. In addition to a stand-alone version, OSKI was also integrated into the TOPS-supported PETSc system.« less

  • Multiscale, multirate scientific and engineering applications in the SciDAC portfolio possess resolution requirements that are practically inexhaustible and demand execution on the highest-capability computers available, which will soon reach the petascale. While the variety of applications is enormous, their needs for mathematical software infrastructure are surprisingly coincident; moreover the chief bottleneck is often the solver. At their current scalability limits, many applications spend a vast majority of their operations in solvers, due to solver algorithmic complexity that is superlinear in the problem size, whereas other phases scale linearly. Furthermore, the solver may be the phase of the simulation with themore » poorest parallel scalability, due to intrinsic global dependencies. This project brings together the providers of some of the world's most widely distributed, freely available, scalable solver software and focuses them on relieving this bottleneck for many specific applications within SciDAC, which are representative of many others outside. Solver software directly supported under TOPS includes: hypre, PETSc, SUNDIALS, SuperLU, TAO, and Trilinos. Transparent access is also provided to other solver software through the TOPS interface. The primary goals of TOPS are the development, testing, and dissemination of solver software, especially for systems governed by PDEs. Upon discretization, these systems possess mathematical structure that must be exploited for optimal scalability; therefore, application-targeted algorithmic research is included. TOPS software development includes attention to high performance as well as interoperability among the solver components. Support for integration of TOPS solvers into SciDAC applications is also directly supported by this proposal. The role of the UCSD PI in this overall CET, is one of direct interaction between the TOPS software partners and various DOE applications scientists' specifically toward magnetohydrodynamics (MHD) simulations with the Center for Extended Magnetohydrodynamic Modeling (CEMM) SciDAC and Applied Partial Differential Equations Center (APDEC) SciDAC, and toward core-collapse supernova simulations with the previous Terascale Supernova Initiative (TSI) SciDAC and in continued work on INCITE projects headed by Doug Swesty, SUNY Stony Brook. In addition to these DOE applications scientists, the UCSD PI works to bring leading-edge DOE solver technology to applications scientists in cosmology and large-scale galactic structure formation. Unfortunately, the funding for this grant ended after only two years of its five-year duration, in August 2008, due to difficulties at DOE in transferring the grant to the PI's new faculty position at Southern Methodist University. Therefore, this report only describes two years' worth of effort.« less

  • The Terascale Optimal PDE Solvers (TOPS) Integrated Software Infrastructure Center (ISIC) was created to develop and implement algorithms and support scientific investigations performed by DOE-sponsored researchers. These simulations often involve the solution of partial differential equations (PDEs) on terascale computers. The TOPS Center researched, developed and deployed an integrated toolkit of open-source, optimal complexity solvers for the nonlinear partial differential equations that arise in many DOE application areas, including fusion, accelerator design, global climate change and reactive chemistry. The algorithms created as part of this project were also designed to reduce current computational bottlenecks by orders of magnitude on terascalemore » computers, enabling scientific simulation on a scale heretofore impossible.« less

  • The Terascale High-Fidelity Simulations of Turbulent Combustion (TSTC) project is a multi-university collaborative effort to develop a high-fidelity turbulent reacting flow simulation capability utilizing terascale, massively parallel computer technology. The main paradigm of the approach is direct numerical simulation (DNS) featuring the highest temporal and spatial accuracy, allowing quantitative observations of the fine-scale physics found in turbulent reacting flows as well as providing a useful tool for development of sub-models needed in device-level simulations. Under this component of the TSTC program the simulation code named S3D, developed and shared with coworkers at Sandia National Laboratories, has been enhanced with newmore » numerical algorithms and physical models to provide predictive capabilities for turbulent liquid fuel spray dynamics. Major accomplishments include improved fundamental understanding of mixing and auto-ignition in multi-phase turbulent reactant mixtures and turbulent fuel injection spray jets.« less