Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Deep Generative Models that Solve PDEs: Distributed Computing for Training Large Data-Free Models

Conference · · Workshop on Machine Learning in HPC Environments (Online)
 [1]; ; ; ; ; ; ;
  1. Iowa State University
+ Show Author Affiliations
Recent progress in scientific machine learning (SciML) has opened up the possibility of training novel neural network architectures that solve complex partial differential equations (PDEs). Several (nearly data free) approaches have been recently reported that successfully solve PDEs, with examples including deep feed forward networks, generative networks, and deep encoder-decoder networks. However, practical adoption of these approaches is limited by the difficulty in training these models, especially to make predictions at large output resolutions (≥1024×1024). Here we report on a software framework for data parallel distributed deep learning that resolves the twin challenges of training these large SciML models - training in reasonable time as well as distributing the storage requirements. Our framework provides several out of the box functionality including (a) loss integrity independent of number of processes, (b) synchronized batch normalization, and (c) distributed higher-order optimization methods. We show excellent scalability of this framework on both cloud as well as HPC clusters, and report on the interplay between bandwidth, network topology and bare metal vs cloud. We deploy this approach to train generative models of sizes hitherto not possible, showing that neural PDE solvers can be viably trained for practical applications. We also demonstrate that distributed higher-order optimization methods are 2-3× faster than stochastic gradient-based methods and provide minimal convergence drift with higher batch-size.
Research Organization:
Iowa State University
Sponsoring Organization:
USDOE Advanced Research Projects Agency - Energy (ARPA-E)
Contributing Organization:
RocketML Inc.
DOE Contract Number:
AR0001215
OSTI ID:
1648524
Report Number(s):
arXiv:2007.12792
Conference Information:
Journal Name: Workshop on Machine Learning in HPC Environments (Online) Journal Volume: 2020
Country of Publication:
United States
Language:
English

References (12)

Learning Particle Physics by Example: Location-Aware Generative Adversarial Networks for Physics Synthesis journal September 2017
Simulator-free solution of high-dimensional stochastic elliptic partial differential equations using deep neural networks journal March 2020
On the control of solidification using magnetic fields and magnetic field gradients journal September 2005
Inverse molecular design using machine learning: Generative models for matter engineering journal July 2018
DGM: A deep learning algorithm for solving partial differential equations journal December 2018
fPINNs: Fractional Physics-Informed Neural Networks journal January 2019
Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data journal October 2019
Solving high-dimensional partial differential equations using deep learning journal August 2018
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations journal February 2019
How do evaporating thin films evolve? Unravelling phase-separation mechanisms during solvent-based fabrication of polymer blends journal October 2014
CaloGAN: Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks journal January 2018
Hidden physics models: Machine learning of nonlinear partial differential equations journal March 2018

Similar Records

Machine learning models for PDE constrained optimization
Technical Report · Mon Sep 01 00:00:00 EDT 2025 · OSTI ID:2589582
Gradient-enhanced physics-informed neural networks for forward and inverse PDE problems
Journal Article · Thu Mar 17 20:00:00 EDT 2022 · Computer Methods in Applied Mechanics and Engineering · OSTI ID:1976976
Model-parallel Fourier neural operators as learned surrogates for large-scale parametric PDEs
Journal Article · Fri Jun 09 20:00:00 EDT 2023 · Computers and Geosciences · OSTI ID:2421940

Related Subjects

PDEs
Cloud vs HPC
Deep generative models
Distributed training
Higher-order optimization
Loss functions