Challenges in Training PINNs: A Loss Landscape Perspective

Conference · Sun Jun 02 00:00:00 EDT 2024

OSTI ID:2528071

Rathore, Pratik ^[1]; Lei, Weimu ^[2]; Frangella, Zachary ^[3]; Lu, Lu ^[4]; Udell, Madeleine ^[5]

---

1. Department of Electrical Engineering, Stanford University, Stanford, CA, USA; Yale University
2. ICME, Stanford University, Stanford, CA, USA
3. Department of Management Science & Engineering, Stan ford University, Stanford, CA, USA
4. Department of Statistics and Data Science, Yale University, New Haven, CT, USA
5. ICME, Stanford University, Stanford, CA, USA; Department of Management Science & Engineering, Stan ford University, Stanford, CA, USA

This paper explores challenges in training Physics Informed Neural Networks (PINNs), emphasizing the role of the loss landscape in the training process. We examine difficulties in minimizing the PINN loss function, particularly due to ill conditioning caused by differential operators in the residual term. We compare gradient-based optimizers Adam, L-BFGS, and their combination Adam+L-FGS, showing the superiority of Adam+L-BFGS, and introduce a novel secondorder optimizer, NysNewton-CG (NNCG), which significantly improves PINN performance. Theoretically, our work elucidates the connection between ill-conditioned differential operators and ill-conditioning in the PINN loss and shows the benefits of combining first- and second-order optimization methods. Our work presents valuable insights and more powerful optimization strategies for training PINNs, which could improve the utility of PINNs for solving difficult partial differential equations.

• Research Organization: University of Pennsylvania
• Sponsoring Organization: USDOE; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: SC0022953
• OSTI ID: 2528071
• Country of Publication: United States
• Language: English