skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Designing accurate emulators for scientific processes using calibration-driven deep models

Abstract

Abstract Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.

Authors:
ORCiD logo; ; ORCiD logo; ORCiD logo; ; ORCiD logo; ORCiD logo
Publication Date:
Research Org.:
Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1706206
Alternate Identifier(s):
OSTI ID: 1738904
Report Number(s):
LLNL-JRNL-808697
Journal ID: ISSN 2041-1723; 5622; PII: 19448
Grant/Contract Number:  
AC52-07NA27344
Resource Type:
Published Article
Journal Name:
Nature Communications
Additional Journal Information:
Journal Name: Nature Communications Journal Volume: 11 Journal Issue: 1; Journal ID: ISSN 2041-1723
Publisher:
Nature Publishing Group
Country of Publication:
United Kingdom
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Computer science; Scientific data

Citation Formats

Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, and Spears, Brian. Designing accurate emulators for scientific processes using calibration-driven deep models. United Kingdom: N. p., 2020. Web. https://doi.org/10.1038/s41467-020-19448-8.
Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, & Spears, Brian. Designing accurate emulators for scientific processes using calibration-driven deep models. United Kingdom. https://doi.org/10.1038/s41467-020-19448-8
Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, and Spears, Brian. Fri . "Designing accurate emulators for scientific processes using calibration-driven deep models". United Kingdom. https://doi.org/10.1038/s41467-020-19448-8.
@article{osti_1706206,
title = {Designing accurate emulators for scientific processes using calibration-driven deep models},
author = {Thiagarajan, Jayaraman J. and Venkatesh, Bindya and Anirudh, Rushil and Bremer, Peer-Timo and Gaffney, Jim and Anderson, Gemma and Spears, Brian},
abstractNote = {Abstract Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.},
doi = {10.1038/s41467-020-19448-8},
journal = {Nature Communications},
number = 1,
volume = 11,
place = {United Kingdom},
year = {2020},
month = {11}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1038/s41467-020-19448-8

Save / Share:

Works referenced in this record:

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing
journal, November 2017

  • Perreault Levasseur, Laurence; Hezaveh, Yashar D.; Wechsler, Risa H.
  • The Astrophysical Journal, Vol. 850, Issue 1
  • DOI: 10.3847/2041-8213/aa9704

Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension
journal, March 2012


Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
journal, October 2019

  • Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
  • Journal of Computational Physics, Vol. 394
  • DOI: 10.1016/j.jcp.2019.05.024

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
journal, January 2020


Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification
journal, August 2018


Learn-By-Calibrating: Using Calibration As A Training Objective
conference, May 2020

  • Thiagarajan, Jayaraman J.; Venkatesh, Bindya; Rajan, Deepta
  • ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • DOI: 10.1109/ICASSP40776.2020.9053195

A data-driven statistical model for predicting the critical temperature of a superconductor
journal, November 2018


CaloGAN: Simulating 3D high energy particle showers in multilayer electromagnetic calorimeters with generative adversarial networks
journal, January 2018


Opportunities and obstacles for deep learning in biology and medicine
journal, April 2018

  • Ching, Travers; Himmelstein, Daniel S.; Beaulieu-Jones, Brett K.
  • Journal of The Royal Society Interface, Vol. 15, Issue 141
  • DOI: 10.1098/rsif.2017.0387

Methods for interpreting and understanding deep neural networks
journal, February 2018

  • Montavon, Grégoire; Samek, Wojciech; Müller, Klaus-Robert
  • Digital Signal Processing, Vol. 73
  • DOI: 10.1016/j.dsp.2017.10.011

Analysis of Strength of Concrete Using Design of Experiments and Neural Networks
journal, August 2006


Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007

  • Gneiting, Tilmann; Raftery, Adrian E.
  • Journal of the American Statistical Association, Vol. 102, Issue 477
  • DOI: 10.1198/016214506000001437

Smoothed state estimates under abrupt changes using sum-of-norms regularization
journal, April 2012


PMLB: a large benchmark suite for machine learning evaluation and comparison
journal, December 2017


Orthogonal Matching Pursuit for Sparse Quantile Regression
conference, December 2014

  • Aravkin, Aleksandr; Lozano, Aurelie; Luss, Ronny
  • 2014 IEEE International Conference on Data Mining (ICDM)
  • DOI: 10.1109/ICDM.2014.134

A second life of the Matthias’s rules
journal, June 2016


Recent Progress on Generative Adversarial Networks (GANs): A Survey
journal, January 2019


Neural networks for variational problems in engineering
journal, September 2008

  • Lopez, R.; Balsa-Canto, E.; Oñate, E.
  • International Journal for Numerical Methods in Engineering, Vol. 75, Issue 11
  • DOI: 10.1002/nme.2304

Leveraging uncertainty information from deep neural networks for disease detection
journal, December 2017


Multimodal Machine Learning: A Survey and Taxonomy
journal, February 2019

  • Baltrusaitis, Tadas; Ahuja, Chaitanya; Morency, Louis-Philippe
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, Issue 2
  • DOI: 10.1109/TPAMI.2018.2798607

Towards Concise Models of Grid Stability
conference, October 2018

  • Arzamasov, Vadim; Bohm, Klemens; Jochem, Patrick
  • 2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)
  • DOI: 10.1109/SmartGridComm.2018.8587498

Searching for exotic particles in high-energy physics with deep learning
journal, July 2014

  • Baldi, P.; Sadowski, P.; Whiteson, D.
  • Nature Communications, Vol. 5, Issue 1
  • DOI: 10.1038/ncomms5308

A HYDRA UQ Workflow for NIF Ignition Experiments
conference, November 2016

  • Langer, Steven H.; Spears, Brian; Peterson, J. Luc
  • 2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV)
  • DOI: 10.1109/ISAV.2016.006

Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences
conference, June 2019

  • Seo, Seonguk; Seo, Paul Hongsuck; Han, Bohyung
  • 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
  • DOI: 10.1109/CVPR.2019.00924

Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests
journal, April 2010

  • Tsanas, A.; Little, M. A.; McSharry, P. E.
  • IEEE Transactions on Biomedical Engineering, Vol. 57, Issue 4
  • DOI: 10.1109/TBME.2009.2036000

Understanding Deep Neural Networks through Input Uncertainties
conference, May 2019

  • Thiagarajan, Jayaraman J.; Kim, Irene; Anirudh, Rushil
  • ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
  • DOI: 10.1109/ICASSP.2019.8682930

A guide to deep learning in healthcare
journal, January 2019


A Procedure for Integrating Geologic Concepts Into History Matching
conference, April 2013

  • Lun, Lisa; Dunn, Paul Alexander; Stern, David
  • SPE Annual Technical Conference and Exhibition
  • DOI: 10.2118/159985-MS

Machine learning for molecular and materials science
journal, July 2018


Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors
journal, April 2020

  • Thiagarajan, Jayaraman J.; Venkatesh, Bindya; Sattigeri, Prasanna
  • Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, Issue 04
  • DOI: 10.1609/aaai.v34i04.6062

Representation Learning: A Review and New Perspectives
journal, August 2013

  • Bengio, Y.; Courville, A.; Vincent, P.
  • IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Issue 8
  • DOI: 10.1109/TPAMI.2013.50

Improved surrogates in inertial confinement fusion with manifold and cycle consistencies
journal, April 2020

  • Anirudh, Rushil; Thiagarajan, Jayaraman J.; Bremer, Peer-Timo
  • Proceedings of the National Academy of Sciences, Vol. 117, Issue 18
  • DOI: 10.1073/pnas.1916634117