Designing accurate emulators for scientific processes using calibration-driven deep models

Thiagarajan, Jayaraman J.; Venkatesh, Bindya; Anirudh, Rushil; Bremer, Peer-Timo; Gaffney, Jim; Anderson, Gemma; Spears, Brian

doi:10.1038/s41467-020-19448-8

Title: Designing accurate emulators for scientific processes using calibration-driven deep models

Abstract

Abstract Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.

Authors:

; Venkatesh, Bindya;

;

; Gaffney, Jim;

;

Publication Date:: 2020-11-06

Research Org.:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1706206

Alternate Identifier(s):: OSTI ID: 1738904

Report Number(s):: LLNL-JRNL-808697
Journal ID: ISSN 2041-1723; 5622; PII: 19448

Grant/Contract Number:: AC52-07NA27344

Resource Type:: Published Article

Journal Name:: Nature Communications

Additional Journal Information:: Journal Name: Nature Communications Journal Volume: 11 Journal Issue: 1; Journal ID: ISSN 2041-1723

Publisher:: Nature Publishing Group

Country of Publication:: United Kingdom

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Computer science; Scientific data

Citation Formats


                    Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, and Spears, Brian. Designing accurate emulators for scientific processes using calibration-driven deep models.  United Kingdom: N. p., 2020. 
Web.  doi:10.1038/s41467-020-19448-8.

Copy to clipboard


                    Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, & Spears, Brian. Designing accurate emulators for scientific processes using calibration-driven deep models.  United Kingdom.  https://doi.org/10.1038/s41467-020-19448-8

Copy to clipboard


                    Thiagarajan, Jayaraman J., Venkatesh, Bindya, Anirudh, Rushil, Bremer, Peer-Timo, Gaffney, Jim, Anderson, Gemma, and Spears, Brian. Fri .  
"Designing accurate emulators for scientific processes using calibration-driven deep models".  United Kingdom.  https://doi.org/10.1038/s41467-020-19448-8.

Copy to clipboard


                    
@article{osti_1706206,

  title        = {Designing accurate emulators for scientific processes using calibration-driven deep models},

  author       = {Thiagarajan, Jayaraman J. and Venkatesh, Bindya and Anirudh, Rushil and Bremer, Peer-Timo and Gaffney, Jim and Anderson, Gemma and Spears, Brian},

  abstractNote = {Abstract Predictive models that accurately emulate complex scientific processes can achieve speed-ups over numerical simulators or experiments and at the same time provide surrogates for improving the subsequent analysis. Consequently, there is a recent surge in utilizing modern machine learning methods to build data-driven emulators. In this work, we study an often overlooked, yet important, problem of choosing loss functions while designing such emulators. Popular choices such as the mean squared error or the mean absolute error are based on a symmetric noise assumption and can be unsuitable for heterogeneous data or asymmetric noise distributions. We propose Learn-by-Calibrating, a novel deep learning approach based on interval calibration for designing emulators that can effectively recover the inherent noise structure without any explicit priors. Using a large suite of use-cases, we demonstrate the efficacy of our approach in providing high-quality emulators, when compared to widely-adopted loss function choices, even in small-data regimes.},

  doi          = {10.1038/s41467-020-19448-8},

  journal      = {Nature Communications},

  number       = 1,

  volume       = 11,

  place        = {United Kingdom},

  year         = {2020},

  month        = {11}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Publisher's Version of Record
https://doi.org/10.1038/s41467-020-19448-8

Other availability

Search WorldCat to find libraries that may hold this journal

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing
journal, November 2017

Perreault Levasseur, Laurence; Hezaveh, Yashar D.; Wechsler, Risa H.
The Astrophysical Journal, Vol. 850, Issue 1
DOI: 10.3847/2041-8213/aa9704

Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension
journal, March 2012

Wang, Lan; Wu, Yichao; Li, Runze
Journal of the American Statistical Association, Vol. 107, Issue 497
DOI: 10.1080/01621459.2012.656014

Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
journal, October 2019

Zhu, Yinhao; Zabaras, Nicholas; Koutsourelakis, Phaedon-Stelios
Journal of Computational Physics, Vol. 394
DOI: 10.1016/j.jcp.2019.05.024

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
journal, January 2020

Jing, Longlong; Tian, Yingli
IEEE Transactions on Pattern Analysis and Machine Intelligence
DOI: 10.1109/TPAMI.2020.2992393

Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification
journal, August 2018

Zhu, Yinhao; Zabaras, Nicholas
Journal of Computational Physics, Vol. 366
DOI: 10.1016/j.jcp.2018.04.018

Learn-By-Calibrating: Using Calibration As A Training Objective
conference, May 2020

Thiagarajan, Jayaraman J.; Venkatesh, Bindya; Rajan, Deepta
ICASSP 2020 - 2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
DOI: 10.1109/ICASSP40776.2020.9053195

Nanophotonic particle simulation and inverse design using artificial neural networks
journal, June 2018

Peurifoy, John; Shen, Yichen; Jing, Li
Science Advances, Vol. 4, Issue 6
DOI: 10.1126/sciadv.aar4206

A data-driven statistical model for predicting the critical temperature of a superconductor
journal, November 2018

Hamidieh, Kam
Computational Materials Science, Vol. 154
DOI: 10.1016/j.commatsci.2018.07.052

Opportunities and obstacles for deep learning in biology and medicine
journal, April 2018

Ching, Travers; Himmelstein, Daniel S.; Beaulieu-Jones, Brett K.
Journal of The Royal Society Interface, Vol. 15, Issue 141
DOI: 10.1098/rsif.2017.0387

Methods for interpreting and understanding deep neural networks
journal, February 2018

Montavon, Grégoire; Samek, Wojciech; Müller, Klaus-Robert
Digital Signal Processing, Vol. 73
DOI: 10.1016/j.dsp.2017.10.011

Analysis of Strength of Concrete Using Design of Experiments and Neural Networks
journal, August 2006

Yeh, I-Cheng
Journal of Materials in Civil Engineering, Vol. 18, Issue 4
DOI: 10.1061/(ASCE)0899-1561(2006)18:4(597)

Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007

Gneiting, Tilmann; Raftery, Adrian E.
Journal of the American Statistical Association, Vol. 102, Issue 477
DOI: 10.1198/016214506000001437

Smoothed state estimates under abrupt changes using sum-of-norms regularization
journal, April 2012

Ohlsson, Henrik; Gustafsson, Fredrik; Ljung, Lennart
Automatica, Vol. 48, Issue 4
DOI: 10.1016/j.automatica.2011.08.063

PMLB: a large benchmark suite for machine learning evaluation and comparison
journal, December 2017

Olson, Randal S.; La Cava, William; Orzechowski, Patryk
BioData Mining, Vol. 10, Issue 1
DOI: 10.1186/s13040-017-0154-4

Orthogonal Matching Pursuit for Sparse Quantile Regression
conference, December 2014

Aravkin, Aleksandr; Lozano, Aurelie; Luss, Ronny
2014 IEEE International Conference on Data Mining (ICDM)
DOI: 10.1109/ICDM.2014.134

A second life of the Matthias’s rules
journal, June 2016

Conder, K.
Superconductor Science and Technology, Vol. 29, Issue 8
DOI: 10.1088/0953-2048/29/8/080502

Recent Progress on Generative Adversarial Networks (GANs): A Survey
journal, January 2019

Pan, Zhaoqing; Yu, Weijie; Yi, Xiaokai
IEEE Access, Vol. 7
DOI: 10.1109/ACCESS.2019.2905015

Neural networks for variational problems in engineering
journal, September 2008

Lopez, R.; Balsa-Canto, E.; Oñate, E.
International Journal for Numerical Methods in Engineering, Vol. 75, Issue 11
DOI: 10.1002/nme.2304

Leveraging uncertainty information from deep neural networks for disease detection
journal, December 2017

Leibig, Christian; Allken, Vaneeda; Ayhan, Murat Seçkin
Scientific Reports, Vol. 7, Issue 1
DOI: 10.1038/s41598-017-17876-z

Multimodal Machine Learning: A Survey and Taxonomy
journal, February 2019

Baltrusaitis, Tadas; Ahuja, Chaitanya; Morency, Louis-Philippe
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 41, Issue 2
DOI: 10.1109/TPAMI.2018.2798607

Towards Concise Models of Grid Stability
conference, October 2018

Arzamasov, Vadim; Bohm, Klemens; Jochem, Patrick
2018 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm)
DOI: 10.1109/SmartGridComm.2018.8587498

Searching for exotic particles in high-energy physics with deep learning
journal, July 2014

Baldi, P.; Sadowski, P.; Whiteson, D.
Nature Communications, Vol. 5, Issue 1
DOI: 10.1038/ncomms5308

A HYDRA UQ Workflow for NIF Ignition Experiments
conference, November 2016

Langer, Steven H.; Spears, Brian; Peterson, J. Luc
2016 Second Workshop on In Situ Infrastructures for Enabling Extreme-Scale Analysis and Visualization (ISAV)
DOI: 10.1109/ISAV.2016.006

Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences
conference, June 2019

Seo, Seonguk; Seo, Paul Hongsuck; Han, Bohyung
2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)
DOI: 10.1109/CVPR.2019.00924

Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests
journal, April 2010

Tsanas, A.; Little, M. A.; McSharry, P. E.
IEEE Transactions on Biomedical Engineering, Vol. 57, Issue 4
DOI: 10.1109/TBME.2009.2036000

Understanding Deep Neural Networks through Input Uncertainties
conference, May 2019

Thiagarajan, Jayaraman J.; Kim, Irene; Anirudh, Rushil
ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
DOI: 10.1109/ICASSP.2019.8682930

A guide to deep learning in healthcare
journal, January 2019

Esteva, Andre; Robicquet, Alexandre; Ramsundar, Bharath
Nature Medicine, Vol. 25, Issue 1
DOI: 10.1038/s41591-018-0316-z

A Procedure for Integrating Geologic Concepts Into History Matching
conference, April 2013

Lun, Lisa; Dunn, Paul Alexander; Stern, David
SPE Annual Technical Conference and Exhibition
DOI: 10.2118/159985-MS

Machine learning for molecular and materials science
journal, July 2018

Butler, Keith T.; Davies, Daniel W.; Cartwright, Hugh
Nature, Vol. 559, Issue 7715
DOI: 10.1038/s41586-018-0337-2

Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors
journal, April 2020

Thiagarajan, Jayaraman J.; Venkatesh, Bindya; Sattigeri, Prasanna
Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, Issue 04
DOI: 10.1609/aaai.v34i04.6062

Representation Learning: A Review and New Perspectives
journal, August 2013

Bengio, Y.; Courville, A.; Vincent, P.
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 35, Issue 8
DOI: 10.1109/TPAMI.2013.50

Improved surrogates in inertial confinement fusion with manifold and cycle consistencies
journal, April 2020

Anirudh, Rushil; Thiagarajan, Jayaraman J.; Bremer, Peer-Timo
Proceedings of the National Academy of Sciences, Vol. 117, Issue 18
DOI: 10.1073/pnas.1916634117

Similar Records in DOE PAGES and OSTI.GOV collections:

Comparing emulation methods for a high‐resolution storm surge model

Journal Article Hutchings, Grant ; Sansó, Bruno ; Gattiker, James ; ... - EnvironMetrics (London)

Abstract Realistic simulations of complex systems are fundamental for climate and environmental studies. Large computer systems are often not sufficient to run sophisticated computational models for large numbers of different input settings. Statistical surrogate models, or emulators, are key tools enabling fast exploration of the simulator input space. Gaussian processes have become standard for computer simulator emulation. However, they require careful implementation to scale appropriately, motivating alternative methods more recently introduced. We present a comparison study of surrogates of the Sea, Lake, and Overland Surges from Hurricanes (SLOSH) simulator—the simulator of choice for government agencies—using four emulation approaches: BASS; BART;more »« less
https://doi.org/10.1002/env.2796
Cholesky-based experimental design for Gaussian process and kernel-based emulation and calibration.

Journal Article Helmut, Harbrecht ; Jakeman, John Davis ; Zaspel, Peter - Communications in Computational Physics

Gaussian processes and other kernel-based methods are used extensively to construct approximations of multivariate data sets. The accuracy of these approximations is dependent on the data used. This paper presents a computationally efficient algorithm to greedily select training samples that minimize the weighted L^p error of kernel-based approximations for a given number of data. The method successively generates nested samples, with the goal of minimizing the error in high probability regions of densities specified by users. The algorithm presented is extremely simple and can be implemented using existing pivoted Cholesky factorization methods. Training samples are generated in batches which allowsmore »« less
https://doi.org/10.4208/cicp.OA-2020-0060

Full Text Available
A comparison of Gaussian processes and neural networks for computer model emulation and calibration

Journal Article Myren, Samuel Thomas Wilkins ; Lawrence, Earl Christopher - Statistical Analysis and Data Mining

Abstract The Department of Energy relies on complex physics simulations for prediction in domains like cosmology, nuclear theory, and materials science. These simulations are often extremely computationally intensive, with some requiring days or weeks for a single simulation. In order to assure their accuracy, these models are calibrated against observational data in order to estimate inputs and systematic biases. Because of their great computational complexity, this process typically requires the construction of an emulator , a fast approximation to the simulation. In this paper, two emulator approaches are compared: Gaussian process regression and neural networks. Their emulation accuracy and calibrationmore »« less
https://doi.org/10.1002/sam.11507

Full Text Available
Deep Learning Emulation of Subgrid-Scale Processes in Turbulent Shear Flows

Journal Article Pal, Anikesh - Geophysical Research Letters

Abstract Deep neural networks (DNNs) are developed from a data set obtained from the dynamic Smagorinsky model to emulate the subgrid‐scale (SGS) viscosity ( ν _{s g s} ) and diffusivity ( κ _{s g s} ) for turbulent stratified shear flows encountered in the oceans and the atmosphere. These DNNs predict ν _{s g s} and κ _{s g s} from velocities, strain rates, and density gradients such that the evolution of the kinetic energy budget and density variance budget terms is similar to the corresponding values obtained from the originalmore »« less
Cited by 9
https://doi.org/10.1029/2020gl087005

Full Text Available
Bayesian calibration of groundwater models with input data uncertainty

Journal Article Xu, Tianfang ; Valocchi, Albert J. ; Ye, Ming ; ... - Water Resources Research

Abstract Effective water resources management typically relies on numerical models to analyze groundwater flow and solute transport processes. Groundwater models are often subject to input data uncertainty, as some inputs (such as recharge and well pumping rates) are estimated and subject to uncertainty. Current practices of groundwater model calibration often overlook uncertainties in input data; this can lead to biased parameter estimates and compromised predictions. Through a synthetic case study of surface‐ground water interaction under changing pumping conditions and land use, we investigate the impacts of uncertain pumping and recharge rates on model calibration and uncertainty analysis. We then presentmore »« less
Cited by 19
https://doi.org/10.1002/2016WR019512

Full Text Available

Similar Records

Title: Designing accurate emulators for scientific processes using calibration-driven deep models

Abstract

Citation Formats

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing journal, November 2017

Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension journal, March 2012

Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data journal, October 2019

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey journal, January 2020

Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification journal, August 2018

Learn-By-Calibrating: Using Calibration As A Training Objective conference, May 2020

Nanophotonic particle simulation and inverse design using artificial neural networks journal, June 2018

A data-driven statistical model for predicting the critical temperature of a superconductor journal, November 2018

Opportunities and obstacles for deep learning in biology and medicine journal, April 2018

Methods for interpreting and understanding deep neural networks journal, February 2018

Analysis of Strength of Concrete Using Design of Experiments and Neural Networks journal, August 2006

Strictly Proper Scoring Rules, Prediction, and Estimation journal, March 2007

Smoothed state estimates under abrupt changes using sum-of-norms regularization journal, April 2012

PMLB: a large benchmark suite for machine learning evaluation and comparison journal, December 2017

Orthogonal Matching Pursuit for Sparse Quantile Regression conference, December 2014

A second life of the Matthias’s rules journal, June 2016

Recent Progress on Generative Adversarial Networks (GANs): A Survey journal, January 2019

Neural networks for variational problems in engineering journal, September 2008

Leveraging uncertainty information from deep neural networks for disease detection journal, December 2017

Multimodal Machine Learning: A Survey and Taxonomy journal, February 2019

Towards Concise Models of Grid Stability conference, October 2018

Searching for exotic particles in high-energy physics with deep learning journal, July 2014

A HYDRA UQ Workflow for NIF Ignition Experiments conference, November 2016

Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences conference, June 2019

Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests journal, April 2010

Understanding Deep Neural Networks through Input Uncertainties conference, May 2019

A guide to deep learning in healthcare journal, January 2019

A Procedure for Integrating Geologic Concepts Into History Matching conference, April 2013

Machine learning for molecular and materials science journal, July 2018

Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors journal, April 2020

Representation Learning: A Review and New Perspectives journal, August 2013

Improved surrogates in inertial confinement fusion with manifold and cycle consistencies journal, April 2020

Uncertainties in Parameters Estimated with Neural Networks: Application to Strong Gravitational Lensing
journal, November 2017

Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension
journal, March 2012

Physics-constrained deep learning for high-dimensional surrogate modeling and uncertainty quantification without labeled data
journal, October 2019

Self-supervised Visual Feature Learning with Deep Neural Networks: A Survey
journal, January 2020

Bayesian deep convolutional encoder–decoder networks for surrogate modeling and uncertainty quantification
journal, August 2018

Learn-By-Calibrating: Using Calibration As A Training Objective
conference, May 2020

Nanophotonic particle simulation and inverse design using artificial neural networks
journal, June 2018

A data-driven statistical model for predicting the critical temperature of a superconductor
journal, November 2018

Opportunities and obstacles for deep learning in biology and medicine
journal, April 2018

Methods for interpreting and understanding deep neural networks
journal, February 2018

Analysis of Strength of Concrete Using Design of Experiments and Neural Networks
journal, August 2006

Strictly Proper Scoring Rules, Prediction, and Estimation
journal, March 2007

Smoothed state estimates under abrupt changes using sum-of-norms regularization
journal, April 2012

PMLB: a large benchmark suite for machine learning evaluation and comparison
journal, December 2017

Orthogonal Matching Pursuit for Sparse Quantile Regression
conference, December 2014

A second life of the Matthias’s rules
journal, June 2016

Recent Progress on Generative Adversarial Networks (GANs): A Survey
journal, January 2019

Neural networks for variational problems in engineering
journal, September 2008

Leveraging uncertainty information from deep neural networks for disease detection
journal, December 2017

Multimodal Machine Learning: A Survey and Taxonomy
journal, February 2019

Towards Concise Models of Grid Stability
conference, October 2018

Searching for exotic particles in high-energy physics with deep learning
journal, July 2014

A HYDRA UQ Workflow for NIF Ignition Experiments
conference, November 2016

Learning for Single-Shot Confidence Calibration in Deep Neural Networks Through Stochastic Inferences
conference, June 2019

Accurate Telemonitoring of Parkinson's Disease Progression by Noninvasive Speech Tests
journal, April 2010

Understanding Deep Neural Networks through Input Uncertainties
conference, May 2019

A guide to deep learning in healthcare
journal, January 2019

A Procedure for Integrating Geologic Concepts Into History Matching
conference, April 2013

Machine learning for molecular and materials science
journal, July 2018

Building Calibrated Deep Models via Uncertainty Matching with Auxiliary Interval Predictors
journal, April 2020

Representation Learning: A Review and New Perspectives
journal, August 2013

Improved surrogates in inertial confinement fusion with manifold and cycle consistencies
journal, April 2020