On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. In conclusion, two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).
Nemani, Venkat, et al. "Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial." Mechanical Systems and Signal Processing, vol. 205, no. 1, Oct. 2023. https://doi.org/10.1016/j.ymssp.2023.110796
Nemani, Venkat, Biggio, Luca, Huan, Xun, Hu, Zhen, Fink, Olga, Tran, Anh, Wang, Yan, Zhang, Xiaoge, & Hu, Chao (2023). Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial. Mechanical Systems and Signal Processing, 205(1). https://doi.org/10.1016/j.ymssp.2023.110796
Nemani, Venkat, Biggio, Luca, Huan, Xun, et al., "Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial," Mechanical Systems and Signal Processing 205, no. 1 (2023), https://doi.org/10.1016/j.ymssp.2023.110796
@article{osti_2320359,
author = {Nemani, Venkat and Biggio, Luca and Huan, Xun and Hu, Zhen and Fink, Olga and Tran, Anh and Wang, Yan and Zhang, Xiaoge and Hu, Chao},
title = {Uncertainty quantification in machine learning for engineering design and health prognostics: A tutorial},
annote = {On top of machine learning (ML) models, uncertainty quantification (UQ) functions as an essential layer of safety assurance that could lead to more principled decision making by enabling sound risk assessment and management. The safety and reliability improvement of ML models empowered by UQ has the potential to significantly facilitate the broad adoption of ML solutions in high-stakes decision settings, such as healthcare, manufacturing, and aviation, to name a few. In this tutorial, we aim to provide a holistic lens on emerging UQ methods for ML models with a particular focus on neural networks and the applications of these UQ methods in tackling engineering design as well as prognostics and health management problems. Towards this goal, we start with a comprehensive classification of uncertainty types, sources, and causes pertaining to UQ of ML models. Next, we provide a tutorial-style description of several state-of-the-art UQ methods: Gaussian process regression, Bayesian neural network, neural network ensemble, and deterministic UQ methods focusing on spectral-normalized neural Gaussian process. Established upon the mathematical formulations, we subsequently examine the soundness of these UQ methods quantitatively and qualitatively (by a toy regression example) to examine their strengths and shortcomings from different dimensions. Then, we review quantitative metrics commonly used to assess the quality of predictive uncertainty in classification and regression problems. Afterward, we discuss the increasingly important role of UQ of ML models in solving challenging problems in engineering design and health prognostics. In conclusion, two case studies with source codes available on GitHub are used to demonstrate these UQ methods and compare their performance in the life prediction of lithium-ion batteries at the early stage (case study 1) and the remaining useful life prediction of turbofan engines (case study 2).},
doi = {10.1016/j.ymssp.2023.110796},
url = {https://www.osti.gov/biblio/2320359},
journal = {Mechanical Systems and Signal Processing},
issn = {ISSN 0888-3270},
number = {1},
volume = {205},
place = {United States},
publisher = {Elsevier},
year = {2023},
month = {10}}
Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States); Univ. of Michigan, Ann Arbor, MI (United States)
Sponsoring Organization:
USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); US Army Engineer Research and Development Center Research (ERDC); Swiss National Science Foundation (SNSF); National Science Foundation (NSF); Automotive Research Center (ARC)
Grant/Contract Number:
NA0003525; SC0021397
OSTI ID:
2320359
Alternate ID(s):
OSTI ID: 2440427
Report Number(s):
SAND--2024-02800J
Journal Information:
Mechanical Systems and Signal Processing, Journal Name: Mechanical Systems and Signal Processing Journal Issue: 1 Vol. 205; ISSN 0888-3270
Proceedings of the Institution of Mechanical Engineers, Part C: Journal of Mechanical Engineering Science, Vol. 235, Issue 19https://doi.org/10.1177/0954406220973233