Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

Shen, Wanggang; Huan, Xun

doi:10.1016/j.cma.2023.116304

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

Journal Article · Fri Aug 11 00:00:00 EDT 2023 · Computer Methods in Applied Mechanics and Engineering

DOI:https://doi.org/10.1016/j.cma.2023.116304· OSTI ID:1996019

^[1]; ^[2]

University of Michigan, Ann Arbor, MI (United States); University of Michigan
University of Michigan, Ann Arbor, MI (United States)

We present a mathematical framework and computational methods for optimally designing a finite sequence of experiments. This sequential optimal experimental design (sOED) problem is formulated as a finite-horizon partially observable Markov decision process (POMDP) under a Bayesian setting and with information-theoretic utilities. The formulation is general and may accommodate continuous random variables, non-Gaussian posteriors, and nonlinear forward models. The sOED design policy incorporates elements of feedback and lookahead simultaneously, and we show it to generalize the commonly-used batch and greedy design strategies. We solve for the sOED policy using the policy gradient (PG) method from reinforcement learning, and provide a derivation for the PG expression in the sOED context. Adopting an actor-critic approach, the policy and value functions are parameterized using deep neural networks and improved via PG estimates produced from simulated episodes of designs and observations. The new PG-sOED algorithm is first validated on a linear-Gaussian benchmark, and then compared against other design baselines on a sensor movement problem for contaminant source inversion in a convection-diffusion field. As a result, we provide explanation for the policy behaviors using knowledge of the underlying physical process.

View Accepted Manuscript (DOE)

Research Organization:: University of Michigan, Ann Arbor, MI (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0021398

OSTI ID:: 1996019

Journal Information:: Computer Methods in Applied Mechanics and Engineering, Journal Name: Computer Methods in Applied Mechanics and Engineering Vol. 416; ISSN 0045-7825

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (37)

Sequential experimental design and response optimisation Pronzato, Luc; Thierry, Éric Statistical Methods & Applications, Vol. 11, Issue 3 https://doi.org/10.1007/BF02509828	journal	October 2002
Simulation Based Optimal Design Müller, Peter Handbook of Statistics https://doi.org/10.1016/S0169-7161(05)25017-4	book	January 2005
A Laplace method for under-determined Bayesian optimal experimental designs Long, Quan; Scavino, Marco; Tempone, Raúl Computer Methods in Applied Mechanics and Engineering, Vol. 285 https://doi.org/10.1016/j.cma.2014.12.008	journal	March 2015
Fast Bayesian experimental design: Laplace-based importance sampling for the expected information gain Beck, Joakim; Dia, Ben Mansour; Espath, Luis F. R. Computer Methods in Applied Mechanics and Engineering, Vol. 334 https://doi.org/10.1016/j.cma.2018.01.053	journal	June 2018
Sequential Monte Carlo for Bayesian sequentially designed experiments for discrete data Drovandi, Christopher C.; McGree, James M.; Pettitt, Anthony N. Computational Statistics & Data Analysis, Vol. 57, Issue 1 https://doi.org/10.1016/j.csda.2012.05.014	journal	January 2013
Bayesian experimental design for the active nitridation of graphite by atomic nitrogen Terejanu, Gabriel; Upadhyay, Rochan R.; Miki, Kenji Experimental Thermal and Fluid Science, Vol. 36 https://doi.org/10.1016/j.expthermflusci.2011.09.012	journal	January 2012
Simulation-based optimal Bayesian experimental design for nonlinear systems Huan, Xun; Marzouk, Youssef M. Journal of Computational Physics, Vol. 232, Issue 1 https://doi.org/10.1016/j.jcp.2012.08.013	journal	January 2013
Simulation-based sequential Bayesian design Müller, Peter; Berry, Don A.; Grieve, Andy P. Journal of Statistical Planning and Inference, Vol. 137, Issue 10 https://doi.org/10.1016/j.jspi.2006.05.021	journal	October 2007
Natural Actor-Critic Peters, Jan; Schaal, Stefan Neurocomputing, Vol. 71, Issue 7-9 https://doi.org/10.1016/j.neucom.2007.11.026	journal	March 2008
Human-level control through deep reinforcement learning Mnih, Volodymyr; Kavukcuoglu, Koray; Silver, David Nature, Vol. 518, Issue 7540 https://doi.org/10.1038/nature14236	journal	February 2015
Recent Advances in Nonlinear Experimental Design Ford, Ian; Titterington, D. M.; Kitsos, Christos P. Technometrics, Vol. 31, Issue 1 https://doi.org/10.1080/00401706.1989.10488475	journal	February 1989
Bayesian Design of Experiments Using Approximate Coordinate Exchange Overstall, Antony M.; Woods, David C. Technometrics, Vol. 59, Issue 4 https://doi.org/10.1080/00401706.2016.1251495	journal	April 2017
Variational Inference: A Review for Statisticians Blei, David M.; Kucukelbir, Alp; McAuliffe, Jon D. Journal of the American Statistical Association, Vol. 112, Issue 518 https://doi.org/10.1080/01621459.2017.1285773	journal	July 2016
George'S Column Box, George E. P. Quality Engineering, Vol. 5, Issue 2 https://doi.org/10.1080/08982119208918971	journal	January 1992
A Sequential Monte Carlo Algorithm to Incorporate Model Uncertainty in Bayesian Sequential Design Drovandi, Christopher C.; McGree,, James M.; Pettitt, Anthony N. Journal of Computational and Graphical Statistics, Vol. 23, Issue 1 https://doi.org/10.1080/10618600.2012.730083	journal	January 2014
Design of Experiments in Non-Linear Situations Box, G. E. P.; Lucas, H. L. Biometrika, Vol. 46, Issue 1-2 https://doi.org/10.1093/biomet/46.1-2.77	journal	January 1959
Bayesian inference in physics von Toussaint, Udo Reviews of Modern Physics, Vol. 83, Issue 3 https://doi.org/10.1103/RevModPhys.83.943	journal	September 2011
Optimal Dynamic Treatment Regimes Murphy, S. A. Journal of the Royal Statistical Society Series B: Statistical Methodology, Vol. 65, Issue 2 https://doi.org/10.1111/1467-9868.00389	journal	April 2003
A Review of Modern Computational Algorithms for Bayesian Optimal Design: A Review of Modern Algorithms for Bayesian Design Ryan, Elizabeth G.; Drovandi, Christopher C.; McGree, James M. International Statistical Review, Vol. 84, Issue 1 https://doi.org/10.1111/insr.12107	journal	June 2015
A Fast and Scalable Method for A-Optimal Design of Experiments for Infinite-dimensional Bayesian Nonlinear Inverse Problems Alexanderian, Alen; Petra, Noemi; Stadler, Georg SIAM Journal on Scientific Computing, Vol. 38, Issue 1 https://doi.org/10.1137/140992564	journal	January 2016
Efficient Bayesian Experimentation Using an Expected Information Gain Lower Bound Tsilifis, Panagiotis; Ghanem, Roger G.; Hajali, Paris SIAM/ASA Journal on Uncertainty Quantification, Vol. 5, Issue 1 https://doi.org/10.1137/15M1043303	journal	January 2017
A Hierarchical Adaptive Approach to Optimal Experimental Design Kim, Woojae; Pitt, Mark A.; Lu, Zhong-Lin Neural Computation, Vol. 26, Issue 11 https://doi.org/10.1162/NECO_a_00654	journal	November 2014
Adaptive Design Optimization: A Mutual Information-Based Approach to Model Discrimination in Cognitive Science Cavagnaro, Daniel R.; Myung, Jay I.; Pitt, Mark A. Neural Computation, Vol. 22, Issue 4 https://doi.org/10.1162/neco.2009.02-09-959	journal	April 2010
Implementation of Backward Induction for Sequentially Adaptive Clinical Trials Wathen, J. Kyle; Christen, J. Andrés Journal of Computational and Graphical Statistics, Vol. 15, Issue 2 https://doi.org/10.1198/016214506X113406	journal	June 2006
Sequential Experimental Designs for Generalized Linear Models Dror, Hovav A.; Steinberg, David M. Journal of the American Statistical Association, Vol. 103, Issue 481 https://doi.org/10.1198/016214507000001346	journal	March 2008
Estimating Expected Information Gains for Experimental Designs With Application to the Random Fatigue-Limit Model Ryan, Kenneth J. Journal of Computational and Graphical Statistics, Vol. 12, Issue 3 https://doi.org/10.1198/1061860032012	journal	September 2003
A Gridding Method for Bayesian Sequential Decision Problems Brockwell, Anthony E.; Kadane, Joseph B. Journal of Computational and Graphical Statistics, Vol. 12, Issue 3 https://doi.org/10.1198/1061860032274	journal	September 2003
Sequential stopping rules for species accumulation Christen, J. Andrés; Nakamura, Miguel Journal of Agricultural, Biological, and Environmental Statistics, Vol. 8, Issue 2 https://doi.org/10.1198/1085711031553	journal	June 2003
Simulation-Based Optimal Design Using a Response Variance Criterion Solonen, Antti; Haario, Heikki; Laine, Marko Journal of Computational and Graphical Statistics, Vol. 21, Issue 1 https://doi.org/10.1198/jcgs.2011.10070	journal	January 2012
On the measure of the information in a statistical experiment Ginebra, Josep Bayesian Analysis, Vol. 2, Issue 1 https://doi.org/10.1214/07-BA207	journal	March 2007
Computational Enhancements to Bayesian Design of Experiments Using Gaussian Processes Weaver, Brian P.; Williams, Brian J.; Anderson-Cook, Christine M. Bayesian Analysis, Vol. 11, Issue 1 https://doi.org/10.1214/15-BA945	journal	March 2016
Sequential Bayesian Experimental Design for Implicit Models via Mutual Information Kleinegesse, Steven; Drovandi, Christopher; Gutmann, Michael U. Bayesian Analysis, Vol. 16, Issue 3 https://doi.org/10.1214/20-BA1225	journal	September 2021
On a Measure of the Information Provided by an Experiment Lindley, D. V. The Annals of Mathematical Statistics, Vol. 27, Issue 4 https://doi.org/10.1214/aoms/1177728069	journal	December 1956
Bayesian Experimental Design: A Review Chaloner, Kathryn; Verdinelli, Isabella Statistical Science, Vol. 10, Issue 3 https://doi.org/10.1214/ss/1177009939	journal	August 1995
Optimal Adaptive Policies for Markov Decision Processes Burnetas, Apostolos N.; Katehakis, Michael N. Mathematics of Operations Research, Vol. 22, Issue 1 https://doi.org/10.1287/moor.22.1.222	journal	February 1997
Approaches for Optimal Sequential Decision Analysis in Clinical Trials Carlin, Bradley P.; Kadane, Joseph B.; Gelfand, Alan E. Biometrics, Vol. 54, Issue 3 https://doi.org/10.2307/2533849	journal	September 1998
Adaptive control for sequential design Gautier, Roland; Pronzato, Luc Discussiones Mathematicae Probability and Statistics, Vol. 20, Issue 1 https://doi.org/10.7151/dmps.1006	journal	January 2000

Similar Records

Practical and Optimal Sequential Bayesian Experimental Design for Complex Systems Incorporating Human Experimenter Preferences (Final Scientific/Technical Report)

Technical Report · Wed Dec 25 23:00:00 EST 2024 · OSTI ID:2483855

Computing optimal policies for partially observable decision processes using compact representations

Conference · Mon Dec 30 23:00:00 EST 1996 · OSTI ID:430799

Sequential ensemble transform for Bayesian inverse problems

Journal Article · Sun Dec 06 19:00:00 EST 2020 · Journal of Computational Physics · OSTI ID:1852513

Related Subjects

97 MATHEMATICS AND COMPUTING
Bayesian experimental design
Markov decision process
adaptive design
expected information gain
sequential experimental design

Bayesian sequential optimal experimental design for nonlinear models using policy gradient reinforcement learning

Citation Formats

References (37)

Similar Records

Related Subjects