Distributionally Robust Partially Observable Markov Decision Process with Moment-Based Ambiguity

Nakao, Hideaki; Jiang, Ruiwei; Shen, Siqian

doi:10.1137/19m1268410

Distributionally Robust Partially Observable Markov Decision Process with Moment-Based Ambiguity

Journal Article · Mon Feb 01 00:00:00 EST 2021 · SIAM Journal on Optimization

DOI:https://doi.org/10.1137/19m1268410· OSTI ID:1785682

Nakao, Hideaki ^[1]; ^[2]; ^[2]

Univ. of Michigan, Ann Arbor, MI (United States); University of Michigan
Univ. of Michigan, Ann Arbor, MI (United States)

In this paper, we consider a distributionally robust partially observable Markov decision process (DR-POMDP), where the distribution of the transition-observation probabilities is unknown at the beginning of each decision period, but their realizations can be inferred using side information at the end of each period after an action being taken. We build an ambiguity set of the joint distribution using bounded moments via conic constraints and seek an optimal policy to maximize the worst-case (minimum) reward for any distribution in the set. We show that the value function of DR-POMDP is piecewise linear convex with respect to the belief state and propose a heuristic search value iteration method for obtaining lower and upper bounds of the value function. We conduct numerical studies and demonstrate the computational performance of our approach via testing instances of a dynamic epidemic control problem. Our results show that DR-POMDP can produce more robust policies under misspecified distributions of transition-observation probabilities as compared to POMDP but has less costly solutions than robust POMDP. The DR-POMDP policies are also insensitive to varying parameter in the ambiguity set and to noise added to the true transition-observation probability values obtained at the end of each decision period.

View Accepted Manuscript (DOE)

Research Organization:: Univ. of Michigan, Ann Arbor, MI (United States)

Sponsoring Organization:: National Science Foundation (NSF); USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0018018

OSTI ID:: 1785682

Journal Information:: SIAM Journal on Optimization, Journal Name: SIAM Journal on Optimization Journal Issue: 1 Vol. 31; ISSN 1052-6234

Publisher:: SIAMCopyright Statement

Country of Publication:: United States

Language:: English

References (22)

Algorithms for singularly perturbed limiting average Markov control problems Abbad, M.; Filar, J. A.; Bielecki, T. R. 29th IEEE Conference on Decision and Control https://doi.org/10.1109/cdc.1990.203841	conference	January 1990
Adaptive Inventory Control for Nonstationary Demand and Partial Information Treharne, James T.; Sox, Charles R. Management Science, Vol. 48, Issue 5 https://doi.org/10.1287/mnsc.48.5.607.7807	journal	May 2002
Distributionally Robust Convex Optimization Wiesemann, Wolfram; Kuhn, Daniel; Sim, Melvyn Operations Research, Vol. 62, Issue 6 https://doi.org/10.1287/opre.2014.1314	journal	December 2014
Monitoring epidemiologic surveillance data using hidden Markov models Le Strat, Yann; Carrat, Fabrice Statistics in Medicine, Vol. 18, Issue 24 https://doi.org/10.1002/(SICI)1097-0258(19991230)18:24<3463::AID-SIM409>3.0.CO;2-I	journal	December 1999
Distributionally robust joint chance constraints with second-order moment information Zymler, Steve; Kuhn, Daniel; Rustem, Berç Mathematical Programming, Vol. 137, Issue 1-2 https://doi.org/10.1007/s10107-011-0494-7	journal	November 2011
Data-driven chance constrained stochastic program Jiang, Ruiwei; Guan, Yongpei Mathematical Programming, Vol. 158, Issue 1-2 https://doi.org/10.1007/s10107-015-0929-7	journal	July 2015
Data-driven distributionally robust optimization using the Wasserstein metric: performance guarantees and tractable reformulations Mohajerin Esfahani, Peyman; Kuhn, Daniel Mathematical Programming, Vol. 171, Issue 1-2 https://doi.org/10.1007/s10107-017-1172-1	journal	July 2017
Planning treatment of ischemic heart disease with partially observable Markov decision processes Hauskrecht, Milos; Fraser, Hamish Artificial Intelligence in Medicine, Vol. 18, Issue 3 https://doi.org/10.1016/S0933-3657(99)00042-1	journal	March 2000
A new distribution-free quantile estimator Harrell, Frank E.; Davis, C. E. Biometrika, Vol. 69, Issue 3 https://doi.org/10.1093/biomet/69.3.635	journal	January 1982
Perturbation and stability theory for Markov control problems Abbad, M.; Filar, J. A. IEEE Transactions on Automatic Control, Vol. 37, Issue 9 https://doi.org/10.1109/9.159584	journal	January 1992
A Convex Optimization Approach to Distributionally Robust Markov Decision Processes With Wasserstein Distance Yang, Insoon IEEE Control Systems Letters, Vol. 1, Issue 1 https://doi.org/10.1109/LCSYS.2017.2711553	journal	July 2017
Distributionally Robust Counterpart in Markov Decision Processes Yu, Pengqian; Xu, Huan IEEE Transactions on Automatic Control, Vol. 61, Issue 9 https://doi.org/10.1109/TAC.2015.2495174	journal	September 2016
The Mathematics of Infectious Diseases Hethcote, Herbert W. SIAM Review, Vol. 42, Issue 4 https://doi.org/10.1137/S0036144500371907	journal	January 2000
Robust Solutions of Optimization Problems Affected by Uncertain Probabilities Ben-Tal, Aharon; den Hertog, Dick; De Waegenaere, Anja Management Science, Vol. 59, Issue 2 https://doi.org/10.1287/mnsc.1120.1641	journal	February 2013
Robust Dynamic Programming Iyengar, Garud N. Mathematics of Operations Research, Vol. 30, Issue 2 https://doi.org/10.1287/moor.1040.0129	journal	May 2005
Distributionally Robust Markov Decision Processes Xu, Huan; Mannor, Shie Mathematics of Operations Research, Vol. 37, Issue 2 https://doi.org/10.1287/moor.1120.0540	journal	May 2012
Robust Markov Decision Processes Wiesemann, Wolfram; Kuhn, Daniel; Rustem, Berç Mathematics of Operations Research, Vol. 38, Issue 1 https://doi.org/10.1287/moor.1120.0566	journal	February 2013
Robust MDPs with k -Rectangular Uncertainty Mannor, Shie; Mebel, Ofir; Xu, Huan Mathematics of Operations Research, Vol. 41, Issue 4 https://doi.org/10.1287/moor.2016.0786	journal	November 2016
Robust Control of Markov Decision Processes with Uncertain Transition Matrices Nilim, Arnab; El Ghaoui, Laurent Operations Research, Vol. 53, Issue 5 https://doi.org/10.1287/opre.1050.0216	journal	October 2005
Percentile Optimization for Markov Decision Processes with Parameter Uncertainty Delage, Erick; Mannor, Shie Operations Research, Vol. 58, Issue 1 https://doi.org/10.1287/opre.1080.0685	journal	February 2010
Distributionally Robust Optimization Under Moment Uncertainty with Application to Data-Driven Problems Delage, Erick; Ye, Yinyu Operations Research, Vol. 58, Issue 3 https://doi.org/10.1287/opre.1090.0741	journal	June 2010
The Optimal Control of Partially Observable Markov Processes over a Finite Horizon Smallwood, Richard D.; Sondik, Edward J. Operations Research, Vol. 21, Issue 5 https://doi.org/10.1287/opre.21.5.1071	journal	October 1973

Similar Records

Multistage distributionally robust mixed-integer programming with decision-dependent moment-based ambiguity sets

Journal Article · Tue Oct 20 20:00:00 EDT 2020 · Mathematical Programming · OSTI ID:1735792

Residuals-based distributionally robust optimization with covariate information

Technical Report · Tue May 03 00:00:00 EDT 2022 · OSTI ID:2377330

Related Subjects

97 MATHEMATICS AND COMPUTING
HSVI
POMDP
distributionally robust optimization
epidemic control
heuristic search value iteration
moment-based ambiguity set
partially observable Markov decision process

Distributionally Robust Partially Observable Markov Decision Process with Moment-Based Ambiguity

Citation Formats

References (22)

Similar Records

Related Subjects