skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach

Abstract

In this paper, we investigate an optimization problem for continuous-time Markov decision processes with both impulsive and continuous controls. We consider the so-called constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.

Authors:
 [1];  [2]
  1. Bordeaux INP, IMB, UMR CNRS 5251 (France)
  2. University of Liverpool, Department of Mathematical Sciences (United Kingdom)
Publication Date:
OSTI Identifier:
22617265
Resource Type:
Journal Article
Resource Relation:
Journal Name: Applied Mathematics and Optimization; Journal Volume: 74; Journal Issue: 1; Other Information: Copyright (c) 2016 Springer Science+Business Media New York; http://www.springer-ny.com; Country of input: International Atomic Energy Agency (IAEA)
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICAL METHODS AND COMPUTING; LINEAR PROGRAMMING; MARKOV PROCESS; OCCUPATIONS; OPTIMAL CONTROL; OPTIMIZATION; PULSES

Citation Formats

Dufour, F., E-mail: dufour@math.u-bordeaux1.fr, and Piunovskiy, A. B., E-mail: piunov@liv.ac.uk. Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach. United States: N. p., 2016. Web. doi:10.1007/S00245-015-9310-8.
Dufour, F., E-mail: dufour@math.u-bordeaux1.fr, & Piunovskiy, A. B., E-mail: piunov@liv.ac.uk. Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach. United States. doi:10.1007/S00245-015-9310-8.
Dufour, F., E-mail: dufour@math.u-bordeaux1.fr, and Piunovskiy, A. B., E-mail: piunov@liv.ac.uk. 2016. "Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach". United States. doi:10.1007/S00245-015-9310-8.
@article{osti_22617265,
title = {Impulsive Control for Continuous-Time Markov Decision Processes: A Linear Programming Approach},
author = {Dufour, F., E-mail: dufour@math.u-bordeaux1.fr and Piunovskiy, A. B., E-mail: piunov@liv.ac.uk},
abstractNote = {In this paper, we investigate an optimization problem for continuous-time Markov decision processes with both impulsive and continuous controls. We consider the so-called constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.},
doi = {10.1007/S00245-015-9310-8},
journal = {Applied Mathematics and Optimization},
number = 1,
volume = 74,
place = {United States},
year = 2016,
month = 8
}
  • We consider a discrete-time constrained discounted Markov decision process (MDP) with Borel state and action spaces, compact action sets, and lower semi-continuous cost functions. We introduce a set of hypotheses related to a positive weight function which allow us to consider cost functions that might not be bounded below by a constant, and which imply the solvability of the linear programming formulation of the constrained MDP. In particular, we establish the existence of a constrained optimal stationary policy. Our results are illustrated with an application to a fishery management problem.
  • Construction of stochastic models that describe the effective dynamics of observables of interest is an useful instrument in various fields of application, such as physics, climate science, and finance. We present a new technique for the construction of such models. From the timeseries of an observable, we construct a discrete-in-time Markov chain and calculate the eigenspectrum of its transition probability (or stochastic) matrix. As a next step we aim to find the generator of a continuous-time Markov chain whose eigenspectrum resembles the observed eigenspectrum as closely as possible, using an appropriate norm. The generator is found by solving a minimizationmore » problem: the norm is chosen such that the object function is quadratic and convex, so that the minimization problem can be solved using quadratic programming techniques. The technique is illustrated on various toy problems as well as on datasets stemming from simulations of molecular dynamics and of atmospheric flows.« less
  • This work develops asymptotically optimal controls for discrete-time singularly perturbed Markov decision processes (MDPs) having weak and strong interactions. The focus is on finite-state-space-MDP problems. The state space of the underlying Markov chain can be decomposed into a number of recurrent classes or a number of recurrent classes and a group of transient states. Using a hierarchical control approach, continuous-time limit problems that are much simpler to handle than the original ones are derived. Based on the optimal solutions for the limit problems, nearly optimal decisions for the original problems are obtained. The asymptotic optimality of such controls is provedmore » and the rate of convergence is provided. Infinite horizon problems are considered; both discounted costs and long-run average costs are examined.« less
  • Impulsive control of continuous-time Markov processes with risk- sensitive long-run average cost is considered. The most general impulsive control problem is studied under the restriction that impulses are in dyadic moments only. In a particular case of additive cost for impulses, the impulsive control problem is solved without restrictions on the moments of impulses.
  • In this paper we discuss MDP with distribution function criterion of first-passage time. Some properties of several kinds of optimal policies are given. Existence results and algorithms for these optimal policies are given in this paper.