Impulsive Control for ContinuousTime Markov Decision Processes: A Linear Programming Approach
Abstract
In this paper, we investigate an optimization problem for continuoustime Markov decision processes with both impulsive and continuous controls. We consider the socalled constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.
 Authors:
 Bordeaux INP, IMB, UMR CNRS 5251 (France)
 University of Liverpool, Department of Mathematical Sciences (United Kingdom)
 Publication Date:
 OSTI Identifier:
 22617265
 Resource Type:
 Journal Article
 Resource Relation:
 Journal Name: Applied Mathematics and Optimization; Journal Volume: 74; Journal Issue: 1; Other Information: Copyright (c) 2016 Springer Science+Business Media New York; http://www.springerny.com; Country of input: International Atomic Energy Agency (IAEA)
 Country of Publication:
 United States
 Language:
 English
 Subject:
 97 MATHEMATICAL METHODS AND COMPUTING; LINEAR PROGRAMMING; MARKOV PROCESS; OCCUPATIONS; OPTIMAL CONTROL; OPTIMIZATION; PULSES
Citation Formats
Dufour, F., Email: dufour@math.ubordeaux1.fr, and Piunovskiy, A. B., Email: piunov@liv.ac.uk. Impulsive Control for ContinuousTime Markov Decision Processes: A Linear Programming Approach. United States: N. p., 2016.
Web. doi:10.1007/S0024501593108.
Dufour, F., Email: dufour@math.ubordeaux1.fr, & Piunovskiy, A. B., Email: piunov@liv.ac.uk. Impulsive Control for ContinuousTime Markov Decision Processes: A Linear Programming Approach. United States. doi:10.1007/S0024501593108.
Dufour, F., Email: dufour@math.ubordeaux1.fr, and Piunovskiy, A. B., Email: piunov@liv.ac.uk. 2016.
"Impulsive Control for ContinuousTime Markov Decision Processes: A Linear Programming Approach". United States.
doi:10.1007/S0024501593108.
@article{osti_22617265,
title = {Impulsive Control for ContinuousTime Markov Decision Processes: A Linear Programming Approach},
author = {Dufour, F., Email: dufour@math.ubordeaux1.fr and Piunovskiy, A. B., Email: piunov@liv.ac.uk},
abstractNote = {In this paper, we investigate an optimization problem for continuoustime Markov decision processes with both impulsive and continuous controls. We consider the socalled constrained problem where the objective of the controller is to minimize a total expected discounted optimality criterion associated with a cost rate function while keeping other performance criteria of the same form, but associated with different cost rate functions, below some given bounds. Our model allows multiple impulses at the same time moment. The main objective of this work is to study the associated linear program defined on a space of measures including the occupation measures of the controlled process and to provide sufficient conditions to ensure the existence of an optimal control.},
doi = {10.1007/S0024501593108},
journal = {Applied Mathematics and Optimization},
number = 1,
volume = 74,
place = {United States},
year = 2016,
month = 8
}

We consider a discretetime constrained discounted Markov decision process (MDP) with Borel state and action spaces, compact action sets, and lower semicontinuous cost functions. We introduce a set of hypotheses related to a positive weight function which allow us to consider cost functions that might not be bounded below by a constant, and which imply the solvability of the linear programming formulation of the constrained MDP. In particular, we establish the existence of a constrained optimal stationary policy. Our results are illustrated with an application to a fishery management problem.

Fitting timeseries by continuoustime Markov chains: A quadratic programming approach
Construction of stochastic models that describe the effective dynamics of observables of interest is an useful instrument in various fields of application, such as physics, climate science, and finance. We present a new technique for the construction of such models. From the timeseries of an observable, we construct a discreteintime Markov chain and calculate the eigenspectrum of its transition probability (or stochastic) matrix. As a next step we aim to find the generator of a continuoustime Markov chain whose eigenspectrum resembles the observed eigenspectrum as closely as possible, using an appropriate norm. The generator is found by solving a minimizationmore » 
Nearly optimal control of singularly perturbed Markov decision processes in discrete time
This work develops asymptotically optimal controls for discretetime singularly perturbed Markov decision processes (MDPs) having weak and strong interactions. The focus is on finitestatespaceMDP problems. The state space of the underlying Markov chain can be decomposed into a number of recurrent classes or a number of recurrent classes and a group of transient states. Using a hierarchical control approach, continuoustime limit problems that are much simpler to handle than the original ones are derived. Based on the optimal solutions for the limit problems, nearly optimal decisions for the original problems are obtained. The asymptotic optimality of such controls is provedmore » 
On RiskSensitive Ergodic Impulsive Control of Markov Processes
Impulsive control of continuoustime Markov processes with risk sensitive longrun average cost is considered. The most general impulsive control problem is studied under the restriction that impulses are in dyadic moments only. In a particular case of additive cost for impulses, the impulsive control problem is solved without restrictions on the moments of impulses. 
Markov Decision Processes with Distribution Function Criterion of FirstPassage Time
In this paper we discuss MDP with distribution function criterion of firstpassage time. Some properties of several kinds of optimal policies are given. Existence results and algorithms for these optimal policies are given in this paper.