skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Extremum seeking for optimal control problems with unknown time-varying systems and unknown objective functions

Abstract

We consider the problem of optimal feedback control of an unknown, noisy, time-varying, dynamic system that is initialized repeatedly. Examples include a robotic manipulator which must perform the same motion, such as assisting a human, repeatedly and accelerating cavities in particle accelerators which are turned on for a fraction of a second with given initial conditions and vary slowly due to temperature fluctuations. In this paper, we present an approach that applies to systems of practical interest. The method presented here is model independent; does not require knowledge of the objective function; is robust to measurement noise; is applicable for any set of initial conditions; is applicable to simultaneously controlling an arbitrary number of parameters; and may be implemented with a broad range of continuous or discontinuous functions such as sine or square waves. For systems with convex cost functions we prove that our algorithm will produce controllers that approach the minimal cost. For linear systems we reproduce the cost minimizing linear quadratic regulator optimal controller that could have been designed analytically had the system and cost function been known. We demonstrate the effectiveness of the algorithm with simulation studies of noisy and time-varying systems.

Authors:
ORCiD logo [1];  [2]
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  2. Stanford Univ., CA (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1688763
Report Number(s):
LA-UR-20-21910
Journal ID: ISSN 0890-6327
Grant/Contract Number:  
89233218CNA000001
Resource Type:
Accepted Manuscript
Journal Name:
International Journal of Adaptive Control and Signal Processing
Additional Journal Information:
Journal Name: International Journal of Adaptive Control and Signal Processing; Journal ID: ISSN 0890-6327
Country of Publication:
United States
Language:
English
Subject:
extremum seeking; industrial applications of optimal control; iterative schemes; optimal control optimal; controller synthesis for systems with uncertainties; time‐varying systems

Citation Formats

Scheinker, Alexander, and Scheinker, David. Extremum seeking for optimal control problems with unknown time-varying systems and unknown objective functions. United States: N. p., 2020. Web. https://doi.org/10.1002/acs.3097.
Scheinker, Alexander, & Scheinker, David. Extremum seeking for optimal control problems with unknown time-varying systems and unknown objective functions. United States. https://doi.org/10.1002/acs.3097
Scheinker, Alexander, and Scheinker, David. Tue . "Extremum seeking for optimal control problems with unknown time-varying systems and unknown objective functions". United States. https://doi.org/10.1002/acs.3097. https://www.osti.gov/servlets/purl/1688763.
@article{osti_1688763,
title = {Extremum seeking for optimal control problems with unknown time-varying systems and unknown objective functions},
author = {Scheinker, Alexander and Scheinker, David},
abstractNote = {We consider the problem of optimal feedback control of an unknown, noisy, time-varying, dynamic system that is initialized repeatedly. Examples include a robotic manipulator which must perform the same motion, such as assisting a human, repeatedly and accelerating cavities in particle accelerators which are turned on for a fraction of a second with given initial conditions and vary slowly due to temperature fluctuations. In this paper, we present an approach that applies to systems of practical interest. The method presented here is model independent; does not require knowledge of the objective function; is robust to measurement noise; is applicable for any set of initial conditions; is applicable to simultaneously controlling an arbitrary number of parameters; and may be implemented with a broad range of continuous or discontinuous functions such as sine or square waves. For systems with convex cost functions we prove that our algorithm will produce controllers that approach the minimal cost. For linear systems we reproduce the cost minimizing linear quadratic regulator optimal controller that could have been designed analytically had the system and cost function been known. We demonstrate the effectiveness of the algorithm with simulation studies of noisy and time-varying systems.},
doi = {10.1002/acs.3097},
journal = {International Journal of Adaptive Control and Signal Processing},
number = ,
volume = ,
place = {United States},
year = {2020},
month = {2}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Reinforcement Learning for Partially Observable Dynamic Processes: Adaptive Dynamic Programming Using Measured Output Data
journal, February 2011

  • Lewis, F. L.; Vamvoudakis, K. G.
  • IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 41, Issue 1
  • DOI: 10.1109/TSMCB.2010.2043839

Constrained extremum seeking stabilization of systems not affine in control: Extremum Seeking Stabilization of Systems Not Affine in Control
journal, July 2017

  • Scheinker, Alexander; Scheinker, David
  • International Journal of Robust and Nonlinear Control, Vol. 28, Issue 2
  • DOI: 10.1002/rnc.3886

Iterative learning control based on extremum seeking
journal, April 2016


On extremum seeking controllers based on the Lie bracket approximation in domains with obstacles
journal, December 2018

  • Grushkovskaya, Victoria; Zuyev, Alexander; Ebenbauer, Christian
  • PAMM, Vol. 18, Issue 1
  • DOI: 10.1002/pamm.201800298

A proportional-integral extremum-seeking controller design technique
journal, March 2017


Adaptive optimal control for continuous-time linear systems based on policy iteration
journal, February 2009


Design and simulation of extremum-seeking open-loop optimal control of current profile in the DIII-D tokamak
journal, September 2008


On a class of generating vector fields for the extremum seeking problem: Lie bracket approximation and stability properties
journal, August 2018


Integrals which are convex functionals. II
journal, January 1971


Extremum seeking control with an adaptive dither signal
journal, March 2019


Reinforcement -learning for optimal tracking control of linear discrete-time systems with unknown dynamics
journal, April 2014


Mixing enhancement in 2D magnetohydrodynamic channel flow by extremum seeking boundary control
conference, June 2009


Swinging up the Stephenson-Kapitza pendulum
conference, December 2013

  • Michalowsky, Simon; Ebenbauer, Christian
  • 2013 IEEE 52nd Annual Conference on Decision and Control (CDC), 52nd IEEE Conference on Decision and Control
  • DOI: 10.1109/CDC.2013.6760498

Adaptive iterative learning control of uncertain robotic systems
journal, March 2000

  • Choi, J. Y.; Lee, J. S.
  • IEE Proceedings - Control Theory and Applications, Vol. 147, Issue 2
  • DOI: 10.1049/ip-cta:20000138

Minimum-Seeking for CLFs: Universal Semiglobally Stabilizing Feedback Under Unknown Control Directions
journal, May 2013

  • Scheinker, Alexander; Krstic, Miroslav
  • IEEE Transactions on Automatic Control, Vol. 58, Issue 5
  • DOI: 10.1109/TAC.2012.2225514

Finite-horizon LQ control for unknown discrete-time linear systems via extremum seeking
journal, September 2013


Newton-based extremum seeking: A second-order Lie bracket approximation approach
journal, July 2019


Extremum seeking-based adaptive control for electromagnetic actuators
journal, September 2014


Bettering operation of Robots by learning
journal, July 1984

  • Arimoto, Suguru; Kawamura, Sadao; Miyazaki, Fumio
  • Journal of Robotic Systems, Vol. 1, Issue 2
  • DOI: 10.1002/rob.4620010203

Formation Shape Control Based on Distance Measurements Using Lie Bracket Approximations
journal, January 2018

  • Suttner, Raik; Sun, Zhiyong
  • SIAM Journal on Control and Optimization, Vol. 56, Issue 6
  • DOI: 10.1137/18M117131X

Multi-parametric extremum seeking-based iterative feedback gains tuning for nonlinear control: ITERATIVE FEEDBACK GAINS TUNING FOR NONLINEAR CONTROL
journal, April 2016

  • Benosman, Mouhacine
  • International Journal of Robust and Nonlinear Control, Vol. 26, Issue 18
  • DOI: 10.1002/rnc.3547

A family of extremum seeking laws for a unicycle model with a moving target: theoretical and experimental studies
conference, June 2018

  • Grushkovskaya, Victoria; Michalowsky, Simon; Zuyev, Alexander
  • 2018 17th European Control Conference (ECC), 2018 European Control Conference (ECC)
  • DOI: 10.23919/ECC.2018.8550280

Simultaneous stabilization and optimization of unknown, time-varying systems
conference, June 2013


Extremum-seeking control for nonlinear systems with periodic steady-state outputs
journal, June 2013


Iterative learning impedance control for rehabilitation robots driven by series elastic actuators
journal, April 2018


Optimal control of unknown nonaffine nonlinear discrete-time systems based on adaptive dynamic programming
journal, August 2012


A constrained extremum-seeking control approach: A CONSTRAINED EXTREMUM-SEEKING CONTROL APPROACH
journal, October 2014

  • Guay, Martin; Moshksar, Ehsan; Dochain, Denis
  • International Journal of Robust and Nonlinear Control, Vol. 25, Issue 16
  • DOI: 10.1002/rnc.3254

Learning-based iterative modular adaptive control for nonlinear systems: Learning-based iterative modular adaptive control for nonlinear systems
journal, June 2018

  • Benosman, Mouhacine; Farahmand, Amir-Massoud; Xia, Meng
  • International Journal of Adaptive Control and Signal Processing, Vol. 33, Issue 2
  • DOI: 10.1002/acs.2892

Flatness-Based Extremum-Seeking Control Over Periodic Orbits
journal, October 2007

  • Guay, M.; Dochain, D.; Perrier, M.
  • IEEE Transactions on Automatic Control, Vol. 52, Issue 10
  • DOI: 10.1109/TAC.2007.904255

Microwave Electronics
journal, October 1946


Adaptive output optimal control algorithm for unknown system dynamics based on policy iteration
conference, June 2010

  • Ohtake, Susumu; Yamakita, Masaki
  • 2010 American Control Conference (ACC 2010), Proceedings of the 2010 American Control Conference
  • DOI: 10.1109/ACC.2010.5531522

Integrals which are convex functionals
journal, January 1968


Linear quadratic optimal learning control (LQL)
journal, January 2000


Batch-to-Batch Finite-Horizon LQ Control for Unknown Discrete-Time Linear Systems Via Stochastic Extremum Seeking
journal, August 2017

  • Liu, Shu-Jun; Krstic, Miroslav; Basar, Tamer
  • IEEE Transactions on Automatic Control, Vol. 62, Issue 8
  • DOI: 10.1109/TAC.2016.2620379

Bounded extremum seeking with discontinuous dithers
journal, July 2016


Lie bracket approximation of extremum seeking systems
journal, June 2013


Iterative Learning Control: Brief Survey and Categorization
journal, November 2007

  • Ahn, Hyo-Sung; Chen, YangQuan; Moore, Kevin L.
  • IEEE Transactions on Systems, Man and Cybernetics, Part C (Applications and Reviews), Vol. 37, Issue 6
  • DOI: 10.1109/TSMCC.2007.905759

Power Optimization for Photovoltaic Microconverters Using Multivariable Newton-Based Extremum Seeking
journal, November 2014

  • Ghaffari, Azad; Krstic, Miroslav; Seshagiri, Sridhar
  • IEEE Transactions on Control Systems Technology, Vol. 22, Issue 6
  • DOI: 10.1109/TCST.2014.2301172

Optimal control of unknown affine nonlinear discrete-time systems using offline-trained neural networks with proof of convergence
journal, July 2009


Newton-based stochastic extremum seeking
journal, March 2014