Model-based and Model-free Designs for an Extended Continuous-time LQR with Exogenous Inputs

Mukherjee, Sayak; Bai, He; Chakrabortty, Aranya

doi:10.1016/j.sysconle.2021.104983

Model-based and Model-free Designs for an Extended Continuous-time LQR with Exogenous Inputs

Journal Article · Sun Aug 01 04:00:00 EDT 2021 · Systems and Control Letters

DOI:https://doi.org/10.1016/j.sysconle.2021.104983· OSTI ID:1819881

Mukherjee, Sayak ^[1]; Bai, He ^[2]; Chakrabortty, Aranya ^[3]

BATTELLE (PACIFIC NW LAB)
Oklahoma State University
North Carolina State University

We present an extended linear quadratic regulator (LQR) design for continuous-time linear time-invariant (LTI) systems in the presence of exogenous inputs. We first propose a model-based solution with cost minimization guarantees for states and inputs using dynamic programming (DP). The control law consists of a combination of the optimal state feedback and an additional optimal term dependent on the exogenous inputs. The control gains for the two components are obtained by solving a set of matrix differential equations. We provide these solutions for both finite horizons and steady-state cases. In the second part of the paper, we formulate a reinforcement learning (RL) based algorithm which does not need any model information except the input matrix, and can compute an approximate steady-state LQR gain using measurements of the states, the control inputs, and the exogenous inputs. Both model-based and data-driven optimal control algorithms are tested with a numerical example under different exogenous inputs showcasing the effectiveness of the designs.

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE

DOE Contract Number:: AC05-76RL01830

OSTI ID:: 1819881

Report Number(s):: PNNL-SA-156098

Journal Information:: Systems and Control Letters, Journal Name: Systems and Control Letters Vol. 154

Country of Publication:: United States

Language:: English

References (27)

Just-in-time control of timed event graphs: update of reference input, presence of uncontrollable input Ferrier, J. -L.; Hardouin, L.; Boimond, J. -L. IEEE Transactions on Automatic Control, Vol. 45, Issue 11 https://doi.org/10.1109/9.887652	journal	January 2000
On an iterative technique for Riccati equation computations Kleinman, D. IEEE Transactions on Automatic Control, Vol. 13, Issue 1 https://doi.org/10.1109/TAC.1968.1098829	journal	February 1968
Reduced-dimensional reinforcement learning control using singular perturbation approximations Mukherjee, Sayak; Bai, He; Chakrabortty, Aranya Automatica, Vol. 126 https://doi.org/10.1016/j.automatica.2020.109451	journal	April 2021
Accomodation of external disturbances in linear regulator and servomechanism problems Johnson, C. IEEE Transactions on Automatic Control, Vol. 16, Issue 6 https://doi.org/10.1109/TAC.1971.1099830	journal	December 1971
Efficient Learning of Distributed Linear-Quadratic Control Policies Fattahi, Salar; Matni, Nikolai; Sojoudi, Somayeh SIAM Journal on Control and Optimization, Vol. 58, Issue 5 https://doi.org/10.1137/19M1291108	journal	January 2020
An Output Regulator With Rejection of Time-Varying Disturbance: Experimental Validation on Clutch Slip Control Gao, Bingzhao; Hong, Jinlong; Qu, Ting IEEE Transactions on Control Systems Technology, Vol. 28, Issue 3 https://doi.org/10.1109/TCST.2019.2900211	journal	May 2020
Batch-to-Batch Finite-Horizon LQ Control for Unknown Discrete-Time Linear Systems Via Stochastic Extremum Seeking Liu, Shu-Jun; Krstic, Miroslav; Basar, Tamer IEEE Transactions on Automatic Control, Vol. 62, Issue 8 https://doi.org/10.1109/TAC.2016.2620379	journal	August 2017
The active disturbance rejection and sliding mode control approach to the stabilization of the Euler–Bernoulli beam equation with boundary input disturbance Guo, Bao-Zhu; Jin, Feng-Fei Automatica, Vol. 49, Issue 9 https://doi.org/10.1016/j.automatica.2013.06.018	journal	September 2013
Finite-horizon LQ control for unknown discrete-time linear systems via extremum seeking Frihauf, Paul; Krstic, Miroslav; Başar, Tamer European Journal of Control, Vol. 19, Issue 5 https://doi.org/10.1016/j.ejcon.2013.05.015	journal	September 2013
Adaptive optimal control for continuous-time linear systems based on policy iteration Vrabie, D.; Pastravanu, O.; Abu-Khalaf, M. Automatica, Vol. 45, Issue 2 https://doi.org/10.1016/j.automatica.2008.08.017	journal	February 2009
Iterative learning control based on extremum seeking Khong, Sei Zhen; Nešić, Dragan; Krstić, Miroslav Automatica, Vol. 66 https://doi.org/10.1016/j.automatica.2015.12.019	journal	April 2016
The generalized H2 control problem Rotea, Mario A. Automatica, Vol. 29, Issue 2 https://doi.org/10.1016/0005-1098(93)90130-L	journal	March 1993
Optimal and Autonomous Control Using Reinforcement Learning: A Survey Kiumarsi, Bahare; Vamvoudakis, Kyriakos G.; Modares, Hamidreza IEEE Transactions on Neural Networks and Learning Systems, Vol. 29, Issue 6 https://doi.org/10.1109/TNNLS.2017.2773458	journal	June 2018
Q-learning for continuous-time linear systems: A model-free infinite horizon optimal control approach Vamvoudakis, Kyriakos G. Systems & Control Letters, Vol. 100 https://doi.org/10.1016/j.sysconle.2016.12.003	journal	February 2017
Computational adaptive optimal control for continuous-time linear systems with completely unknown dynamics Jiang, Yu; Jiang, Zhong-Ping Automatica, Vol. 48, Issue 10 https://doi.org/10.1016/j.automatica.2012.06.096	journal	October 2012
Off-Policy Reinforcement Learning for $ H_\infty $ Control Design Luo, Biao; Wu, Huai-Ning; Huang, Tingwen IEEE Transactions on Cybernetics, Vol. 45, Issue 1 https://doi.org/10.1109/TCYB.2014.2319577	journal	January 2015
From PID to Active Disturbance Rejection Control Han, Jingqing IEEE Transactions on Industrial Electronics, Vol. 56, Issue 3 https://doi.org/10.1109/TIE.2008.2011621	journal	March 2009
Robust Stabilization and Performance Recovery of Nonlinear Systems With Unmodeled Dynamics Chakrabortty, A.; Arcak, M. IEEE Transactions on Automatic Control, Vol. 54, Issue 6 https://doi.org/10.1109/TAC.2009.2015554	journal	June 2009
Scalable Designs for Reinforcement Learning-Based Wide-Area Damping Control Mukherjee, Sayak; Chakrabortty, Aranya; Bai, He IEEE Transactions on Smart Grid, Vol. 12, Issue 3 https://doi.org/10.1109/TSG.2021.3050419	journal	May 2021
On the convergence of an extended state observer for nonlinear systems with uncertainty Guo, Bao-Zhu; Zhao, Zhi-liang Systems & Control Letters, Vol. 60, Issue 6 https://doi.org/10.1016/j.sysconle.2011.03.008	journal	June 2011
Disturbance-Observer-Based Control and Related Methods—An Overview Chen, Wen-Hua; Yang, Jun; Guo, Lei IEEE Transactions on Industrial Electronics, Vol. 63, Issue 2 https://doi.org/10.1109/TIE.2015.2478397	journal	February 2016
Formulas for Data-Driven Control: Stabilization, Optimality, and Robustness De Persis, Claudio; Tesi, Pietro IEEE Transactions on Automatic Control, Vol. 65, Issue 3 https://doi.org/10.1109/TAC.2019.2959924	journal	March 2020
Wind Speed and Direction Detection by Means of Solid-state Anemometers Embedded on Small Quadcopters Bruschi, P.; Piotto, M.; Dell’Agnello, F. Procedia Engineering, Vol. 168 https://doi.org/10.1016/j.proeng.2016.11.274	journal	January 2016
Distributed adaptive dynamic programming for data-driven optimal control Tang, Wentao; Daoutidis, Prodromos Systems & Control Letters, Vol. 120 https://doi.org/10.1016/j.sysconle.2018.08.002	journal	October 2018
An extended linear quadratic regulator for LTI systems with exogenous inputs Singh, Abhinav Kumar; Pal, Bikash C. Automatica, Vol. 76 https://doi.org/10.1016/j.automatica.2016.10.014	journal	February 2017
Corrective control of asynchronous machines with uncontrollable inputs: application to single-event-upset error counters Kwak, S. W.; Yang, J. -M. IET Control Theory & Applications, Vol. 4, Issue 11 https://doi.org/10.1049/iet-cta.2009.0397	journal	November 2010
Disturbance Observer Based Control for Nonlinear Systems Chen, W. -H. IEEE/ASME Transactions on Mechatronics, Vol. 9, Issue 4 https://doi.org/10.1109/TMECH.2004.839034	journal	December 2004

Similar Records

Reinforcement Learning of Structured Stabilizing Control for Linear Systems With Unknown State Matrix

Journal Article · Tue Feb 28 19:00:00 EST 2023 · IEEE Transactions on Automatic Control · OSTI ID:1971347

Online Optimization as a Feedback Controller: Stability and Tracking

Journal Article · Sun Mar 24 20:00:00 EDT 2019 · IEEE Transactions on Control of Network Systems · OSTI ID:1526200

A Secure Learning Control Strategy via Dynamic Camouflaging for Unknown Dynamical Systems under Attacks

Conference · Thu Dec 30 23:00:00 EST 2021 · OSTI ID:1856244

Model-based and Model-free Designs for an Extended Continuous-time LQR with Exogenous Inputs

Citation Formats

References (27)

Similar Records

Related Subjects