A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability

Zhao, Tianqiao; Wang, Jianhui; Yue, Meng

doi:10.1109/tpwrs.2022.3233770

A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability

Journal Article · Wed Jan 04 00:00:00 EST 2023 · IEEE Transactions on Power Systems

DOI:https://doi.org/10.1109/tpwrs.2022.3233770· OSTI ID:2396603

^[1]; ^[2]; ^[1]

Brookhaven National Laboratory (BNL), Upton, NY (United States)
Southern Methodist Univ., Dallas, TX (United States)

Increasing integration of renewable resources brings more flexibility and poses new challenges to modern power systems, leading to highly nonlinear and complex dynamics. Here, this paper aims to provide a general solution framework to traditional control problems, such as frequency control and voltage control, which attempt to maintain the stability of either synchronous generators-governed or inverter-governed systems when subjected to a disturbance and simultaneously guarantee operational constraints, providing a complete complement to existing works on control design. Building on reinforcement learning (RL) and control barrier functions, the framework includes two subsystems, i.e., a model-free controller and a barrier-certification system, which discover RL-based control actions and sequentially filter them using a barrier certificate to satisfy operational constraints. Calculating a barrier function is generally challenging for a complex power system. This is addressed by representing the barrier function using neural networks (NNs) and data-based approaches. An adaptive method is introduced to certify the neural barrier function that perseveres barrier conditions, which is more compatible with online implementation. The proposed framework synthesizes a stabilizing controller that satisfies predefined safety regions. The effectiveness of the proposed framework is demonstrated via several comparative case studies.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE), Renewable Power Office

Grant/Contract Number:: SC0012704

OSTI ID:: 2396603

Report Number(s):: BNL-225797-2024-JAAM

Journal Information:: IEEE Transactions on Power Systems, Vol. 38, Issue 6; ISSN 0885-8950

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (38)

Definition and Classification of Power System Stability IEEE/CIGRE Joint Task Force on Stability Terms and Definitions Kundur, P.; Paserba, J.; Ajjarapu, V. IEEE Transactions on Power Systems, Vol. 19, Issue 3, p. 1387-1401 https://doi.org/10.1109/TPWRS.2004.825981	journal	August 2004
Transient Stability Assessment of Prone-to-Trip Renewable Generation Rich Power Systems Using Lyapunov's Direct Method Mishra, Chetan; Pal, Anamitra; Thorp, James S. IEEE Transactions on Sustainable Energy, Vol. 10, Issue 3 https://doi.org/10.1109/TSTE.2019.2905608	journal	July 2019
Distributed transient frequency control for power networks with stability and performance guarantees Zhang, Yifu; Cortés, Jorge Automatica, Vol. 105 https://doi.org/10.1016/j.automatica.2019.03.017	journal	July 2019
Distributed Stability Conditions for Power Systems With Heterogeneous Nonlinear Bus Dynamics Yang, Peng; Liu, Feng; Wang, Zhaojian IEEE Transactions on Power Systems, Vol. 35, Issue 3 https://doi.org/10.1109/TPWRS.2019.2951202	journal	May 2020
Distributed Frequency Control With Operational Constraints, Part I: Per-Node Power Balance Wang, Zhaojian; Liu, Feng; Low, Steven H. IEEE Transactions on Smart Grid, Vol. 10, Issue 1 https://doi.org/10.1109/TSG.2017.2731810	journal	January 2019
Lyapunov Functions Family Approach to Transient Stability Assessment Vu, Thanh Long; Turitsyn, Konstantin IEEE Transactions on Power Systems, Vol. 31, Issue 2 https://doi.org/10.1109/TPWRS.2015.2425885	journal	March 2016
Global transient stability and voltage regulation for power systems Guo, Y.; Hill, D. J.; Wang, Y. IEEE Transactions on Power Systems, Vol. 16, Issue 4 https://doi.org/10.1109/59.962413	journal	January 2001
A Neural Lyapunov Approach to Transient Stability Assessment of Power Electronics-Interfaced Networked Microgrids Huang, Tong; Gao, Sicun; Xie, Le IEEE Transactions on Smart Grid, Vol. 13, Issue 1 https://doi.org/10.1109/TSG.2021.3117889	journal	January 2022
A Transient Stability Assessment Framework in Power Electronic-Interfaced Distribution Systems Zhang, Yun; Xie, Le IEEE Transactions on Power Systems, Vol. 31, Issue 6 https://doi.org/10.1109/TPWRS.2016.2531745	journal	November 2016
Conditions for stability of droop-controlled inverter-based microgrids Schiffer, Johannes; Ortega, Romeo; Astolfi, Alessandro Automatica, Vol. 50, Issue 10 https://doi.org/10.1016/j.automatica.2014.08.009	journal	October 2014
Energy Management in Autonomous Microgrid Using Stability-Constrained Droop Control of Inverters Barklund, E.; Pogaku, N.; Prodanovic, M. IEEE Transactions on Power Electronics, Vol. 23, Issue 5 https://doi.org/10.1109/TPEL.2008.2001910	journal	September 2008
Stability Region of Droop-Controlled Distributed Generation in Autonomous Microgrids Pan, Yanfei; Chen, Laijun; Lu, Xiaonan IEEE Transactions on Smart Grid, Vol. 10, Issue 2 https://doi.org/10.1109/TSG.2018.2849084	journal	March 2019
Distributed Barrier Certificates for Safe Operation of Inverter-Based Microgrids Kundu, Soumya; Geng, Sijia; Nandanoori, Sai Pushpak 2019 American Control Conference (ACC) https://doi.org/10.23919/ACC.2019.8815296	conference	July 2019
Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems Wang, Wei; Yu, Nanpeng; Gao, Yuanqi IEEE Transactions on Smart Grid, Vol. 11, Issue 4 https://doi.org/10.1109/TSG.2019.2962625	journal	July 2020
Transient stability-constrained optimal power flow for online dispatch and nodal price evaluation in power systems with flexible AC transmission system devices Nguyen, T. T.; Nguyen, V. L.; Karimishad, A. IET Generation, Transmission & Distribution, Vol. 5, Issue 3 https://doi.org/10.1049/iet-gtd.2008.0527	journal	January 2011
Partial Feedback Linearizing Excitation Controller for Multimachine Power Systems to Improve Transient Stability Mahmud, M. A.; Pota, H. R.; Aldeen, M. IEEE Transactions on Power Systems, Vol. 29, Issue 2 https://doi.org/10.1109/TPWRS.2013.2283867	journal	March 2014
Control Barrier Function Based Quadratic Programs for Safety Critical Systems Ames, Aaron D.; Xu, Xiangru; Grizzle, Jessy W. IEEE Transactions on Automatic Control, Vol. 62, Issue 8 https://doi.org/10.1109/TAC.2016.2638961	journal	August 2017
Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations Duan, Jiajun; Shi, Di; Diao, Ruisheng IEEE Transactions on Power Systems, Vol. 35, Issue 1 https://doi.org/10.1109/TPWRS.2019.2941134	journal	January 2020
Safe Reinforcement Learning: Learning with Supervision Using a Constraint-Admissible Set Li, Zhaojian; Kalabic, Uros; Chu, Tianshu 2018 Annual American Control Conference (ACC) https://doi.org/10.23919/ACC.2018.8430770	conference	June 2018
Learning to Operate Distribution Networks With Safe Deep Reinforcement Learning Li, Hepeng; He, Haibo IEEE Transactions on Smart Grid, Vol. 13, Issue 3 https://doi.org/10.1109/TSG.2022.3142961	journal	May 2022
Deep-Reinforcement-Learning-Based Capacity Scheduling for PV-Battery Storage System Huang, Bin; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 12, Issue 3 https://doi.org/10.1109/TSG.2020.3047890	journal	May 2021
Learning-Based Model Predictive Control for Safe Exploration Koller, Torsten; Berkenkamp, Felix; Turchetta, Matteo 2018 IEEE Conference on Decision and Control (CDC) https://doi.org/10.1109/CDC.2018.8619572	conference	December 2018
End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks Cheng, Richard; Orosz, Gábor; Murray, Richard M. Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 33 https://doi.org/10.1609/aaai.v33i01.33013387	journal	July 2019
Synthesizing barrier certificates using neural networks Zhao, Hengjun; Zeng, Xia; Chen, Taolue Proceedings of the 23rd International Conference on Hybrid Systems: Computation and Control https://doi.org/10.1145/3365365.3382222	conference	April 2020
Compositional safety analysis using barrier certificates Sloth, Christoffer; Pappas, George J.; Wisniewski, Rafael Proceedings of the 15th ACM international conference on Hybrid Systems: Computation and Control https://doi.org/10.1145/2185632.2185639	conference	April 2012
Control Barrier Functions: Theory and Applications Ames, Aaron D.; Coogan, Samuel; Egerstedt, Magnus 2019 18th European Control Conference (ECC) https://doi.org/10.23919/ECC.2019.8796030	conference	June 2019
Learning Deep Neural Network Controllers for Dynamical Systems with Safety Guarantees: Invited Paper Deshmukh, Jyotirmoy V.; Kapinski, James P.; Yamaguchi, Tomoya 2019 IEEE/ACM International Conference on Computer-Aided Design (ICCAD) https://doi.org/10.1109/ICCAD45719.2019.8942130	conference	November 2019
Reinforcement Learning for Safety-Critical Control under Model Uncertainty, using Control Lyapunov Functions and Control Barrier Functions Choi, Jason; Castañeda, Fernando; Tomlin, Claire Robotics: Science and Systems 2020, Robotics: Science and Systems XVI https://doi.org/10.15607/RSS.2020.XVI.088	conference	July 2020
Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search Yan, Ziming; Xu, Yan IEEE Transactions on Power Systems, Vol. 34, Issue 2 https://doi.org/10.1109/TPWRS.2018.2881359	journal	March 2019
Reinforcement learning of motor skills with policy gradients Peters, Jan; Schaal, Stefan Neural Networks, Vol. 21, Issue 4 https://doi.org/10.1016/j.neunet.2008.02.003	journal	May 2008
Continuous control for robot based on deep reinforcement learning Zhang, Shansi Nanyang Technological University https://doi.org/10.32657/10356/90191	dissertation	January 2019
Safe Reinforcement Learning via Shielding Alshiekh, Mohammed; Bloem, Roderick; Ehlers, Rüdiger Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, Issue 1 https://doi.org/10.1609/aaai.v32i1.11797	journal	April 2018
Neural Lyapunov Control for Power System Transient Stability: A Deep Learning-Based Approach Zhao, Tianqiao; Wang, Jianhui; Lu, Xiaonan IEEE Transactions on Power Systems, Vol. 37, Issue 2 https://doi.org/10.1109/TPWRS.2021.3102857	journal	March 2022
Lyapunov-Regularized Reinforcement Learning for Power System Transient Stability Cui, Wenqi; Zhang, Baosen IEEE Control Systems Letters, Vol. 6 https://doi.org/10.1109/LCSYS.2021.3088068	journal	January 2022
Breaking the Hierarchy: Distributed Control and Economic Optimality in Microgrids Dorfler, Florian; Simpson-Porco, John W.; Bullo, Francesco IEEE Transactions on Control of Network Systems, Vol. 3, Issue 3 https://doi.org/10.1109/TCNS.2015.2459391	journal	September 2016
Secondary Frequency and Voltage Control of Islanded Microgrids via Distributed Averaging Simpson-Porco, John W.; Shafiee, Qobad; Dorfler, Florian IEEE Transactions on Industrial Electronics, Vol. 62, Issue 11 https://doi.org/10.1109/TIE.2015.2436879	journal	November 2015
Distributed Secondary Control Strategy for Microgrid Operation with Dynamic Boundaries Du, Yuhua; Lu, Xiaonan; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 10, Issue 5 https://doi.org/10.1109/TSG.2018.2879793	journal	September 2019
A Modified IEEE 118-Bus Test Case for Geomagnetic Disturbance Studies–Part I: Model Data Haddadi, Aboutaleb; Rezaei-Zare, Afshin; Gerin-Lajoie, Luc IEEE Transactions on Electromagnetic Compatibility, Vol. 62, Issue 3 https://doi.org/10.1109/TEMC.2019.2920271	journal	June 2020

Similar Records

Barrier Function-based Reinforcement Learning for Emergency Control of Power Systems

Conference · Mon Dec 13 00:00:00 EST 2021 · OSTI ID:1862295

Vu, Thanh Long; Mukherjee, Sayak; Huang, Renke; +1 more

Neural Lyapunov Differentiable Predictive Control

Conference · Wed Dec 07 00:00:00 EST 2022 · OSTI ID:1924633

Mukherjee, Sayak; Drgona, Jan; Tuor, Aaron R.; +2 more

Machine‐learning‐based construction of barrier functions and models for safe model predictive control

Journal Article · Wed Oct 06 00:00:00 EDT 2021 · AIChE Journal · OSTI ID:1824904

Chen, Scarlett; Wu, Zhe; Christofides, Panagiotis D.

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION
Power system stability
safe reinforcement learning
control barrier functions
neural networks
power system control

A Barrier-Certificated Reinforcement Learning Approach for Enhancing Power System Transient Stability

Citation Formats

References (38)

Similar Records

Related Subjects