Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control

Zhang, Ying; Yue, Meng; Wang, Jianhui

doi:10.1016/j.epsr.2023.109136

Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control

Journal Article · Wed Jan 25 00:00:00 EST 2023 · Electric Power Systems Research

DOI:https://doi.org/10.1016/j.epsr.2023.109136· OSTI ID:1992867

^[1]; Yue, Meng ^[1]; Wang, Jianhui ^[2]

Brookhaven National Laboratory (BNL), Upton, NY (United States)
Southern Methodist Univ., Dallas, TX (United States)

Electric overloading conditions and contingencies put modern power systems at risk of voltage collapse and blackouts. Load shedding is crucial to maintain voltage stability for grid emergency control. However, the rule- or model-based schemes rely on accurate dynamic system models and face considerable challenges in adapting to various operating conditions and uncertain event occurrences. Here, to address these issues, this paper proposes a novel deep reinforcement learning (DRL)-based voltage stability control algorithm with automatic entropy adjustment (AEA) for grid emergency control. Various dynamic network components for complex system operations are modeled to construct the DRL environment. An off-policy soft actor-critic architecture is developed to maximize the expected reward and policy entropy simultaneously. The AEA mechanism is proposed to facilitate the policy maximum entropy procedure, and the proposed method can automatically provide effective discrete and continuous actions against various fault scenarios. Our approach accomplishes high sampling efficiency, scalability, and auto-adaptivity of the control policies under high uncertainties. Comparative studies with the existing DRL-based control methods in IEEE benchmarks indicate salient performance improvement of the proposed method for dynamic system emergency control.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE; USDOE Office of Electricity (OE), Advanced Grid Research & Development. Power Systems Engineering Research

Grant/Contract Number:: SC0012704

OSTI ID:: 1992867

Alternate ID(s):: OSTI ID: 1960881

Report Number(s):: BNL-224619-2023-JAAM

Journal Information:: Electric Power Systems Research, Journal Name: Electric Power Systems Research Vol. 217; ISSN 0378-7796

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (17)

A critical review of the state‐of‐art schemes for under voltage load shedding Larik, Raja Masood; Mustafa, Mohd Wazir; Aman, Muhammad Naveed International Transactions on Electrical Energy Systems, Vol. 29, Issue 5 https://doi.org/10.1002/2050-7038.2828	journal	January 2019
A Hierarchical Data-Driven Method for Event-Based Load Shedding Against Fault-Induced Delayed Voltage Recovery in Power Systems Li, Qiaoqiao; Xu, Yan; Ren, Chao IEEE Transactions on Industrial Informatics, Vol. 17, Issue 1 https://doi.org/10.1109/TII.2020.2993807	journal	January 2021
Assessing Short-Term Voltage Stability of Electric Power Systems by a Hierarchical Intelligent System Xu, Yan; Zhang, Rui; Zhao, Junhua IEEE Transactions on Neural Networks and Learning Systems, Vol. 27, Issue 8 https://doi.org/10.1109/TNNLS.2015.2441706	journal	August 2016
Adaptation in load shedding under vulnerable operating conditions Jung, J.; Tanimoto, S. L. IEEE Transactions on Power Systems, Vol. 17, Issue 4 https://doi.org/10.1109/TPWRS.2002.805023	journal	November 2002
Coordinated system protection scheme against voltage collapse using heuristic search and predictive control Larsson, M.; Karlsson, D. IEEE Transactions on Power Systems, Vol. 18, Issue 3 https://doi.org/10.1109/TPWRS.2003.814852	journal	August 2003
Power Systems Stability Control: Reinforcement Learning Framework Ernst, D.; Glavic, M.; Wehenkel, L. IEEE Transactions on Power Systems, Vol. 19, Issue 1 https://doi.org/10.1109/TPWRS.2003.821457	journal	February 2004
Optimal Coordinated Voltage Control for Power System Voltage Stability Wen, J. Y.; Wu, Q. H.; Turner, D. R. IEEE Transactions on Power Systems, Vol. 19, Issue 2 https://doi.org/10.1109/TPWRS.2004.825897	journal	May 2004
Model Predictive Control-Based Real-Time Power System Protection Schemes Jin, Licheng; Kumar, Ratnesh; Elia, Nicola IEEE Transactions on Power Systems, Vol. 25, Issue 2 https://doi.org/10.1109/TPWRS.2009.2034748	journal	May 2010
Decision Tree-Based Preventive and Corrective Control Applications for Dynamic Security Enhancement in Power Systems Genc, Istemihan; Diao, Ruisheng; Vittal, Vijay IEEE Transactions on Power Systems, Vol. 25, Issue 3 https://doi.org/10.1109/TPWRS.2009.2037006	journal	August 2010
An Efficient Optimal Control Method for Open-Loop Transient Stability Emergency Control Li, Zhihao; Yao, Guoqiang; Geng, Guangchao IEEE Transactions on Power Systems, Vol. 32, Issue 4 https://doi.org/10.1109/TPWRS.2016.2629620	journal	July 2017
Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search Yan, Ziming; Xu, Yan IEEE Transactions on Power Systems, Vol. 34, Issue 2 https://doi.org/10.1109/TPWRS.2018.2881359	journal	March 2019
Accelerated Derivative-Free Deep Reinforcement Learning for Large-Scale Grid Emergency Voltage Control Huang, Renke; Chen, Yujiao; Yin, Tianzhixi IEEE Transactions on Power Systems, Vol. 37, Issue 1 https://doi.org/10.1109/TPWRS.2021.3095179	journal	January 2022
Adaptive Power System Emergency Control Using Deep Reinforcement Learning Huang, Qiuhua; Huang, Renke; Hao, Weituo IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2933191	journal	March 2020
Safe Off-Policy Deep Reinforcement Learning Algorithm for Volt-VAR Control in Power Distribution Systems Wang, Wei; Yu, Nanpeng; Gao, Yuanqi IEEE Transactions on Smart Grid, Vol. 11, Issue 4 https://doi.org/10.1109/TSG.2019.2962625	journal	July 2020
Graph-Based Faulted Line Identification Using Micro-PMU Data in Distribution Systems Zhang, Ying; Wang, Jianhui; Khodayar, Mohammad E. IEEE Transactions on Smart Grid, Vol. 11, Issue 5 https://doi.org/10.1109/TSG.2020.2988349	journal	September 2020
Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems Zhang, Ying; Wang, Xinan; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 12, Issue 1 https://doi.org/10.1109/TSG.2020.3010130	journal	January 2021
Decentralized Voltage Control of PowerSystems Using Multi-agent Systems Shahbazi, Hamidreza; Karbalaei, Farid Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 2 https://doi.org/10.35833/MPCE.2018.000628	journal	January 2020

Similar Records

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Journal Article · Sun Nov 05 19:00:00 EST 2023 · Machine Learning · OSTI ID:2345752

Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Journal Article · Wed Jan 24 19:00:00 EST 2024 · IEEE Transactions on Neural Networks and Learning Systems · OSTI ID:2433976

Adaptive Load Shedding for Grid Emergency Control via Deep Reinforcement Learning

Journal Article · Sun Jul 25 20:00:00 EDT 2021 · IEEE Power & Energy Society General Meeting · OSTI ID:1890215

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION
Deep reinforcement learning
Grid emergency control
Soft actor-critic
Voltage stability

Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control

Citation Formats

References (17)

Similar Records

Related Subjects