Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Zhang, Ying; Yue, Meng; Wang, Jianhui; Yoo, Shinjae

doi:10.1109/tnnls.2023.3341334

Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Journal Article · Thu Jan 25 00:00:00 EST 2024 · IEEE Transactions on Neural Networks and Learning Systems

DOI:https://doi.org/10.1109/tnnls.2023.3341334· OSTI ID:2433976

^[1]; ^[2]; ^[3]; ^[2]

Oklahoma State Univ., Stillwater, OK (United States)
Brookhaven National Laboratory (BNL), Upton, NY (United States)
Southern Methodist Univ., Dallas, TX (United States)

Grid emergency voltage control (GEVC) is paramount in electric power systems to improve voltage stability and prevent cascading outages and blackouts in case of contingencies. While most deep reinforcement learning (DRL)-based paradigms perform single agents in a static environment, real-world agents for GEVC are expected to cooperate in a dynamically shifting grid. Moreover, due to high uncertainties from combinatory natures of various contingencies and load consumption, along with the complexity of dynamic grid operation, the data efficiency and control performance of the existing DRL-based methods are challenged. To address these limitations, we propose a multi-agent graph-attention (GATT)-based DRL algorithm for GEVC in multi-area power systems. Here, we develop graph convolutional network (GCN)-based agents for feature representation of the graph-structured voltages to improve the decision accuracy in a data-efficient manner. Furthermore, a cutting-edge attention mechanism concentrates on effective information sharing among multiple agents, synergizing different-sized subnetworks in the grid for cooperative learning. We address several key challenges in the existing DRL-based GEVC approaches, including low scalability and poor stability against high uncertainties. Test results in the IEEE benchmark system verify the advantages of the proposed method over several recent multi-agent DRL-based algorithms.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Electricity (OE), Advanced Grid Research & Development. Power Systems Engineering Research

Grant/Contract Number:: SC0012704

OSTI ID:: 2433976

Report Number(s):: BNL--225967-2024-JAAM

Journal Information:: IEEE Transactions on Neural Networks and Learning Systems, Journal Name: IEEE Transactions on Neural Networks and Learning Systems Journal Issue: 3 Vol. 35; ISSN 2162-237X

Publisher:: IEEE Computational Intelligence SocietyCopyright Statement

Country of Publication:: United States

Language:: English

References (27)

A critical review of the state‐of‐art schemes for under voltage load shedding Larik, Raja Masood; Mustafa, Mohd Wazir; Aman, Muhammad Naveed International Transactions on Electrical Energy Systems, Vol. 29, Issue 5 https://doi.org/10.1002/2050-7038.2828	journal	January 2019
Leader–follower formation control of nonholonomic mobile robots with input constraints Consolini, Luca; Morbidi, Fabio; Prattichizzo, Domenico Automatica, Vol. 44, Issue 5 https://doi.org/10.1016/j.automatica.2007.09.019	journal	May 2008
Voltage instability in interconnected power systems: a simulation approach Lachs, W. R.; Sutanto, D. IEEE Transactions on Power Systems, Vol. 7, Issue 2 https://doi.org/10.1109/59.141782	journal	May 1992
Load Shedding Scheme with Deep Reinforcement Learning to Improve Short-term Voltage Stability Zhang, Jingyi; Lu, Chao; Fang, Chen 2018 IEEE Innovative Smart Grid Technologies - Asia (ISGT Asia) https://doi.org/10.1109/ISGT-Asia.2018.8467877	conference	May 2018
Adaptive Load Shedding for Grid Emergency Control via Deep Reinforcement Learning Zhang, Ying; Yue, Meng; Wang, Jianhui 2021 IEEE Power & Energy Society General Meeting (PESGM) https://doi.org/10.1109/PESGM46819.2021.9638058	conference	July 2021
Leader-to-Formation Stability Tanner, H. G.; Pappas, G. J.; Kumar, V. IEEE Transactions on Robotics and Automation, Vol. 20, Issue 3 https://doi.org/10.1109/TRA.2004.825275	journal	June 2004
Emergency Voltage Stability Controls: an Overview Van Cutsem, T.; Vournas, C. D. 2007 IEEE Power Engineering Society General Meeting https://doi.org/10.1109/pes.2007.386089	conference	June 2007
A Multi-agent approach to coordination of different emergency control devices against voltage collapse Panasetsky, D. A.; Voropai, N. I. 2009 IEEE Bucharest PowerTech https://doi.org/10.1109/ptc.2009.5281995	conference	June 2009
A Hierarchical Data-Driven Method for Event-Based Load Shedding Against Fault-Induced Delayed Voltage Recovery in Power Systems Li, Qiaoqiao; Xu, Yan; Ren, Chao IEEE Transactions on Industrial Informatics, Vol. 17, Issue 1 https://doi.org/10.1109/tii.2020.2993807	journal	January 2021
Toward a Reinforcement Learning Environment Toolbox for Intelligent Electric Motor Control Traue, Arne; Book, Gerrit; Kirchgassner, Wilhelm IEEE Transactions on Neural Networks and Learning Systems, Vol. 33, Issue 3 https://doi.org/10.1109/tnnls.2020.3029573	journal	March 2022
Resilient Optimal Defensive Strategy of TSK Fuzzy-Model-Based Microgrids’ System via a Novel Reinforcement Learning Approach Zhang, Huifeng; Yue, Dong; Dou, Chunxia IEEE Transactions on Neural Networks and Learning Systems, Vol. 34, Issue 4 https://doi.org/10.1109/tnnls.2021.3105668	journal	April 2023
Adaptation in load shedding under vulnerable operating conditions Jung, J.; Tanimoto, S. L. IEEE Transactions on Power Systems, Vol. 17, Issue 4 https://doi.org/10.1109/tpwrs.2002.805023	journal	November 2002
Coordinated system protection scheme against voltage collapse using heuristic search and predictive control Larsson, M.; Karlsson, D. IEEE Transactions on Power Systems, Vol. 18, Issue 3 https://doi.org/10.1109/tpwrs.2003.814852	journal	August 2003
Power Systems Stability Control: Reinforcement Learning Framework Ernst, D.; Glavic, M.; Wehenkel, L. IEEE Transactions on Power Systems, Vol. 19, Issue 1 https://doi.org/10.1109/tpwrs.2003.821457	journal	February 2004
Optimal Coordinated Voltage Control for Power System Voltage Stability Wen, J. Y.; Wu, Q. H.; Turner, D. R. IEEE Transactions on Power Systems, Vol. 19, Issue 2 https://doi.org/10.1109/tpwrs.2004.825897	journal	May 2004
Model Predictive Control-Based Real-Time Power System Protection Schemes Jin, Licheng; Kumar, Ratnesh; Elia, Nicola IEEE Transactions on Power Systems, Vol. 25, Issue 2 https://doi.org/10.1109/tpwrs.2009.2034748	journal	May 2010
Adaptive Coordinated Voltage Control—Part I: Basic Scheme Ma, Haomin; Hill, David J. IEEE Transactions on Power Systems, Vol. 29, Issue 4 https://doi.org/10.1109/tpwrs.2013.2293577	journal	July 2014
An Efficient Optimal Control Method for Open-Loop Transient Stability Emergency Control Li, Zhihao; Yao, Guoqiang; Geng, Guangchao IEEE Transactions on Power Systems, Vol. 32, Issue 4 https://doi.org/10.1109/tpwrs.2016.2629620	journal	July 2017
Data-Driven Load Frequency Control for Stochastic Power Systems: A Deep Reinforcement Learning Method With Continuous Action Search Yan, Ziming; Xu, Yan IEEE Transactions on Power Systems, Vol. 34, Issue 2 https://doi.org/10.1109/tpwrs.2018.2881359	journal	March 2019
A Data-Driven Multi-Agent Autonomous Voltage Control Framework Using Deep Reinforcement Learning Wang, Shengyi; Duan, Jiajun; Shi, Di IEEE Transactions on Power Systems, Vol. 35, Issue 6 https://doi.org/10.1109/tpwrs.2020.2990179	journal	November 2020
A Deep Reinforcement Learning-Based Multi-Agent Framework to Enhance Power System Resilience Using Shunt Resources Kamruzzaman, Md.; Duan, Jiajun; Shi, Di IEEE Transactions on Power Systems, Vol. 36, Issue 6 https://doi.org/10.1109/tpwrs.2021.3078446	journal	November 2021
Graph Convolutional Network-Based Topology Embedded Deep Reinforcement Learning for Voltage Stability Control Hossain, Ramij R.; Huang, Qiuhua; Huang, Renke IEEE Transactions on Power Systems, Vol. 36, Issue 5 https://doi.org/10.1109/tpwrs.2021.3084469	journal	September 2021
Adaptive Power System Emergency Control Using Deep Reinforcement Learning Huang, Qiuhua; Huang, Renke; Hao, Weituo IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/tsg.2019.2933191	journal	March 2020
Deep Reinforcement Learning Based Volt-VAR Optimization in Smart Distribution Systems Zhang, Ying; Wang, Xinan; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 12, Issue 1 https://doi.org/10.1109/tsg.2020.3010130	journal	January 2021
Cooperative Control and Potential Games Marden, J. R.; Arslan, G.; Shamma, J. S. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), Vol. 39, Issue 6 https://doi.org/10.1109/tsmcb.2009.2017273	journal	December 2009
Attention Enabled Multi-Agent DRL for Decentralized Volt-VAR Control of Active Distribution System Using PV Inverters and SVCs Cao, Di; Zhao, Junbo; Hu, Weihao IEEE Transactions on Sustainable Energy, Vol. 12, Issue 3 https://doi.org/10.1109/tste.2021.3057090	journal	July 2021
Decentralized Voltage Control of PowerSystems Using Multi-agent Systems Shahbazi, Hamidreza; Karbalaei, Farid Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 2 https://doi.org/10.35833/mpce.2018.000628	journal	January 2020

Similar Records

Off-policy deep reinforcement learning with automatic entropy adjustment for adaptive online grid emergency control

Journal Article · Tue Jan 24 19:00:00 EST 2023 · Electric Power Systems Research · OSTI ID:1992867

Graph Convolutional Network-Based Topology Embedded Deep Reinforcement Learning for Voltage Stability Control

Journal Article · Wed Sep 01 00:00:00 EDT 2021 · IEEE Transactions on Power Systems · OSTI ID:1824163

Topology-Aware Reinforcement Learning for Voltage Control: Centralized and Decentralized Strategies

Journal Article · Thu Feb 27 19:00:00 EST 2025 · IEEE Transactions on Industry Applications · OSTI ID:2574179

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION

Multi-Agent Graph-Attention Deep Reinforcement Learning for Post-Contingency Grid Emergency Voltage Control

Citation Formats

References (27)

Similar Records

Related Subjects