Adaptive Deep Reinforcement Learning Algorithm for Distribution System Cyber Attack Defense With High Penetration of DERs

Selim, Alaa; Zhao, Junbo; Ding, Fei; Miao, Fei; Park, Sung-Yeul

doi:10.1109/tsg.2023.3345314

Adaptive Deep Reinforcement Learning Algorithm for Distribution System Cyber Attack Defense With High Penetration of DERs

Journal Article · Wed Dec 20 23:00:00 EST 2023 · IEEE Transactions on Smart Grid

DOI:https://doi.org/10.1109/tsg.2023.3345314· OSTI ID:2284045

^[1]; ^[1]; ^[2]; ^[1]; ^[1]

University of Connecticut, Storrs, CT (United States)
National Renewable Energy Laboratory (NREL), Golden, CO (United States)

With grid modernization, smart inverters are increasingly used to execute advanced controls for distribution network reliability. However, this also increases the cyber-attack space. Here this paper focuses on the defense approaches to restore the system to normal operation circumstances in the presence of cyber-attacks. A unique deep reinforcement learning (DRL) method is developed to minimize voltage violations and reduce power losses for impacted feeders. The defense problem is reformulated as a Markov decision-making process to dynamically control DERs while minimizing load shedding. This is achieved via an improved soft actor-critic (SAC)-based DRL algorithm, which can govern DER set points and load-shedding scenarios in discrete and continuous modes via the auto-tune entropy and Gaussian policy features. Numerical comparison results on the modified IEEE 123-node system with other control approaches, such as Volt-VAR (VV), Volt-Watt (VW), and model predictive control (MPC) show that the proposed method can eliminate voltage violations and provide feasible control actions that perform complete mitigation of cyber-threats.

View Accepted Manuscript (DOE)

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Cybersecurity, Energy Security, and Emergency Response (CESER)

Grant/Contract Number:: AC36-08GO28308

OSTI ID:: 2284045

Report Number(s):: NREL/JA-5D00-88726; MainId:89505; UUID:d57f7e6a-8e78-4f8b-b394-aad89e358580; MainAdminId:71738

Journal Information:: IEEE Transactions on Smart Grid, Journal Name: IEEE Transactions on Smart Grid Journal Issue: 4 Vol. 15; ISSN 1949-3053

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (32)

A stacked deep learning approach to cyber-attacks detection in industrial systems: application to power system and gas pipeline systems Wang, Wu; Harrou, Fouzi; Bouyeddou, Benamar Cluster Computing, Vol. 25, Issue 1 https://doi.org/10.1007/s10586-021-03426-w	journal	October 2021
A tri-level optimization model to mitigate coordinated attacks on electric power systems in a cyber-physical environment Lai, Kexing; Illindala, Mahesh; Subramaniam, Karthikeyan Applied Energy, Vol. 235 https://doi.org/10.1016/j.apenergy.2018.10.077	journal	February 2019
Cyber security in the Smart Grid: Survey and challenges Wang, Wenye; Lu, Zhuo Computer Networks, Vol. 57, Issue 5 https://doi.org/10.1016/j.comnet.2012.12.017	journal	April 2013
Cyber-security in smart grid: Survey and challenges Mrabet, Zakaria El; Kaabouch, Naima; Ghazi, Hassan El Computers & Electrical Engineering, Vol. 67 https://doi.org/10.1016/j.compeleceng.2018.01.015	journal	April 2018
Multi-agent-based hierarchical detection and mitigation of cyber attacks in power systems Zhou, T. L.; Xiahou, K. S.; Zhang, L. L. International Journal of Electrical Power & Energy Systems, Vol. 125 https://doi.org/10.1016/j.ijepes.2020.106516	journal	February 2021
Cybersecurity for distributed energy resources and smart inverters Qi, Junjian; Hahn, Adam; Lu, Xiaonan IET Cyber-Physical Systems: Theory & Applications, Vol. 1, Issue 1 https://doi.org/10.1049/iet-cps.2016.0018	journal	December 2016
Defending Against Data Integrity Attacks in Smart Grid: A Deep Reinforcement Learning-Based Approach An, Dou; Yang, Qingyu; Liu, Wenmao IEEE Access, Vol. 7 https://doi.org/10.1109/ACCESS.2019.2933020	journal	January 2019
Cyber-Physical Energy Systems Security: Threat Modeling, Risk Assessment, Resources, Metrics, and Case Studies Zografopoulos, Ioannis; Ospina, Juan; Liu, Xiaorui IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3058403	journal	January 2021
Simulation of smart functionalities of photovoltaic inverters by interfacing OpenDSS and Matlab No authors listed 2015 IEEE 16th Workshop on Control and Modeling for Power Electronics (COMPEL) https://doi.org/10.1109/COMPEL.2015.7236443	conference	July 2015
Deep Reinforcement Learning for Distribution System Cyber Attack Defense with DERs Selim, Alaa; Zhao, Junbo; Ding, Fei 2023 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT) https://doi.org/10.1109/ISGT51731.2023.10066375	conference	January 2023
Detection and Mitigation of Cyber-Attacks on AGC Systems of Low Inertia Power Grid Roy, Siddhartha Deb; Debbarma, Sanjoy IEEE Systems Journal, Vol. 14, Issue 2 https://doi.org/10.1109/JSYST.2019.2943921	journal	June 2020
Deep Reinforcement Learning for Distribution System Restoration using Distributed Energy Resources and Tie-Switches Selim, Alaa; Zhao, Junbo; Zhang, Xiangyu 2022 IEEE Power & Energy Society General Meeting (PESGM) https://doi.org/10.1109/PESGM48719.2022.9916740	conference	July 2022
Deep Reinforcement Learning for DER Cyber-Attack Mitigation Roberts, Ciaran; Ngo, Sy-Toan; Milesi, Alexandre 2020 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) https://doi.org/10.1109/SmartGridComm47815.2020.9302997	conference	November 2020
Analysis of Malware-Induced Cyber Attacks in Cyber-Physical Power Systems Xu, Sheng; Xia, Yongxiang; Shen, Hui-Liang IEEE Transactions on Circuits and Systems II: Express Briefs, Vol. 67, Issue 12 https://doi.org/10.1109/TCSII.2020.2999875	journal	December 2020
Optimal Data Injection Attacks in Cyber-Physical Systems Wu, Guangyu; Sun, Jian; Chen, Jie IEEE Transactions on Cybernetics, Vol. 48, Issue 12 https://doi.org/10.1109/TCYB.2018.2846365	journal	December 2018
Distributed Model-Predictive Control Strategy for Distribution Network Volt/VAR Control: A Smart-Building-Based Approach Dhulipala, Surya Chandan; Monteiro, Raul Vitor Arantes; Silva Teixeira, Raoni Florentino da IEEE Transactions on Industry Applications, Vol. 55, Issue 6 https://doi.org/10.1109/TIA.2019.2941179	journal	November 2019
End-to-End Communication Architecture for Smart Grids Sauter, Thilo; Lobashov, Maksim IEEE Transactions on Industrial Electronics, Vol. 58, Issue 4 https://doi.org/10.1109/TIE.2010.2070771	journal	April 2011
Classification of Disturbances and Cyber-Attacks in Power Systems Using Heterogeneous Time-Synchronized Data Pan, Shengyi; Morris, Thomas; Adhikari, Uttam IEEE Transactions on Industrial Informatics, Vol. 11, Issue 3 https://doi.org/10.1109/TII.2015.2420951	journal	June 2015
Vulnerability Identification and Remediation of FDI Attacks in Islanded DC Microgrids Using Multiagent Reinforcement Learning Abianeh, Ali Jafarian; Wan, Yihao; Ferdowsi, Farzad IEEE Transactions on Power Electronics, Vol. 37, Issue 6 https://doi.org/10.1109/TPEL.2021.3132028	journal	June 2022
The 2015 Ukraine Blackout: Implications for False Data Injection Attacks Liang, Gaoqi; Weller, Steven R.; Zhao, Junhua IEEE Transactions on Power Systems, Vol. 32, Issue 4 https://doi.org/10.1109/TPWRS.2016.2631891	journal	July 2017
A DER Attack-Mitigation Differential Game for Smart Grid Security Analysis Srikantha, Pirathayini; Kundur, Deepa IEEE Transactions on Smart Grid, Vol. 7, Issue 3 https://doi.org/10.1109/TSG.2015.2466611	journal	May 2016
A Review of False Data Injection Attacks Against Modern Power Systems Liang, Gaoqi; Zhao, Junhua; Luo, Fengji IEEE Transactions on Smart Grid, Vol. 8, Issue 4 https://doi.org/10.1109/TSG.2015.2495133	journal	July 2017
Stochastic Optimization of Economic Dispatch for Microgrid Based on Approximate Dynamic Programming Shuai, Hang; Fang, Jiakun; Ai, Xiaomeng IEEE Transactions on Smart Grid, Vol. 10, Issue 3 https://doi.org/10.1109/TSG.2018.2798039	journal	May 2019
Cyber-Attack Recovery Strategy for Smart Grid Based on Deep Reinforcement Learning Wei, Fanrong; Wan, Zhiqiang; He, Haibo IEEE Transactions on Smart Grid, Vol. 11, Issue 3 https://doi.org/10.1109/TSG.2019.2956161	journal	May 2020
Countering FDI Attacks on DERs Coordinated Control System Using FMI-Compatible Cosimulation Jafarigiv, Danial; Sheshyekani, Keyhan; Kassouf, Marthe IEEE Transactions on Smart Grid, Vol. 12, Issue 2 https://doi.org/10.1109/TSG.2020.3034745	journal	March 2021
Cyber Anomaly-Aware Distributed Voltage Control With Active Power Curtailment and DERs Sarker, Partha S.; Rafy, Md. Fazley; Srivastava, Anurag K. IEEE Transactions on Industry Applications, Vol. 60, Issue 1 https://doi.org/10.1109/tia.2023.3328850	journal	January 2024
A survey of distributed denial-of-service attack, prevention, and mitigation techniques Mahjabin, Tasnuva; Xiao, Yang; Sun, Guang International Journal of Distributed Sensor Networks, Vol. 13, Issue 12 https://doi.org/10.1177/1550147717741463	journal	December 2017
A review of cyber security risks of power systems: from static to dynamic false data attacks Xu, Yan Protection and Control of Modern Power Systems, Vol. 5, Issue 1 https://doi.org/10.1186/s41601-020-00164-w	journal	September 2020
Deep Reinforcement Learning for Mitigating Cyber-Physical DER Voltage Unbalance Attacks Roberts, Ciaran; Ngo, Sy-Toan; Milesi, Alexandre 2021 American Control Conference (ACC) https://doi.org/10.23919/ACC50511.2021.9482815	conference	May 2021
A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning Zhou, Yuhao; Zhang, Bei; Xu, Chunlei Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 6 https://doi.org/10.35833/MPCE.2020.000522	journal	January 2020
Deep Reinforcement Learning Based Approach for Optimal Power Flow of Distribution Networks Embedded with Renewable Energy and Storage Devices Cao, Di; Hu, Weihao; Xu, Xiao Journal of Modern Power Systems and Clean Energy, Vol. 9, Issue 5 https://doi.org/10.35833/MPCE.2020.000557	journal	January 2021
Soft Actor-Critic Algorithms and Applications Haarnoja, Tuomas; Zhou, Aurick; Hartikainen, Kristian arXiv https://doi.org/10.48550/arXiv.1812.05905	preprint	January 2018

Similar Records

Deep Reinforcement Learning for Distribution System Cyber Attack Defense with DERs

Conference · Wed Mar 22 00:00:00 EDT 2023 · OSTI ID:1974035

Federated Deep Reinforcement Learning for Decentralized VVO of BTM DERs

Conference · Tue Oct 01 00:00:00 EDT 2024 · OSTI ID:2477510

Soft Actor Critic Based Volt-VAR Co-optimization in Active Distribution Grids

Conference · Thu Jul 21 00:00:00 EDT 2022 · OSTI ID:1894924

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION
active distribution systems
cyber attack
deep reinforcement learning
renewable generation

Adaptive Deep Reinforcement Learning Algorithm for Distribution System Cyber Attack Defense With High Penetration of DERs

Citation Formats

References (32)

Similar Records

Related Subjects