Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration

Zhang, Xiangyu; Eseye, Abinet Tesfaye; Knueven, Bernard; Liu, Weijia; Reynolds, Matthew; Jones, Wesley

doi:10.1109/tpwrs.2022.3209919

Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration

Journal Article · Mon Sep 26 00:00:00 EDT 2022 · IEEE Transactions on Power Systems

DOI:https://doi.org/10.1109/tpwrs.2022.3209919· OSTI ID:1890146

^[1]; Eseye, Abinet Tesfaye ^[1]; Knueven, Bernard ^[1]; ^[1]; Reynolds, Matthew ^[1]; Jones, Wesley ^[1]

National Renewable Energy Lab. (NREL), Golden, CO (United States)

This paper focuses on the critical load restoration problem in distribution systems following major outages. To provide fast online response and optimal sequential decision-making support, a reinforcement learning (RL) based approach is proposed to optimize the restoration. Due to the complexities stemming from the large policy search space, renewable uncertainty, and nonlinearity in a complex grid control problem, directly applying RL algorithms to train a satisfactory policy requires extensive tuning to be successful. To address this challenge, this paper leverages the curriculum learning (CL) technique to design a training curriculum involving a simpler steppingstone problem that guides the RL agent to learn to solve the original hard problem in a progressive and more effective manner. We demonstrate that compared with direct learning, CL facilitates controller training to achieve better performance. To study realistic scenarios where renewable forecasts used for decision-making are in general imperfect, the experiments compare the trained RL controllers against two model predictive controllers (MPCs) using renewable forecasts with different error levels and observe how these controllers can hedge against the uncertainty. Results show that RL controllers are less susceptible to forecast errors than the baseline MPCs and can provide a more reliable restoration process.

View Accepted Manuscript (DOE)

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Electricity, Advanced Grid Modeling (AGM) Program

Grant/Contract Number:: AC36-08GO28308

OSTI ID:: 1890146

Report Number(s):: NREL/JA--2C00-84156; MainId:84929; UUID:5d6600ff-9336-4343-9444-230bd1795691; MainAdminID:67658

Journal Information:: IEEE Transactions on Power Systems, Journal Name: IEEE Transactions on Power Systems Journal Issue: 5 Vol. 38; ISSN 0885-8950

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (29)

The Wind Integration National Dataset (WIND) Toolkit Draxl, Caroline; Clifton, Andrew; Hodge, Bri-Mathias Applied Energy, Vol. 151, p. 355-366 https://doi.org/10.1016/j.apenergy.2015.03.121	journal	August 2015
Influence of extreme weather and climate change on the resilience of power systems: Impacts and possible mitigation strategies Panteli, Mathaios; Mancarella, Pierluigi Electric Power Systems Research, Vol. 127 https://doi.org/10.1016/j.epsr.2015.06.012	journal	October 2015
Resilient Operation of Power Distribution Systems Using MPC-based Critical Service Restoration Eseye, Abinet Tesfaye; Knueven, Bernard; Zhang, Xiangyu 2021 IEEE Green Technologies Conference (GreenTech) https://doi.org/10.1109/GreenTech48523.2021.00054	conference	April 2021
Residual Reinforcement Learning for Robot Control Johannink, Tobias; Bahl, Shikhar; Nair, Ashvin 2019 International Conference on Robotics and Automation (ICRA) https://doi.org/10.1109/ICRA.2019.8794127	conference	May 2019
The Grid: Stronger, Bigger, Smarter?: Presenting a Conceptual Framework of Power System Resilience Panteli, Mathaios; Mancarella, Pierluigi IEEE Power and Energy Magazine, Vol. 13, Issue 3 https://doi.org/10.1109/MPE.2015.2397334	journal	May 2015
Restoring Critical Loads In Resilient Distribution Systems using A Curriculum Learned Controller Zhang, Xiangyu; Eseye, Abinet Tesfaye; Reynolds, Matthew 2021 IEEE Power & Energy Society General Meeting (PESGM) https://doi.org/10.1109/PESGM46819.2021.9637834	conference	July 2021
Convex relaxations and linear approximation for optimal power flow in multiphase radial networks Gan, Lingwen; Low, Steven H. 2014 Power Systems Computation Conference (PSCC) https://doi.org/10.1109/PSCC.2014.7038399	conference	August 2014
Distribution System Restoration With Microgrids Using Spanning Tree Search Li, Juan; Ma, Xi-Yuan; Liu, Chen-Ching IEEE Transactions on Power Systems, Vol. 29, Issue 6 https://doi.org/10.1109/TPWRS.2014.2312424	journal	November 2014
Self-Healing Resilient Distribution Systems Based on Sectionalization Into Microgrids Wang, Zhaoyu; Wang, Jianhui IEEE Transactions on Power Systems, Vol. 30, Issue 6 https://doi.org/10.1109/TPWRS.2015.2389753	journal	November 2015
Robust Restoration Method for Active Distribution Networks Chen, Xin; Wu, Wenchuan; Zhang, Boming IEEE Transactions on Power Systems, Vol. 31, Issue 5 https://doi.org/10.1109/TPWRS.2015.2503426	journal	September 2016
A Model Predictive Control Based Generator Start-Up Optimization Strategy for Restoration With Microgrids as Black-Start Resources Zhao, Yuxuan; Lin, Zhenzhi; Ding, Yi IEEE Transactions on Power Systems, Vol. 33, Issue 6 https://doi.org/10.1109/TPWRS.2018.2849265	journal	November 2018
Critical Load Restoration Using Distributed Energy Resources for Resilient Power Distribution System Poudel, Shiva; Dubey, Anamika IEEE Transactions on Power Systems, Vol. 34, Issue 1 https://doi.org/10.1109/TPWRS.2018.2860256	journal	January 2019
A Bi-Level Interval Robust Optimization Model for Service Restoration in Flexible Distribution Networks Liu, Wenxia; Fu, Mengdi; Yang, Mengyao IEEE Transactions on Power Systems, Vol. 36, Issue 3 https://doi.org/10.1109/TPWRS.2020.3038400	journal	May 2021
Collaborative Distribution System Restoration Planning and Real-Time Dispatch Considering Behind-the-Meter DERS Liu, Weijia; Ding, Fei IEEE Transactions on Power Systems, Vol. 36, Issue 4 https://doi.org/10.1109/TPWRS.2020.3048089	journal	July 2021
Distribution System Resilience Under Asynchronous Information Using Deep Reinforcement Learning Bedoya, Juan Carlos; Wang, Yubo; Liu, Chen-Ching IEEE Transactions on Power Systems, Vol. 36, Issue 5 https://doi.org/10.1109/TPWRS.2021.3056543	journal	September 2021
Learning Sequential Distribution System Restoration via Graph-Reinforcement Learning Zhao, Tianqiao; Wang, Jianhui IEEE Transactions on Power Systems, Vol. 37, Issue 2 https://doi.org/10.1109/TPWRS.2021.3102870	journal	March 2022
Resilience-Oriented Critical Load Restoration Using Microgrids in Distribution Systems Gao, Haixiang; Chen, Ying; Xu, Yin IEEE Transactions on Smart Grid, Vol. 7, Issue 6 https://doi.org/10.1109/TSG.2016.2550625	journal	November 2016
Microgrids for Service Restoration to Critical Load in a Resilient Distribution System Xu, Yin; Liu, Chen-Ching; Schneider, Kevin P. IEEE Transactions on Smart Grid, Vol. 9, Issue 1 https://doi.org/10.1109/TSG.2016.2591531	journal	January 2018
Microgrid Scheduling With Uncertainty: The Quest for Resilience Gholami, Amin; Shekari, Tohid; Aminifar, Farrokh IEEE Transactions on Smart Grid, Vol. 7, Issue 6 https://doi.org/10.1109/TSG.2016.2598802	journal	November 2016
Multi-Time Step Service Restoration for Advanced Distribution Systems and Microgrids Chen, Bo; Chen, Chen; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 9, Issue 6 https://doi.org/10.1109/TSG.2017.2723798	journal	November 2018
Risk-Limiting Load Restoration for Resilience Enhancement With Intermittent Energy Resources Wang, Zhiwen; Shen, Chen; Xu, Yin IEEE Transactions on Smart Grid, Vol. 10, Issue 3 https://doi.org/10.1109/TSG.2018.2803141	journal	May 2019
Coordinating Multiple Sources for Service Restoration to Enhance Resilience of Distribution Systems Wang, Ying; Xu, Yin; He, Jinghan IEEE Transactions on Smart Grid, Vol. 10, Issue 5 https://doi.org/10.1109/TSG.2019.2891515	journal	September 2019
Rolling Optimization of Mobile Energy Storage Fleets for Resilient Service Restoration Yao, Shuhan; Wang, Peng; Liu, Xiaochuan IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2930012	journal	March 2020
Distributed Risk-Limiting Load Restoration in Unbalanced Distribution Systems With Networked Microgrids Shen, Feifan; Wu, Qiuwei; Zhao, Jin IEEE Transactions on Smart Grid, Vol. 11, Issue 6 https://doi.org/10.1109/TSG.2020.2995099	journal	November 2020
A Two-Level Simulation-Assisted Sequential Distribution System Restoration Model With Frequency Dynamics Constraints Zhang, Qianzhi; Ma, Zixiao; Zhu, Yongli IEEE Transactions on Smart Grid, Vol. 12, Issue 5 https://doi.org/10.1109/TSG.2021.3088006	journal	September 2021
Curriculum learning Bengio, Yoshua; Louradour, Jérôme; Collobert, Ronan Proceedings of the 26th Annual International Conference on Machine Learning - ICML '09 https://doi.org/10.1145/1553374.1553380	conference	January 2009
Deep Reinforcement Learning for Building HVAC Control Wei, Tianshu; Wang, Yanzhi; Zhu, Qi DAC '17: The 54th Annual Design Automation Conference 2017, Proceedings of the 54th Annual Design Automation Conference 2017 https://doi.org/10.1145/3061639.3062224	conference	June 2017
Grid-Interactive Multi-Zone Building Control Using Reinforcement Learning with Global-Local Policy Search Zhang, Xiangyu; Chintala, Rohit; Bernstein, Andrey 2021 American Control Conference (ACC) https://doi.org/10.23919/ACC50511.2021.9482917	conference	May 2021
Automatic Curriculum Learning For Deep RL: A Short Survey Portelas, Rémy; Colas, Cédric; Weng, Lilian Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence https://doi.org/10.24963/ijcai.2020/671	conference	July 2020

Cited By (1)

Error-Level-Controlled Synthetic Forecasts for Renewable Generation Zhang, Xiangyu; Knueven, Bernard; Eseye, Abinet Tesfaye DOE Open Energy Data Initiative (OEDI); National Renewable Energy Laboratory (NREL) https://doi.org/10.25984/2222585	dataset	January 2021

Similar Records

A Hybrid Reinforcement Learning-MPC Approach for Distribution System Critical Load Restoration

Conference · Fri Jul 22 00:00:00 EDT 2022 · OSTI ID:1878557

A Hybrid Reinforcement Learning-MPC Approach for Distribution System Critical Load Restoration: Preprint

Conference · Tue Feb 01 23:00:00 EST 2022 · OSTI ID:1844201

Hybrid-RL-MPC4CLR (Hybird-Reinforcement-Learning-Model-Predictive-Control-for-Reserve-Policy-Assisted-Critical-Load-Restoration-in-Distribution-Grids)

Software · Thu Mar 17 20:00:00 EDT 2022 · OSTI ID:code-72315

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION
critical load restoration
curriculum learning
distribution system
grid resilience
reinforcement learning

Curriculum-based Reinforcement Learning for Distribution System Critical Load Restoration

Citation Formats

References (29)

Cited By (1)

Similar Records

Related Subjects