Bi-Level Adaptive Storage Expansion Strategy for Microgrids Using Deep Reinforcement Learning

Huang, Bin; Zhao, Tianqiao; Yue, Meng; Wang, Jianhui

doi:10.1109/tsg.2023.3312225

Bi-Level Adaptive Storage Expansion Strategy for Microgrids Using Deep Reinforcement Learning

Journal Article · Mon Sep 11 00:00:00 EDT 2023 · IEEE Transactions on Smart Grid

DOI:https://doi.org/10.1109/tsg.2023.3312225· OSTI ID:2440517

^[1]; ^[2]; ^[2]; ^[1]

Southern Methodist Univ., Dallas, TX (United States)
Brookhaven National Laboratory (BNL), Upton, NY (United States)

Battery energy storage (BES) is a versatile resource for the secure and economic operation of microgrids (MGs). Prevailing stochastic optimization-based approaches for BES expansion planning for MGs are computationally complicated. This work proposes a data-driven bi-level multi-period BES expansion planning framework to determine the siting, sizing, and timing of BES installations. The proposed planning framework unifies deep reinforcement learning (DRL) and linear programming, thereby decoupling the determinations for the integer and continuous decision variables in two time scales, respectively. In the upper level, a rainbow DRL agent with quantile regression is trained to provide dynamic planning policies to accommodate stochastic renewable energy resources (RESs), load, and battery price changes efficiently. Further, the lower level computes the optimal operation of MGs with frequency constraints to hedge the islanding contingency. The two levels communicate with one another by exchanging storage configuration and operating expenses in order to accomplish the shared goal of minimizing investment and operation costs. Comparative case studies on an MG are carried out to demonstrate the superiority of the proposed DRL-based solution to the mixed-integer linear programming counterpart on efficiency, scalability, and adaptability.

View Accepted Manuscript (DOE)

Research Organization:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Organization:: USDOE Office of Electricity (OE), Advanced Grid Research & Development. Power Systems Engineering Research

Grant/Contract Number:: SC0012704

OSTI ID:: 2440517

Report Number(s):: BNL-226081-2024-JAAM

Journal Information:: IEEE Transactions on Smart Grid, Vol. 15, Issue 2; ISSN 1949-3053

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (20)

Integrated Planning of a Solar/Storage Collective Contreras-Ocana, Jesus E.; Singh, Arshpreet; Besanger, Yvon IEEE Transactions on Smart Grid, Vol. 12, Issue 1 https://doi.org/10.1109/TSG.2020.3020402	journal	January 2021
Deep reinforcement learning for resilient microgrid expansion planning with multiple energy resource Pang, Kexin; Zhou, Jian; Tsianikas, Stamatis Quality and Reliability Engineering International https://doi.org/10.1002/qre.3203	journal	October 2022
Utility-Scale Portable Energy Storage Systems He, Guannan; Michalek, Jeremy; Kar, Soummya Joule, Vol. 5, Issue 2 https://doi.org/10.1016/j.joule.2020.12.005	journal	February 2021
Joint Distribution Network and Renewable Energy Expansion Planning Considering Demand Response and Energy Storage—Part I: Stochastic Programming Model Asensio, Miguel; Meneses de Quevedo, Pilar; Munoz-Delgado, Gregorio IEEE Transactions on Smart Grid, Vol. 9, Issue 2 https://doi.org/10.1109/TSG.2016.2560339	journal	March 2018
Co-Optimization of Battery Storage Investment and Grid Expansion in Integrated Energy Systems Karimianfard, Hossein; Haghighat, Hossein; Zeng, Bo IEEE Systems Journal, Vol. 16, Issue 4 https://doi.org/10.1109/JSYST.2021.3130057	journal	December 2022
Cost Projections for Utility-Scale Battery Storage: 2021 Update Cole, Wesley; Frazier, A.; Augustine, Chad https://doi.org/10.2172/1786976	report	June 2021
Frequency Dynamics Constrained Unit Commitment With Battery Energy Storage Wen, Yunfeng; Li, Wenyuan; Huang, Gang IEEE Transactions on Power Systems, Vol. 31, Issue 6 https://doi.org/10.1109/TPWRS.2016.2521882	journal	November 2016
A Comprehensive Battery Energy Storage Optimal Sizing Model for Microgrid Applications Alsaidan, Ibrahim; Khodaei, Amin; Gao, Wenzhong IEEE Transactions on Power Systems, Vol. 33, Issue 4 https://doi.org/10.1109/TPWRS.2017.2769639	journal	July 2018
Stochastic expansion planning of battery energy storage for the interconnected distribution and data networks Alobaidi, Abdulraheem Hassan; Khodayar, Mahdi; Vafamehr, Ali International Journal of Electrical Power & Energy Systems, Vol. 133 https://doi.org/10.1016/j.ijepes.2021.107231	journal	December 2021
Rainbow: Combining Improvements in Deep Reinforcement Learning Hessel, Matteo; Modayil, Joseph; Van Hasselt, Hado Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, Issue 1 https://doi.org/10.1609/aaai.v32i1.11796	journal	April 2018
Microgrid Dispatch With Frequency-Aware Islanding Constraints Wen, Yunfeng; Chung, C. Y.; Liu, Xuan IEEE Transactions on Power Systems, Vol. 34, Issue 3 https://doi.org/10.1109/TPWRS.2019.2895573	journal	May 2019
On representative day selection for capacity expansion planning of power systems under extreme operating conditions Li, Can; Conejo, Antonio J.; Siirola, John D. International Journal of Electrical Power & Energy Systems, Vol. 137 https://doi.org/10.1016/j.ijepes.2021.107697	journal	May 2022
Convex Relaxation of Optimal Power Flow—Part I: Formulations and Equivalence Low, Steven H. IEEE Transactions on Control of Network Systems, Vol. 1, Issue 1, p. 15-27 https://doi.org/10.1109/TCNS.2014.2309732	journal	March 2014
Distributional Reinforcement Learning With Quantile Regression Dabney, Will; Rowland, Mark; Bellemare, Marc Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32, Issue 1 https://doi.org/10.1609/aaai.v32i1.11791	journal	April 2018
Governor Rate-Constrained OPF for Primary Frequency Control Adequacy Chavez, Hector; Baldick, Ross; Sharma, Sandip IEEE Transactions on Power Systems, Vol. 29, Issue 3 https://doi.org/10.1109/TPWRS.2014.2298838	journal	May 2014
Deep-Reinforcement-Learning-Based Capacity Scheduling for PV-Battery Storage System Huang, Bin; Wang, Jianhui IEEE Transactions on Smart Grid, Vol. 12, Issue 3 https://doi.org/10.1109/TSG.2020.3047890	journal	May 2021
Energy Storage Planning for Enhanced Resilience of Power Distribution Networks Against Earthquakes Nazemi, Mostafa; Moeini-Aghtaie, Moein; Fotuhi-Firuzabad, Mahmud IEEE Transactions on Sustainable Energy, Vol. 11, Issue 2 https://doi.org/10.1109/TSTE.2019.2907613	journal	April 2020
A Stochastic-Robust Approach for Resilient Microgrid Investment Planning Under Static and Transient Islanding Security Constraints Nakiganda, Agnes Marjorie; Dehghan, Shahab; Markovic, Uros IEEE Transactions on Smart Grid, Vol. 13, Issue 3 https://doi.org/10.1109/TSG.2022.3146193	journal	May 2022
Machine learning for combinatorial optimization: A methodological tour d’horizon Bengio, Yoshua; Lodi, Andrea; Prouvost, Antoine European Journal of Operational Research, Vol. 290, Issue 2 https://doi.org/10.1016/j.ejor.2020.07.063	journal	April 2021
A storage expansion planning framework using reinforcement learning and simulation-based optimization Tsianikas, Stamatis; Yousefi, Nooshin; Zhou, Jian Applied Energy, Vol. 290 https://doi.org/10.1016/j.apenergy.2021.116778	journal	May 2021

Similar Records

An Evaluation of the Economic and Resilience Benefits of a Microgrid in Northampton, Massachusetts

Journal Article · Mon Sep 14 00:00:00 EDT 2020 · Energies (Basel) · OSTI ID:1660320

Balducci, Patrick; Mongird, Kendall; Wu, Di; +3 more

Networked Microgrids Planning Through Chance Constrained Stochastic Conic Programming

Journal Article · Tue Apr 02 00:00:00 EDT 2019 · IEEE Transactions on Smart Grid · OSTI ID:1870059

Cao, Xiaoyu; Wang, Jianxue; Zeng, Bo

Related Subjects

24 POWER TRANSMISSION AND DISTRIBUTION
planning
costs
uncertainty
indexes
investment
stochastic processes
batteries
battery storage systems
deep reinforcement learning
microgrid planning
two-timescale

Bi-Level Adaptive Storage Expansion Strategy for Microgrids Using Deep Reinforcement Learning

Citation Formats

References (20)

Similar Records

Related Subjects