Model-Free Voltage Control of Active Distribution System with PVs Using Surrogate Model-Based Deep Reinforcement Learning

Cao, Di; Zhao, Junbo; Hu, Weihao; Ding, Fei; Yu, Nanpeng; Huang, Qi; Chen, Zhe

doi:10.1016/j.apenergy.2021.117982

Model-Free Voltage Control of Active Distribution System with PVs Using Surrogate Model-Based Deep Reinforcement Learning

Journal Article · 06 November 2021 · Applied Energy

DOI:https://doi.org/10.1016/j.apenergy.2021.117982· OSTI ID:1834046

Cao, Di; Zhao, Junbo; Hu, Weihao; Ding, Fei; Yu, Nanpeng; Huang, Qi; Chen, Zhe

Accurate knowledge of the distribution system topology and parameters is required to achieve good voltage control performance, but this is difficult to obtain in practice. This paper proposes a physical-model-free voltage control method based on a surrogate-model-enabled deep reinforcement learning approach. Specifically, a surrogate model is trained in a supervised manner using the recorded limited number of historical data to learn the relationship between the power injections and voltage fluctuations of each node. Then, the deep reinforcement learning algorithm is applied to learn an optimal control strategy from the experiences obtained by continuous interactions with the surrogate model. The proposed method can achieve physical-model-free control of unbalanced distribution network and inform real-time decisions to deal with fast voltage fluctuations caused by the rapid variation of PV generation. Simulation results on an unbalance IEEE 123-bus system show that the proposed method can achieve similar performance as that of perfect physical-model-based approaches while being advantageous over other traditional methods.

Research Organization:: National Renewable Energy Lab. (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE); National Key Research and Development Program of China

DOE Contract Number:: AC36-08GO28308

OSTI ID:: 1834046

Report Number(s):: NREL/JA-5D00-81589; MainId:82362; UUID:6a889dbc-f16a-4888-8ca7-1e4897d276f3; MainAdminID:63387

Journal Information:: Applied Energy, Vol. 306, Issue Part A

Country of Publication:: United States

Language:: English

References (37)

Voltage regulation challenges with unbalanced PV integration in low voltage distribution systems and the corresponding solution Wang, Licheng; Yan, Ruifeng; Saha, Tapan Kumar Applied Energy, Vol. 256 https://doi.org/10.1016/j.apenergy.2019.113927	journal	December 2019
Active power control to mitigate voltage and frequency deviations for the smart grid using smart PV inverters Howlader, Abdul Motin; Sadoyama, Staci; Roose, Leon R. Applied Energy, Vol. 258 https://doi.org/10.1016/j.apenergy.2019.114000	journal	January 2020
An Effective Bi-Stage Method for Renewable Energy Sources Integration into Unbalanced Distribution Systems Considering Uncertainty Ali, Eman S.; El-Sehiemy, Ragab A.; Abou El-Ela, Adel A. Processes, Vol. 9, Issue 3 https://doi.org/10.3390/pr9030471	journal	March 2021
Efficient Control of Active Transformers for Increasing the PV Hosting Capacity of LV Grids Hashemi, Seyedmostafa; Ostergaard, Jacob; Degner, Thomas IEEE Transactions on Industrial Informatics, Vol. 13, Issue 1 https://doi.org/10.1109/TII.2016.2619065	journal	February 2017
Double-Time-Scale Coordinated Voltage Control in Active Distribution Networks Based on MPC Guo, Yifei; Wu, Qiuwei; Gao, Houlei IEEE Transactions on Sustainable Energy, Vol. 11, Issue 1 https://doi.org/10.1109/TSTE.2018.2890621	journal	January 2020
Online reconfiguration scheme of self-sufficient distribution network based on a reinforcement learning approach Oh, Seok Hwa; Yoon, Yong Tae; Kim, Seung Wan Applied Energy, Vol. 280 https://doi.org/10.1016/j.apenergy.2020.115900	journal	December 2020
Voltage Quality Improvement in Low Voltage Distribution Networks Using Reactive Power Capability of Single-Phase PV Inverters Zeraati, Mehdi; Golshan, Mohamad Esmail Hamedani; Guerrero, Josep M. IEEE Transactions on Smart Grid, Vol. 10, Issue 5 https://doi.org/10.1109/TSG.2018.2874381	journal	September 2019
Distributed Control of Battery Energy Storage Systems for Voltage Regulation in Distribution Networks With High PV Penetration Zeraati, Mehdi; Hamedani Golshan, Mohamad Esmail; Guerrero, Josep M. IEEE Transactions on Smart Grid, Vol. 9, Issue 4 https://doi.org/10.1109/TSG.2016.2636217	journal	July 2018
Cooperation-Driven Distributed Model Predictive Control for Energy Storage Systems Meng, Ke; Dong, Zhao Yang; Xu, Zhao IEEE Transactions on Smart Grid, Vol. 6, Issue 6 https://doi.org/10.1109/TSG.2015.2449760	journal	November 2015
Novel Control Strategy for Enhancing Microgrid Operation Connected to Photovoltaic Generation and Energy Storage Systems Emara, Dina; Ezzat, Mohamed; Abdelaziz, Almoataz Y. Electronics, Vol. 10, Issue 11 https://doi.org/10.3390/electronics10111261	journal	May 2021
Improved Low Voltage Grid-Integration of Photovoltaic Systems in Germany Stetz, Thomas; Marten, Frank; Braun, Martin IEEE Transactions on Sustainable Energy, Vol. 4, Issue 2 https://doi.org/10.1109/TSTE.2012.2198925	journal	April 2013
Multi-Timescale Coordinated Voltage/Var Control of High Renewable-Penetrated Distribution Systems Xu, Yan; Dong, Zhao Yang; Zhang, Rui IEEE Transactions on Power Systems, Vol. 32, Issue 6 https://doi.org/10.1109/TPWRS.2017.2669343	journal	November 2017
Stochastic Reactive Power Management in Microgrids With Renewables Kekatos, Vassilis; Wang, Gang; Conejo, Antonio J. IEEE Transactions on Power Systems, Vol. 30, Issue 6 https://doi.org/10.1109/TPWRS.2014.2369452	journal	November 2015
Three-Stage Robust Inverter-Based Voltage/Var Control for Distribution Networks With High-Level PV Zhang, Cuo; Xu, Yan; Dong, Zhaoyang IEEE Transactions on Smart Grid, Vol. 10, Issue 1 https://doi.org/10.1109/TSG.2017.2752234	journal	January 2019
A Two-Stage Robust Optimization for Centralized-Optimal Dispatch of Photovoltaic Inverters in Active Distribution Networks Ding, Tao; Li, Cheng; Yang, Yongheng IEEE Transactions on Sustainable Energy, Vol. 8, Issue 2 https://doi.org/10.1109/TSTE.2016.2605926	journal	April 2017
Distributed Adaptive Robust Voltage/VAR Control With Network Partition in Active Distribution Networks Li, Peishuai; Zhang, Cuo; Wu, Zaijun IEEE Transactions on Smart Grid, Vol. 11, Issue 3 https://doi.org/10.1109/TSG.2019.2950120	journal	May 2020
Robust Coordinated Optimization of Active and Reactive Power in Active Distribution Systems Gao, Hongjun; Liu, Junyong; Wang, Lingfeng IEEE Transactions on Smart Grid, Vol. 9, Issue 5 https://doi.org/10.1109/TSG.2017.2657782	journal	September 2018
Improving Supervised Phase Identification Through the Theory of Information Losses Foggo, Brandon; Yu, Nanpeng IEEE Transactions on Smart Grid, Vol. 11, Issue 3 https://doi.org/10.1109/TSG.2019.2952080	journal	May 2020
Maximum Marginal Likelihood Estimation of Phase Connections in Power Distribution Systems Wang, Wenyu; Yu, Nanpeng IEEE Transactions on Power Systems, Vol. 35, Issue 5 https://doi.org/10.1109/TPWRS.2020.2977071	journal	September 2020
Robust Medium-Voltage Distribution System State Estimation using Multi-Source Data Zhao, Junbo; Huang, Can; Mili, Lamine 2020 IEEE Power & Energy Society Innovative Smart Grid Technologies Conference (ISGT) https://doi.org/10.1109/ISGT45199.2020.9087787	conference	February 2020
Robust Ensemble Kalman Filter for Medium-Voltage Distribution System State Estimation Huang, Manyun; Wei, Zhinong; Zhao, Junbo IEEE Transactions on Instrumentation and Measurement, Vol. 69, Issue 7 https://doi.org/10.1109/TIM.2019.2945743	journal	July 2020
Ergodic Energy Management Leveraging Resource Variability in Distribution Grids Wang, Gang; Kekatos, Vassilis; Conejo, Antonio J. IEEE Transactions on Power Systems, Vol. 31, Issue 6 https://doi.org/10.1109/TPWRS.2016.2524679	journal	November 2016
Data-Driven Local Control Design for Active Distribution Grids Using Off-Line Optimal Power Flow and Machine Learning Techniques Karagiannopoulos, Stavros; Aristidou, Petros; Hug, Gabriela IEEE Transactions on Smart Grid, Vol. 10, Issue 6 https://doi.org/10.1109/TSG.2019.2905348	journal	November 2019
Designing Reactive Power Control Rules for Smart Inverters Using Support Vector Machines Jalali, Mana; Kekatos, Vassilis; Gatsis, Nikolaos IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2942850	journal	March 2020
Model-Free Optimal Control of VAR Resources in Distribution Systems: An Extremum Seeking Approach Arnold, Daniel B.; Negrete-Pincetic, Matias; Sankur, Michael D. IEEE Transactions on Power Systems, Vol. 31, Issue 5 https://doi.org/10.1109/TPWRS.2015.2502554	journal	September 2016
Two-Timescale Voltage Control in Distribution Grids Using Deep Reinforcement Learning Yang, Qiuling; Wang, Gang; Sadeghi, Alireza IEEE Transactions on Smart Grid, Vol. 11, Issue 3 https://doi.org/10.1109/TSG.2019.2951769	journal	May 2020
Double Deep $Q$ -Learning-Based Distributed Operation of Battery Energy Storage System Considering Uncertainties Bui, Yan-Hai; Hussain, Akhtar; Kim, Hak-Man IEEE Transactions on Smart Grid, Vol. 11, Issue 1 https://doi.org/10.1109/TSG.2019.2924025	journal	January 2020
A Multi-Agent Deep Reinforcement Learning Based Voltage Regulation Using Coordinated PV Inverters Cao, Di; Hu, Weihao; Zhao, Junbo IEEE Transactions on Power Systems, Vol. 35, Issue 5 https://doi.org/10.1109/TPWRS.2020.3000652	journal	September 2020
Attention Enabled Multi-Agent DRL for Decentralized Volt-VAR Control of Active Distribution System Using PV Inverters and SVCs Cao, Di; Zhao, Junbo; Hu, Weihao IEEE Transactions on Sustainable Energy, Vol. 12, Issue 3 https://doi.org/10.1109/TSTE.2021.3057090	journal	July 2021
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review Cao, Di; Hu, Weihao; Zhao, Junbo Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 6 https://doi.org/10.35833/MPCE.2020.000552	journal	January 2020
Dynamic Distribution Network Reconfiguration Using Reinforcement Learning Gao, Yuanqi; Shi, Jie; Wang, Wei 2019 IEEE International Conference on Communications, Control, and Computing Technologies for Smart Grids (SmartGridComm) https://doi.org/10.1109/SmartGridComm.2019.8909777	conference	October 2019
Batch-Constrained Reinforcement Learning for Dynamic Distribution Network Reconfiguration Gao, Yuanqi; Wang, Wei; Shi, Jie IEEE Transactions on Smart Grid, Vol. 11, Issue 6 https://doi.org/10.1109/TSG.2020.3005270	journal	November 2020
Experimental Setup for Online Fault Diagnosis of Induction Machines via Promising IoT and Machine Learning: Towards Industry 4.0 Empowerment Tran, Minh-Quang; Elsisi, Mahmoud; Mahmoud, Karar IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3105297	journal	January 2021
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Graph Algorithms for Topology Identification Using Power Grid Probing Cavraro, Guido; Kekatos, Vassilis IEEE Control Systems Letters, Vol. 2, Issue 4 https://doi.org/10.1109/LCSYS.2018.2846801	journal	October 2018
Comprehensive Modeling of Three-Phase Distribution Systems via the Bus Admittance Matrix Bazrafshan, Mohammadhafez; Gatsis, Nikolaos IEEE Transactions on Power Systems, Vol. 33, Issue 2 https://doi.org/10.1109/TPWRS.2017.2728618	journal	March 2018
Two-Stage Deep Reinforcement Learning for Inverter-Based Volt-VAR Control in Active Distribution Networks Liu, Haotian; Wu, Wenchuan IEEE Transactions on Smart Grid, Vol. 12, Issue 3 https://doi.org/10.1109/TSG.2020.3041620	journal	May 2021

Similar Records

Data-Driven Multi-agent Deep Reinforcement Learning for Distribution System Decentralized Voltage Control with High Penetration of PVs

Journal Article · 2021 · IEEE Transactions on Smart Grid · OSTI ID:1781623

Cao, Di; Zhao, Junbo; Hu, Weihao; +4 more

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Journal Article · 2023 · Machine Learning · OSTI ID:2345752

Hossain, Ramij Raja; Yin, Tianzhixi; Du, Yan; +5 more

Deep Reinforcement Scheduling of Energy Storage Systems for Real-time Voltage Regulation in Unbalanced LV Networks with High PV Penetration

Journal Article · 2021 · IEEE Transactions on Sustainable Energy · OSTI ID:1821608

Wang, Shengyi; Du, Liang; Fan, Xiaoyuan; +1 more

Related Subjects

POWER TRANSMISSION AND DISTRIBUTION
active distribution network
deep reinforcement learning
model-free
optimization
solar PVs
voltage regulation

Model-Free Voltage Control of Active Distribution System with PVs Using Surrogate Model-Based Deep Reinforcement Learning

Citation Formats

References (37)

Similar Records

Related Subjects