Deep Reinforcement Learning Based Control of Wind Turbines for Fast Frequency Response

Gao, Wei; Fan, Rui; Qiao, Wei; Wang, Shaobu; Gao, David Wenzhong

doi:10.1109/TIA.2025.3574047

Deep Reinforcement Learning Based Control of Wind Turbines for Fast Frequency Response

Journal Article · Mon May 26 20:00:00 EDT 2025 · IEEE Transactions on Industry Applications

DOI:https://doi.org/10.1109/TIA.2025.3574047· OSTI ID:3008524

Gao, Wei ^[1]; ^[2]; ^[3]; ^[4]; ^[2]

Argonne National Laboratory (ANL), Argonne, IL (United States)
Univ. of Denver, CO (United States)
Univ. of Nebraska, Lincoln, NE (United States)
Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

In order to fulfill vital auxiliary grid services, such as load regulation, spin and non-spin reserve provision, and frequency support during emergencies, there is often a requirement for certain wind farms to operate in de-loaded modes. Leveraging the swift response capabilities of wind farms, this study demonstrates that reserving power in de-loaded modes can significantly enhance power grid stability and reliability during system contingencies. Controlling wind farms optimally for frequency support is intricate due to the nonlinearity of models and controllers and the complexity of wind farm interactions with power systems. Here, to address this challenge, this paper introduces a novel approach that integrates wind turbines into reinforcement learning-based solutions for frequency response. This innovative methodology utilizes the state-of-the-art reinforcement learning algorithm known as the surrogate-gradient-based evolutionary strategy. The proposed learning-based algorithm provides continuous control of wind farm output to rapidly stabilize system frequency and prevent unnecessary trips of under-frequency load shedding relays. To facilitate efficient training, parallel computing techniques are employed. The proposed methodology is evaluated on a modified IEEE-39 bus system, and simulation results reveal its efficacy in reliably supporting power system frequency and preventing the need for unnecessary load shedding.

View Accepted Manuscript (DOE)

Research Organization:: Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE Office of Electricity (OE)

Grant/Contract Number:: AC05-76RL01830

OSTI ID:: 3008524

Report Number(s):: PNNL-SA--193706

Journal Information:: IEEE Transactions on Industry Applications, Journal Name: IEEE Transactions on Industry Applications Journal Issue: 6 Vol. 61; ISSN 0093-9994; ISSN 1939-9367

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (28)

Damping inter‐area oscillation using reinforcement learning controlled TCSC Huang, Renke; Gao, Wei; Fan, Rui IET Generation, Transmission & Distribution, Vol. 16, Issue 11 https://doi.org/10.1049/gtd2.12441	journal	March 2022
Improving DFIG performance under fault scenarios through evolutionary reinforcement learning based control Gao, Wei; Fan, Rui; Huang, Renke IET Generation, Transmission & Distribution, Vol. 16, Issue 19 https://doi.org/10.1049/gtd2.12563	journal	August 2022
Optimal controller design of a doubly-fed induction generator wind turbine system for small signal stability enhancement Yang, L.; Yang, G. Y.; Xu, Z. IET Generation, Transmission & Distribution, Vol. 4, Issue 5 https://doi.org/10.1049/iet-gtd.2009.0553	journal	January 2010
Coordinated frequency regulation by doubly fed induction generator-based wind power plants Zhang, Z. -S.; Sun, Y. -Z.; Lin, J. IET Renewable Power Generation, Vol. 6, Issue 1 https://doi.org/10.1049/iet-rpg.2010.0208	journal	January 2012
Frequency support from doubly fed induction generator wind turbines Ramtharan, G.; Jenkins, N.; Ekanayake, J. B. IET Renewable Power Generation, Vol. 1, Issue 1 https://doi.org/10.1049/iet-rpg:20060019	journal	January 2007
Fast Frequency Response Using Reinforcement Learning-Controlled Wind Turbines Gao, Wei; Fan, Rui; Qiao, Wei 2023 IEEE Industry Applications Society Annual Meeting (IAS) https://doi.org/10.1109/IAS54024.2023.10406378	conference	October 2023
Linearization Approach for Modeling Power Electronics Devices in Power Systems No authors listed IEEE Journal of Emerging and Selected Topics in Power Electronics, Vol. 2, Issue 4 https://doi.org/10.1109/JESTPE.2014.2320223	journal	December 2014
Primary frequency control by wind turbines Erlich, I.; Wilch, M. IEEE PES General Meeting https://doi.org/10.1109/PES.2010.5589911	conference	July 2010
Dynamic state estimation and parameter calibration of a DFIG using the ensemble Kalman filter No authors listed 2015 IEEE Power & Energy Society General Meeting https://doi.org/10.1109/PESGM.2015.7285990	conference	July 2015
Control of IPM Synchronous Generator for Maximum Wind Power Generation Considering Magnetic Saturation Qiao, Wei; Qu, Liyan; Harley, Ronald G. IEEE Transactions on Industry Applications, Vol. 45, Issue 3 https://doi.org/10.1109/TIA.2009.2018914	journal	January 2009
Case Study: An Adaptive Underfrequency Load-Shedding System Manson, Scott; Zweigle, Greg; Yedidi, Vinod IEEE Transactions on Industry Applications, Vol. 50, Issue 3 https://doi.org/10.1109/TIA.2013.2288432	journal	May 2014
Emerging Power Quality Challenges Due to Integration of Renewable Energy Sources Liang, Xiaodong IEEE Transactions on Industry Applications, Vol. 53, Issue 2 https://doi.org/10.1109/TIA.2016.2626253	journal	March 2017
Frequency Regulation at a Wind Farm Using Time-Varying Inertia and Droop Controls Wu, Yuan-Kang; Yang, Wu-Han; Hu, Yi-Liang IEEE Transactions on Industry Applications, Vol. 55, Issue 1 https://doi.org/10.1109/TIA.2018.2868644	journal	January 2019
Frequency Regulation and Control of Grid-Connected Wind Farms Based on Online Reduced-Order Modeling and Adaptive Control Thakallapelli, Abilash; Nair, Anuprabha R.; Biswas, Biswajit Dipan IEEE Transactions on Industry Applications, Vol. 56, Issue 2 https://doi.org/10.1109/TIA.2020.2965507	journal	March 2020
Self-Adaptive Secondary Frequency Regulation Strategy of Micro-Grid With Multiple Virtual Synchronous Generators Zhang, Qian; Li, Yan; Ding, Zhuwei IEEE Transactions on Industry Applications, Vol. 56, Issue 5 https://doi.org/10.1109/TIA.2020.2975144	journal	September 2020
Overview of Control Systems for the Operation of DFIGs in Wind Energy Applications Cardenas, Roberto; Pena, Rubén; Alepuz, Salvador IEEE Transactions on Industrial Electronics, Vol. 60, Issue 7 https://doi.org/10.1109/TIE.2013.2243372	journal	July 2013
Reinforcement Learning Controllers for Enhancement of Low Voltage Ride Through Capability in Hybrid Power Systems Zhou, Lv; Swain, Akshya; Ukil, Abhisek IEEE Transactions on Industrial Informatics, Vol. 16, Issue 8 https://doi.org/10.1109/TII.2019.2956509	journal	August 2020
A Guided Evolutionary Strategy Based Static Var Compensator Control Approach for Inter-area Oscillation Damping Huang, Renke; Gao, Wei; Fan, Rui IEEE Transactions on Industrial Informatics https://doi.org/10.1109/TII.2022.3177430	journal	January 2022
Participation of Doubly Fed Induction Wind Generators in System Frequency Regulation de Almeida, RogÉrio G.; Pecas Lopes, J. A. IEEE Transactions on Power Systems, Vol. 22, Issue 3 https://doi.org/10.1109/TPWRS.2007.901096	journal	August 2007
Dynamic Contribution of DFIG-Based Wind Plants to System Frequency Disturbances Kayikci, M.; Milanovic, J. V. IEEE Transactions on Power Systems, Vol. 24, Issue 2 https://doi.org/10.1109/TPWRS.2009.2016062	journal	May 2009
Enhancing Frequency Response Control by DFIGs in the High Wind Penetrated Power Systems Chang-Chien, Le-Ren; Lin, Wei-Ting; Yin, Yao-Ching IEEE Transactions on Power Systems, Vol. 26, Issue 2 https://doi.org/10.1109/TPWRS.2010.2052402	journal	May 2011
Distributed Torque Control of Deloaded Wind DFIGs for Wind Farm Power Output Regulation Baros, Stefanos; Ilic, Marija D. IEEE Transactions on Power Systems, Vol. 32, Issue 6 https://doi.org/10.1109/TPWRS.2017.2663661	journal	November 2017
A Multi-Agent Deep Reinforcement Learning Method for Cooperative Load Frequency Control of a Multi-Area Power System Yan, Ziming; Xu, Yan IEEE Transactions on Power Systems, Vol. 35, Issue 6 https://doi.org/10.1109/TPWRS.2020.2999890	journal	November 2020
Reinforcement Learning for Optimal Primary Frequency Control: A Lyapunov Approach Cui, Wenqi; Jiang, Yan; Zhang, Baosen IEEE Transactions on Power Systems, Vol. 38, Issue 2 https://doi.org/10.1109/TPWRS.2022.3176525	journal	March 2023
Adaptive Power System Emergency Control Using Deep Reinforcement Learning Huang, Qiuhua; Huang, Renke; Hao, Weituo IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2933191	journal	March 2020
An Analysis of the Effects and Dependency of Wind Power Penetration on System Frequency Regulation Nguyen, Nga; Mitra, Joydeep IEEE Transactions on Sustainable Energy, Vol. 7, Issue 1 https://doi.org/10.1109/TSTE.2015.2496970	journal	January 2016
A Practical Equivalent Method for DFIG Wind Farms Li, Weixing; Chao, Pupu; Liang, Xiaodong IEEE Transactions on Sustainable Energy, Vol. 9, Issue 2 https://doi.org/10.1109/TSTE.2017.2749761	journal	April 2018
Reinforcement Learning and Its Applications in Modern Power and Energy Systems: A Review Cao, Di; Hu, Weihao; Zhao, Junbo Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 6 https://doi.org/10.35833/MPCE.2020.000552	journal	January 2020

Similar Records

Fast Frequency Response using Reinforcement Learning-Controlled Wind Turbines

Conference · Tue Oct 31 00:00:00 EDT 2023 · OSTI ID:2407018

Adaptive PV Frequency Control Strategy Based on Real-time Inertia Estimation

Journal Article · Wed Dec 16 23:00:00 EST 2020 · IEEE Transactions on Smart Grid · OSTI ID:1764455

Tri-Level Linear Programming Model for Automatic Load Shedding Using Spectral Clustering

Conference · Sun Mar 03 23:00:00 EST 2024 · OSTI ID:2319236

Deep Reinforcement Learning Based Control of Wind Turbines for Fast Frequency Response

Citation Formats

References (28)

Similar Records

Related Subjects