Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control

Dai, Yuxin; Chen, Qimei; Zhang, Jun; Wang, Xiaohui; Chen, Yilin; Gao, Tianlu; Xu, Peidong; Chen, Siyuan; Liao, Siyang; Jiang, Huaiguang; Gao, David Wenzhong

doi:10.1016/j.epsr.2022.107932

Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control

Journal Article · Sat Apr 23 04:00:00 EDT 2022 · Electric Power Systems Research

DOI:https://doi.org/10.1016/j.epsr.2022.107932· OSTI ID:1869799

Dai, Yuxin; Chen, Qimei; Zhang, Jun; Wang, Xiaohui; Chen, Yilin; Gao, Tianlu; Xu, Peidong; Chen, Siyuan; Liao, Siyang; Jiang, Huaiguang; Gao, David Wenzhong

Deep reinforcement learning (DRL) algorithms have successfully solved many challenging problems in various power system control scenarios. However, their decision-making process is usually regarded as black-boxes. Furthermore, how DRL models interact with human intelligence remains an open problem. Thus, this paper proposes a policy extraction framework to extract a complex DRL model into an explainable policy. This framework includes three parts: 1) DRL training and data generation. We train an agent for a specific control task and generate data, which contains the control policy of the agent. 2) Policy extraction. We propose an information gain rate based weighted oblique decision tree (IGR-WODT) for DRL policy extraction. 3) Policy evaluation. We define three metrics to evaluate the performance of the proposed approach. A case study for the under-voltage load shedding problem shows that the IGR-WODT presents a performance enhancement compared with DRL, weighted oblique decision tree, and univariate decision tree. The proposed policy extraction method could provide an intuitive explanation of the neural network decision-making process to the dispatchers when making final decisions on power grid operation. Also, the resulted rule-based controller could replace the deep neural network-based controller in many field edge devices with limited computing resources, providing comparable performance.

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Office of Energy Efficiency and Renewable Energy (EERE)

DOE Contract Number:: AC36-08GO28308

OSTI ID:: 1869799

Report Number(s):: NREL/JA-5D00-82997; MainId:83770; UUID:877e7b74-8c59-4b28-bef6-6374d8f86a91; MainAdminID:64574

Journal Information:: Electric Power Systems Research, Journal Name: Electric Power Systems Research Vol. 209

Country of Publication:: United States

Language:: English

References (16)

Sparse Oblique Decision Tree for Power System Security Rules Extraction and Embedding Hou, Qingchun; Zhang, Ning; Kirschen, Daniel S. IEEE Transactions on Power Systems, Vol. 36, Issue 2 https://doi.org/10.1109/TPWRS.2020.3019383	journal	March 2021
Integral reinforcement learning and experience replay for adaptive optimal control of partially-unknown constrained-input continuous-time systems Modares, Hamidreza; Lewis, Frank L.; Naghibi-Sistani, Mohammad-Bagher Automatica, Vol. 50, Issue 1 https://doi.org/10.1016/j.automatica.2013.09.043	journal	January 2014
What is a Tabby? Interpretable Model Decisions by Learning Attribute-Based Classification Criteria Liu, Haomiao; Wang, Ruiping; Shan, Shiguang IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 43, Issue 5 https://doi.org/10.1109/TPAMI.2019.2954501	journal	May 2021
Deep Reinforcement Learning Method for Demand Response Management of Interruptible Load Wang, Biao; Li, Yan; Ming, Weiyu IEEE Transactions on Smart Grid, Vol. 11, Issue 4 https://doi.org/10.1109/TSG.2020.2967430	journal	July 2020
A Strategic Day-ahead bidding strategy and operation for battery energy storage system by reinforcement learning Dong, Yi; Dong, Zhen; Zhao, Tianqiao Electric Power Systems Research, Vol. 196 https://doi.org/10.1016/j.epsr.2021.107229	journal	July 2021
Model Predictive Control-Based Real-Time Power System Protection Schemes Jin, Licheng; Kumar, Ratnesh; Elia, Nicola IEEE Transactions on Power Systems, Vol. 25, Issue 2 https://doi.org/10.1109/TPWRS.2009.2034748	journal	May 2010
Adaptive Power System Emergency Control Using Deep Reinforcement Learning Huang, Qiuhua; Huang, Renke; Hao, Weituo IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2933191	journal	March 2020
Sequential topology recovery of complex power systems based on reinforcement learning Wu, Jiajing; Fang, Biaoyan; Fang, Junyuan Physica A: Statistical Mechanics and its Applications, Vol. 535 https://doi.org/10.1016/j.physa.2019.122487	journal	December 2019
Reinforcement learning approach for congestion management and cascading failure prevention with experimental application Zarrabian, Sina; Belkacemi, Rabie; Babalola, Adeniyi A. Electric Power Systems Research, Vol. 141 https://doi.org/10.1016/j.epsr.2016.06.041	journal	December 2016
Energy optimal dispatching of ship's integrated power system based on deep reinforcement learning Shang, Chengya; Fu, Lijun; Bao, Xianqiang Electric Power Systems Research, Vol. 208 https://doi.org/10.1016/j.epsr.2022.107885	journal	July 2022
Toward self‐driving processes: A deep reinforcement learning approach to control Spielberg, Steven; Tulsyan, Aditya; Lawrence, Nathan P. AIChE Journal, Vol. 65, Issue 10 https://doi.org/10.1002/aic.16689	journal	June 2019
Transient Stability Enhancement of Power Grid With Integrated Wide Area Control of Wind Farms and Synchronous Generators Yousefian, Reza; Bhattarai, Rojan; Kamalasadan, S. IEEE Transactions on Power Systems, Vol. 32, Issue 6 https://doi.org/10.1109/TPWRS.2017.2676138	journal	November 2017
Deep-Reinforcement-Learning-Based Autonomous Voltage Control for Power Grid Operations Duan, Jiajun; Shi, Di; Diao, Ruisheng IEEE Transactions on Power Systems, Vol. 35, Issue 1 https://doi.org/10.1109/TPWRS.2019.2941134	journal	January 2020
Real-Time metadata-driven routing optimization for electric vehicle energy consumption minimization using deep reinforcement learning and Markov chain model Aljohani, Tawfiq M.; Ebrahim, Ahmed; Mohammed, Osama Electric Power Systems Research, Vol. 192 https://doi.org/10.1016/j.epsr.2020.106962	journal	March 2021
Multi-task deep reinforcement learning for intelligent multi-zone residential HVAC control Du, Yan; Li, Fangxing; Munk, Jeffrey Electric Power Systems Research, Vol. 192 https://doi.org/10.1016/j.epsr.2020.106959	journal	March 2021
Induction of decision trees Quinlan, J. R. Machine Learning, Vol. 1, Issue 1, p. 81-106 https://doi.org/10.1007/BF00116251	journal	March 1986

Similar Records

Efficient learning of power grid voltage control strategies via model-based deep reinforcement learning

Journal Article · Sun Nov 05 19:00:00 EST 2023 · Machine Learning · OSTI ID:2345752

pnnl/MBDRL

Software · Tue Nov 14 19:00:00 EST 2023 · OSTI ID:code-115961

Distributed Power Allocation for 6-GHz Unlicensed Spectrum Sharing via Multi-agent Deep Reinforcement Learning

Conference · Wed Apr 05 00:00:00 EDT 2023 · OSTI ID:1975104

Related Subjects

IGR-WODT
POWER TRANSMISSION AND DISTRIBUTION
deep reinforcement learning
explainability
knowledge distillation
policy extraction
power system emergency control

Enhanced Oblique Decision Tree Enabled Policy Extraction for Deep Reinforcement Learning in Power System Emergency Control

Citation Formats

References (16)

Similar Records

Related Subjects