ARM-IRL: Adaptive Resilience Metric Quantification Using Inverse Reinforcement Learning

Sahu, Abhijeet; Venkatramanan, Venkatesh; Macwan, Richard

doi:10.3390/ai6050103

ARM-IRL: Adaptive Resilience Metric Quantification Using Inverse Reinforcement Learning

Journal Article · Sun May 18 00:00:00 EDT 2025 · AI

DOI:https://doi.org/10.3390/ai6050103· OSTI ID:2573884

^[1]; ^[1]; Macwan, Richard ^[1]

National Renewable Energy Laboratory (NREL), Golden, CO (United States)

The resilience of safety-critical systems is gaining importance due to the rise in cyber and physical threats, especially within critical infrastructure. Traditional static resilience metrics may not capture dynamic system states, leading to inaccurate assessments and ineffective responses to cyber threats. This work aims to develop a data-driven, adaptive method for resilience metric learning. We propose a data-driven approach using inverse reinforcement learning (IRL) to learn a single, adaptive resilience metric. The method infers a reward function from expert control actions. Unlike previous approaches using static weights or fuzzy logic, this work applies adversarial inverse reinforcement learning (AIRL), training a generator and discriminator in parallel to learn the reward structure and derive an optimal policy. The proposed approach is evaluated on multiple scenarios: optimal communication network rerouting, power distribution network reconfiguration, and cyber–physical restoration of critical loads using the IEEE 123-bus system. The adaptive, learned resilience metric enables faster critical load restoration in comparison to conventional RL approaches.

View Accepted Manuscript (DOE)

Research Organization:: National Renewable Energy Laboratory (NREL), Golden, CO (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program

Grant/Contract Number:: AC36-08GO28308

OSTI ID:: 2573884

Report Number(s):: NREL/JA--5R00-86269

Journal Information:: AI, Journal Name: AI Journal Issue: 5 Vol. 6; ISSN 2673-2688

Publisher:: MDPI AGCopyright Statement

Country of Publication:: United States

Language:: English

References (27)

A survey of inverse reinforcement learning Adams, Stephen; Cody, Tyler; Beling, Peter A. Artificial Intelligence Review, Vol. 55, Issue 6 https://doi.org/10.1007/s10462-021-10108-x	journal	February 2022
Resilience metrics for cyber systems Linkov, Igor; Eisenberg, Daniel A.; Plourde, Kenton Environment Systems and Decisions, Vol. 33, Issue 4 https://doi.org/10.1007/s10669-013-9485-y	journal	November 2013
Microgrids as a resilience resource and strategies used by microgrids for enhancing resilience Hussain, Akhtar; Bui, Van-Hai; Kim, Hak-Man Applied Energy, Vol. 240 https://doi.org/10.1016/j.apenergy.2019.02.055	journal	April 2019
Reinforcement Learning for feedback-enabled cyber resilience Huang, Yunhan; Huang, Linan; Zhu, Quanyan Annual Reviews in Control, Vol. 53 https://doi.org/10.1016/j.arcontrol.2022.01.001	journal	January 2022
GAIL-PT: An intelligent penetration testing framework with generative adversarial imitation learning Chen, Jinyin; Hu, Shulong; Zheng, Haibin Computers & Security, Vol. 126 https://doi.org/10.1016/j.cose.2022.103055	journal	March 2023
An imitation learning-based energy management strategy for electric vehicles considering battery aging Ye, Yiming; Wang, Hanchen; Xu, Bin Energy, Vol. 283 https://doi.org/10.1016/j.energy.2023.128537	journal	November 2023
Real-time power system dispatch scheme using grid expert strategy-based imitation learning Xu, Siyang; Zhu, Jiebei; Li, Bingsen International Journal of Electrical Power & Energy Systems, Vol. 161 https://doi.org/10.1016/j.ijepes.2024.110148	journal	October 2024
Quantitative analysis of power systems resilience: Standardization, categorizations, and challenges Umunnakwe, A.; Huang, H.; Oikonomou, K. Renewable and Sustainable Energy Reviews, Vol. 149 https://doi.org/10.1016/j.rser.2021.111252	journal	October 2021
Modelling penetration testing with reinforcement learning using capture‐the‐flag challenges: Trade‐offs between model‐free learning and a priori knowledge Zennaro, Fabio Massimo; Erdődi, László IET Information Security, Vol. 17, Issue 3 https://doi.org/10.1049/ise2.12107	journal	January 2023
Optimizing dynamics of integrated food–energy–water systems under the risk of climate change Memarzadeh, Milad; Moura, Scott; Horvath, Arpad Environmental Research Letters, Vol. 14, Issue 7 https://doi.org/10.1088/1748-9326/ab2104	journal	July 2019
Power System Resilience: Current Practices, Challenges, and Future Directions Bhusal, Narayan; Abdelmalak, Michael; Kamruzzaman, Md IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.2968586	journal	January 2020
Resilient Scheduling of Networked Microgrids Against Real-Time Failures Fesagandis, Hadi Safari; Jalali, Mehdi; Zare, Kazem IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3052653	journal	January 2021
Emotional Deep Learning Programming Controller for Automatic Voltage Control of Power Systems Yin, Linfei; Zhang, Chenwei; Wang, Yaoxiong IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3060620	journal	January 2021
Reinforcement Learning Environment for Cyber-Resilient Power Distribution System Sahu, Abhijeet; Venkatraman, Venkatesh; Macwan, Richard IEEE Access, Vol. 11 https://doi.org/10.1109/ACCESS.2023.3282182	journal	January 2023
Performance-Based Cyber Resilience Metrics: An Applied Demonstration Toward Moving Target Defense Hossain-McKenzie, S.; Lai, C.; Chavez, A. IECON 2018 - 44th Annual Conference of the IEEE Industrial Electronics Society https://doi.org/10.1109/IECON.2018.8591764	conference	October 2018
Quantifying the System-Level Resilience of Thermal Power Generation to Extreme Temperatures and Water Scarcity Zhou, Yutian; Panteli, Mathaios; Wang, Bowen IEEE Systems Journal, Vol. 14, Issue 1 https://doi.org/10.1109/JSYST.2019.2938332	journal	March 2020
To Improve Cyber Resilience, Measure It Kott, Alexander; Linkov, Igor Computer, Vol. 54, Issue 2 https://doi.org/10.1109/MC.2020.3038411	journal	February 2021
Distribution System Restoration With Microgrids Using Spanning Tree Search Li, Juan; Ma, Xi-Yuan; Liu, Chen-Ching IEEE Transactions on Power Systems, Vol. 29, Issue 6 https://doi.org/10.1109/TPWRS.2014.2312424	journal	November 2014
Metrics and Quantification of Operational and Infrastructure Resilience in Power Systems Panteli, Mathaios; Mancarella, Pierluigi; Trakas, Dimitris N. IEEE Transactions on Power Systems, Vol. 32, Issue 6 https://doi.org/10.1109/TPWRS.2017.2664141	journal	November 2017
CP-SAM: Cyber-Physical Security Assessment Metric for Monitoring Microgrid Resiliency Venkataramanan, Venkatesh; Hahn, Adam; Srivastava, Anurag IEEE Transactions on Smart Grid, Vol. 11, Issue 2 https://doi.org/10.1109/TSG.2019.2930241	journal	March 2020
CP-TRAM: Cyber-Physical Transmission Resiliency Assessment Metric No authors listed IEEE Transactions on Smart Grid, Vol. 11, Issue 6, p. 5114-5123 https://doi.org/10.1109/TSG.2020.2996137	journal	May 2020
Online Learning and Distributed Control for Residential Demand Response Chen, Xin; Li, Yingying; Shimada, Jun IEEE Transactions on Smart Grid, Vol. 12, Issue 6 https://doi.org/10.1109/TSG.2021.3090039	journal	November 2021
Self-Organizing Map-Based Resilience Quantification and Resilient Control of Distribution Systems Under Extreme Events Utkarsh, Kumar; Ding, Fei IEEE Transactions on Smart Grid, Vol. 13, Issue 3 https://doi.org/10.1109/TSG.2022.3150226	journal	May 2022
Towards a Resilience Metric Framework for Cyber-Physical Systems Friedberg, Ivo; McLaughlin, Kieran; Smith, Paul Electronic Workshops in Computing https://doi.org/10.14236/ewic/ICS2016.3	conference	October 2016
An IRL Approach for Cyber-Physical Attack Intention Prediction and Recovery Elnaggar, Mahmoud; Bezzo, Nicola 2018 Annual American Control Conference (ACC) https://doi.org/10.23919/ACC.2018.8430922	conference	June 2018
Resilience of Cyber-Physical Systems: an Experimental Appraisal of Quantitative Measures Murino, Giuseppina; Armando, Alessandro; Tacchella, Armando 2019 11th International Conference on Cyber Conflict (CyCon) https://doi.org/10.23919/CYCON.2019.8757010	conference	May 2019
A Data-driven Method for Fast AC Optimal Power Flow Solutions via Deep Reinforcement Learning Zhou, Yuhao; Zhang, Bei; Xu, Chunlei Journal of Modern Power Systems and Clean Energy, Vol. 8, Issue 6 https://doi.org/10.35833/MPCE.2020.000522	journal	January 2020

Similar Records

Reinforcement Learning for feedback-enabled cyber resilience

Journal Article · Sun Jan 30 19:00:00 EST 2022 · Annual Reviews in Control · OSTI ID:1976876

Planning for Resilient Power Distribution Systems using Risk-Based Quantification and Q-Learning

Conference · Sun Dec 19 23:00:00 EST 2021 · OSTI ID:1844605

Enhancing Distribution System Resilience: A First-Order Meta-RL Algorithm for Critical Load Restoration

Conference · Sun Nov 03 23:00:00 EST 2024 · OSTI ID:2497732

Related Subjects

97 MATHEMATICS AND COMPUTING
cyber-physical systems
inverse reinforcement learning
reconfiguration
reinforcement learning
resilience

ARM-IRL: Adaptive Resilience Metric Quantification Using Inverse Reinforcement Learning

Citation Formats

References (27)

Similar Records

Related Subjects