Reinforcement Learning for feedback-enabled cyber resilience

Huang, Yunhan; Huang, Linan; Zhu, Quanyan

doi:10.1016/j.arcontrol.2022.01.001

Reinforcement Learning for feedback-enabled cyber resilience

Journal Article · Sun Jan 30 23:00:00 EST 2022 · Annual Reviews in Control

DOI:https://doi.org/10.1016/j.arcontrol.2022.01.001· OSTI ID:1976876

Huang, Yunhan ^[1]; Huang, Linan ^[2]; Zhu, Quanyan ^[2]

New York University (NYU), NY (United States); OSTI
New York University (NYU), NY (United States)

The rapid growth in the number of devices and their connectivity has enlarged the attack surface and made cyber systems more vulnerable. As attackers become increasingly sophisticated and resourceful, mere reliance on traditional cyber protection, such as intrusion detection, firewalls, and encryption, is insufficient to secure the cyber systems. Cyber resilience provides a new security paradigm that complements inadequate protection with resilience mechanisms. A Cyber-Resilient Mechanism (CRM) adapts to the known or zero-day threats and uncertainties in real-time and strategically responds to them to maintain the critical functions of the cyber systems in the event of successful attacks. Feedback architectures play a pivotal role in enabling the online sensing, reasoning, and actuation process of the CRM. Reinforcement Learning (RL) is an important gathering of algorithms that epitomize the feedback architectures for cyber resilience. It allows the CRM to provide dynamic and sequential responses to attacks with limited or without prior knowledge of the environment and the attacker. In this work, we review the literature on RL for cyber resilience and discuss the cyber-resilient defenses against three major types of vulnerabilities, i.e., posture-related, information-related, and human-related vulnerabilities. Here we introduce moving target defense, defensive cyber deception, and assistive human security technologies as three application domains of CRMs to elaborate on their designs. The RL algorithms also have vulnerabilities themselves. We explain the major vulnerabilities of RL and present develop several attack models where the attacker target the information exchanged between the environment and the agent: the rewards, the state observations, and the action commands. We show that the attacker can trick the RL agent into learning a nefarious policy with minimum attacking effort. The paper introduces several defense methods to secure the RL-enabled systems from these attacks. However, there is still a lack of works that focuses on the defensive mechanisms for RL-enabled systems. Last but not least, we discuss the future challenges of RL for cyber security and resilience and emerging applications of RL-based CRMs.

View Accepted Manuscript (DOE)

Research Organization:: The Ohio State University, Columbus, OH (United States); New York University (NYU), NY (United States)

Sponsoring Organization:: USDOE Office of Nuclear Energy (NE); National Science Foundation (NSF); Army Research Office (ARO)

Grant/Contract Number:: NE0008986

OSTI ID:: 1976876

Journal Information:: Annual Reviews in Control, Journal Name: Annual Reviews in Control Journal Issue: C Vol. 53; ISSN 1367-5788

Publisher:: International Federation of Automatic Control - ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (64)

Metalearning: a survey of trends and technologies Lemke, Christiane; Budka, Marcin; Gabrys, Bogdan Artificial Intelligence Review, Vol. 44, Issue 1 https://doi.org/10.1007/s10462-013-9406-y	journal	July 2013
On the rewards of self-adaptive IoT honeypots Pauna, Adrian; Bica, Ion; Pop, Florin Annals of Telecommunications, Vol. 74, Issue 7-8 https://doi.org/10.1007/s12243-018-0695-7	journal	January 2019
Theory and applications of adaptive control—A survey Åström, K. J. Automatica, Vol. 19, Issue 5 https://doi.org/10.1016/0005-1098(83)90002-X	journal	September 1983
Decentralized control: An overview Bakule, Lubomír Annual Reviews in Control, Vol. 32, Issue 1 https://doi.org/10.1016/j.arcontrol.2008.03.004	journal	April 2008
Resilient control under Denial-of-Service: Robust design Feng, Shuai; Tesi, Pietro Automatica, Vol. 79 https://doi.org/10.1016/j.automatica.2017.01.031	journal	May 2017
Control structure design for complete chemical plants Skogestad, Sigurd Computers & Chemical Engineering, Vol. 28, Issue 1-2 https://doi.org/10.1016/j.compchemeng.2003.08.002	journal	January 2004
A dynamic games approach to proactive defense strategies against Advanced Persistent Threats in cyber-physical systems Huang, Linan; Zhu, Quanyan Computers & Security, Vol. 89 https://doi.org/10.1016/j.cose.2019.101660	journal	February 2020
Detection of online phishing email using dynamic evolving neural network based on reinforcement learning Smadi, Sami; Aslam, Nauman; Zhang, Li Decision Support Systems, Vol. 107 https://doi.org/10.1016/j.dss.2018.01.001	journal	March 2018
Distributed response to network intrusions using multiagent reinforcement learning Malialis, Kleanthis; Kudenko, Daniel Engineering Applications of Artificial Intelligence, Vol. 41 https://doi.org/10.1016/j.engappai.2015.01.013	journal	May 2015
Application of deep reinforcement learning to intrusion detection for supervised problems Lopez-Martin, Manuel; Carro, Belen; Sanchez-Esguevillas, Antonio Expert Systems with Applications, Vol. 141 https://doi.org/10.1016/j.eswa.2019.112963	journal	March 2020
Securing oil and gas infrastructure Bajpai, S.; Gupta, J. P. Journal of Petroleum Science and Engineering, Vol. 55, Issue 1-2 https://doi.org/10.1016/j.petrol.2006.04.007	journal	January 2007
Finite-horizon semi-Markov game for time-sensitive attack response and probabilistic risk assessment in nuclear power plants Zhao, Yunfei; Huang, Linan; Smidts, Carol Reliability Engineering & System Safety, Vol. 201 https://doi.org/10.1016/j.ress.2020.106878	journal	September 2020
Power system structure optimization based on reinforcement learning and sparse constraints under DoS attacks in cloud environments Zhu, Zhiqin; Zeng, Fancheng; Qi, Guanqiu Simulation Modelling Practice and Theory, Vol. 110 https://doi.org/10.1016/j.simpat.2021.102272	journal	July 2021
Mastering the game of Go with deep neural networks and tree search Silver, David; Huang, Aja; Maddison, Chris J. Nature, Vol. 529, Issue 7587 https://doi.org/10.1038/nature16961	journal	January 2016
Fast reinforcement learning with generalized policy updates Barreto, André; Hou, Shaobo; Borsa, Diana Proceedings of the National Academy of Sciences, Vol. 117, Issue 48 https://doi.org/10.1073/pnas.1907370117	journal	August 2020
Event-triggered resilient control for cyber-physical system under denial-of-service attacks Liu, Shan; Li, Shanbin; Xu, Bugong International Journal of Control, Vol. 93, Issue 8 https://doi.org/10.1080/00207179.2018.1537518	journal	October 2018
Improving adaptive honeypot functionality with efficient reinforcement learning parameters for automated malware Dowling, Seamus; Schukat, Michael; Barrett, Enda Journal of Cyber Security Technology, Vol. 2, Issue 2 https://doi.org/10.1080/23742917.2018.1495375	journal	April 2018
Dynamic games for secure and resilient control system design Huang, Yunhan; Chen, Juntao; Huang, Linan National Science Review, Vol. 7, Issue 7 https://doi.org/10.1093/nsr/nwz218	journal	January 2020
Stabilizing and destabilizing control for a piecewise-linear circuit Tsubone, T.; Saijo, T. IEEE Transactions on Circuits and Systems I: Fundamental Theory and Applications, Vol. 45, Issue 2 https://doi.org/10.1109/81.661686	journal	January 1998
An Intelligent Deployment Policy for Deception Resources Based on Reinforcement Learning Wang, Shuo; Pei, Qingqi; Wang, Jianhua IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.2974786	journal	January 2020
CPSS LR-DDoS Detection and Defense in Edge Computing Utilizing DCNN Q-Learning Liu, Zengguang; Yin, Xiaochun; Hu, Yuemei IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.2976706	journal	January 2020
Adversarial Attacks Against Reinforcement Learning-Based Portfolio Management Strategy Chen, Yu-Ying; Chen, Chiao-Ting; Sang, Chuan-Yun IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3068768	journal	January 2021
DESOLATER: Deep Reinforcement Learning-Based Resource Allocation and Moving Target Defense Deployment Framework Yoon, Seunghyun; Cho, Jin-Hee; Kim, Dong Seong IEEE Access, Vol. 9 https://doi.org/10.1109/ACCESS.2021.3076599	journal	January 2021
A Survey of Data Mining and Machine Learning Methods for Cyber Security Intrusion Detection Buczak, Anna L.; Guven, Erhan IEEE Communications Surveys & Tutorials, Vol. 18, Issue 2 https://doi.org/10.1109/COMST.2015.2494502	journal	July 2016
Deep Reinforcement Learning for Partially Observable Data Poisoning Attack in Crowdsensing Systems Li, Mohan; Sun, Yanbin; Lu, Hui IEEE Internet of Things Journal, Vol. 7, Issue 7 https://doi.org/10.1109/JIOT.2019.2962914	journal	July 2020
A Comprehensive Survey on Transfer Learning Zhuang, Fuzhen; Qi, Zhiyuan; Duan, Keyu Proceedings of the IEEE, Vol. 109, Issue 1 https://doi.org/10.1109/JPROC.2020.3004555	journal	January 2021
Interference Aware Routing Game for Cognitive Radio Multi-Hop Networks Zhu, Quanyan; Yuan, Zhou; Song, Ju Bin IEEE Journal on Selected Areas in Communications, Vol. 30, Issue 10 https://doi.org/10.1109/JSAC.2012.121115	journal	November 2012
A Bi-Level Game Approach to Attack-Aware Cyber Insurance of Computer Networks Zhang, Rui; Zhu, Quanyan; Hayel, Yezekael IEEE Journal on Selected Areas in Communications, Vol. 35, Issue 3 https://doi.org/10.1109/JSAC.2017.2672378	journal	March 2017
Deep Reinforcement Learning for Mobile Edge Caching: Review, New Features, and Open Issues Zhu, Hao; Cao, Yang; Wang, Wei IEEE Network, Vol. 32, Issue 6 https://doi.org/10.1109/MNET.2018.1800109	journal	November 2018
The Ransomware Threat to Energy-Delivery Systems Nicol, David M. IEEE Security & Privacy, Vol. 19, Issue 3 https://doi.org/10.1109/MSEC.2021.3063678	journal	May 2021
Anti-honeypot technology Krawetz, N. IEEE Security & Privacy Magazine, Vol. 2, Issue 1 https://doi.org/10.1109/MSECP.2004.1264861	journal	January 2004
Combating the Insider Cyber Threat Greitzer, Frank L.; Moore, Andrew P.; Cappelli, Dawn M. IEEE Security & Privacy Magazine, Vol. 6, Issue 1 https://doi.org/10.1109/MSP.2008.8	journal	January 2008
Stuxnet: Dissecting a Cyberwarfare Weapon Langner, Ralph IEEE Security & Privacy Magazine, Vol. 9, Issue 3 https://doi.org/10.1109/MSP.2011.67	journal	May 2011
IoT Security Techniques Based on Machine Learning: How Do IoT Devices Use AI to Enhance Security? Xiao, Liang; Wan, Xiaoyue; Lu, Xiaozhen IEEE Signal Processing Magazine, Vol. 35, Issue 5 https://doi.org/10.1109/MSP.2018.2825478	journal	September 2018
Security in Mobile Edge Caching with Reinforcement Learning Xiao, Liang; Wan, Xiaoyue; Dai, Canhuang IEEE Wireless Communications, Vol. 25, Issue 3 https://doi.org/10.1109/MWC.2018.1700291	journal	June 2018
Markov decision processes with delays and asynchronous cost collection Katsikopoulos, K. V.; Engelbrecht, S. E. IEEE Transactions on Automatic Control, Vol. 48, Issue 4 https://doi.org/10.1109/TAC.2003.809799	journal	April 2003
Design of Networked Control Systems With Packet Dropouts Wu, Jing; Chen, Tongwen IEEE Transactions on Automatic Control, Vol. 52, Issue 7 https://doi.org/10.1109/TAC.2007.900839	journal	July 2007
Secure Estimation and Control for Cyber-Physical Systems Under Adversarial Attacks Fawzi, Hamza; Tabuada, Paulo; Diggavi, Suhas IEEE Transactions on Automatic Control, Vol. 59, Issue 6 https://doi.org/10.1109/TAC.2014.2303233	journal	June 2014
LQ Secure Control for Cyber-Physical Systems Against Sparse Sensor and Actuator Attacks An, Liwei; Yang, Guang-Hong IEEE Transactions on Control of Network Systems, Vol. 6, Issue 2 https://doi.org/10.1109/TCNS.2018.2878507	journal	June 2019
A Differential Game Approach to Decentralized Virus-Resistant Weight Adaptation Policy Over Complex Networks Huang, Yunhan; Zhu, Quanyan IEEE Transactions on Control of Network Systems, Vol. 7, Issue 2 https://doi.org/10.1109/TCNS.2019.2931862	journal	June 2020
Reliable Control Policy of Cyber-Physical Systems Against a Class of Frequency-Constrained Sensor and Actuator Attacks Huang, Xin; Dong, Jiuxiang IEEE Transactions on Cybernetics, Vol. 48, Issue 12 https://doi.org/10.1109/TCYB.2018.2815758	journal	December 2018
$\mathtt{FlipIn}$ : A Game-Theoretic Cyber Insurance Framework for Incentive-Compatible Cyber Risk Management of Internet of Things Zhang, Rui; Zhu, Quanyan IEEE Transactions on Information Forensics and Security, Vol. 15 https://doi.org/10.1109/TIFS.2019.2955891	journal	January 2020
DRL-FAS: A Novel Framework Based on Deep Reinforcement Learning for Face Anti-Spoofing Cai, Rizhao; Li, Haoliang; Wang, Shiqi IEEE Transactions on Information Forensics and Security, Vol. 16 https://doi.org/10.1109/TIFS.2020.3026553	journal	January 2021
Duplicity Games for Deception Design With an Application to Insider Threat Mitigation Huang, Linan; Zhu, Quanyan IEEE Transactions on Information Forensics and Security, Vol. 16 https://doi.org/10.1109/TIFS.2021.3118886	journal	January 2021
A Multistage Game in Smart Grid Security: A Reinforcement Learning Solution Ni, Zhen; Paul, Shuva IEEE Transactions on Neural Networks and Learning Systems, Vol. 30, Issue 9 https://doi.org/10.1109/TNNLS.2018.2885530	journal	September 2019
Distributed Reinforcement Learning for Cyber-Physical System With Multiple Remote State Estimation Under DoS Attacker Dai, Pengcheng; Yu, Wenwu; Wang, He IEEE Transactions on Network Science and Engineering, Vol. 7, Issue 4 https://doi.org/10.1109/TNSE.2020.3018871	journal	October 2020
Dependable Demand Response Management in the Smart Grid: A Stackelberg Game Approach Maharjan, Sabita; Zhu, Quanyan; Zhang, Yan IEEE Transactions on Smart Grid, Vol. 4, Issue 1 https://doi.org/10.1109/TSG.2012.2223766	journal	March 2013
Evaluation of Reinforcement Learning-Based False Data Injection Attack to Automatic Voltage Control Chen, Ying; Huang, Shaowei; Liu, Feng IEEE Transactions on Smart Grid, Vol. 10, Issue 2 https://doi.org/10.1109/TSG.2018.2790704	journal	March 2019
Online Cyber-Attack Detection in Smart Grid: A Reinforcement Learning Approach Kurt, Mehmet Necip; Ogundijo, Oyetunji; Li, Chong IEEE Transactions on Smart Grid, Vol. 10, Issue 5 https://doi.org/10.1109/TSG.2018.2878570	journal	September 2019
PHY-Layer Spoofing Detection With Reinforcement Learning in Wireless Networks Xiao, Liang; Li, Yan; Han, Guoan IEEE Transactions on Vehicular Technology, Vol. 65, Issue 12 https://doi.org/10.1109/TVT.2016.2524258	journal	December 2016
Attack models and scenarios for networked control systems Teixeira, André; Pérez, Daniel; Sandberg, Henrik Proceedings of the 1st international conference on High Confidence Networked Systems - HiCoNS '12 https://doi.org/10.1145/2185505.2185515	conference	January 2012
Game theory meets network security and privacy Manshaei, Mohammad Hossein; Zhu, Quanyan; Alpcan, Tansu ACM Computing Surveys, Vol. 45, Issue 3 https://doi.org/10.1145/2480741.2480742	journal	June 2013
Compliance Control Casey, William Austin; Zhu, Quanyan; Morales, Jose Andre Proceedings of the 7th ACM CCS International Workshop on Managing Insider Security Threats https://doi.org/10.1145/2808783.2808788	conference	October 2015
Forecasting Zero-Day Vulnerabilities Last, David Proceedings of the 11th Annual Cyber and Information Security Research Conference https://doi.org/10.1145/2897795.2897813	conference	April 2016
A Game-Theoretic Approach to Secure Control of Communication-Based Train Control Systems Under Jamming Attacks Xu, Zhiheng; Zhu, Quanyan Proceedings of the 1st International Workshop on Safe Control of Connected and Autonomous Vehicles https://doi.org/10.1145/3055378.3055381	conference	April 2017
Detecting Stealthy Botnets in a Resource-Constrained Environment using Reinforcement Learning Venkatesan, Sridhar; Albanese, Massimiliano; Shah, Ankit Proceedings of the 2017 Workshop on Moving Target Defense https://doi.org/10.1145/3140549.3140552	conference	October 2017
Adaptive Strategic Cyber Defense for Advanced Persistent Threats in Critical Infrastructure Networks Huang, Linan; Zhu, Quanyan ACM SIGMETRICS Performance Evaluation Review, Vol. 46, Issue 2 https://doi.org/10.1145/3305218.3305239	journal	January 2019
A Game-theoretic Taxonomy and Survey of Defensive Deception for Cybersecurity and Privacy Pawlick, Jeffrey; Colbert, Edward; Zhu, Quanyan ACM Computing Surveys, Vol. 52, Issue 4 https://doi.org/10.1145/3337772	journal	August 2019
Survey of review spam detection using machine learning techniques Crawford, Michael; Khoshgoftaar, Taghi M.; Prusa, Joseph D. Journal of Big Data, Vol. 2, Issue 1 https://doi.org/10.1186/s40537-015-0029-9	journal	October 2015
A Tutorial on Thompson Sampling Russo, Daniel J.; Van Roy, Benjamin; Kazerouni, Abbas Foundations and Trends® in Machine Learning, Vol. 11, Issue 1 https://doi.org/10.1561/2200000070	journal	January 2018
An Introduction to Deep Reinforcement Learning François-Lavet, Vincent; Henderson, Peter; Islam, Riashat Foundations and Trends® in Machine Learning, Vol. 11, Issue 3-4 https://doi.org/10.1561/2200000071	journal	January 2018
On the Control of Multi-Agent Systems: A Survey Chen, Fei; Ren, Wei Foundations and Trends® in Systems and Control, Vol. 6, Issue 4 https://doi.org/10.1561/2600000019	journal	January 2019
Reinforcement Learning with Perturbed Rewards Wang, Jingkang; Liu, Yang; Li, Bo Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 34, Issue 04 https://doi.org/10.1609/aaai.v34i04.6086	journal	April 2020
Social Engineering Attacks: A Survey Salahdine, Fatima; Kaabouch, Naima Future Internet, Vol. 11, Issue 4 https://doi.org/10.3390/fi11040089	journal	April 2019

Similar Records

Defense on the Move: Ant-Based Cyber Defense

Journal Article · Tue Apr 15 00:00:00 EDT 2014 · IEEE Security & Privacy, 12(2):36-43 · OSTI ID:1158971

Automated Adversary Emulation for Cyber-Physical Systems via Reinforcement Learning

Conference · Sun Nov 15 23:00:00 EST 2020 · OSTI ID:1760319

Deception used for Cyber Defense of Control Systems

Conference · Fri May 01 00:00:00 EDT 2009 · OSTI ID:957557

Related Subjects

97 MATHEMATICS AND COMPUTING
advanced persistent threats
cyber deception
cyber vulnerabilities
feedback control systems
honeypots
human inattention
moving target defense
optimal control theory
reinforcement Learning
resilience
security

Reinforcement Learning for feedback-enabled cyber resilience

Citation Formats

References (64)

Similar Records

Related Subjects