Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Self-Taught Anomaly Detection With Hybrid Unsupervised/Supervised Machine Learning in Optical Networks

Journal Article · · Journal of Lightwave Technology
 [1];  [2];  [3];  [2];  [3]
  1. University of California, Davis, CA (United States); DOE/OSTI
  2. University of Science and Technology of China, Hefei (China)
  3. University of California, Davis, CA (United States)
+ Show Author Affiliations
Here this paper proposes a self-taught anomaly detection framework for optical networks. The proposed framework makes use of a hybrid unsupervised and supervised machine learning scheme. First, it employs an unsupervised data clustering module (DCM) to analyze the patterns of monitoring data. The DCM enables a self-learning capability that eliminates the requirement of prior knowledge of abnormal network behaviors and therefore can potentially detect unforeseen anomalies. Second, we introduce a self-taught mechanism that transfers the patterns learned by the DCM to a supervised data regression and classification module (DRCM). The DRCM, whose complexity is mainly related to the scale of the applied supervised learning model, can potentially facilitate more scalable and time-efficient online anomaly detection by avoiding excessively traversing the original dataset. We designed the DCM and DRCM based on the density-based clustering algorithm and the deep neural network structure, respectively. Evaluations with experimental data from two use cases (i.e., single-point detection and end-to-end detection) demonstrate that up to 99% anomaly detection accuracy can be achieved with a false positive rate below 1%.
Research Organization:
University of California, Davis, CA (United States)
Sponsoring Organization:
National Science Foundation (NSF); USDOE Office of Science (SC)
Grant/Contract Number:
SC0016700
OSTI ID:
1612677
Journal Information:
Journal of Lightwave Technology, Journal Name: Journal of Lightwave Technology Journal Issue: 7 Vol. 37; ISSN 0733-8724
Publisher:
IEEECopyright Statement
Country of Publication:
United States
Language:
English

References (34)

A Survey of Distance and Similarity Measures Used Within Network Intrusion Anomaly Detection journal January 2015
An Overview on Application of Machine Learning Techniques in Optical Networks journal July 2019
Leveraging Deep Learning to Achieve Knowledge-based Autonomous Service Provisioning in Broker-based Multi-Domain SD-EONs with Proactive and Intelligent Predictions of Multi-Domain Traffic conference September 2017
Field Demonstration of Machine-Learning-Aided Detection and Identification of Jamming Attacks in Optical Networks conference September 2018
On Real-Time and Self-Taught Anomaly Detection in Optical Networks Using Hybrid Unsupervised/Supervised Learning conference September 2018
Toward profit-seeking virtual network embedding algorithm via global resource capacity conference April 2014
Dynamic Service Provisioning in Elastic Optical Networks With Hybrid Single-/Multi-Path Routing journal January 2013
Virtual Optical Network Embedding (VONE) Over Elastic Optical Networks journal February 2014
Dynamic and Adaptive Bandwidth Defragmentation in Spectrum-Sliced Elastic Optical Networks With Time-Varying Traffic journal March 2014
On Spectrum Efficient Failure-Independent Path Protection p-Cycle Design in Elastic Optical Networks journal September 2015
Flexible Availability-Aware Differentiated Protection in Software-Defined Elastic Optical Networks journal September 2015
A Learning Living Network With Open ROADMs journal April 2017
BER Degradation Detection and Failure Identification in Elastic Optical Networks journal November 2017
Cognitive Assurance Architecture for Optical Network Fault Management journal April 2018
Network Telemetry Streaming Services in SDN-Based Disaggregated Optical Networks journal August 2018
When Deep Learning Meets Inter-Datacenter Optical Network Management: Advantages and Vulnerabilities journal October 2018
Failure location algorithm for transparent optical networks journal August 2005
Fault and attack management in all-optical networks journal November 2006
Knowledge-Based Autonomous Service Provisioning in Multi-Domain Elastic Optical Networks journal August 2018
A review of fault management in WDM mesh networks: basic concepts and research challenges journal March 2004
Highly efficient data migration and backup for big data applications in elastic optical inter-data-center networks journal September 2015
Anomaly Detection for Discrete Sequences: A Survey journal May 2012
Leveraging Statistical Machine Learning to Address Failure Localization in Optical Networks journal February 2018
T-Trail: Link Failure Monitoring in Software-Defined Optical Networks journal March 2018
Soft Failure Localization During Commissioning Testing and Lightpath Operation journal November 2017
Deep-Learning-Assisted Network Orchestration for On-Demand and Cost-Effective vNF Service Chaining in Inter-DC Elastic Optical Networks journal May 2018
Experimental Demonstration of Machine-Learning-Aided QoT Estimation in Multi-Domain Elastic Optical Networks with Alien Wavelengths journal September 2018
Efficient Resource Allocation for All-Optical Multicasting Over Spectrum-Sliced Elastic Optical Networks journal January 2013
Design of Low-Margin Optical Networks journal August 2016
Machine-Learning-Based Soft-Failure Detection and Identification in Optical Networks conference January 2018
The First Testbed Demonstration of Cognitive End-to-End Optical Service Provisioning with Hierarchical Learning across Multiple Autonomous Systems conference January 2018
Deep-RMSA: A Deep-Reinforcement-Learning Routing, Modulation and Spectrum Assignment Agent for Elastic Optical Networks conference January 2018
Multi-Agent Deep Reinforcement Learning in Cognitive Inter-Domain Networking with Multi-Broker Orchestration conference January 2019
Algorithm AS 136: A K-Means Clustering Algorithm journal January 1979

Cited By (1)

On Detecting and Preventing Jamming Attacks with Machine Learning in Optical Networks text January 2019

Figures / Tables (11)

Fig. 1(p. 2)figure Fig. 1
Fig. 2(p. 3)figure Fig. 2
Fig. 3(p. 4)figure Fig. 3
Fig. 4(p. 5)figure Fig. 4
Fig. 5(p. 5)figure Fig. 5
Fig. 6(p. 6)figure Fig. 6
Fig. 7(p. 6)figure Fig. 7
TABLE I(p. 6)table TABLE I
TABLE II(p. 6)table TABLE II
TABLE III(p. 6)table TABLE III
TABLE IV(p. 7)table TABLE IV

Similar Records

Future SDN-HPON Control Plane Architecture and Protocol for On-Demand Terabit End-to-End Extreme-Scale Science Applications
Technical Report · Mon Nov 25 23:00:00 EST 2019 · OSTI ID:1579715
Unsupervised anomaly clustering via offset alignment in multivariate grid sensing data
Conference · Sun Jun 01 00:00:00 EDT 2025 · OSTI ID:2575316
MAD: Self-Supervised Masked Anomaly Detection Task for Multivariate Time Series
Conference · Mon Jul 18 00:00:00 EDT 2022 · 2022 International Joint Conference on Neural Networks (IJCNN) · OSTI ID:1905117

Related Subjects

42 ENGINEERING
DCM
DRCM
data clustering module
data regression and classification module
hybrid unsupervised and supervised machine learning
self-taught anomaly detection