Online anomaly detection for multi‐source VMware using a distributed streaming framework

Journal Article · Mon Jan 11 00:00:00 EST 2016 · Software, Practice and Experience

DOI:https://doi.org/10.1002/spe.2390· OSTI ID:1400533

Solaimani, Mohiuddin ^[1]; Iftekhar, Mohammed ^[1]; Khan, Latifur ^[1]; Thuraisingham, Bhavani ^[1]; Ingram, Joe ^[2]; Seker, Sadi Evren ^[3]

---

1. Department of Computer Science The University of Texas at Dallas Dallas TX 75080 USA
2. Sandia National Laboratories Albuquerque NM 87123 USA
3. Department of Business Istanbul Medeniyet University Istanbul Turkey

Summary

Anomaly detection refers to the identification of patterns in a dataset that do not conform to expected patterns. Such non‐conformant patterns typically correspond to samples of interest and are assigned to different labels in different domains, such as outliers, anomalies, exceptions, and malware. A daunting challenge is to detect anomalies in rapid voluminous streams of data.

This paper presents a novel, generic real‐time distributed anomaly detection framework for multi‐source stream data. As a case study, we investigate anomaly detection for a multi‐source VMware‐based cloud data center, which maintains a large number of virtual machines (VMs). This framework continuously monitors VMware performance stream data related to CPU statistics (e.g., load and usage). It collects data simultaneously from all of the VMs connected to the network and notifies the resource manager to reschedule its CPU resources dynamically when it identifies any abnormal behavior from its collected data. A semi‐supervised clustering technique is used to build a model from benign training data only. During testing, if a data instance deviates significantly from the model, then it is flagged as an anomaly.

Effective anomaly detection in this case demands a distributed framework with high throughput and low latency. Distributed streaming frameworks like Apache Storm, Apache Spark, S4, and others are designed for a lower data processing time and a higher throughput than standard centralized frameworks. We have experimentally compared the average processing latency of a tuple during clustering and prediction in both Spark and Storm and demonstrated that Spark processes a tuple much quicker than storm on average. Copyright © 2016 John Wiley & Sons, Ltd.

• Sponsoring Organization: USDOE
• Grant/Contract Number: AC04-94AL85000
• OSTI ID: 1400533
• Journal Information: Software, Practice and Experience, Journal Name: Software, Practice and Experience Journal Issue: 11 Vol. 46; ISSN 0038-0644
• Publisher: Wiley Blackwell (John Wiley & Sons)
• Country of Publication: United Kingdom
• Language: English