skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Multivariate network traffic analysis using clustered patterns

Abstract

Traffic analysis is a core element in network operations and management for various purposes including change detection, traffic prediction, and anomaly detection. In this paper, we introduce a new approach to online traffic analysis based on a pattern-based representation for high-level summarization of the traffic measurement data. Unlike the past online analysis techniques limited to a single variable to summarize (e.g., sketch), the focus of this study is on capturing the network state from the multivariate attributes under consideration. To this end, we employ clustering with its benefit of the aggregation of multidimensional variables. The clustered result represents the state of the network with regard to the monitored variables, which can also be compared with the observed patterns from previous time windows enabling intuitive analysis. Finally, we demonstrate the proposed method with two popular use cases, one for estimating state changes and the other for identifying anomalous states, to confirm its feasibility. Our extensive experimental results with public traces and collected monitoring measurements from ESnet traffic traces show that our pattern-based approach is effective for multivariate analysis of online network traffic with visual and quantitative tools.

Authors:
ORCiD logo [1];  [2];  [3];  [1];  [4]
  1. Texas A & M Univ., Commerce, TX (United States)
  2. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  3. Energy Sciences Network, Berkeley, CA (United States)
  4. Electronics and Telecommunications Research Inst., Daejon (Korea, Republic of)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
OSTI Identifier:
1498687
DOE Contract Number:  
AC02-05CH11231
Resource Type:
Journal Article
Journal Name:
Computing: Archiv fuer Informatik und Numerik
Additional Journal Information:
Journal Volume: 101; Journal Issue: 4; Journal ID: ISSN 0010-485X
Publisher:
Springer Nature
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Network traffic analysis; Clustered patterns; Change detection; Anomaly detection; Multivariate analysis

Citation Formats

Kim, Jinoh, Sim, Alex, Tierney, Brian, Suh, Sang, and Kim, Ikkyun. Multivariate network traffic analysis using clustered patterns. United States: N. p., 2018. Web. doi:10.1007/s00607-018-0619-4.
Kim, Jinoh, Sim, Alex, Tierney, Brian, Suh, Sang, & Kim, Ikkyun. Multivariate network traffic analysis using clustered patterns. United States. doi:10.1007/s00607-018-0619-4.
Kim, Jinoh, Sim, Alex, Tierney, Brian, Suh, Sang, and Kim, Ikkyun. Sat . "Multivariate network traffic analysis using clustered patterns". United States. doi:10.1007/s00607-018-0619-4. https://www.osti.gov/servlets/purl/1498687.
@article{osti_1498687,
title = {Multivariate network traffic analysis using clustered patterns},
author = {Kim, Jinoh and Sim, Alex and Tierney, Brian and Suh, Sang and Kim, Ikkyun},
abstractNote = {Traffic analysis is a core element in network operations and management for various purposes including change detection, traffic prediction, and anomaly detection. In this paper, we introduce a new approach to online traffic analysis based on a pattern-based representation for high-level summarization of the traffic measurement data. Unlike the past online analysis techniques limited to a single variable to summarize (e.g., sketch), the focus of this study is on capturing the network state from the multivariate attributes under consideration. To this end, we employ clustering with its benefit of the aggregation of multidimensional variables. The clustered result represents the state of the network with regard to the monitored variables, which can also be compared with the observed patterns from previous time windows enabling intuitive analysis. Finally, we demonstrate the proposed method with two popular use cases, one for estimating state changes and the other for identifying anomalous states, to confirm its feasibility. Our extensive experimental results with public traces and collected monitoring measurements from ESnet traffic traces show that our pattern-based approach is effective for multivariate analysis of online network traffic with visual and quantitative tools.},
doi = {10.1007/s00607-018-0619-4},
journal = {Computing: Archiv fuer Informatik und Numerik},
issn = {0010-485X},
number = 4,
volume = 101,
place = {United States},
year = {2018},
month = {4}
}

Works referenced in this record:

Measuring IP and TCP behavior on edge nodes with Tstat
journal, January 2005


A multivariate Kolmogorov-Smirnov test of goodness of fit
journal, October 1997


A survey of network anomaly detection techniques
journal, January 2016

  • Ahmed, Mohiuddin; Naser Mahmood, Abdun; Hu, Jiankun
  • Journal of Network and Computer Applications, Vol. 60
  • DOI: 10.1016/j.jnca.2015.11.016

Maintaining Stream Statistics over Sliding Windows
journal, January 2002


A survey of network flow applications
journal, March 2013

  • Li, Bingdong; Springer, Jeff; Bebis, George
  • Journal of Network and Computer Applications, Vol. 36, Issue 2
  • DOI: 10.1016/j.jnca.2012.12.020

GT: picking up the truth from the ground for internet traffic
journal, October 2009

  • Gringoli, F.; Salgarelli, Luca; Dusi, M.
  • ACM SIGCOMM Computer Communication Review, Vol. 39, Issue 5
  • DOI: 10.1145/1629607.1629610

NeTraMark: a network traffic classification benchmark
journal, January 2011

  • Lee, Suchul; Kim, Hyunchul; Barman, Dhiman
  • ACM SIGCOMM Computer Communication Review, Vol. 41, Issue 1
  • DOI: 10.1145/1925861.1925865

BLINC: multilevel traffic classification in the dark
journal, October 2005

  • Karagiannis, Thomas; Papagiannaki, Konstantina; Faloutsos, Michalis
  • ACM SIGCOMM Computer Communication Review, Vol. 35, Issue 4
  • DOI: 10.1145/1090191.1080119

High Throughput Sketch Based Online Heavy Hitter Detection on FPGA
journal, April 2016


A new intrusion detection system using support vector machines and hierarchical clustering
journal, August 2006


Scalable k-means++
journal, March 2012

  • Bahmani, Bahman; Moseley, Benjamin; Vattani, Andrea
  • Proceedings of the VLDB Endowment, Vol. 5, Issue 7
  • DOI: 10.14778/2180912.2180915

Finite population corrections for the Kolmogorov–Smirnov tests
journal, June 2012


Anomaly-based network intrusion detection: Techniques, systems and challenges
journal, February 2009

  • García-Teodoro, P.; Díaz-Verdejo, J.; Maciá-Fernández, G.
  • Computers & Security, Vol. 28, Issue 1-2
  • DOI: 10.1016/j.cose.2008.08.003

Network anomaly detection using IP flows with Principal Component Analysis and Ant Colony Optimization
journal, April 2016

  • Fernandes, Gilberto; Carvalho, Luiz F.; Rodrigues, Joel J. P. C.
  • Journal of Network and Computer Applications, Vol. 64
  • DOI: 10.1016/j.jnca.2015.11.024

Monitoring abnormal network traffic based on blind source separation approach
journal, September 2011

  • Qin, Tao; Guan, Xiaohong; Li, Wei
  • Journal of Network and Computer Applications, Vol. 34, Issue 5
  • DOI: 10.1016/j.jnca.2011.06.006