Multivariate network traffic analysis using clustered patterns

Kim, Jinoh; Sim, Alex; Tierney, Brian; Suh, Sang; Kim, Ikkyun

doi:10.1007/s00607-018-0619-4

Multivariate network traffic analysis using clustered patterns

Journal Article · 28 April 2018 · Computing: Archiv fuer Informatik und Numerik

DOI:https://doi.org/10.1007/s00607-018-0619-4· OSTI ID:1498687

^[1]; Sim, Alex ^[2]; Tierney, Brian ^[3]; Suh, Sang ^[1]; Kim, Ikkyun ^[4]

Texas A & M Univ., Commerce, TX (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Energy Sciences Network, Berkeley, CA (United States)
Electronics and Telecommunications Research Inst., Daejon (Korea, Republic of)

Traffic analysis is a core element in network operations and management for various purposes including change detection, traffic prediction, and anomaly detection. In this paper, we introduce a new approach to online traffic analysis based on a pattern-based representation for high-level summarization of the traffic measurement data. Unlike the past online analysis techniques limited to a single variable to summarize (e.g., sketch), the focus of this study is on capturing the network state from the multivariate attributes under consideration. To this end, we employ clustering with its benefit of the aggregation of multidimensional variables. The clustered result represents the state of the network with regard to the monitored variables, which can also be compared with the observed patterns from previous time windows enabling intuitive analysis. Finally, we demonstrate the proposed method with two popular use cases, one for estimating state changes and the other for identifying anomalous states, to confirm its feasibility. Our extensive experimental results with public traces and collected monitoring measurements from ESnet traffic traces show that our pattern-based approach is effective for multivariate analysis of online network traffic with visual and quantitative tools.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1498687

Journal Information:: Computing: Archiv fuer Informatik und Numerik, Journal Name: Computing: Archiv fuer Informatik und Numerik Journal Issue: 4 Vol. 101; ISSN 0010-485X

Publisher:: Springer NatureCopyright Statement

Country of Publication:: United States

Language:: English

References (35)

Network Anomaly Detection Using Co-clustering Papalexakis, Evangelos E.; Beutel, Alex; Steenkiste, Peter Encyclopedia of Social Network Analysis and Mining https://doi.org/10.1007/978-1-4614-6170-8_354	book	January 2014
Network Anomaly Detection Using Co-clustering Papalexakis, Evangelos E.; Beutel, Alex; Steenkiste, Peter Encyclopedia of Social Network Analysis and Mining https://doi.org/10.1007/978-1-4614-7163-9_354-1	book	January 2017
Randomized Algorithms Che, Maolin; Wei, Yimin Theory and Computation of Complex Tensors and its Applications https://doi.org/10.1007/978-981-15-2059-4_8	book	January 2020
A new intrusion detection system using support vector machines and hierarchical clustering Khan, Latifur; Awad, Mamoun; Thuraisingham, Bhavani The VLDB Journal, Vol. 16, Issue 4 https://doi.org/10.1007/s00778-006-0002-5	journal	August 2006
A multivariate Kolmogorov-Smirnov test of goodness of fit Justel, Ana; Peña, Daniel; Zamar, Rubén Statistics & Probability Letters, Vol. 35, Issue 3 https://doi.org/10.1016/S0167-7152(97)00020-5	journal	October 1997
Measuring IP and TCP behavior on edge nodes with Tstat Mellia, M.; Locigno, R.; Neri, F. Computer Networks, Vol. 47, Issue 1 https://doi.org/10.1016/S1389-1286(04)00201-4	journal	January 2005
Anomaly-based network intrusion detection: Techniques, systems and challenges García-Teodoro, P.; Díaz-Verdejo, J.; Maciá-Fernández, G. Computers & Security, Vol. 28, Issue 1-2 https://doi.org/10.1016/j.cose.2008.08.003	journal	February 2009
Monitoring abnormal network traffic based on blind source separation approach Qin, Tao; Guan, Xiaohong; Li, Wei Journal of Network and Computer Applications, Vol. 34, Issue 5 https://doi.org/10.1016/j.jnca.2011.06.006	journal	September 2011
A survey of network flow applications Li, Bingdong; Springer, Jeff; Bebis, George Journal of Network and Computer Applications, Vol. 36, Issue 2 https://doi.org/10.1016/j.jnca.2012.12.020	journal	March 2013
A survey of network anomaly detection techniques Ahmed, Mohiuddin; Naser Mahmood, Abdun; Hu, Jiankun Journal of Network and Computer Applications, Vol. 60 https://doi.org/10.1016/j.jnca.2015.11.016	journal	January 2016
Network anomaly detection using IP flows with Principal Component Analysis and Ant Colony Optimization Fernandes, Gilberto; Carvalho, Luiz F.; Rodrigues, Joel J. P. C. Journal of Network and Computer Applications, Vol. 64 https://doi.org/10.1016/j.jnca.2015.11.024	journal	April 2016
Finite population corrections for the Kolmogorov–Smirnov tests O'Neill, Terence J.; Stern, Steven E. Journal of Nonparametric Statistics, Vol. 24, Issue 2 https://doi.org/10.1080/10485252.2011.650169	journal	June 2012
Network Anomaly Detection Using Co-clustering Papalexakis, E. E.; Beutel, A.; Steenkiste, P. 2012 International Conference on Advances in Social Networks Analysis and Mining (ASONAM 2012), 2012 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining https://doi.org/10.1109/ASONAM.2012.72	conference	August 2012
A Cluster-Based Intrusion Detection Framework for Monitoring the Traffic of Cloud Environments Li, Bo; Liu, Peng; Lin, Li 2016 IEEE 3rd International Conference on Cyber Security and Cloud Computing (CSCloud) https://doi.org/10.1109/CSCloud.2016.43	conference	June 2016
Unsupervised Labeling for Supervised Anomaly Detection in Enterprise and Cloud Networks Baek, Sunhee; Kwon, Donghwoon; Kim, Jinoh 2017 IEEE 4th International Conference on Cyber Security and Cloud Computing (CSCloud) https://doi.org/10.1109/CSCloud.2017.26	conference	June 2017
Using GMM and SVM-Based Techniques for the Classification of SSH-Encrypted Traffic Dusi, M.; Este, A.; Gringoli, F. ICC 2009 - 2009 IEEE International Conference on Communications https://doi.org/10.1109/ICC.2009.5199557	conference	June 2009
CoTS: A Scalable Framework for Parallelizing Frequency Counting over Data Streams Das, Sudipto; Antony, Shyam; Agrawal, Divyakant 2009 IEEE 25th International Conference on Data Engineering (ICDE) https://doi.org/10.1109/ICDE.2009.231	conference	March 2009
Maintaining Stream Statistics over Sliding Windows Datar, Mayur; Gionis, Aristides; Indyk, Piotr SIAM Journal on Computing, Vol. 31, Issue 6 https://doi.org/10.1137/S0097539701398363	journal	January 2002
BLINC: multilevel traffic classification in the dark Karagiannis, Thomas; Papagiannaki, Konstantina; Faloutsos, Michalis Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications - SIGCOMM '05 https://doi.org/10.1145/1080091.1080119	conference	January 2005
BLINC: multilevel traffic classification in the dark Karagiannis, Thomas; Papagiannaki, Konstantina; Faloutsos, Michalis ACM SIGCOMM Computer Communication Review, Vol. 35, Issue 4 https://doi.org/10.1145/1090191.1080119	journal	October 2005
Network monitoring using traffic dispersion graphs (tdgs) Iliofotou, Marios; Pappu, Prashanth; Faloutsos, Michalis Proceedings of the 7th ACM SIGCOMM conference on Internet measurement - IMC '07 https://doi.org/10.1145/1298306.1298349	conference	January 2007
Observing slow crustal movement in residential user traffic Cho, Kenjiro; Fukuda, Kensuke; Esaki, Hiroshi Proceedings of the 2008 ACM CoNEXT Conference on - CONEXT '08 https://doi.org/10.1145/1544012.1544024	conference	January 2008
GT: picking up the truth from the ground for internet traffic Gringoli, F.; Salgarelli, Luca; Dusi, M. ACM SIGCOMM Computer Communication Review, Vol. 39, Issue 5 https://doi.org/10.1145/1629607.1629610	journal	October 2009
NeTraMark: a network traffic classification benchmark Lee, Suchul; Kim, Hyunchul; Barman, Dhiman ACM SIGCOMM Computer Communication Review, Vol. 41, Issue 1 https://doi.org/10.1145/1925861.1925865	journal	January 2011
Randomized algorithms Motwani, Rajeev; Raghavan, Prabhakar ACM Computing Surveys, Vol. 28, Issue 1 https://doi.org/10.1145/234313.234327	journal	March 1996
The Science DMZ: a network design pattern for data-intensive science Dart, Eli; Rotman, Lauren; Tierney, Brian Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '13 https://doi.org/10.1145/2503210.2503245	conference	January 2013
Visualizing Traffic Causality for Analyzing Network Anomalies Zhang, Hao; Sun, Maoyuan; Yao, Danfeng (Daphne) Proceedings of the 2015 ACM International Workshop on International Workshop on Security and Privacy Analytics - IWSPA '15 https://doi.org/10.1145/2713579.2713583	conference	January 2015
High Throughput Sketch Based Online Heavy Hitter Detection on FPGA Tong, Da; Prasanna, Viktor ACM SIGARCH Computer Architecture News, Vol. 43, Issue 4 https://doi.org/10.1145/2927964.2927977	journal	April 2016
One Sketch to Rule Them All: Rethinking Network Flow Monitoring with UnivMon Liu, Zaoxing; Manousis, Antonis; Vorsanger, Gregory Proceedings of the 2016 conference on ACM SIGCOMM 2016 Conference - SIGCOMM '16 https://doi.org/10.1145/2934872.2934906	conference	January 2016
Heavy-Hitter Detection Entirely in the Data Plane Sivaraman, Vibhaalakshmi; Narayana, Srinivas; Rottenstreich, Ori Proceedings of the Symposium on SDN Research - SOSR '17 https://doi.org/10.1145/3050220.3063772	conference	January 2017
Data-streams and histograms Guha, Sudipto; Koudas, Nick; Shim, Kyuseok Proceedings of the thirty-third annual ACM symposium on Theory of computing - STOC '01 https://doi.org/10.1145/380752.380841	conference	January 2001
Sketch-based change detection: methods, evaluation, and applications Krishnamurthy, Balachander; Sen, Subhabrata; Zhang, Yin Proceedings of the 2003 ACM SIGCOMM conference on Internet measurement - IMC '03 https://doi.org/10.1145/948205.948236	conference	January 2003
Sketch-based change detection: methods, evaluation, and applications Krishnamurthy, Balachander; Sen, Subhabrata; Zhang, Yin Proceedings of the conference on Internet measurement conference - IMC '03 https://doi.org/10.1145/948234.948236	conference	January 2003
Scalable k-means++ Bahmani, Bahman; Moseley, Benjamin; Vattani, Andrea Proceedings of the VLDB Endowment, Vol. 5, Issue 7 https://doi.org/10.14778/2180912.2180915	journal	March 2012
Heavy-Hitter Detection Entirely in the Data Plane Sivaraman, Vibhaalakshmi; Narayana, Srinivas; Rottenstreich, Ori arXiv https://doi.org/10.48550/arxiv.1611.04825	text	January 2016

Similar Records

A New Approach to Multivariate Network Traffic Analysis

Journal Article · 2019 · Journal of Computer Science and Technology · OSTI ID:1526584

An approach to online network monitoring using clustered patterns

Journal Article · 2017 · 2017 International Conference on Computing, Networking and Communications, ICNC 2017 · OSTI ID:1379769

Peeking Network States with Clustered Patterns

Technical Report · 2015 · OSTI ID:1237063

Related Subjects

97 MATHEMATICS AND COMPUTING
Anomaly detection
Change detection
Clustered patterns
Multivariate analysis
Network traffic analysis

Multivariate network traffic analysis using clustered patterns

Citation Formats

References (35)

Similar Records

Related Subjects