Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Kiran, Mariam; Wang, Cong; Papadimitriou, George; Mandal, Anirban; Deelman, Ewa

doi:10.1007/s10994-020-05870-y

Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Journal Article · Thu Mar 12 00:00:00 EDT 2020 · Machine Learning

DOI:https://doi.org/10.1007/s10994-020-05870-y· OSTI ID:1787027

Kiran, Mariam ^[1]; Wang, Cong ^[2]; Papadimitriou, George ^[3]; Mandal, Anirban ^[2]; Deelman, Ewa ^[3]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); University of Southern California
Univ. of North Carolina, Chapel Hill, NC (United States)
Univ. of Southern California, Los Angeles, CA (United States). Information Sciences Institute

Large-scale scientific workflows rely heavily on high-performance file transfers. These transfers require strict quality parameters such as guaranteed bandwidth, no packet loss or data duplication. To have successful file transfers, methods such as predetermined thresholds and statistical analysis need to be done to determine abnormal patterns. Network administrators routinely monitor and analyze network data for diagnosing and alleviating these, making decisions based on their experience. However, as networks grow and become complex, monitoring large data files and quickly processing them, makes it improbable to identify errors and rectify these. Abnormal file transfers have been classified by simply setting alert thresholds, via tools such as PerfSonar and TCP statistics (Tstat). This paper investigates the feasibility of unsupervised feature extraction methods for identifying network anomaly patterns with three unsupervised classification methods—principal component analysis, autoencoder and isolation forest. Here, we collect file transfer statistics from two experiment sets—synthetic iPerf generated traffic and 1000 Genome workflow runs, with synthetically introduced anomalies. Our results show that while PCA and a simple autoencoder finds it difficult to detect clusters, the tree-variant isolation forest is able to identify anomalous packets by breaking down TCP traces into tree classes early.

View Accepted Manuscript (DOE)

Research Organization:: Univ. of Southern California, Los Angeles, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: SC0012636

OSTI ID:: 1787027

Journal Information:: Machine Learning, Journal Name: Machine Learning Journal Issue: 5 Vol. 109; ISSN 0885-6125

Publisher:: Springer NatureCopyright Statement

Country of Publication:: United States

Language:: English

References (28)

Principal Component Analysis Jolliffe, Ian International Encyclopedia of Statistical Science https://doi.org/10.1007/978-3-642-04898-2_455	book	January 2011
Passive analysis of TCP anomalies Mellia, Marco; Meo, Michela; Muscariello, Luca Computer Networks, Vol. 52, Issue 14 https://doi.org/10.1016/j.comnet.2008.05.010	journal	October 2008
Network anomaly detection through nonlinear analysis Palmieri, Francesco; Fiore, Ugo Computers & Security, Vol. 29, Issue 7 https://doi.org/10.1016/j.cose.2010.05.002	journal	October 2010
A transform domain-based anomaly detection approach to network-wide traffic Jiang, Dingde; Xu, Zhengzheng; Zhang, Peng Journal of Network and Computer Applications, Vol. 40 https://doi.org/10.1016/j.jnca.2013.09.014	journal	April 2014
A global reference for human genetic variation Consortium, The 1000 Genomes Project; Auton, Adam; Abecasis, Gonçalo R. Nature, Vol. 526, Issue 7571, p. 68-74 https://doi.org/10.1038/nature15393	journal	January 2015
Mining and modeling web trajectories from passive traces Vassio, Luca; Mellia, Marco; Figueiredo, Flavio 2017 IEEE International Conference on Big Data (Big Data 2017), 2017 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData.2017.8258416	conference	December 2017
Flow Monitoring Explained: From Packet Capture to Data Analysis With NetFlow and IPFIX Hofstede, Rick; Celeda, Pavel; Trammell, Brian IEEE Communications Surveys & Tutorials, Vol. 16, Issue 4 https://doi.org/10.1109/COMST.2014.2321898	journal	January 2014
Unusual internet traffic detection at network edge Bansal, Neha; Kaushal, Rishabh 2015 International Conference on Computing and Network Communications (CoCoNet) https://doi.org/10.1109/CoCoNet.2015.7411184	conference	December 2015
User patience and the Web: a hands-on investigation Rossi, D.; Mellia, M.; Casetti, C. GLOBECOM '03. IEEE Global Telecommunications Conference (IEEE Cat. No.03CH37489) https://doi.org/10.1109/GLOCOM.2003.1259011	conference	January 2003
Log summarization and anomaly detection for troubleshooting distributed systems Gunter, Dan; Tierney, Brian L.; Brown, Aaron 2007 8th IEEE/ACM International Conference on Grid Computing https://doi.org/10.1109/GRID.2007.4354137	conference	September 2007
Anomaly detection for scientific workflow applications on networked clouds Gaikwad, Prathamesh; Mandal, Anirban; Ruth, Paul 2016 International Conference on High Performance Computing & Simulation (HPCS) https://doi.org/10.1109/HPCSim.2016.7568396	conference	July 2016
Anomaly Based Network Intrusion Detection with Unsupervised Outlier Detection Zhang, Jiong; Zulkernine, Mohammad 2006 IEEE International Conference on Communications https://doi.org/10.1109/ICC.2006.255127	conference	June 2006
Isolation Forest Liu, Fei Tony; Ting, Kai Ming; Zhou, Zhi-Hua 2008 Eighth IEEE International Conference on Data Mining (ICDM) https://doi.org/10.1109/ICDM.2008.17	conference	December 2008
Detection Network Anomalies Based on Packet and Flow Analysis Wang, Hong; Gong, Zhenghu; Guan, Qing 2008 Seventh International Conference on Networking (ICN '08), Seventh International Conference on Networking (icn 2008) https://doi.org/10.1109/ICN.2008.83	conference	April 2008
Passive TCP stream estimation of RTT and jitter parameters But, J.; Keller, U.; Kennedy, D. The IEEE Conference on Local Computer Networks 30th Anniversary (LCN'05)l https://doi.org/10.1109/LCN.2005.101	conference	January 2005
Automated traffic classification and application identification using machine learning Zander, S.; Nguyen, T.; Armitage, G. The IEEE Conference on Local Computer Networks 30th Anniversary (LCN'05)l https://doi.org/10.1109/LCN.2005.35	conference	January 2005
Unveiling network and service performance degradation in the wild with mplane Casas, Pedro; Fiadino, Pierdomenico; Wassermann, Sarah IEEE Communications Magazine, Vol. 54, Issue 3 https://doi.org/10.1109/MCOM.2016.7432151	journal	March 2016
Traffic Analysis with Off-the-Shelf Hardware: Challenges and Lessons Learned Trevisan, Martino; Finamore, Alessandro; Mellia, Marco IEEE Communications Magazine, Vol. 55, Issue 3 https://doi.org/10.1109/MCOM.2017.1600756CM	journal	March 2017
Experiences of Internet traffic monitoring with tstat Finamore, Alessandro; Mellia, Marco; Meo, Michela IEEE Network, Vol. 25, Issue 3 https://doi.org/10.1109/MNET.2011.5772055	journal	May 2011
A Machine Learning Approach to TCP Throughput Prediction Mirza, Mariyam; Sommers, Joel; Barford, Paul IEEE/ACM Transactions on Networking, Vol. 18, Issue 4 https://doi.org/10.1109/TNET.2009.2037812	journal	August 2010
Diagnosing network-wide traffic anomalies Lakhina, Anukool; Crovella, Mark; Diot, Christophe Proceedings of the 2004 conference on Applications, technologies, architectures, and protocols for computer communications - SIGCOMM '04 https://doi.org/10.1145/1015467.1015492	conference	January 2004
Mining anomalies using traffic feature distributions Lakhina, Anukool; Crovella, Mark; Diot, Christophe Proceedings of the 2005 conference on Applications, technologies, architectures, and protocols for computer communications - SIGCOMM '05 https://doi.org/10.1145/1080091.1080118	conference	January 2005
Boosting for transfer learning Dai, Wenyuan; Yang, Qiang; Xue, Gui-Rong Proceedings of the 24th international conference on Machine learning - ICML '07 https://doi.org/10.1145/1273496.1273521	conference	January 2007
Self-taught learning: transfer learning from unlabeled data Raina, Rajat; Battle, Alexis; Lee, Honglak Proceedings of the 24th international conference on Machine learning - ICML '07 https://doi.org/10.1145/1273496.1273592	conference	January 2007
Measuring Web Speed From Passive Traces Trevisan, Martino; Drago, Idilio; Mellia, Marco ANRW '18: Applied Networking Research Workshop, Proceedings of the Applied Networking Research Workshop https://doi.org/10.1145/3232755.3232780	conference	July 2018
A signal analysis of network traffic anomalies Barford, Paul; Kline, Jeffery; Plonka, David Proceedings of the second ACM SIGCOMM Workshop on Internet measurment - IMW '02 https://doi.org/10.1145/637201.637210	conference	January 2002
Learning Deep Architectures for AI Bengio, Y. Foundations and Trends® in Machine Learning, Vol. 2, Issue 1 https://doi.org/10.1561/2200000006	journal	January 2009
Measurement Analysis of TCP Congestion Control Algorithms in LTE Uplink Parichehreh, Ali; Alfredsson, Stefan; Brunstrom, Anna 2018 Network Traffic Measurement and Analysis Conference (TMA) https://doi.org/10.23919/TMA.2018.8506522	conference	June 2018

Similar Records

Detecting Outliers in Network Transfers with Feature Extraction

Conference · Sun Jul 01 00:00:00 EDT 2018 · OSTI ID:1468101

An investigation of packet reordering in TCP traces (extended abstract)

Conference · Wed Dec 31 23:00:00 EST 2003 · OSTI ID:977651

Experiences with TCP/IP over an ATM OC12 WAN

Technical Report · Wed Dec 22 23:00:00 EST 1999 · OSTI ID:764365

Related Subjects

97 MATHEMATICS AND COMPUTING
Autoencoders
File transfers
Isolation forest
PCA
TCP anomalies
Unsupervised feature extraction

Detecting anomalous packets in network transfers: investigations using PCA, autoencoder and isolation forest in TCP

Citation Formats

References (28)

Similar Records

Related Subjects