Designing Size Consistent Statistics for Accurate Anomaly Detection in Dynamic Networks

Fond, Timothy La; Neville, Jennifer; Gallagher, Brian

doi:10.1145/3185059

Title: Designing Size Consistent Statistics for Accurate Anomaly Detection in Dynamic Networks

Journal Article · 16 April 2018 · ACM Transactions on Knowledge Discovery from Data

DOI: https://doi.org/10.1145/3185059 · OSTI ID:1497323

Fond, Timothy La ^[1]; Neville, Jennifer ^[2]; Gallagher, Brian ^[1]

Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
Purdue Univ., West Lafayette, IN (United States)

An important task in network analysis is the detection of anomalous events in a network time series. These events could merely be times of interest in the network timeline or they could be examples of malicious activity or network malfunction. Hypothesis testing using network statistics to summarize the behavior of the network provides a robust framework for the anomaly detection decision process. Unfortunately, choosing network statistics that are dependent on confounding factors like the total number of nodes or edges can lead to incorrect conclusions (e.g., false positives and false negatives). In this article, we describe the challenges that face anomaly detection in dynamic network streams regarding confounding factors. We also provide two solutions to avoiding error due to confounding factors: the first is a randomization testing method that controls for confounding factors, and the second is a set of size-consistent network statistics that avoid confounding due to the most common factors, edge count and node count.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)

Sponsoring Organization:: USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: AC52-07NA27344

OSTI ID:: 1497323

Report Number(s):: LLNL-JRNL-738004; 889202

Journal Information:: ACM Transactions on Knowledge Discovery from Data, Vol. 20, Issue 2; ISSN 1556-4681

Publisher:: Association for Computing MachineryCopyright Statement

Country of Publication:: United States

Language:: English

References (34)

Graph based anomaly detection and description: a survey Akoglu, Leman; Tong, Hanghang; Koutra, Danai Data Mining and Knowledge Discovery, Vol. 29, Issue 3 https://doi.org/10.1007/s10618-014-0365-y	journal	July 2014
Randomization tests for distinguishing social influence and homophily effects La Fond, Timothy; Neville, Jennifer Proceedings of the 19th international conference on World wide web - WWW '10 https://doi.org/10.1145/1772690.1772752	conference	January 2010
Networks in transition: how industry events (re)shape interfirm relationships Madhavan, Ravindranath; Koka, Balaji R.; Prescott, John E. Strategic Management Journal, Vol. 19, Issue 5 https://doi.org/10.1002/(SICI)1097-0266(199805)19:5<439::AID-DIA952>3.0.CO;2-2	journal	May 1998
How Do They ‘End Up Together’? A Social Network Analysis of Self-Control, Homophily, and Adolescent Relationships Young, Jacob T. N. Journal of Quantitative Criminology, Vol. 27, Issue 3 https://doi.org/10.1007/s10940-010-9105-7	journal	August 2010
Introduction to stochastic actor-based models for network dynamics Snijders, Tom A. B.; van de Bunt, Gerhard G.; Steglich, Christian E. G. Social Networks, Vol. 32, Issue 1 https://doi.org/10.1016/j.socnet.2009.02.004	journal	January 2010
Testing Multitheoretical, Multilevel Hypotheses About Organizational Networks: An Analytic Framework and Empirical Example Contractor, Noshir S.; Wasserman, Stanley; Faust, Katherine Academy of Management Review, Vol. 31, Issue 3 https://doi.org/10.5465/amr.2006.21318925	journal	July 2006
Predicting with networks: Nonparametric multiple regression analysis of dyadic data Krackhardt, David Social Networks, Vol. 10, Issue 4 https://doi.org/10.1016/0378-8733(88)90004-4	journal	December 1988
Group formation in large social networks: membership, growth, and evolution Backstrom, Lars; Huttenlocher, Dan; Kleinberg, Jon Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '06 https://doi.org/10.1145/1150402.1150412	conference	January 2006
Comparison of distance matrices in studies of population structure and genetic microdifferentiation: Quadratic assignment Dow, Malcolm M.; Cheverud, James M. American Journal of Physical Anthropology, Vol. 68, Issue 3 https://doi.org/10.1002/ajpa.1330680307	journal	November 1985
Models for evolving fixed node networks: Model fitting and model testing Sanil, Ashish; Banks, David; Carley, Kathleen Social Networks, Vol. 17, Issue 1 https://doi.org/10.1016/0378-8733(94)00249-A	journal	January 1995
8. Comparing Networks across Space and Time, Size and Species Faust, Katherine; Skvoretz, John Sociological Methodology, Vol. 32, Issue 1 https://doi.org/10.1111/1467-9531.00118	journal	August 2002
RolX: structural role extraction & mining in large graphs Henderson, Keith; Gallagher, Brian; Eliassi-Rad, Tina Proceedings of the 18th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '12 https://doi.org/10.1145/2339530.2339723	conference	January 2012
D elta C on : A Principled Massive-Graph Similarity Function Koutra, Danai; Vogelstein, Joshua T.; Faloutsos, Christos Proceedings of the 2013 SIAM International Conference on Data Mining https://doi.org/10.1137/1.9781611972832.18	conference	December 2013
A statistical infinite feature cascade-based approach to anomaly detection for dynamic social networks Yasami, Yasser; Safaei, Farshad Computer Communications, Vol. 100 https://doi.org/10.1016/j.comcom.2016.11.010	journal	March 2017
Scan Statistics on Enron Graphs Priebe, Carey E.; Conroy, John M.; Marchette, David J. Computational and Mathematical Organization Theory, Vol. 11, Issue 3, p. 229-247 https://doi.org/10.1007/s10588-005-5378-z	journal	October 2005
Fast Generation of Large Scale Social Networks While Incorporating Transitive Closures Pfeiffer, Joseph J.; La Fond, Timothy; Moreno, Sebastian 2012 International Conference on Privacy, Security, Risk and Trust (PASSAT), 2012 International Conference on Privacy, Security, Risk and Trust and 2012 International Confernece on Social Computing https://doi.org/10.1109/SocialCom-PASSAT.2012.130	conference	September 2012
Decay functions Burt, Ronald S. Social Networks, Vol. 22, Issue 1 https://doi.org/10.1016/S0378-8733(99)00015-5	journal	May 2000
Web graph similarity for anomaly detection Papadimitriou, Panagiotis; Dasdan, Ali; Garcia-Molina, Hector Journal of Internet Services and Applications, Vol. 1, Issue 1 https://doi.org/10.1007/s13174-010-0003-x	journal	February 2010
Micro-Level Interpretation of Exponential Random Graph Models with Application to Estuary Networks: Desmarais/Cranmer: Micro-Level Interpretation of ERGM Desmarais, Bruce A.; Cranmer, Skyler J. Policy Studies Journal, Vol. 40, Issue 3 https://doi.org/10.1111/j.1541-0072.2012.00459.x	journal	August 2012
Computing core/periphery structures and permutation tests for social relations data Boyd, John P.; Fitzgerald, William J.; Beck, Robert J. Social Networks, Vol. 28, Issue 2 https://doi.org/10.1016/j.socnet.2005.06.003	journal	May 2006
A random graph model for massive graphs Aiello, William; Chung, Fan; Lu, Linyuan Proceedings of the thirty-second annual ACM symposium on Theory of computing - STOC '00 https://doi.org/10.1145/335305.335326	conference	January 2000
Influence and correlation in social networks Anagnostopoulos, Aris; Kumar, Ravi; Mahdian, Mohammad Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 08 https://doi.org/10.1145/1401890.1401897	conference	January 2008
Predicting defects using network analysis on dependency graphs Zimmermann, Thomas; Nagappan, Nachiappan Proceedings of the 13th international conference on Software engineering - ICSE '08 https://doi.org/10.1145/1368088.1368161	conference	January 2008
Discrete temporal models of social networks Hanneke, Steve; Fu, Wenjie; Xing, Eric P. Electronic Journal of Statistics, Vol. 4, Issue 0 https://doi.org/10.1214/09-EJS548	journal	January 2010
On the Exact Covariance of Products of Random Variables Bohrnstedt, George W.; Goldberger, Arthur S. Journal of the American Statistical Association, Vol. 64, Issue 328 https://doi.org/10.1080/01621459.1969.10501069	journal	December 1969
The Coevolution of Networks and Political Attitudes Lazer, David; Rubineau, Brian; Chetkovich, Carol Political Communication, Vol. 27, Issue 3 https://doi.org/10.1080/10584609.2010.500187	journal	August 2010
Network science Börner, Katy; Sanyal, Soma; Vespignani, Alessandro Annual Review of Information Science and Technology, Vol. 41, Issue 1 https://doi.org/10.1002/aris.2007.1440410119	journal	January 2007
New Specifications for Exponential Random Graph Models Snijders, Tom A. B.; Pattison, Philippa E.; Robins, Garry L. Sociological Methodology, Vol. 36, Issue 1 https://doi.org/10.1111/j.1467-9531.2006.00176.x	journal	August 2006
A survey of graph edit distance Gao, Xinbo; Xiao, Bing; Tao, Dacheng Pattern Analysis and Applications, Vol. 13, Issue 1 https://doi.org/10.1007/s10044-008-0141-y	journal	January 2009
Korean university life in a network perspective: Dynamics of a large affiliation network Holme, Petter; Min Park, Sung; Kim, Beom Jun Physica A: Statistical Mechanics and its Applications, Vol. 373 https://doi.org/10.1016/j.physa.2006.04.066	journal	January 2007
The simultaneous evolution of author and paper networks Borner, K.; Maru, J. T.; Goldstone, R. L. Proceedings of the National Academy of Sciences, Vol. 101, Issue Supplement 1 https://doi.org/10.1073/pnas.0307625100	journal	February 2004
Distinguishing influence-based contagion from homophily-driven diffusion in dynamic networks Aral, S.; Muchnik, L.; Sundararajan, A. Proceedings of the National Academy of Sciences, Vol. 106, Issue 51 https://doi.org/10.1073/pnas.0908800106	journal	December 2009
Computer Network Monitoring and Abnormal Event Detection Using Graph Matching and Multidimensional Scaling Bunke, H.; Dickinson, P.; Humm, A. Advances in Data Mining. Applications in Medicine, Web Mining, Marketing, Image and Signal Mining https://doi.org/10.1007/11790853_45	book	January 2006
Learning mixed kronecker product graph models with simulated method of moments Moreno, Sebastian I.; Neville, Jennifer; Kirshner, Sergey Proceedings of the 19th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '13 https://doi.org/10.1145/2487575.2487675	conference	January 2013