skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Correcting evaluation bias of relational classifiers with network cross validation

Journal Article · · Knowledge and Information Systems
 [1];  [2];  [3];  [1]
  1. Purdue Univ., West Lafayette, IN (United States)
  2. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
  3. Rutgers Univ., Piscataway, NJ (United States)
+ Show Author Affiliations

Recently, a number of modeling techniques have been developed for data mining and machine learning in relational and network domains where the instances are not independent and identically distributed (i.i.d.). These methods specifically exploit the statistical dependencies among instances in order to improve classification accuracy. However, there has been little focus on how these same dependencies affect our ability to draw accurate conclusions about the performance of the models. More specifically, the complex link structure and attribute dependencies in relational data violate the assumptions of many conventional statistical tests and make it difficult to use these tests to assess the models in an unbiased manner. In this work, we examine the task of within-network classification and the question of whether two algorithms will learn models that will result in significantly different levels of performance. We show that the commonly used form of evaluation (paired t-test on overlapping network samples) can result in an unacceptable level of Type I error. Furthermore, we show that Type I error increases as (1) the correlation among instances increases and (2) the size of the evaluation set increases (i.e., the proportion of labeled nodes in the network decreases). Lastly, we propose a method for network cross-validation that combined with paired t-tests produces more acceptable levels of Type I error while still providing reasonable levels of statistical power (i.e., 1–Type II error).

Research Organization:
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
Sponsoring Organization:
USDOE
Grant/Contract Number:
AC52-07NA27344
OSTI ID:
1343835
Report Number(s):
LLNL-JRNL-455699
Journal Information:
Knowledge and Information Systems, Vol. 30, Issue 1; ISSN 0219-1377
Publisher:
SpringerCopyright Statement
Country of Publication:
United States
Language:
English
Citation Metrics:
Cited by: 4 works
Citation information provided by
Web of Science

References (19)

Enhanced hypertext categorization using hyperlinks journal June 1998
Cautious Inference in Collective Classification report July 2007
Learning relational probability trees
  • Neville, Jennifer; Jensen, David; Friedland, Lisa
  • Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '03 https://doi.org/10.1145/956750.956830
conference January 2003
Leveraging Relational Autocorrelation with Latent Group Models conference January 2005
Distribution-based aggregation for relational learning with identifier attributes journal January 2006
An Examination of Experimental Methodology for Classifiers of Relational Data
  • Gallagher, Brian; Eliassi-Rad, Tina
  • 2007 Seventh IEEE International Conference on Data Mining - Workshops (ICDM Workshops), Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007) https://doi.org/10.1109/ICDMW.2007.27
conference October 2007
Exploiting Cluster-Structure to Predict the Labeling of a Graph book January 2008
A Simple Relational Classifier report January 2003
Using ghost edges for classification in sparsely labeled networks
  • Gallagher, Brian; Tong, Hanghang; Eliassi-Rad, Tina
  • Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 08 https://doi.org/10.1145/1401890.1401925
conference January 2008
Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms journal October 1998
Simple estimators for relational Bayesian classifiers conference January 2003
Why collective inference improves relational classification
  • Jensen, David; Neville, Jennifer; Gallagher, Brian
  • Proceedings of the 2004 ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '04 https://doi.org/10.1145/1014052.1014125
conference January 2004
Collective Classification in Network Data journal September 2008
Evaluating Statistical Tests for Within-Network Classifiers of Relational Data conference December 2009
National Longitudinal Study of Adolescent to Adult Health (Add Health), 1994-2008 [Public Use] (ICPSR 21600). Version 18 dataset October 2017
Enhanced hypertext categorization using hyperlinks conference January 1998
Leveraging relational autocorrelation with latent group models conference January 2005
Leveraging relational autocorrelation with latent group models text January 2006
Learning Probabilistic Relational Models book January 2001

Cited By (1)

Bayesian Model Selection on Random Networks preprint January 2020

Linked Research (24)

Similar Records

Related Subjects

99 GENERAL AND MISCELLANEOUS
97 MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE
relational learning
collective classification
statistical tests
methodology