DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Correcting evaluation bias of relational classifiers with network cross validation

Abstract

Recently, a number of modeling techniques have been developed for data mining and machine learning in relational and network domains where the instances are not independent and identically distributed (i.i.d.). These methods specifically exploit the statistical dependencies among instances in order to improve classification accuracy. However, there has been little focus on how these same dependencies affect our ability to draw accurate conclusions about the performance of the models. More specifically, the complex link structure and attribute dependencies in relational data violate the assumptions of many conventional statistical tests and make it difficult to use these tests to assess the models in an unbiased manner. In this work, we examine the task of within-network classification and the question of whether two algorithms will learn models that will result in significantly different levels of performance. We show that the commonly used form of evaluation (paired t-test on overlapping network samples) can result in an unacceptable level of Type I error. Furthermore, we show that Type I error increases as (1) the correlation among instances increases and (2) the size of the evaluation set increases (i.e., the proportion of labeled nodes in the network decreases). Lastly, we propose a method for networkmore » cross-validation that combined with paired t-tests produces more acceptable levels of Type I error while still providing reasonable levels of statistical power (i.e., 1–Type II error).« less

Authors:
 [1];  [2];  [3];  [1]
  1. Purdue Univ., West Lafayette, IN (United States)
  2. Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
  3. Rutgers Univ., Piscataway, NJ (United States)
Publication Date:
Research Org.:
Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1343835
Report Number(s):
LLNL-JRNL-455699
Journal ID: ISSN 0219-1377
Grant/Contract Number:  
AC52-07NA27344
Resource Type:
Accepted Manuscript
Journal Name:
Knowledge and Information Systems
Additional Journal Information:
Journal Volume: 30; Journal Issue: 1; Journal ID: ISSN 0219-1377
Publisher:
Springer
Country of Publication:
United States
Language:
English
Subject:
99 GENERAL AND MISCELLANEOUS; 97 MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; relational learning; collective classification; statistical tests; methodology

Citation Formats

Neville, Jennifer, Gallagher, Brian, Eliassi-Rad, Tina, and Wang, Tao. Correcting evaluation bias of relational classifiers with network cross validation. United States: N. p., 2011. Web. doi:10.1007/s10115-010-0373-1.
Neville, Jennifer, Gallagher, Brian, Eliassi-Rad, Tina, & Wang, Tao. Correcting evaluation bias of relational classifiers with network cross validation. United States. https://doi.org/10.1007/s10115-010-0373-1
Neville, Jennifer, Gallagher, Brian, Eliassi-Rad, Tina, and Wang, Tao. Tue . "Correcting evaluation bias of relational classifiers with network cross validation". United States. https://doi.org/10.1007/s10115-010-0373-1. https://www.osti.gov/servlets/purl/1343835.
@article{osti_1343835,
title = {Correcting evaluation bias of relational classifiers with network cross validation},
author = {Neville, Jennifer and Gallagher, Brian and Eliassi-Rad, Tina and Wang, Tao},
abstractNote = {Recently, a number of modeling techniques have been developed for data mining and machine learning in relational and network domains where the instances are not independent and identically distributed (i.i.d.). These methods specifically exploit the statistical dependencies among instances in order to improve classification accuracy. However, there has been little focus on how these same dependencies affect our ability to draw accurate conclusions about the performance of the models. More specifically, the complex link structure and attribute dependencies in relational data violate the assumptions of many conventional statistical tests and make it difficult to use these tests to assess the models in an unbiased manner. In this work, we examine the task of within-network classification and the question of whether two algorithms will learn models that will result in significantly different levels of performance. We show that the commonly used form of evaluation (paired t-test on overlapping network samples) can result in an unacceptable level of Type I error. Furthermore, we show that Type I error increases as (1) the correlation among instances increases and (2) the size of the evaluation set increases (i.e., the proportion of labeled nodes in the network decreases). Lastly, we propose a method for network cross-validation that combined with paired t-tests produces more acceptable levels of Type I error while still providing reasonable levels of statistical power (i.e., 1–Type II error).},
doi = {10.1007/s10115-010-0373-1},
journal = {Knowledge and Information Systems},
number = 1,
volume = 30,
place = {United States},
year = {Tue Jan 04 00:00:00 EST 2011},
month = {Tue Jan 04 00:00:00 EST 2011}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 4 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Enhanced hypertext categorization using hyperlinks
journal, June 1998

  • Chakrabarti, Soumen; Dom, Byron; Indyk, Piotr
  • ACM SIGMOD Record, Vol. 27, Issue 2
  • DOI: 10.1145/276305.276332

Cautious Inference in Collective Classification
report, July 2007

  • McDowell, Luke K.; Gupta, Kalyan M.; Aha, David W.
  • AAAI Conference on Artificial Intelligence
  • DOI: 10.21236/ADA479429

Learning relational probability trees
conference, January 2003

  • Neville, Jennifer; Jensen, David; Friedland, Lisa
  • Proceedings of the ninth ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '03
  • DOI: 10.1145/956750.956830

Leveraging Relational Autocorrelation with Latent Group Models
conference, January 2005

  • Neville, J.; Jensen, D.
  • Fifth IEEE International Conference on Data Mining (ICDM'05)
  • DOI: 10.1109/ICDM.2005.89

Distribution-based aggregation for relational learning with identifier attributes
journal, January 2006


An Examination of Experimental Methodology for Classifiers of Relational Data
conference, October 2007

  • Gallagher, Brian; Eliassi-Rad, Tina
  • 2007 Seventh IEEE International Conference on Data Mining - Workshops (ICDM Workshops), Seventh IEEE International Conference on Data Mining Workshops (ICDMW 2007)
  • DOI: 10.1109/ICDMW.2007.27

Exploiting Cluster-Structure to Predict the Labeling of a Graph
book, January 2008


A Simple Relational Classifier
report, January 2003


Using ghost edges for classification in sparsely labeled networks
conference, January 2008

  • Gallagher, Brian; Tong, Hanghang; Eliassi-Rad, Tina
  • Proceeding of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 08
  • DOI: 10.1145/1401890.1401925

Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
journal, October 1998


Simple estimators for relational Bayesian classifiers
conference, January 2003


Why collective inference improves relational classification
conference, January 2004

  • Jensen, David; Neville, Jennifer; Gallagher, Brian
  • Proceedings of the 2004 ACM SIGKDD international conference on Knowledge discovery and data mining - KDD '04
  • DOI: 10.1145/1014052.1014125

Collective Classification in Network Data
journal, September 2008


Evaluating Statistical Tests for Within-Network Classifiers of Relational Data
conference, December 2009

  • Neville, Jennifer; Gallagher, Brian; Eliassi-Rad, Tina
  • 2009 Ninth IEEE International Conference on Data Mining (ICDM)
  • DOI: 10.1109/ICDM.2009.50

National Longitudinal Study of Adolescent to Adult Health (Add Health), 1994-2008 [Public Use] (ICPSR 21600). Version 18
dataset, October 2017

  • Harris, Kathleen Mullan; Udry, J. Richard
  • Data Sharing for Demographic Research , multi-year/versions
  • DOI: 10.3886/icpsr21600.v18

Enhanced hypertext categorization using hyperlinks
conference, January 1998

  • Chakrabarti, Soumen; Dom, Byron; Indyk, Piotr
  • Proceedings of the 1998 ACM SIGMOD international conference on Management of data - SIGMOD '98
  • DOI: 10.1145/276304.276332

Leveraging relational autocorrelation with latent group models
conference, January 2005

  • Neville, Jennifer; Jensen, David
  • Proceedings of the 4th international workshop on Multi-relational mining - MRDM '05
  • DOI: 10.1145/1090193.1090201

Leveraging relational autocorrelation with latent group models
text, January 2006


Learning Probabilistic Relational Models
book, January 2001


Enhanced hypertext categorization using hyperlinks
conference, January 1998

  • Chakrabarti, Soumen; Dom, Byron; Indyk, Piotr
  • Proceedings of the 1998 ACM SIGMOD international conference on Management of data - SIGMOD '98
  • DOI: 10.1145/276304.276332

Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms
journal, October 1998


Works referencing / citing this record:

Bayesian Model Selection on Random Networks
preprint, January 2020