skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Bias in meta-analyses using Hedges’ d

Abstract

The type of metric and weighting method used in meta-analysis can create bias and alter coverage of confidence intervals when the estimated effect size and its weight are correlated. Here, we investigate bias associated with the common metric, Hedges’ d, under conditions common in ecological meta-analyses. We simulated data from experiments, computed effect sizes and their variances, and performed meta-analyses applying three weighting schemes (inverse variance, sample size, and unweighted) for varying levels of effect size, within-study replication, number of studies in the meta-analysis, and among-study variance. Unweighted analyses, and those using weights based on sample size, were close to unbiased and yielded coverages close to the nominal level of 0.95. In contrast, the inverse-variance weighting scheme led to bias and low coverage, especially for meta-analyses based on studies with low replication. This bias arose because of a correlation between the estimated effect and its weight when using the inverse-variance method. In many cases, the sample size weighting scheme was most efficient, and, when not, the differences in efficiency among the three methods were relatively minor. Thus, if using Hedges’ d, we recommend using weights based upon sample size that do not involve individual study estimates of the effect size.

Authors:
ORCiD logo [1]; ORCiD logo [1];  [2];  [3]; ORCiD logo [1]
  1. Odum School of Ecology, University of Georgia, 140 E. Green Street Athens Georgia 30602 USA
  2. Quantitative Fisheries Center, Department of Fisheries and Wildlife, Michigan State University, 480 Wilson Road #13 East Lansing Michigan 48824 USA
  3. Department of Fisheries and Wildlife, Michigan State University, 375 Wilson Road, 101 UPLA Building East Lansing Michigan 48824 USA
Publication Date:
Research Org.:
Univ. of Georgia, Athens, GA (United States); Michigan State Univ., East Lansing, MI (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Biological and Environmental Research (BER); National Science Foundation (NSF)
OSTI Identifier:
1473748
Alternate Identifier(s):
OSTI ID: 1476656; OSTI ID: 1483359
Grant/Contract Number:  
SC0010632; DEB-1655426; DEB-1655394
Resource Type:
Published Article
Journal Name:
Ecosphere
Additional Journal Information:
Journal Name: Ecosphere Journal Volume: 9 Journal Issue: 9; Journal ID: ISSN 2150-8925
Publisher:
Ecological Society of America
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; bias; coverage; effect size; Hedges' d; meta-analysis; sample size; weights

Citation Formats

Hamman, Elizabeth A., Pappalardo, Paula, Bence, James R., Peacor, Scott D., and Osenberg, Craig W. Bias in meta-analyses using Hedges’ d. United States: N. p., 2018. Web. doi:10.1002/ecs2.2419.
Hamman, Elizabeth A., Pappalardo, Paula, Bence, James R., Peacor, Scott D., & Osenberg, Craig W. Bias in meta-analyses using Hedges’ d. United States. doi:10.1002/ecs2.2419.
Hamman, Elizabeth A., Pappalardo, Paula, Bence, James R., Peacor, Scott D., and Osenberg, Craig W. Tue . "Bias in meta-analyses using Hedges’ d". United States. doi:10.1002/ecs2.2419.
@article{osti_1473748,
title = {Bias in meta-analyses using Hedges’ d},
author = {Hamman, Elizabeth A. and Pappalardo, Paula and Bence, James R. and Peacor, Scott D. and Osenberg, Craig W.},
abstractNote = {The type of metric and weighting method used in meta-analysis can create bias and alter coverage of confidence intervals when the estimated effect size and its weight are correlated. Here, we investigate bias associated with the common metric, Hedges’ d, under conditions common in ecological meta-analyses. We simulated data from experiments, computed effect sizes and their variances, and performed meta-analyses applying three weighting schemes (inverse variance, sample size, and unweighted) for varying levels of effect size, within-study replication, number of studies in the meta-analysis, and among-study variance. Unweighted analyses, and those using weights based on sample size, were close to unbiased and yielded coverages close to the nominal level of 0.95. In contrast, the inverse-variance weighting scheme led to bias and low coverage, especially for meta-analyses based on studies with low replication. This bias arose because of a correlation between the estimated effect and its weight when using the inverse-variance method. In many cases, the sample size weighting scheme was most efficient, and, when not, the differences in efficiency among the three methods were relatively minor. Thus, if using Hedges’ d, we recommend using weights based upon sample size that do not involve individual study estimates of the effect size.},
doi = {10.1002/ecs2.2419},
journal = {Ecosphere},
number = 9,
volume = 9,
place = {United States},
year = {2018},
month = {9}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
DOI: 10.1002/ecs2.2419

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Conducting Meta-Analyses in R with the metafor Package
journal, January 2010


Statistical Issues in Ecological Meta-Analyses
journal, June 1999


Effect Size in Ecological Experiments: The Application of Biological Models in Meta‐Analysis
journal, December 1997

  • Osenberg, Craig W.; Sarnelle, Orlando; Cooper, Scott D.
  • The American Naturalist, Vol. 150, Issue 6
  • DOI: 10.1086/286095

Resolving Ecological Questions Through Meta-Analysis: Goals, Metrics, and Models
journal, June 1999


Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications
journal, December 2016

  • Senior, Alistair M.; Grueber, Catherine E.; Kamiya, Tsukushi
  • Ecology, Vol. 97, Issue 12
  • DOI: 10.1002/ecy.1591

Estimation of effect size from a series of independent experiments.
journal, January 1982


Gauging the impact of meta-analysis on ecology
journal, June 2012

  • Cadotte, Marc W.; Mehrkens, Lea R.; Menge, Duncan N. L.
  • Evolutionary Ecology, Vol. 26, Issue 5
  • DOI: 10.1007/s10682-012-9585-z

Weighting by Inverse Variance or by Sample Size in Meta-Analysis: A Simulation Study
journal, April 1998

  • Sanchez-Meca, Julio; Marín-Martínez, Fulgencio
  • Educational and Psychological Measurement, Vol. 58, Issue 2
  • DOI: 10.1177/0013164498058002005

Assessing the effect of elevated carbon dioxide on soil carbon: a comparison of four meta-analyses
journal, August 2009


Modelling data from different sites, times or studies: weighted vs. unweighted regression: Combining information
journal, July 2011


Meta-analysis of magnitudes, differences and variation in evolutionary parameters
journal, October 2016

  • Morrissey, M. B.
  • Journal of Evolutionary Biology, Vol. 29, Issue 10
  • DOI: 10.1111/jeb.12950

Distribution Theory for Glass's Estimator of Effect size and Related Estimators
journal, June 1981


Bias and correction for the log response ratio in ecological meta-analysis
journal, August 2015


Estimating the mean effect size in meta-analysis: Bias, precision, and mean squared error of different weighting methods
journal, November 2003

  • Van Den Noortgate, Wim; Onghena, Patrick
  • Behavior Research Methods, Instruments, & Computers, Vol. 35, Issue 4
  • DOI: 10.3758/BF03195529

Uses and misuses of meta-analysis in plant ecology
journal, June 2014


Correction for bias in meta-analysis of little-replicated studies
journal, November 2017

  • Doncaster, C. Patrick; Spake, Rebecca
  • Methods in Ecology and Evolution, Vol. 9, Issue 3
  • DOI: 10.1111/2041-210X.12927

Methodological issues and advances in biological meta-analysis
journal, February 2012


Confidence intervals for the overall effect size in random-effects meta-analysis.
journal, January 2008


The Problem of Pattern and Scale in Ecology: The Robert H. MacArthur Award Lecture
journal, December 1992


A random effects model for effect sizes.
journal, January 1983


Achieving synthesis with meta-analysis by combining and comparing all available studies
journal, September 2010


Meta-analysis and the science of research synthesis
journal, March 2018

  • Gurevitch, Jessica; Koricheva, Julia; Nakagawa, Shinichi
  • Nature, Vol. 555, Issue 7695
  • DOI: 10.1038/nature25753

    Works referencing / citing this record:

    Estimation in meta‐analyses of mean difference and standardized mean difference
    journal, November 2019

    • Bakbergenuly, Ilyas; Hoaglin, David C.; Kulinskaya, Elena
    • Statistics in Medicine, Vol. 39, Issue 2
    • DOI: 10.1002/sim.8422

    Fitness consequences of different migratory strategies in partially migratory populations: A multi‐taxa meta‐analysis
    journal, December 2019

    • Buchan, Claire; Gilroy, James J.; Catry, Inês
    • Journal of Animal Ecology, Vol. 89, Issue 3
    • DOI: 10.1111/1365-2656.13155

    A mother’s legacy: the strength of maternal effects in animal populations
    journal, July 2019

    • Moore, Michael P.; Whiteman, Howard H.; Martin, Ryan A.
    • Ecology Letters, Vol. 22, Issue 10
    • DOI: 10.1111/ele.13351

    The magnitude of small-study effects in the Cochrane Database of Systematic Reviews : an empirical study of nearly 30 000 meta-analyses
    journal, July 2019


    Can linear transportation infrastructure verges constitute a habitat and/or a corridor for vertebrates in temperate ecosystems? A systematic review
    journal, June 2020

    • Ouédraogo, Dakis-Yaoba; Villemey, Anne; Vanpeene, Sylvie
    • Environmental Evidence, Vol. 9, Issue 1
    • DOI: 10.1186/s13750-020-00196-7