skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Analysis of Repeatability and Reproducibility Studies With Ordinal Measurements

Abstract

A Bayesian inferential approach with a noninformative prior is introduced to analyze ordinal repeatability and reproducibility (R&R) data using the De Mast–Van Wieringen model. This approach is extended with a weakly informative prior and random effects to allow for the consideration of a population of raters and prediction of a new rater. This random-effects approach is also shown to result in partial pooling of estimates across raters. In addition, match-probability-based measures to decompose ordinal R&R study data into contributions due to repeatability and due to reproducibility are defined. All extensions involving Bayesian inference (for fixed or random effects) and measures are illustrated on real and simulated ordinal R&R study data and are applicable in business and industry settings. This methodology can be implemented using the supplemental R package ordinalRR available from CRAN. Additional supplementary material for this article is available online.

Authors:
 [1];  [1];  [2];  [3]
  1. West Virginia Univ., Morgantown, WV (United States). Dept. of Statistics
  2. Hubei Univ. of Technology, Wuhan, China. Dept. of Statistics
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1460636
Report Number(s):
LA-UR-17-20022
Journal ID: ISSN 0040-1706
Grant/Contract Number:  
AC52-06NA25396
Resource Type:
Accepted Manuscript
Journal Name:
Technometrics
Additional Journal Information:
Journal Volume: 60; Journal Issue: 4; Journal ID: ISSN 0040-1706
Publisher:
Taylor & Francis
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Dirichlet distribution; Fixed effects; Markov chain Monte Carlo (MCMC); Random effects

Citation Formats

Culp, Stacey L., Ryan, Kenneth J., Chen, Juan, and Hamada, Michael S. Analysis of Repeatability and Reproducibility Studies With Ordinal Measurements. United States: N. p., 2018. Web. doi:10.1080/00401706.2018.1429317.
Culp, Stacey L., Ryan, Kenneth J., Chen, Juan, & Hamada, Michael S. Analysis of Repeatability and Reproducibility Studies With Ordinal Measurements. United States. doi:10.1080/00401706.2018.1429317.
Culp, Stacey L., Ryan, Kenneth J., Chen, Juan, and Hamada, Michael S. Mon . "Analysis of Repeatability and Reproducibility Studies With Ordinal Measurements". United States. doi:10.1080/00401706.2018.1429317. https://www.osti.gov/servlets/purl/1460636.
@article{osti_1460636,
title = {Analysis of Repeatability and Reproducibility Studies With Ordinal Measurements},
author = {Culp, Stacey L. and Ryan, Kenneth J. and Chen, Juan and Hamada, Michael S.},
abstractNote = {A Bayesian inferential approach with a noninformative prior is introduced to analyze ordinal repeatability and reproducibility (R&R) data using the De Mast–Van Wieringen model. This approach is extended with a weakly informative prior and random effects to allow for the consideration of a population of raters and prediction of a new rater. This random-effects approach is also shown to result in partial pooling of estimates across raters. In addition, match-probability-based measures to decompose ordinal R&R study data into contributions due to repeatability and due to reproducibility are defined. All extensions involving Bayesian inference (for fixed or random effects) and measures are illustrated on real and simulated ordinal R&R study data and are applicable in business and industry settings. This methodology can be implemented using the supplemental R package ordinalRR available from CRAN. Additional supplementary material for this article is available online.},
doi = {10.1080/00401706.2018.1429317},
journal = {Technometrics},
number = 4,
volume = 60,
place = {United States},
year = {2018},
month = {6}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

A Novel Approach to Evaluate Repeatability and Reproducibility for Ordinal Data
journal, January 2014


A Generalized Partial Credit Model: Application of an EM Algorithm
journal, June 1992


Measurement System Analysis for Categorical Measurements: Agreement and Kappa-Type Indices
journal, July 2007


The Selection of Prior Distributions by Formal Rules
journal, September 1996


Bayesian Measures of Explained Variance and Pooling in Multilevel (Hierarchical) Models
journal, May 2006


Modeling and Evaluating Repeatability and Reproducibility of Ordinal Classifications
journal, February 2010


The Statistical Evaluation of Categorical Measurements: “Simple Scales, but Treacherous Complexity Underneath”
journal, December 2013


Two-Way Random-Effects Analyses and Gauge R&R Studies
journal, August 1999