skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms

Abstract

When evaluating the performance of several forensic classification algorithms, it is desirable to construct a design that considers a variety of performance levels for each of the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different classification algorithms as a function of multiple explanatory variables and use this to choose a design about which units to test. Bayesian design of experiments has been successful for generalized linear models, including logistic regression models. Here, we develop methodology for the case where there are several potentially nonoverlapping priors for anticipated performance under consideration. The weighted priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other candidate design choices. Additionally, we show how this can be applied in the multivariate input case and provide some useful summary measures. The shared information plot is used to evaluate design point allocation, and the D-value difference plot allows for the comparison of design performance across multiple potential parameter values in higher dimensions. Here, we illustrate the methods with several examples.

Authors:
 [1]; ORCiD logo [2]
  1. Pennsylvania State Univ., University Park, PA (United States). Dept. of Statistics
  2. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1481136
Report Number(s):
LA-UR-17-30911
Journal ID: ISSN 1524-1904
Grant/Contract Number:  
AC52-06NA25396
Resource Type:
Accepted Manuscript
Journal Name:
Applied Stochastic Models in Business and Industry
Additional Journal Information:
Journal Volume: 34; Journal Issue: 6; Journal ID: ISSN 1524-1904
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Mathematics; Bayesian D‐optimal designs; design of experiments; D‐value difference plot; forensic statistics; shared information plot

Citation Formats

Quinlan, Kevin R., and Anderson-Cook, Christine Michaela. Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms. United States: N. p., 2018. Web. doi:10.1002/asmb.2359.
Quinlan, Kevin R., & Anderson-Cook, Christine Michaela. Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms. United States. doi:10.1002/asmb.2359.
Quinlan, Kevin R., and Anderson-Cook, Christine Michaela. Tue . "Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms". United States. doi:10.1002/asmb.2359. https://www.osti.gov/servlets/purl/1481136.
@article{osti_1481136,
title = {Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms},
author = {Quinlan, Kevin R. and Anderson-Cook, Christine Michaela},
abstractNote = {When evaluating the performance of several forensic classification algorithms, it is desirable to construct a design that considers a variety of performance levels for each of the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different classification algorithms as a function of multiple explanatory variables and use this to choose a design about which units to test. Bayesian design of experiments has been successful for generalized linear models, including logistic regression models. Here, we develop methodology for the case where there are several potentially nonoverlapping priors for anticipated performance under consideration. The weighted priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other candidate design choices. Additionally, we show how this can be applied in the multivariate input case and provide some useful summary measures. The shared information plot is used to evaluate design point allocation, and the D-value difference plot allows for the comparison of design performance across multiple potential parameter values in higher dimensions. Here, we illustrate the methods with several examples.},
doi = {10.1002/asmb.2359},
journal = {Applied Stochastic Models in Business and Industry},
number = 6,
volume = 34,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 2 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Selecting an Informative/Discriminating Multivariate Response for Inverse Prediction
journal, July 2017


Optimal Bayesian design applied to logistic regression experiments
journal, February 1989


The Coordinate-Exchange Algorithm for Constructing Exact Optimal Experimental Designs
journal, February 1995


Fingerprint-Based Recognition
journal, August 2007


Experimental Design for Binary Data
journal, March 1983


Design of experiments and data analysis challenges in calibration for forensics applications
journal, December 2015

  • Anderson-Cook, Christine; Burr, Tom; Hamada, Michael S.
  • Chemometrics and Intelligent Laboratory Systems, Vol. 149
  • DOI: 10.1016/j.chemolab.2015.07.008

Quantifying the weight of evidence from a forensic fingerprint comparison: a new paradigm: Quantifying Weight of Evidence from Fingerprint Comparison
journal, March 2012

  • Neumann, C.; Evett, I. W.; Skerrett, J.
  • Journal of the Royal Statistical Society: Series A (Statistics in Society), Vol. 175, Issue 2
  • DOI: 10.1111/j.1467-985X.2011.01027.x

Monte Carlo and quasi-Monte Carlo methods
journal, January 1998


Optimum Designs in Regression Problems
journal, June 1959


The weighted priors approach for combining expert opinions in logistic regression experiments
journal, April 2017


    Works referencing / citing this record:

    How to Host An Effective Data Competition: Statistical Advice for Competition Design and Analysis
    journal, February 2019

    • Anderson‐Cook, Christine M.; Myers, Kary L.; Lu, Lu
    • Statistical Analysis and Data Mining: The ASA Data Science Journal, Vol. 12, Issue 4
    • DOI: 10.1002/sam.11404