Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms

doi:10.1002/asmb.2359

Title: Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms

Full Record
Other Related Research

Abstract

When evaluating the performance of several forensic classification algorithms, it is desirable to construct a design that considers a variety of performance levels for each of the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different classification algorithms as a function of multiple explanatory variables and use this to choose a design about which units to test. Bayesian design of experiments has been successful for generalized linear models, including logistic regression models. Here, we develop methodology for the case where there are several potentially nonoverlapping priors for anticipated performance under consideration. The weighted priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other candidate design choices. Additionally, we show how this can be applied in the multivariate input case and provide some useful summary measures. The shared information plot is used to evaluate design point allocation, and the D-value difference plot allows for the comparison of design performance across multiple potential parameter values in higher dimensions. Here, we illustrate the methods with several examples.

Authors:

Quinlan, Kevin R. ^[1];

^[2]

Pennsylvania State Univ., University Park, PA (United States). Dept. of Statistics
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Publication Date:: 2018-07-17

Research Org.:: Los Alamos National Lab. (LANL), Los Alamos, NM (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1481136

Report Number(s):: LA-UR-17-30911
Journal ID: ISSN 1524-1904

Grant/Contract Number:: AC52-06NA25396

Resource Type:: Accepted Manuscript

Journal Name:: Applied Stochastic Models in Business and Industry

Additional Journal Information:: Journal Volume: 34; Journal Issue: 6; Journal ID: ISSN 1524-1904

Publisher:: Wiley

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Mathematics; Bayesian D‐optimal designs; design of experiments; D‐value difference plot; forensic statistics; shared information plot

Citation Formats


                    Quinlan, Kevin R., and Anderson-Cook, Christine Michaela. Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms.  United States: N. p., 2018. 
        Web.  doi:10.1002/asmb.2359.

Copy to clipboard


                    Quinlan, Kevin R., & Anderson-Cook, Christine Michaela. Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms.  United States.  doi:10.1002/asmb.2359.

Copy to clipboard


                    Quinlan, Kevin R., and Anderson-Cook, Christine Michaela. Tue .  
        "Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms".  United States.  doi:10.1002/asmb.2359.  https://www.osti.gov/servlets/purl/1481136.

Copy to clipboard


                    
@article{osti_1481136,

  title        = {Bayesian Design of Experiments for Logistic Regression to Evaluate Multiple Forensic Algorithms},

  author       = {Quinlan, Kevin R. and Anderson-Cook, Christine Michaela},

  abstractNote = {When evaluating the performance of several forensic classification algorithms, it is desirable to construct a design that considers a variety of performance levels for each of the algorithms. We describe a strategy to use Bayesian design of experiments with multiple prior estimates to capture anticipated performance. Our goal is to characterize results from the different classification algorithms as a function of multiple explanatory variables and use this to choose a design about which units to test. Bayesian design of experiments has been successful for generalized linear models, including logistic regression models. Here, we develop methodology for the case where there are several potentially nonoverlapping priors for anticipated performance under consideration. The weighted priors method performs well for a broad range of true underlying model parameter choices and is more robust when compared to other candidate design choices. Additionally, we show how this can be applied in the multivariate input case and provide some useful summary measures. The shared information plot is used to evaluate design point allocation, and the D-value difference plot allows for the comparison of design performance across multiple potential parameter values in higher dimensions. Here, we illustrate the methods with several examples.},

  doi          = {10.1002/asmb.2359},

  journal      = {Applied Stochastic Models in Business and Industry},

  number       = 6,

  volume       = 34,

  place        = {United States},

  year         = {2018},

  month        = {7}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

DOI: 10.1002/asmb.2359

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 1 work

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

The weighted priors approach for combining expert opinions in logistic regression experiments

Journal Article Quinlan, Kevin R. ; Anderson-Cook, Christine M. ; Myers, Kary L. - Quality Engineering

When modeling the reliability of a system or component, it is not uncommon for more than one expert to provide very different prior estimates of the expected reliability as a function of an explanatory variable such as age or temperature. Our goal in this paper is to incorporate all information from the experts when choosing a design about which units to test. Bayesian design of experiments has been shown to be very successful for generalized linear models, including logistic regression models. We use this approach to develop methodology for the case where there are several potentially non-overlapping priors under consideration.more »« less
DOI: 10.1080/08982112.2017.1319956

Full Text Available
Evaluating penalized logistic regression models to predict Heat-Related Electric grid stress days

Journal Article Bramer, Lisa M. ; Rounds, J. ; Burleyson, C. D. ; ... - Applied Energy

Understanding the conditions associated with stress on the electricity grid is important in the development of contingency plans for maintaining reliability during periods when the grid is stressed. In this paper, heat-related grid stress and the relationship with weather conditions were examined using data from the eastern United States. Penalized logistic regression models were developed and applied to predict stress on the electric grid using weather data. The inclusion of other weather variables, such as precipitation, in addition to temperature improved model performance. Several candidate models and combinations of predictive variables were examined. A penalized logistic regression model which wasmore »« less
Cited by 2
DOI: 10.1016/J.APENERGY.2017.09.087

Full Text Available
Comparing D-optimal designs with common mixture experimental designs for logistic regression

Journal Article Mancenido, Michelle V. ; Pan, Rong ; Montgomery, Douglas C. ; ... - Chemometrics and Intelligent Laboratory Systems

Mixture experiments are used in applications where the proportion of mixing components affects a response variable, such as in studies involving chemical formulations. In many applications, the response is dichotomous or binary (e.g., pass or fail) and a concern for researchers is how to efficiently and informatively design such experiments. A naive approach is to use design recommendations derived from linear normal-theory models with constant variance. In this research, we investigate the potential risks of such designs by comparing them to D-optimal mixture designs for binary responses and evaluating the D-efficiency of these design alternatives for several parameter subspaces. Standardmore »« less
DOI: 10.1016/j.chemolab.2019.02.003
A note on the Bayesian analysis of experiments with correlated multiple responses using a matrix-logarithmic covariance model

Journal Article Hamada, Michael S. ; Jaramillo, Brandon M. ; Chiao, Chih‐Hua - Quality and Reliability Engineering International

In the literature, analysis of multiple responses from experiments with replicates has modeled the covariance matrix directly as linear models of the transformed variances and correlations, i.e., covariance modeling. This article considers models based on the matrix- logarithm of the covariance matrix. This so-called log-covariance modeling is illustrated with data from actual experiments and compared with the traditional covariance modeling.
DOI: 10.1002/qre.2540
Discussion of “Bayesian design of experiments for industrial and scientific applications via gaussian processes”

Journal Article Anderson-Cook, Christine M. ; Burke, Sarah E. - Quality Engineering

First, we would like to commend Dr. Woods on his thought-provoking paper and insightful presentation at the 4th Annual Stu Hunter conference. We think that the material presented highlights some important needs in the area of design of experiments for generalized linear models (GLMs). In addition, we agree with Dr. Woods that design of experiements of GLMs does implicitly require expert judgement about model parameters, and hence using a Bayesian approach to capture this knowledge is a natural strategy to summarize what is known with the opportunity to incorporate associated uncertainty about that information.
DOI: 10.1080/08982112.2016.1246049

Full Text Available

Similar Records