skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Selecting a near-optimal design for multiple criteria with improved robustness to different user priorities

Abstract

In a decision-making process, relying on only one objective can often lead to oversimplified decisions that ignore important considerations. Incorporating multiple, and likely competing, objectives is critical for balancing trade-offs on different aspects of performance. When multiple objectives are considered, it is often hard to make a precise decision on how to weight the different objectives when combining their performance for ranking and selecting designs. We show that there are situations when selecting a design with near-optimality for a broad range of weight combinations of the criteria is a better test selection strategy compared with choosing a design that is strictly optimal under very restricted conditions. Here, we propose a new design selection strategy that identifies several top-ranked solutions across broad weight combinations using layered Pareto fronts and then selects the final design that offers the best robustness to different user priorities. This method involves identifying multiple leading solutions based on the primary objectives and comparing the alternatives using secondary objectives to make the final decision. We focus on the selection of screening designs because they are widely used both in industrial research, development, and operational testing. The method is illustrated with an example of selecting a single design frommore » a catalog of designs of a fixed size. However, the method can be adapted to more general designed experiment selection problems that involve searching through a large design space.« less

Authors:
 [1];  [2];  [3];  [4]
  1. Scientific Test and Analysis Techniques Center of Excellence, Dayton, OH (United States)
  2. Univ. of South Florida, Tampa, FL (United States). Dept. of Mathematics and Statistics
  3. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  4. Arizona State Univ., Tempe, AZ (United States). School of Computing, Informatics and Decisions Systems, Engineering Dept. of Industrial Engineering
Publication Date:
Research Org.:
Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
US Department of Homeland Security (DHS); USDOE
OSTI Identifier:
1481982
Report Number(s):
LA-UR-18-20532
Journal ID: ISSN 0748-8017
Grant/Contract Number:  
AC52-06NA25396; PSA‐ASU‐ODEX‐16‐04
Resource Type:
Accepted Manuscript
Journal Name:
Quality and Reliability Engineering International
Additional Journal Information:
Journal Volume: 35; Journal Issue: 3; Journal ID: ISSN 0748-8017
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Mathematics

Citation Formats

Burke, Sarah E., Lu, Lu, Anderson-Cook, Christine M., and Montgomery, Douglas C. Selecting a near-optimal design for multiple criteria with improved robustness to different user priorities. United States: N. p., 2018. Web. doi:10.1002/qre.2413.
Burke, Sarah E., Lu, Lu, Anderson-Cook, Christine M., & Montgomery, Douglas C. Selecting a near-optimal design for multiple criteria with improved robustness to different user priorities. United States. doi:10.1002/qre.2413.
Burke, Sarah E., Lu, Lu, Anderson-Cook, Christine M., and Montgomery, Douglas C. Mon . "Selecting a near-optimal design for multiple criteria with improved robustness to different user priorities". United States. doi:10.1002/qre.2413. https://www.osti.gov/servlets/purl/1481982.
@article{osti_1481982,
title = {Selecting a near-optimal design for multiple criteria with improved robustness to different user priorities},
author = {Burke, Sarah E. and Lu, Lu and Anderson-Cook, Christine M. and Montgomery, Douglas C.},
abstractNote = {In a decision-making process, relying on only one objective can often lead to oversimplified decisions that ignore important considerations. Incorporating multiple, and likely competing, objectives is critical for balancing trade-offs on different aspects of performance. When multiple objectives are considered, it is often hard to make a precise decision on how to weight the different objectives when combining their performance for ranking and selecting designs. We show that there are situations when selecting a design with near-optimality for a broad range of weight combinations of the criteria is a better test selection strategy compared with choosing a design that is strictly optimal under very restricted conditions. Here, we propose a new design selection strategy that identifies several top-ranked solutions across broad weight combinations using layered Pareto fronts and then selects the final design that offers the best robustness to different user priorities. This method involves identifying multiple leading solutions based on the primary objectives and comparing the alternatives using secondary objectives to make the final decision. We focus on the selection of screening designs because they are widely used both in industrial research, development, and operational testing. The method is illustrated with an example of selecting a single design from a catalog of designs of a fixed size. However, the method can be adapted to more general designed experiment selection problems that involve searching through a large design space.},
doi = {10.1002/qre.2413},
journal = {Quality and Reliability Engineering International},
number = 3,
volume = 35,
place = {United States},
year = {2018},
month = {10}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Save / Share:

Works referenced in this record:

Simultaneous Optimization of Several Response Variables
journal, October 1980


Balancing Multiple Criteria Incorporating Cost using Pareto Front Optimization for Split-Plot Designed Experiments: Pareto Front Optimization for Split-Plot Designs
journal, December 2012

  • Lu, Lu; Anderson-Cook, Christine M.
  • Quality and Reliability Engineering International, Vol. 30, Issue 1
  • DOI: 10.1002/qre.1476

A case study to demonstrate a Pareto Frontier for selecting a best response surface design while simultaneously optimizing multiple criteria: L. LU, C. M. ANDERSON-COOK AND T. J. ROBINSON
journal, March 2012

  • Lu, Lu; Anderson-Cook, Christine M.; Robinson, Timothy J.
  • Applied Stochastic Models in Business and Industry, Vol. 28, Issue 3
  • DOI: 10.1002/asmb.940

Complete enumeration of pure-level and mixed-level orthogonal arrays
journal, August 2009

  • Schoen, Eric D.; Eendebak, Pieter T.; Nguyen, Man V. M.
  • Journal of Combinatorial Designs, Vol. 18, Issue 2
  • DOI: 10.1002/jcd.20236

A comparison of two-level designs to estimate all main effects and two-factor interactions
journal, May 2016


Fraction of Design Space to Assess Prediction Capability of Response Surface Designs
journal, October 2003


A Case Study to Select an Optimal Split-Plot Design for a Mixture-Process Experiment Based on Multiple Objectives
journal, August 2014


Adapting the Hypervolume Quality Indicator to Quantify Trade-offs and Search Efficiency for Multiple Criteria Decision Making Using Pareto Fronts
journal, September 2012

  • Lu, Lu; Anderson-Cook, Christine M.
  • Quality and Reliability Engineering International, Vol. 29, Issue 8
  • DOI: 10.1002/qre.1464

A fast and elitist multiobjective genetic algorithm: NSGA-II
journal, April 2002

  • Deb, K.; Pratap, A.; Agarwal, S.
  • IEEE Transactions on Evolutionary Computation, Vol. 6, Issue 2
  • DOI: 10.1109/4235.996017

Optimal designed experiments using a Pareto front search for focused preference of multiple objectives
journal, March 2014

  • Lu, Lu; Anderson-Cook, Christine M.; Lin, Dennis K. J.
  • Computational Statistics & Data Analysis, Vol. 71
  • DOI: 10.1016/j.csda.2013.04.008