Conservative Estimation of Tail Probabilities from Limited Sample Data

Jekel, Charles F.; Romero, Vincente J.

doi:10.2172/1605343

Title: Conservative Estimation of Tail Probabilities from Limited Sample Data

Full Record
Other Related Research

Abstract

Several sparse-sample uncertainty quantification (UQ) methods are compared for conservative but not overly conservative estimation of small tail probabilities involving responses that lay beyond specified thresholds in the tails of probability distributions. Sixteen very differently shaped distributions (or probability density functions, PDFs) and tail probability magnitudes ranging from 10^-5 to 10^-1 are considered in order for the study to be relevant to a wide range of risk analysis and quantification of margins and uncertainty (QMU) problems. The emphasis of the study is on limited data regimes ranging from N = 2 to 20 samples, reflective of most experimental and some expensive computational situations. Relatively simple sparse-sample UQ methods tested for this regime involve statistical tolerance interval "Equivalent Normal and related "Ensemble of Normals" and "Superdistribution (SD) approaches. (The independently derived SD is effectively equivalent to the Bayesian posterior predictive distribution given the assumptions of the derivation.) The performance of the methods was generally improved for N ≥ 5 samples with a generalized Jackknife resampling technique, which determines a tail probability estimate by averaging estimates from smaller sub-samples. Several quantitative metrics for method conservatism and accuracy of tail probability estimation are used to assess and rank the methods' performance over manymore »« less

Authors:

Jekel, Charles F. ^[1]; Romero, Vincente J. ^[1]

Sandia National Laboratories (SNL), Albuquerque, NM, and Livermore, CA (United States)

Publication Date:: 2020-03-01

Research Org.:: Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA)

OSTI Identifier:: 1605343

Report Number(s):: SAND-2020-2828
684818

DOE Contract Number:: AC04-94AL85000; NA0003525

Resource Type:: Technical Report

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING

Citation Formats


                    Jekel, Charles F., and Romero, Vincente J.. Conservative Estimation of Tail Probabilities from Limited Sample Data.  United States: N. p., 2020. 
        Web.  doi:10.2172/1605343.

Copy to clipboard


                    Jekel, Charles F., & Romero, Vincente J.. Conservative Estimation of Tail Probabilities from Limited Sample Data.  United States.  https://doi.org/10.2172/1605343

Copy to clipboard


                    Jekel, Charles F., and Romero, Vincente J.. 2020.  
        "Conservative Estimation of Tail Probabilities from Limited Sample Data".  United States.  https://doi.org/10.2172/1605343.  https://www.osti.gov/servlets/purl/1605343.

Copy to clipboard


                    
@article{osti_1605343,

  title        = {Conservative Estimation of Tail Probabilities from Limited Sample Data},

  author       = {Jekel, Charles F. and Romero, Vincente J.},

  abstractNote = {Several sparse-sample uncertainty quantification (UQ) methods are compared for conservative but not overly conservative estimation of small tail probabilities involving responses that lay beyond specified thresholds in the tails of probability distributions. Sixteen very differently shaped distributions (or probability density functions, PDFs) and tail probability magnitudes ranging from 10-5 to 10-1 are considered in order for the study to be relevant to a wide range of risk analysis and quantification of margins and uncertainty (QMU) problems. The emphasis of the study is on limited data regimes ranging from N = 2 to 20 samples, reflective of most experimental and some expensive computational situations. Relatively simple sparse-sample UQ methods tested for this regime involve statistical tolerance interval "Equivalent Normal and related "Ensemble of Normals" and "Superdistribution (SD) approaches. (The independently derived SD is effectively equivalent to the Bayesian posterior predictive distribution given the assumptions of the derivation.) The performance of the methods was generally improved for N ≥ 5 samples with a generalized Jackknife resampling technique, which determines a tail probability estimate by averaging estimates from smaller sub-samples. Several quantitative metrics for method conservatism and accuracy of tail probability estimation are used to assess and rank the methods' performance over many random trials for each test PDF and probability magnitude. A variant of Bootstrap resampling was also tried, but did not significantly improve tail probability estimates in most cases. Detailed results are presented from over 100-million tests over the above factors that provide useful granular information on which methods or combination of methods perform best in various areas of the factor space.},

  doi          = {10.2172/1605343},

  url          = {https://www.osti.gov/biblio/1605343},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2020},

  month        = {3}

}

Copy to clipboard

Technical Report:

View Technical Report (49.75 MB)

https://doi.org/10.2172/1605343

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Evaluation of a Class of Simple and Effective Uncertainty Methods for Sparse Samples of Random Variables and Functions

Technical Report Romero, Vicente; Bonney, Matthew; Schroeder, Benjamin; ...

When very few samples of a random quantity are available from a source distribution of unknown shape, it is usually not possible to accurately infer the exact distribution from which the data samples come. Under-estimation of important quantities such as response variance and failure probabilities can result. For many engineering purposes, including design and risk analysis, we attempt to avoid under-estimation with a strategy to conservatively estimate (bound) these types of quantities -- without being overly conservative -- when only a few samples of a random quantity are available from model predictions or replicate experiments. This report examines a classmore »« less
https://doi.org/10.2172/1409725

Full Text Available
Discrete-Direct Model Calibration and Uncertainty Propagation Method Confirmed on Multi-Parameter Plasticity Model Calibrated to Sparse Random Field Data

Journal Article Romero, Vicente; Winokur, Justin; Orient, George; ... - ASCE-ASME Journal of Risk and Uncertainty in Engineering Systems. Part B. Mechanical Engineering

A discrete direct (DD) model calibration and uncertainty propagation approach is explained and demonstrated on a 4-parameter Johnson-Cook (J-C) strain-rate dependent material strength model for an aluminum alloy. The methodology's performance is characterized in many trials involving four random realizations of strain-rate dependent material-test data curves per trial, drawn from a large synthetic population. The J-C model is calibrated to particular combinations of the data curves to obtain calibration parameter sets which are then propagated to “Can Crush” structural model predictions to produce samples of predicted response variability. These are processed with appropriate sparse-sample uncertainty quantification (UQ) methods to estimatemore »« less
https://doi.org/10.1115/1.4050371

Full Text Available
A comparison of methods for representing sparsely sampled random quantities.

Technical Report Romero, Vicente; Swiler, Laura; Urbina, Angel; ...

This report discusses the treatment of uncertainties stemming from relatively few samples of random quantities. The importance of this topic extends beyond experimental data uncertainty to situations involving uncertainty in model calibration, validation, and prediction. With very sparse data samples it is not practical to have a goal of accurately estimating the underlying probability density function (PDF). Rather, a pragmatic goal is that the uncertainty representation should be conservative so as to bound a specified percentile range of the actual PDF, say the range between 0.025 and .975 percentiles, with reasonable reliability. A second, opposing objective is that the representationmore »« less
https://doi.org/10.2172/1096268

Full Text Available
Improved statistical models for limited datasets in uncertainty quantification using stochastic collocation

Journal Article Alwan, Aravind; Aluru, N. - Journal of Computational Physics

This paper presents a data-driven framework for performing uncertainty quantification (UQ) by choosing a stochastic model that accurately describes the sources of uncertainty in a system. This model is propagated through an appropriate response surface function that approximates the behavior of this system using stochastic collocation. Given a sample of data describing the uncertainty in the inputs, our goal is to estimate a probability density function (PDF) using the kernel moment matching (KMM) method so that this PDF can be used to accurately reproduce statistics like mean and variance of the response surface function. Instead of constraining the PDF tomore »« less
https://doi.org/10.1016/J.JCP.2013.08.024
Quantification of the Variability of Diaphragm Motion and Implications for Treatment Margin Construction

Journal Article Rit, Simon; Herk, Marcel; Zijp, Lambert; ... - International Journal of Radiation Oncology, Biology and Physics

Purpose: To quantify the variability of diaphragm motion during free-breathing radiotherapy of lung patients and its effect on treatment margins to account for geometric uncertainties. Methods and Materials: Thirty-three lung cancer patients were analyzed. Each patient had 5-19 cone-beam scans acquired during different treatment fractions. The craniocaudal position of the diaphragm dome on the same side as the tumor was tracked over 2 min in the projection images, because it is both easily visible and a suitable surrogate to study the variability of the tumor motion and its impact on treatment margins. Intra-acquisition, inter-acquisition, and inter-patient variability of the respiratorymore »« less
https://doi.org/10.1016/J.IJROBP.2011.06.1986

Similar Records