skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Analyst-to-Analyst Variability in Simulation-Based Prediction

Abstract

This report describes findings from the culminating experiment of the LDRD project entitled, "Analyst-to-Analyst Variability in Simulation-Based Prediction". For this experiment, volunteer participants solving a given test problem in engineering and statistics were interviewed at different points in their solution process. These interviews are used to trace differing solutions to differing solution processes, and differing processes to differences in reasoning, assumptions, and judgments. The issue that the experiment was designed to illuminate -- our paucity of understanding of the ways in which humans themselves have an impact on predictions derived from complex computational simulations -- is a challenging and open one. Although solution of the test problem by analyst participants in this experiment has taken much more time than originally anticipated, and is continuing past the end of this LDRD, this project has provided a rare opportunity to explore analyst-to-analyst variability in significant depth, from which we derive evidence-based insights to guide further explorations in this important area.

Authors:
 [1];  [1]
  1. Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Publication Date:
Research Org.:
Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
Sponsoring Org.:
USDOE National Nuclear Security Administration (NNSA)
OSTI Identifier:
1343367
Report Number(s):
SAND2017-1263
651021
DOE Contract Number:
AC04-94AL85000
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Glickman, Matthew R., and Romero, Vicente J.. Analyst-to-Analyst Variability in Simulation-Based Prediction. United States: N. p., 2017. Web. doi:10.2172/1343367.
Glickman, Matthew R., & Romero, Vicente J.. Analyst-to-Analyst Variability in Simulation-Based Prediction. United States. doi:10.2172/1343367.
Glickman, Matthew R., and Romero, Vicente J.. Wed . "Analyst-to-Analyst Variability in Simulation-Based Prediction". United States. doi:10.2172/1343367. https://www.osti.gov/servlets/purl/1343367.
@article{osti_1343367,
title = {Analyst-to-Analyst Variability in Simulation-Based Prediction},
author = {Glickman, Matthew R. and Romero, Vicente J.},
abstractNote = {This report describes findings from the culminating experiment of the LDRD project entitled, "Analyst-to-Analyst Variability in Simulation-Based Prediction". For this experiment, volunteer participants solving a given test problem in engineering and statistics were interviewed at different points in their solution process. These interviews are used to trace differing solutions to differing solution processes, and differing processes to differences in reasoning, assumptions, and judgments. The issue that the experiment was designed to illuminate -- our paucity of understanding of the ways in which humans themselves have an impact on predictions derived from complex computational simulations -- is a challenging and open one. Although solution of the test problem by analyst participants in this experiment has taken much more time than originally anticipated, and is continuing past the end of this LDRD, this project has provided a rare opportunity to explore analyst-to-analyst variability in significant depth, from which we derive evidence-based insights to guide further explorations in this important area.},
doi = {10.2172/1343367},
journal = {},
number = ,
volume = ,
place = {United States},
year = {Wed Feb 01 00:00:00 EST 2017},
month = {Wed Feb 01 00:00:00 EST 2017}
}

Technical Report:

Save / Share:
  • This is Volume 1 of a two-volume set of reports describing work conducted at North Carolina State University sponsored by Grant Number DE-FG05-95ER30250 by the U.S. Department of Energy. The title of the project is “Quantitative Analysis of Variability and Uncertainty in Acid Rain Assessments.” The work conducted under sponsorship of this grant pertains primarily to two main topics: (1) development of new methods for quantitative analysis of variability and uncertainty applicable to any type of model; and (2) analysis of variability and uncertainty in the performance, emissions, and cost of electric power plant combustion-based NOx control technologies. These twomore » main topics are reported separately in Volumes 1 and 2.« less
  • The primary outcome of the project was the development of the Regional Arctic System Model (RASM) and evaluation of its individual model components, coupling among them and fully coupled model results. Overall, we have demonstrated that RASM produces realistic mean and seasonal surface climate as well as its interannual and decadal variability and trends.
  • This project aims to develop, apply and evaluate a regional Arctic System model (RASM) for enhanced decadal predictions. Its overarching goal is to advance understanding of the past and present states of arctic climate and to facilitate improvements in seasonal to decadal predictions. In particular, it will focus on variability and long-term change of energy and freshwater flows through the arctic climate system. The project will also address modes of natural climate variability as well as extreme and rapid climate change in a region of the Earth that is: (i) a key indicator of the state of global climate throughmore » polar amplification and (ii) which is undergoing environmental transitions not seen in instrumental records. RASM will readily allow the addition of other earth system components, such as ecosystem or biochemistry models, thus allowing it to facilitate studies of climate impacts (e.g., droughts and fires) and of ecosystem adaptations to these impacts. As such, RASM is expected to become a foundation for more complete Arctic System models and part of a model hierarchy important for improving climate modeling and predictions.« less
  • RASM is a multi-disciplinary project, which brings together researchers from six state universities, one military postgraduate school, and one DoE laboratory to address the core modeling objectives of the arctic research community articulated in the Arctic System Modeling report by Roberts et al. (2010b). This report advocates the construction of a regional downscaling tool to generate probabilistic decadal projections of Greenland ice sheet retreat, evolution of arctic sea ice cover, changes in land surface vegetation, and regional processes leading to arctic amplification. Unified coupled models such as RASM are ideal for this purpose because they simulate fine-scale physics, essential formore » the realistic representation of intra-annual variability, in addition to processes fundamental to long term climatic shifts (Hurrell et al. 2009). By using RASM with boundary conditions from a global model, we can generate many-member ensembles essential for understanding uncertainty in regional climate projections (Hawkins and Sutton 2009). This probabilistic approach is computationally prohibitive for high-resolution global models in the foreseeable future, and also for regional models interactively nested within global simulations. Yet it is fundamental for quantifying uncertainty in decadal forecasts to make them useful for decision makers (Doherty et al. 2009). For this reason, we have targeted development of ensemble generation techniques as a core project task (Task 4.5). Environmental impact assessment specialists need high-fidelity regional ensemble projections to improve the accuracy of their work (Challinor et al. 2009; Moss et al. 2010). This is especially true of the Arctic, where economic, social and national interests are rapidly reshaping the high north in step with regional climate change. During the next decade, considerable oil and gas discoveries are expected across many parts of the marine and terrestrial Arctic (Gautier et al. 2009), the economics of the Northern Sea Route will steadily improve (Arctic Council 2009), and sovereign claims over the Arctic Ocean will increasingly be subject to international negotiations (Proelss 2009). Issues such as these have led to an expanding demand for Arctic climate projections to aid national and commercial decisions. However, detailed information from existing models is lacking. RASM will enhance the existing Arctic system modeling capabilities and align them with the scientific and societal needs outlined above. The science involved in the development of RASM will be integrated with teaching and training at a few different levels. RASM PIs will supervise at least six postdoctoral and doctoral students with additional involvement of undergraduate research assistants. RASM postdoctoral fellows and graduate trainees will benefit from being a part of a large, collaborative, and multi-disciplinary research projects employing state-of-the-art modeling and computational tools. They will also have the opportunity to contribute to the UTEP-led education and outreach effort developed specifically for RASM. The UTEP team will use RASM as a platform for education and outreach. First, we will develop products for public dissemination, such as curriculum units, lesson plans and other materials (simulations, movies, and images) for use by students and teachers in high school and university classrooms. Second, we will facilitate bringing RASM PIs, postdocs, and graduate students into the classroom, through electronic mentorship and by contributing content via online lectures and presentations. Third, by the use of problem-based learning (PBL) approaches, we will provide real-world scenarios and problems enabling students to do research and develop position papers or presentations on topics related to RASM. (Problem-based learning is a student-centered, inquiry-based approach in which students work in teams to solve challenging, open-ended problems.) Fourth, we will develop teacher-training materials that will be developed into workshops (face-to-face or online) to help teachers understand how to use the materials we supply. Overall, we will provide a broad, and long-term RASM program for education and outreach. RASM education and outreach activities will target underrepresented populations - in particular, Hispanic students and teachers. For example, the UTEP student body is more than 70 percent Hispanic and 55 percent female, and the majority of students are the first in their families to attend college. Students from underrepresented groups will be given the chance to expand their understanding of the different scientific knowledge and processes associated with RASM. We will develop an eye-catching RASM website to include information about the project, model results and educational materials, and lists of RASM publications and presentations.« less