skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models

Abstract

Experts in data and physical sciences have to regularly grapple with the problem of competing models. Be it analytical or physics-based models, a cross-cutting challenge for experts is to reliably diagnose which model outcomes appropriately predict or simulate real-world phenomena. Expert judgment involves reconciling information across many, and often, conflicting criteria that describe the quality of model outcomes. In this paper, through a design study with climate scientists, we develop a deeper understanding of the problem and solution space of model diagnostics, resulting in the following contributions: i) a problem and task characterization using which we map experts’ model diagnostics goals to multi-way visual comparison tasks, ii) a design space of comparative visual cues for letting experts quickly understand the degree of disagreement among competing models and gauge the degree of stability of model outputs with respect to alternative criteria, and iii) design and evaluation of MyriadCues, an interactive visualization interface for exploring alternative hypotheses and insights about good and bad models by leveraging comparative visual cues. Here, we present case studies and subjective feedback by experts, which validate how MyriadCues enables more transparent model diagnostic mechanisms, as compared to the state of the art.

Authors:
 [1];  [2];  [3]; ORCiD logo [3]
  1. New Jersey Institute of Technology, Newark, NJ (United States)
  2. Arizona State Univ., Tempe, AZ (United States)
  3. Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Publication Date:
Research Org.:
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1577980
Report Number(s):
PNNL-SA-129640
Journal ID: ISSN 1077-2626
Grant/Contract Number:  
AC05-76RL01830
Resource Type:
Journal Article: Accepted Manuscript
Journal Name:
IEEE Transactions on Visualization and Computer Graphics
Additional Journal Information:
Journal Volume: 26; Journal Issue: 1; Journal ID: ISSN 1077-2626
Publisher:
IEEE
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; climate model analytics; Climate Model Evaluation; visual analytics; model benchmarking; climate model diagnostics; visual comparison; visual cues; model evaluation; transparency; simulation

Citation Formats

Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., and Burrows, Susannah M. Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models. United States: N. p., 2019. Web. doi:10.1109/TVCG.2019.2934540.
Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., & Burrows, Susannah M. Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models. United States. https://doi.org/10.1109/TVCG.2019.2934540
Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., and Burrows, Susannah M. 2019. "Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models". United States. https://doi.org/10.1109/TVCG.2019.2934540. https://www.osti.gov/servlets/purl/1577980.
@article{osti_1577980,
title = {Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models},
author = {Dasgupta, Aritra and Wang, Hong X. and O'Brien, Nancy D. and Burrows, Susannah M.},
abstractNote = {Experts in data and physical sciences have to regularly grapple with the problem of competing models. Be it analytical or physics-based models, a cross-cutting challenge for experts is to reliably diagnose which model outcomes appropriately predict or simulate real-world phenomena. Expert judgment involves reconciling information across many, and often, conflicting criteria that describe the quality of model outcomes. In this paper, through a design study with climate scientists, we develop a deeper understanding of the problem and solution space of model diagnostics, resulting in the following contributions: i) a problem and task characterization using which we map experts’ model diagnostics goals to multi-way visual comparison tasks, ii) a design space of comparative visual cues for letting experts quickly understand the degree of disagreement among competing models and gauge the degree of stability of model outputs with respect to alternative criteria, and iii) design and evaluation of MyriadCues, an interactive visualization interface for exploring alternative hypotheses and insights about good and bad models by leveraging comparative visual cues. Here, we present case studies and subjective feedback by experts, which validate how MyriadCues enables more transparent model diagnostic mechanisms, as compared to the state of the art.},
doi = {10.1109/TVCG.2019.2934540},
url = {https://www.osti.gov/biblio/1577980}, journal = {IEEE Transactions on Visualization and Computer Graphics},
issn = {1077-2626},
number = 1,
volume = 26,
place = {United States},
year = {2019},
month = {8}
}