Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models

Dasgupta, Aritra; Wang, Hong X.; O'Brien, Nancy D.; Burrows, Susannah M.

doi:10.1109/TVCG.2019.2934540

Title: Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models

Full Record
Other Related Research

Abstract

Experts in data and physical sciences have to regularly grapple with the problem of competing models. Be it analytical or physics-based models, a cross-cutting challenge for experts is to reliably diagnose which model outcomes appropriately predict or simulate real-world phenomena. Expert judgment involves reconciling information across many, and often, conflicting criteria that describe the quality of model outcomes. In this paper, through a design study with climate scientists, we develop a deeper understanding of the problem and solution space of model diagnostics, resulting in the following contributions: i) a problem and task characterization using which we map experts’ model diagnostics goals to multi-way visual comparison tasks, ii) a design space of comparative visual cues for letting experts quickly understand the degree of disagreement among competing models and gauge the degree of stability of model outputs with respect to alternative criteria, and iii) design and evaluation of MyriadCues, an interactive visualization interface for exploring alternative hypotheses and insights about good and bad models by leveraging comparative visual cues. Here, we present case studies and subjective feedback by experts, which validate how MyriadCues enables more transparent model diagnostic mechanisms, as compared to the state of the art.

Authors:

Dasgupta, Aritra ^[1]; Wang, Hong X. ^[2]; O'Brien, Nancy D. ^[3];

^[3]

New Jersey Institute of Technology, Newark, NJ (United States)
Arizona State Univ., Tempe, AZ (United States)
Pacific Northwest National Lab. (PNNL), Richland, WA (United States)

Publication Date:: 2019-08-26

Research Org.:: Pacific Northwest National Lab. (PNNL), Richland, WA (United States)

Sponsoring Org.:: USDOE

OSTI Identifier:: 1577980

Report Number(s):: PNNL-SA-129640
Journal ID: ISSN 1077-2626

Grant/Contract Number:: AC05-76RL01830

Resource Type:: Journal Article: Accepted Manuscript

Journal Name:: IEEE Transactions on Visualization and Computer Graphics

Additional Journal Information:: Journal Volume: 26; Journal Issue: 1; Journal ID: ISSN 1077-2626

Publisher:: IEEE

Country of Publication:: United States

Language:: English

Subject:: 54 ENVIRONMENTAL SCIENCES; climate model analytics; Climate Model Evaluation; visual analytics; model benchmarking; climate model diagnostics; visual comparison; visual cues; model evaluation; transparency; simulation

Citation Formats


                    Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., and Burrows, Susannah M.. Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models.  United States: N. p., 2019. 
        Web.  doi:10.1109/TVCG.2019.2934540.

Copy to clipboard


                    Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., & Burrows, Susannah M.. Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models.  United States.  https://doi.org/10.1109/TVCG.2019.2934540

Copy to clipboard


                    Dasgupta, Aritra, Wang, Hong X., O'Brien, Nancy D., and Burrows, Susannah M.. 2019.  
        "Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models".  United States.  https://doi.org/10.1109/TVCG.2019.2934540.  https://www.osti.gov/servlets/purl/1577980.

Copy to clipboard


                    
@article{osti_1577980,

  title        = {Separating the Wheat from the Chaff: Comparative Visual Cues for Transparent Diagnostics of Competing Models},

  author       = {Dasgupta, Aritra and Wang, Hong X. and O'Brien, Nancy D. and Burrows, Susannah M.},

  abstractNote = {Experts in data and physical sciences have to regularly grapple with the problem of competing models. Be it analytical or physics-based models, a cross-cutting challenge for experts is to reliably diagnose which model outcomes appropriately predict or simulate real-world phenomena. Expert judgment involves reconciling information across many, and often, conflicting criteria that describe the quality of model outcomes. In this paper, through a design study with climate scientists, we develop a deeper understanding of the problem and solution space of model diagnostics, resulting in the following contributions: i) a problem and task characterization using which we map experts’ model diagnostics goals to multi-way visual comparison tasks, ii) a design space of comparative visual cues for letting experts quickly understand the degree of disagreement among competing models and gauge the degree of stability of model outputs with respect to alternative criteria, and iii) design and evaluation of MyriadCues, an interactive visualization interface for exploring alternative hypotheses and insights about good and bad models by leveraging comparative visual cues. Here, we present case studies and subjective feedback by experts, which validate how MyriadCues enables more transparent model diagnostic mechanisms, as compared to the state of the art.},

  doi          = {10.1109/TVCG.2019.2934540},

  url          = {https://www.osti.gov/biblio/1577980},
  journal      = {IEEE Transactions on Visualization and Computer Graphics},

  issn         = {1077-2626},

  number       = 1,

  volume       = 26,

  place        = {United States},

  year         = {2019},

  month        = {8}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1109/TVCG.2019.2934540

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 8 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

An expert system for sensor data validation and malfunction detection

Conference Hashemi, S; Hajek, B; Miller, D

Nuclear power plant operation and monitoring in general is a complex task which requires a large number of sensors, alarms and displays. At any instant in time, the operator is required to make a judgment about the state of the plant and to react accordingly. During abnormal situations, operators are further burdened with time constraints. The possibility of an undetected faulty instrumentation line, adds to the complexity of operators' reasoning tasks. Failure of human operators to cope with the conceptual complexity of abnormal situations often leads to more serious malfunctions and further damages to plant (TMI-2 as an example). Duringmore »« less
Full Text Available
Haven't a Cue? Mapping the CUE Space as an Aid to HRA Modeling

Conference Gertman, David; Boring, Ronald; Hugo, Jacques; ...

Advances in automation present a new modeling environment for the human reliability analysis (HRA) practitioner. Many, if not most, current day HRA methods have their origin in characterizing and quantifying human performance in analog environments where mode awareness and system status indications are potentially less comprehensive, but simpler to comprehend at a glance when compared to advanced presentation systems. The introduction of highly complex automation has the potential to lead to: decreased levels of situation awareness caused by the need for increased monitoring; confusion regarding the often non-obvious causes of automation failures, and emergent system dependencies that formerly may havemore »« less
Full Text Available
MO-DE-BRA-05: EUTEMPE-RX: Combining E-Learning and Face-To-Face Training to Build Expert Knowledge, Skills and Competences for Medical Physicists in Diagnostic and Interventional Radiology

Journal Article Bosmans, H; Van Peteghem, N; Creten, S; ... - Medical Physics

Purpose: In 2013, the EURATOM authorities of the European Commission decided to support the Horizon2020 project submission ‘EUTEMPE-RX’ that aimed for a new set of course modules to train medical physicists in diagnostic and interventional radiology to expert level with small group deep learning. Each module would consist of 2 phases: an e-learning and a face-to-face phase, each phase requiring typically 40h of participant time. Methods: The European Federation (EFOMP) and 13 European partners, all of them selected for their excellent scientific and/or educational skills, led the 12 course modules. A quality manual ensured the quality of course content andmore »« less
https://doi.org/10.1118/1.4957222
What do you want? : how perceivers use cues to make goal inferences about others.

Journal Article Guss, Dominik; Britt, M; Skowronski, John; ... - Proposed for publication in Cognitive Psychology.

Variables influencing inferences about a stranger's goal during an unsolicited social interaction were explored. Experiment 1 developed a procedure for identifying cues. Experiments 2 and 3 assessed the relative importance of various cues (space, time, characteristics of oneself, characteristics of the stranger, and the stranger's behavior) for goal judgments. Results indicated that situational context cues informed goal judgments in ways that were consistent with diagnosticity ratings and typicality ratings of those cues. Stranger characteristics and stranger behaviors affected goal judgments more than would be expected from these quantitative measures of their informativeness. Nonetheless, the results are consistent with a mentalmore »« less
Quantitative evaluation of existing viewpoints on the origin of mercury mineralization

Journal Article Rayevskiy, V - Int. Geol. Rev.; (United States)

As yet, no final conclusion has been reached on the origin of mercury. Over 16 suggestions have been made in the literature on sources of mercury. The task here is to isolate the soundest hypothesis on the origin of mercury. The author has used a set of advanced mathematical methods providing a rational tool for handling complex problems and providing systematic evidence substantially free from subjective components. This includes the following: (1) systems analysis, which provides information on the detail and degree of uniformity in their knowledge even in the first stages of the research; (2) a wide range ofmore »« less
https://doi.org/10.1080/00206818609466288

Similar Records