A Software Developer’s Guide to Informal Evaluation of Visual Analytics Environments Using VAST Challenge Information

Conference · Mon Dec 07 04:00:00 EST 2015

DOI:https://doi.org/10.1109/VAST.2015.7347674· OSTI ID:1334902

Cook, Kris A.; Scholtz, Jean; Whiting, Mark A.

The VAST Challenge has been a popular venue for academic and industry participants for over ten years. Many participants comment that the majority of their time in preparing VAST Challenge entries is discovering elements in their software environments that need to be redesigned in order to solve the given task. Fortunately, there is no need to wait until the VAST Challenge is announced to test out software systems. The Visual Analytics Benchmark Repository contains all past VAST Challenge tasks, data, solutions and submissions. This paper details the various types of evaluations that may be conducted using the Repository information. In this paper we describe how developers can do informal evaluations of various aspects of their visual analytics environments using VAST Challenge information. Aspects that can be evaluated include the appropriateness of the software for various tasks, the various data types and formats that can be accommodated, the effectiveness and efficiency of the process supported by the software, and the intuitiveness of the visualizations and interactions. Researchers can compare their visualizations and interactions to those submitted to determine novelty. In addition, the paper provides pointers to various guidelines that software teams can use to evaluate the usability of their software. While these evaluations are not a replacement for formal evaluation methods, this information can be extremely useful during the development of visual analytics environments.

• Research Organization: Pacific Northwest National Laboratory (PNNL), Richland, WA (US)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-76RL01830
• OSTI ID: 1334902
• Report Number(s): PNNL-SA-108733
• Country of Publication: United States
• Language: English

Related Subjects

Visual Analytics Benchmark Repository
Visual analytics environments
evaluation