Developing Guidelines for Assessing Visual Analytics Environments
Visual analytic systems can be evaluated from a user perspective with quantitative metrics (i.e., time to complete the analysis or the accuracy of the solution found). However, qualitative measures are also useful in a user assessment. These include such measures as the utility of the interactive visualizations in the analysis process and the user's assessment of the efficiency of the analytic process. Quantitative measures can be found if data sets with embedded ground truth are used for analysis. Qualitative measures are more elusive. In this paper we report on an experiment with professional analysts who ranked five of submissions to the VAST 2009 Challenge and provided the rationale for their rankings. Their comments were used in conjunction with a meta-analysis of the 2009 VAST Challenge reviews to produce a set of guidelines for visual analytic systems. As visual analytic software is expected to eventually help in all aspects of analysis, we expect to see future systems provide more help with generating the final report. Hence, researchers also need to have an understanding of what makes a good analytic product. Therefore we asked the analysts to rank the situational assessments of four grand challenge entries and to provide comments on those assessments. We used these comments to produce guidelines for researchers to use in evaluating their analytic reports.
- Research Organization:
- Pacific Northwest National Lab. (PNNL), Richland, WA (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-76RL01830
- OSTI ID:
- 1030863
- Report Number(s):
- PNNL-SA-71830; 400904120; TRN: US201124%%494
- Journal Information:
- Information Visualization, Vol. 10, Issue 3
- Country of Publication:
- United States
- Language:
- English
Similar Records
Evaluation of Visual Analytics Environments: The Road to the Visual Analytics Science and Technology Challenge Evaluation Methodology
A Software Developer’s Guide to Informal Evaluation of Visual Analytics Environments Using VAST Challenge Information