Evaluating Emulation-based Models of Distributed Computing Systems
Abstract
Emulation-based models of distributed computing systems are collections of virtual ma- chines, virtual networks, and other emulation components configured to stand in for oper- ational systems when performing experimental science, training, analysis of design alterna- tives, test and evaluation, or idea generation. As with any tool, we should carefully evaluate whether our uses of emulation-based models are appropriate and justified. Otherwise, we run the risk of using a model incorrectly and creating meaningless results. The variety of uses of emulation-based models each have their own goals and deserve thoughtful evaluation. In this paper, we enumerate some of these uses and describe approaches that one can take to build an evidence-based case that a use of an emulation-based model is credible. Predictive uses of emulation-based models, where we expect a model to tell us something true about the real world, set the bar especially high and the principal evaluation method, called validation , is comensurately rigorous. We spend the majority of our time describing and demonstrating the validation of a simple predictive model using a well-established methodology inherited from decades of development in the compuational science and engineering community.
- Authors:
-
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States). Cyber Initiatives
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States). Emulytics Initiatives
- Publication Date:
- Research Org.:
- Sandia National Lab. (SNL-NM), Albuquerque, NM (United States)
- Sponsoring Org.:
- USDOE National Nuclear Security Administration (NNSA)
- OSTI Identifier:
- 1398865
- Report Number(s):
- SAND-2017-10634
657469
- DOE Contract Number:
- AC04-94AL85000
- Resource Type:
- Technical Report
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING
Citation Formats
Jones, Stephen T., Gabert, Kasimir G., and Tarman, Thomas D. Evaluating Emulation-based Models of Distributed Computing Systems. United States: N. p., 2017.
Web. doi:10.2172/1398865.
Jones, Stephen T., Gabert, Kasimir G., & Tarman, Thomas D. Evaluating Emulation-based Models of Distributed Computing Systems. United States. https://doi.org/10.2172/1398865
Jones, Stephen T., Gabert, Kasimir G., and Tarman, Thomas D. Tue .
"Evaluating Emulation-based Models of Distributed Computing Systems". United States. https://doi.org/10.2172/1398865. https://www.osti.gov/servlets/purl/1398865.
@article{osti_1398865,
title = {Evaluating Emulation-based Models of Distributed Computing Systems},
author = {Jones, Stephen T. and Gabert, Kasimir G. and Tarman, Thomas D.},
abstractNote = {Emulation-based models of distributed computing systems are collections of virtual ma- chines, virtual networks, and other emulation components configured to stand in for oper- ational systems when performing experimental science, training, analysis of design alterna- tives, test and evaluation, or idea generation. As with any tool, we should carefully evaluate whether our uses of emulation-based models are appropriate and justified. Otherwise, we run the risk of using a model incorrectly and creating meaningless results. The variety of uses of emulation-based models each have their own goals and deserve thoughtful evaluation. In this paper, we enumerate some of these uses and describe approaches that one can take to build an evidence-based case that a use of an emulation-based model is credible. Predictive uses of emulation-based models, where we expect a model to tell us something true about the real world, set the bar especially high and the principal evaluation method, called validation , is comensurately rigorous. We spend the majority of our time describing and demonstrating the validation of a simple predictive model using a well-established methodology inherited from decades of development in the compuational science and engineering community.},
doi = {10.2172/1398865},
url = {https://www.osti.gov/biblio/1398865},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2017},
month = {8}
}