Tackling the Reproducibility Problem in Systems Research with Declarative Experiment Specifications
- Univ. of California, Santa Cruz, CA (United States)
- Sandia National Lab. (SNL-CA), Livermore, CA (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Univ. of Wisconsin, Madison, WI (United States)
Validating experimental results in the field of computer systems is a challenging task, mainly due to the many changes in software and hardware that computational environments go through. Determining if an experiment is reproducible entails two separate tasks: re-executing the experiment and validating the results. Existing reproducibility efforts have focused on the former, envisioning techniques and infrastructures that make it easier to re-execute an experiment. In this work we focus on the latter by analyzing the validation workflow that an experiment re-executioner goes through. We notice that validating results is done on the basis of experiment design and high-level goals, rather than exact quantitative metrics. Based on this insight, we introduce a declarative format for specifying the high-level components of an experiment as well as describing generic, testable conditions that serve as the basis for validation. We present a use case in the area of storage systems to illustrate the usefulness of this approach. We also discuss limitations and potential benefits of using this approach in other areas of experimental systems research.
- Research Organization:
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC52-07NA27344
- OSTI ID:
- 1251056
- Report Number(s):
- LLNL-TR-670295
- Country of Publication:
- United States
- Language:
- English
Similar Records
The Ghost of Performance Reproducibility Past
Modular performance prediction for scientific workflows using Machine Learning