Science Capsule: Towards Sharing and Reproducibility of Scientific Workflows

Conference · Fri Dec 03 04:00:00 EST 2021 · Workshop on Workflows in Support of Large-Scale Science.

DOI:https://doi.org/10.1109/WORKS54523.2021.00014· OSTI ID:1833998

Ghoshal, Devarshi; Bianchi, Ludovico; Essiari, Abdelilah; Paine, Drew; Poon, Sarah; Beach, Michael; N'Diaye, Alpha; Huck, Patrick; Ramakrishnan, Lavanya

Workflows are increasingly processing large volumes of data from scientific instruments, experiments and sensors. These workflows often consist of complex data processing and analysis steps that might include a diverse ecosystem of tools and also often involve human-in-the-loop steps. Sharing and reproducing these workflows with collaborators and the larger community is critical but hard to do without the entire context of the workflow including user notes and execution environment. In this paper, we describe Science Capsule, which is a framework to capture, share, and reproduce scientific workflows. Science Capsule captures, manages and represents both computational and human elements of a workflow. It automatically captures and processes events associated with the execution and data life cycle of workflows, and lets users add other types and forms of scientific artifacts. Science Capsule also allows users to create `workflow snapshots' that keep track of the different versions of a workflow and their lineage, allowing scientists to incrementally share and extend workflows between users. Our results show that Science Capsule is capable of processing and organizing events in near real-time for high-throughput experimental and data analysis workflows without incurring any significant performance overheads.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)
• DOE Contract Number: AC02-05CH11231
• OSTI ID: 1833998
• Conference Information: Journal Name: Workshop on Workflows in Support of Large-Scale Science. Journal Volume: 2021
• Country of Publication: United States
• Language: English