skip to main content


Title: The MPO system for automatic workflow documentation

Data from large-scale experiments and extreme-scale computing is expensive to produce and may be used for critical applications. However, it is not the mere existence of data that is important, but our ability to make use of it. Experience has shown that when metadata is better organized and more complete, the underlying data becomes more useful. Traditionally, capturing the steps of scientific workflows and metadata was the role of the lab notebook, but the digital era has resulted instead in the fragmentation of data, processing, and annotation. Here, this article presents the Metadata, Provenance, and Ontology (MPO) System, the software that can automate the documentation of scientific workflows and associated information. Based on recorded metadata, it provides explicit information about the relationships among the elements of workflows in notebook form augmented with directed acyclic graphs. A set of web-based graphical navigation tools and Application Programming Interface (API) have been created for searching and browsing, as well as programmatically accessing the workflows and data. We describe the MPO concepts and its software architecture. We also report the current status of the software as well as the initial deployment experience.
 [1] ;  [1] ;  [1] ;  [2] ;  [1] ;  [3] ;  [1] ;  [3] ;  [2] ;  [2] ;  [3]
  1. General Atomics, San Diego, CA (United States)
  2. Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
  3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Publication Date:
Grant/Contract Number:
FC02-04ER54698; SC0008697; AC02-05CH11231; SC0008736; DEAC02-05CH11231
Accepted Manuscript
Journal Name:
Fusion Engineering and Design
Additional Journal Information:
Journal Volume: 112; Journal Issue: C; Journal ID: ISSN 0920-3796
Research Org:
General Atomics, San Diego, CA (United States)
Sponsoring Org:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE Office of Science (SC), Fusion Energy Sciences (FES) (SC-24)
Country of Publication:
United States
70 PLASMA PHYSICS AND FUSION TECHNOLOGY; Workflow; Provenance; Metadata; Ontology; DIII-D; EFIT
OSTI Identifier:
Alternate Identifier(s):
OSTI ID: 1399140