The MPO system for automatic workflow documentation

Abla, G.; Coviello, E. N.; Flanagan, S. M.; Greenwald, M.; Lee, X.; Romosan, A.; Schissel, D. P.; Shoshani, A.; Stillerman, J.; Wright, J.; Wu, K. J.

doi:10.1016/j.fusengdes.2016.04.023

Title: The MPO system for automatic workflow documentation

Full Record
Other Related Research

Abstract

Data from large-scale experiments and extreme-scale computing is expensive to produce and may be used for critical applications. However, it is not the mere existence of data that is important, but our ability to make use of it. Experience has shown that when metadata is better organized and more complete, the underlying data becomes more useful. Traditionally, capturing the steps of scientific workflows and metadata was the role of the lab notebook, but the digital era has resulted instead in the fragmentation of data, processing, and annotation. Here, this article presents the Metadata, Provenance, and Ontology (MPO) System, the software that can automate the documentation of scientific workflows and associated information. Based on recorded metadata, it provides explicit information about the relationships among the elements of workflows in notebook form augmented with directed acyclic graphs. A set of web-based graphical navigation tools and Application Programming Interface (API) have been created for searching and browsing, as well as programmatically accessing the workflows and data. We describe the MPO concepts and its software architecture. We also report the current status of the software as well as the initial deployment experience.

Authors:

Abla, G. ^[1]; Coviello, E. N. ^[1]; Flanagan, S. M. ^[1]; Greenwald, M. ^[2]; Lee, X. ^[1]; Romosan, A. ^[3]; Schissel, D. P. ^[1]; Shoshani, A. ^[3]; Stillerman, J. ^[2]; Wright, J. ^[2]; Wu, K. J. ^[3]

General Atomics, San Diego, CA (United States)
Massachusetts Inst. of Technology (MIT), Cambridge, MA (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Publication Date:: Mon Apr 18 00:00:00 EDT 2016

Research Org.:: General Atomics, San Diego, CA (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Fusion Energy Sciences (FES)

OSTI Identifier:: 1371571

Alternate Identifier(s):: OSTI ID: 1399140

Grant/Contract Number:: FC02-04ER54698; SC0008697; AC02-05CH11231; SC0008736; DEAC02-05CH11231

Resource Type:: Accepted Manuscript

Journal Name:: Fusion Engineering and Design

Additional Journal Information:: Journal Volume: 112; Journal Issue: C; Journal ID: ISSN 0920-3796

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 70 PLASMA PHYSICS AND FUSION TECHNOLOGY; Workflow; Provenance; Metadata; Ontology; DIII-D; EFIT

Citation Formats


                    Abla, G., Coviello, E. N., Flanagan, S. M., Greenwald, M., Lee, X., Romosan, A., Schissel, D. P., Shoshani, A., Stillerman, J., Wright, J., and Wu, K. J. The MPO system for automatic workflow documentation.  United States: N. p., 2016. 
Web.  doi:10.1016/j.fusengdes.2016.04.023.

Copy to clipboard


                    Abla, G., Coviello, E. N., Flanagan, S. M., Greenwald, M., Lee, X., Romosan, A., Schissel, D. P., Shoshani, A., Stillerman, J., Wright, J., & Wu, K. J. The MPO system for automatic workflow documentation.  United States.  https://doi.org/10.1016/j.fusengdes.2016.04.023

Copy to clipboard


                    Abla, G., Coviello, E. N., Flanagan, S. M., Greenwald, M., Lee, X., Romosan, A., Schissel, D. P., Shoshani, A., Stillerman, J., Wright, J., and Wu, K. J. Mon .  
"The MPO system for automatic workflow documentation".  United States.  https://doi.org/10.1016/j.fusengdes.2016.04.023.  https://www.osti.gov/servlets/purl/1371571.

Copy to clipboard


                    
@article{osti_1371571,

  title        = {The MPO system for automatic workflow documentation},

  author       = {Abla, G. and Coviello, E. N. and Flanagan, S. M. and Greenwald, M. and Lee, X. and Romosan, A. and Schissel, D. P. and Shoshani, A. and Stillerman, J. and Wright, J. and Wu, K. J.},

  abstractNote = {Data from large-scale experiments and extreme-scale computing is expensive to produce and may be used for critical applications. However, it is not the mere existence of data that is important, but our ability to make use of it. Experience has shown that when metadata is better organized and more complete, the underlying data becomes more useful. Traditionally, capturing the steps of scientific workflows and metadata was the role of the lab notebook, but the digital era has resulted instead in the fragmentation of data, processing, and annotation. Here, this article presents the Metadata, Provenance, and Ontology (MPO) System, the software that can automate the documentation of scientific workflows and associated information. Based on recorded metadata, it provides explicit information about the relationships among the elements of workflows in notebook form augmented with directed acyclic graphs. A set of web-based graphical navigation tools and Application Programming Interface (API) have been created for searching and browsing, as well as programmatically accessing the workflows and data. We describe the MPO concepts and its software architecture. We also report the current status of the software as well as the initial deployment experience.},

  doi          = {10.1016/j.fusengdes.2016.04.023},

  journal      = {Fusion Engineering and Design},

  number       = C,

  volume       = 112,

  place        = {United States},

  year         = {Mon Apr 18 00:00:00 EDT 2016},

  month        = {Mon Apr 18 00:00:00 EDT 2016}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.fusengdes.2016.04.023

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Automated metadata, provenance cataloging and navigable interfaces: ensuring the usefulness of extreme-scale data

Technical Report Schissel, David ; Greenwald, Martin

The MPO (Metadata, Provenance, Ontology) Project successfully addressed the goal of improving the usefulness and traceability of scientific data by building a system that could capture and display all steps in the process of creating, analyzing and disseminating that data. Throughout history, scientists have generated handwritten logbooks to keep track of data, their hypotheses, assumptions, experimental setup, and computational processes as well as reflections on observations and issues encountered. Over the last several decades, with the growth of personal computers, handheld devices, and the World Wide Web, the handwritten logbook has begun to be replaced by electronic logbooks. This transitionmore »« less
Automated metadata--final project report

Technical Report Schissel, David

This report summarizes the work of the Automated Metadata, Provenance Cataloging, and Navigable Interfaces: Ensuring the Usefulness of Extreme-Scale Data Project (MPO Project) funded by the United States Department of Energy (DOE), Offices of Advanced Scientific Computing Research and Fusion Energy Sciences. Initially funded for three years starting in 2012, it was extended for 6 months with additional funding. The project was a collaboration between scientists at General Atomics, Lawrence Berkley National Laboratory (LBNL), and Massachusetts Institute of Technology (MIT). The group leveraged existing computer science technology where possible, and extended or created new capabilities where required. The MPO projectmore »« less
https://doi.org/10.2172/1297137

Full Text Available
A High-Throughput Computing Infrastructure to Generate Custom, Open Community Geothermal Datasets

Technical Report Zaffos, Andrew A.

The most significant challenge facing geothermal research, development, and deployment is a lack of comprehensive datasets describing the geological and economical properties of North America. Automated knowledge base construction, the process of designing algorithms to analyze text and images to programmatically build new datasets, is one possible solution to this problem. The xDD library of full-text scientific articles (https://xdd.wisc.edu) is one of the largest collections of open and controlled-access scientific documents available for knowledge base construction in the world, but it has been underutilized by experts in geothermal research. The xDD development team attributed the lack of engagement by softwaremore »« less
https://doi.org/10.2172/1856435

Full Text Available
A Simple Standard for Sharing Ontological Mappings (SSSOM)

Journal Article Matentzoglu, Nicolas ; Balhoff, James P. ; Bello, Susan M. ; ... - Database

Abstract Despite progress in the development of standards for describing and exchanging scientific information, the lack of easy-to-use standards for mapping between different representations of the same or similar objects in different databases poses a major impediment to data integration and interoperability. Mappings often lack the metadata needed to be correctly interpreted and applied. For example, are two terms equivalent or merely related? Are they narrow or broad matches? Or are they associated in some other way? Such relationships between the mapped terms are often not documented, which leads to incorrect assumptions and makes them hard to use in scenariosmore »« less
https://doi.org/10.1093/database/baac035
Enhancing Monte Carlo Workflows for Nuclear Reactor Analysis with Metamodel-Driven Modeling

Journal Article Kowal, Peter J. ; Blake, Camden E. ; Dominesey, Kurt A. ; ... - Nuclear Science and Engineering

Monte Carlo codes are essential components of many reactor physics simulation workflows as high-fidelity continuous-energy neutron transport solvers. Among Monte Carlo radiation transport codes, MCNP is particularly notable due to its diverse simulation capabilities, large user base, and long validation history. Despite being a powerful simulation tool, MCNP provides limited capabilities to allow automated execution, model transformation, or support for user-defined logic and abstractions that limit its compatibility with modern workflows. Here, to better integrate MCNP into a modern scientific workflow, we have developed an intuitive yet full-featured MCNP Application Program Interface (API) in Python, named MCNPy, which provides amore »« less
https://doi.org/10.1080/00295639.2022.2153617

Full Text Available

Similar Records