Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Godoy, William; Peterson, Peter; Hahn, Steven; Hetrick, John; Doucet, Mathieu; Billings, Jay Jay

Title: Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Full Record
Other Related Research

Abstract

Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.

Authors:

^[1];

^[1]; Hetrick, John ^[1];

^[1];

^[1]

ORNL

Publication Date:: 2020-12-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Basic Energy Sciences (BES)

OSTI Identifier:: 1755304

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: Smoky Mountains Computational Sciences and Engineering Conference (SMC) - Kingsport, Tennessee, United States of America - 8/25/2020 4:00:00 PM-8/28/2020 4:00:00 PM

Country of Publication:: United States

Language:: English

Citation Formats


                    Godoy, William, Peterson, Peter, Hahn, Steven, Hetrick, John, Doucet, Mathieu, and Billings, Jay Jay. Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid.  United States: N. p., 2020. 
        Web.

Copy to clipboard


                    Godoy, William, Peterson, Peter, Hahn, Steven, Hetrick, John, Doucet, Mathieu, & Billings, Jay Jay. Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid.  United States.

Copy to clipboard


                    Godoy, William, Peterson, Peter, Hahn, Steven, Hetrick, John, Doucet, Mathieu, and Billings, Jay Jay. 2020.  
        "Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid".  United States.  https://www.osti.gov/servlets/purl/1755304.

Copy to clipboard


                    
@article{osti_1755304,

  title        = {Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid},

  author       = {Godoy, William and Peterson, Peter and Hahn, Steven and Hetrick, John and Doucet, Mathieu and Billings, Jay Jay},

  abstractNote = {Performance of data reduction workflows at the High Flux Isotope Reactor (HFIR) and the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory (ORNL) is mainly determined by the time spent loading raw measurement events stored in large and sparse datasets. This paper describes: (1) our long-term view to leverage SNS and HFIR data management needs with our experience at ORNL’s world-class high performance computing (HPC) facilities, and (2) our short-term efforts to speed up current workflows using Mantid, a data analysis and reduction community framework used across several neutron scattering facilities. We show that minimally invasive short-term improvements in metadata management have a moderate impact in speeding up current production workflows. We propose a more disruptive domain-specific solution: the No Cost Input Output (NCIO) framework, we provide an overview, the risks and challenges in NCIO’s adoption by HFIR and SNS stakeholders.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1755304},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2020},

  month        = {12}

}

Copy to clipboard

Conference:

View Conference (2.49 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

Conference Godoy, William; Savici, Andrei; Hahn, Steven; ...

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-sourcemore »« less
https://doi.org/10.1109/BigData52589.2021.9671354

Full Text Available
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Conference Godoy, William; Peterson, Peter; Hahn, Steven; ...

Oak Ridge National Laboratory (ORNL) experimental neutron science facilities produce 1.2 TB a day of raw event-based data that is stored using the standard metadata-rich NeXus schema built on top of the HDF5 file format. Performance of several data reduction workflows is largely determined by the amount of time spent on the loading and processing algorithms in Mantid, an open-source data analysis framework used across several neutron sciences facilities around the world. The present work introduces new data management algorithms to address identified input output (I/O) bottlenecks on Mantid. First, we introduce an in-memory binary-tree metadata index that resemble NeXusmore »« less
Full Text Available
The SNS/HFIR Web Portal System for SANS

Conference Campbell, Stuart; Miller, Stephen; Bilheux, Jean-Christophe; ...

In a busy world, continuing with the status-quo, to do things the way we are already familiar, often seems to be the most efficient way to conduct our work. We look for the value-add to decide if investing in a new method is worth the effort. How shall we evaluate if we have reached this tipping point for change? For contemporary researchers, understanding the properties of the data is a good starting point. The new generation of neutron scattering instruments being built are higher resolution and produce one or more orders of magnitude larger data than the previous generation ofmore »« less
https://doi.org/10.1088/1742-6596/247/1/012013

Full Text Available
The SNS/HFIR Web Portal System – How Can it Help Me?

Conference Miller, Stephen; Geist II, Al; Herwig, Kenneth; ...

Abstract. In a busy world, continuing with the status-quo, to do things the way we are already familiar, often seems to be the most efficient way to conduct our work. We look for the value-add to decide if investing in a new method is worth the effort. How shall we evaluate if we have reached this tipping point for change? For contemporary researchers, understanding the properties of the data is a good starting point. The new generation of neutron scattering instruments being built are higher resolution and produce one or more orders of magnitude larger data than the previous generationmore »« less
https://doi.org/10.1088/1742-6596/251/1/012096

Full Text Available
Cross-Cutting Software Solutions in Support of Experimental Analysis Challenges at National Scattering Facilities

Book Proffen, Thomas; Granroth, Garrett

Rapid improvements in instrumentation at neutron and light source combined with new capabilities in computing, data analytics, and machine learning offer unprecedented scientific capabilities to address fundamental and applied science questions in a wide area of subjects from quantum materials to biological systems. In order to make these capabilities accessible to the broader scientific community, significant developments are needed related to workflow spanning from experimental control to advanced materials simulations as well as the underpinning scientific software. In this chapter, we explore these needs and recent developments at the Spallation Neutron Source (SNS) at Oak Ridge National Laboratory. A centralmore »« less
https://doi.org/10.1142/9789811204579_0011

Similar Records