Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

Godoy, William; Savici, Andrei T.; Hahn, Steven; Peterson, Peter

doi:10.1109/BigData52589.2021.9671354

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

Conference · Wed Dec 01 04:00:00 EST 2021

DOI:https://doi.org/10.1109/BigData52589.2021.9671354· OSTI ID:1841479

^[1]; ^[1]; ^[1]; ^[1]

ORNL

We present algorithmic improvements to the loading operations of certain reduced data ensembles produced from neutron scattering experiments at Oak Ridge National Laboratory (ORNL) facilities. Ensembles from multiple measurements are required to cover a wide range of the phase space of a sample material of interest. They are stored using the standard NeXus schema on individual HDF5 files. This makes it a scalability challenge, as the number of experiments stored increases in a single ensemble file. The present work follows up on our previous efforts on data management algorithms, to address identified input output (I/O) bottlenecks in Mantid, an open-source data analysis framework used across several neutron science facilities around the world. We reuse an in-memory binary-tree metadata index that resembles data access patterns, to provide a scalable search and extraction mechanism. In addition, several memory operations are refactored and optimized for the current common use cases, ranging most frequently from 10 to 180, and up to 360 separate measurement configurations. Results from this work show consistent speed ups in wall-clock time on the Mantid LoadMD routine, ranging from 19% to 23% on average, on ORNL production computing systems. The latter depends on the complexity of the targeted instrument-specific data and the system I/O and compute variability for the shared computational resources available to users of ORNL’s Spallation Neutron Source (SNS) and the High Flux Isotope Reactor (HFIR) instruments. Nevertheless, we continue to highlight the need for more research to address reduction challenges as experimental data volumes, user time and processing costs increase.

View Conference

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; USDOE Office of Science (SC), Basic Energy Sciences (BES) (SC-22)

DOE Contract Number:: AC05-00OR22725

OSTI ID:: 1841479

Country of Publication:: United States

Language:: English

References (15)

Horace : Software for the analysis of data from single crystal spectroscopy experiments at time-of-flight neutron instruments Ewings, R. A.; Buts, A.; Le, M. D. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 834 https://doi.org/10.1016/j.nima.2016.07.036	journal	October 2016
A View from ORNL: Scientific Data Research Opportunities in the Big Data Age Klasky, Scott; Wolf, Matthew; Ainsworth, Mark 2018 IEEE 38th International Conference on Distributed Computing Systems (ICDCS) https://doi.org/10.1109/ICDCS.2018.00136	conference	July 2018
The NeXus data format Könnecke, Mark; Akeroyd, Frederick A.; Bernstein, Herbert J. Journal of Applied Crystallography, Vol. 48, Issue 1 https://doi.org/10.1107/S1600576714027575	journal	January 2015
The SNS/HFIR Web Portal System for SANS Campbell, Stuart I.; Miller, Stephen D.; Bilheux, Jean-Christophe Journal of Physics: Conference Series, Vol. 247 https://doi.org/10.1088/1742-6596/247/1/012013	journal	October 2010
Design and operation of the wide angular-range chopper spectrometer ARCS at the Spallation Neutron Source Abernathy, D. L.; Stone, M. B.; Loguillo, M. J. Review of Scientific Instruments, Vol. 83, Issue 1 https://doi.org/10.1063/1.3680104	journal	January 2012
Cold neutron sources at ILL Ageron, Paul Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 284, Issue 1 https://doi.org/10.1016/0168-9002(89)90281-7	journal	November 1989
Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. Communications in Computer and Information Science https://doi.org/10.1007/978-3-030-63393-6_12	book	January 2020
Event-based processing of neutron scattering data at the Spallation Neutron Source Granroth, Garrett E.; An, Ke; Smith, Hillary L. Journal of Applied Crystallography, Vol. 51, Issue 3 https://doi.org/10.1107/S1600576718004727	journal	May 2018
A comparison of four direct geometry time-of-flight spectrometers at the Spallation Neutron Source Stone, M. B.; Niedziela, J. L.; Abernathy, D. L. Review of Scientific Instruments, Vol. 85, Issue 4 https://doi.org/10.1063/1.4870050	journal	April 2014
Mantid—Data analysis and visualization package for neutron scattering and μ SR experiments Arnold, O.; Bilheux, J. C.; Borreguero, J. M. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 764 https://doi.org/10.1016/j.nima.2014.07.029	journal	November 2014
The ISIS Spallation Neutron and Muon Source—The first thirty-three years Thomason, J. W. G. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 917 https://doi.org/10.1016/j.nima.2018.11.129	journal	February 2019
Exploring Metadata Search Essentials for Scientific Data Management Zhang, Wei; Byna, Suren; Niu, Chenxu 2019 IEEE 26th International Conference on High Performance Computing, Data, and Analytics (HiPC) https://doi.org/10.1109/HiPC.2019.00021	conference	December 2019
Event-based processing of neutron scattering data Peterson, Peter F.; Campbell, Stuart I.; Reuter, Michael A. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 803 https://doi.org/10.1016/j.nima.2015.09.016	journal	December 2015
Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL Godoy, William F.; Peterson, Peter F.; Hahn, Steven E. 2020 IEEE International Conference on Big Data (Big Data) https://doi.org/10.1109/BigData50022.2020.9377836	conference	December 2020
The flame graph Gregg, Brendan Communications of the ACM, Vol. 59, Issue 6 https://doi.org/10.1145/2909476	journal	May 2016

Similar Records

Efficient Data Management in Neutron Scattering Data Reduction Workflows at ORNL

Conference · Mon Nov 30 23:00:00 EST 2020 · OSTI ID:1772865

SNAP diffraction dataset for 2023 SMC data challenge

Dataset · Fri Apr 14 00:00:00 EDT 2023 · OSTI ID:1968472

Performance Improvements on SNS and HFIR Instrument Data Reduction Workflows Using Mantid

Conference · Mon Nov 30 23:00:00 EST 2020 · OSTI ID:1755304

Efficient loading of reduced data ensembles produced at ORNL SNS/HFIR neutron time-of-flight facilities

Citation Formats

References (15)

Similar Records

Related Subjects