Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source

Campbell, Stuart I; Kohl, James Arthur; Granroth, Garrett E; Miller, Ross G; Doucet, Mathieu; Stansberry, Dale V; Proffen, Thomas E; Taylor, Russell J; Dillow, David

Title: Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source

Full Record
Other Related Research

Abstract

ORNL operates the world's brightest neutron source, the Spallation Neutron Source (SNS). Funded by the US DOE Office of Basic Energy Science, this national user facility hosts hundreds of scientists from around the world, providing a platform to enable break-through research in materials science, sustainable energy, and basic science. While the SNS provides scientists with advanced experimental instruments, the deluge of data generated from these instruments represents both a big data challenge and a big data opportunity. For example, instruments at the SNS can now generate multiple millions of neutron events per second providing unprecedented experiment fidelity but leaving the user with a dataset that cannot be processed and analyzed in a timely fashion using legacy techniques. To address this big data challenge, ORNL has developed a near real-time streaming data reduction and analysis infrastructure. The Accelerating Data Acquisition, Reduction, and Analysis (ADARA) system provides a live streaming data infrastructure based on a high-performance publish subscribe system, in situ data reduction, visualization, and analysis tools, and integration with a high-performance computing and data storage infrastructure. ADARA allows users of the SNS instruments to analyze their experiment as it is run and make changes to the experiment in real-time and visualizemore »« less

Authors:

Campbell, Stuart I ^[1]; Kohl, James Arthur ^[1]; Granroth, Garrett E ^[1]; Miller, Ross G ^[1]; Doucet, Mathieu ^[1]; Stansberry, Dale V ^[1]; Proffen, Thomas E ^[1]; Taylor, Russell J ^[1]; Dillow, David ^[2]

ORNL
None

Publication Date:: Wed Jan 01 00:00:00 EST 2014

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1154872

DOE Contract Number:: DE-AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: The 10th IEEE International Conference on e-Science, Guaruj , Brazil, 20141020, 20141024

Country of Publication:: United States

Language:: English

Citation Formats


                    Campbell, Stuart I, Kohl, James Arthur, Granroth, Garrett E, Miller, Ross G, Doucet, Mathieu, Stansberry, Dale V, Proffen, Thomas E, Taylor, Russell J, and Dillow, David. Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source.  United States: N. p., 2014. 
        Web.

Copy to clipboard


                    Campbell, Stuart I, Kohl, James Arthur, Granroth, Garrett E, Miller, Ross G, Doucet, Mathieu, Stansberry, Dale V, Proffen, Thomas E, Taylor, Russell J, & Dillow, David. Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source.  United States.

Copy to clipboard


                    Campbell, Stuart I, Kohl, James Arthur, Granroth, Garrett E, Miller, Ross G, Doucet, Mathieu, Stansberry, Dale V, Proffen, Thomas E, Taylor, Russell J, and Dillow, David. 2014.  
        "Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source".  United States.

Copy to clipboard


                    
@article{osti_1154872,

  title        = {Accelerating Data Acquisition, Reduction, and Analysis at the Spallation Neutron Source},

  author       = {Campbell, Stuart I and Kohl, James Arthur and Granroth, Garrett E and Miller, Ross G and Doucet, Mathieu and Stansberry, Dale V and Proffen, Thomas E and Taylor, Russell J and Dillow, David},

  abstractNote = {ORNL operates the world's brightest neutron source, the Spallation Neutron Source (SNS). Funded by the US DOE Office of Basic Energy Science, this national user facility hosts hundreds of scientists from around the world, providing a platform to enable break-through research in materials science, sustainable energy, and basic science. While the SNS provides scientists with advanced experimental instruments, the deluge of data generated from these instruments represents both a big data challenge and a big data opportunity. For example, instruments at the SNS can now generate multiple millions of neutron events per second providing unprecedented experiment fidelity but leaving the user with a dataset that cannot be processed and analyzed in a timely fashion using legacy techniques. To address this big data challenge, ORNL has developed a near real-time streaming data reduction and analysis infrastructure. The Accelerating Data Acquisition, Reduction, and Analysis (ADARA) system provides a live streaming data infrastructure based on a high-performance publish subscribe system, in situ data reduction, visualization, and analysis tools, and integration with a high-performance computing and data storage infrastructure. ADARA allows users of the SNS instruments to analyze their experiment as it is run and make changes to the experiment in real-time and visualize the results of these changes. In this paper we describe ADARA, provide a high-level architectural overview of the system, and present a set of use-cases and real-world demonstrations of the technology.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1154872},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Wed Jan 01 00:00:00 EST 2014},

  month        = {Wed Jan 01 00:00:00 EST 2014}

}

Copy to clipboard

Conference:

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Data and Communications in Basic Energy Sciences: Creating a Pathway for Scientific Discovery

Program Document Nugent, Peter; Simonson, J.

This report is based on the Department of Energy (DOE) Workshop on “Data and Communications in Basic Energy Sciences: Creating a Pathway for Scientific Discovery” that was held at the Bethesda Marriott in Maryland on October 24-25, 2011. The workshop brought together leading researchers from the Basic Energy Sciences (BES) facilities and Advanced Scientific Computing Research (ASCR). The workshop was co-sponsored by these two Offices to identify opportunities and needs for data analysis, ownership, storage, mining, provenance and data transfer at light sources, neutron sources, microscopy centers and other facilities. Their charge was to identify current and anticipated issues inmore »« less
https://doi.org/10.2172/1291136

Full Text Available
The Neutron Science TeraGrid Gateway, a TeraGrid Science Gateway to Support the Spallation Neutron Source

Journal Article Cobb, John; Geist, Al; Kohl, James; ... - Concurrency and Computation: Practice and Experience

The National Science Foundation's (NSF's) Extensible Terascale Facility (ETF), or TeraGrid [1] is entering its operational phase. An ETF science gateway effort is the Neutron Science TeraGrid Gateway (NSTG.) The Oak Ridge National Laboratory (ORNL) resource provider effort (ORNL-RP) during construction and now in operations is bridging a large scale experimental community and the TeraGrid as a large-scale national cyberinfrastructure. Of particular emphasis is collaboration with the Spallation Neutron Source (SNS) at ORNL. The U.S. Department of Energy's (DOE's) SNS [2] at ORNL will be commissioned in spring of 2006 as the world's brightest source of neutrons. Neutron science usersmore »« less
Boosting Big National Lab Data

Journal Article Kleese van Dam, Kerstin - Datanami

Introduction: Big data. Love it or hate it, solving the world’s most intractable problems requires the ability to make sense of huge and complex sets of data and do it quickly. Speeding up the process – from hours to minutes or from weeks to days – is key to our success. One major source of such big data are physical experiments. As many will know, these physical experiments are commonly used to solve challenges in fields such as energy security, manufacturing, medicine, pharmacology, environmental protection and national security. Experiments use different instruments and sensor types to research for example themore »« less
Neutron and X-ray Detectors

Program Document Carini, Gabriella; Denes, Peter; Gruener, Sol; ...

The Basic Energy Sciences (BES) X-ray and neutron user facilities attract more than 12,000 researchers each year to perform cutting-edge science at these state-of-the-art sources. While impressive breakthroughs in X-ray and neutron sources give us the powerful illumination needed to peer into the nano- to mesoscale world, a stumbling block continues to be the distinct lag in detector development, which is slowing progress toward data collection and analysis. Urgently needed detector improvements would reveal chemical composition and bonding in 3-D and in real time, allow researchers to watch “movies” of essential life processes as they happen, and make much moremore »« less
https://doi.org/10.2172/1287494

Full Text Available
2010 Neutron Review: ORNL Neutron Sciences Progress Report

Technical Report Bardoel, Agatha; Counce, Deborah; Ekkebus, Allen; ...

During 2010, the Neutron Sciences Directorate focused on producing world-class science, while supporting the needs of the scientific community. As the instrument, sample environment, and data analysis tools at High Flux Isotope Reactor (HFIR ) and Spallation Neutron Source (SNS) have grown over the last year, so has promising neutron scattering research. This was an exciting year in science, technology, and operations. Some topics discussed are: (1) HFIR and SNS Experiments Take Gordon Battelle Awards for Scientific Discovery - Battelle Memorial Institute presented the inaugural Gordon Battelle Prizes for scientific discovery and technology impact in 2010. Battelle awards the prizesmore »« less
https://doi.org/10.2172/1018612

Full Text Available

Similar Records