Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

Distributed X-ray photon correlation spectroscopy data reduction using Hadoop MapReduce

Journal Article · · Journal of Synchrotron Radiation (Online)
 [1];  [1];  [1];  [1];  [1]
  1. Argonne National Lab. (ANL), Lemont, IL (United States)
+ Show Author Affiliations

Multi-speckle X-ray photon correlation spectroscopy (XPCS) is a powerful technique for characterizing the dynamic nature of complex materials over a range of time scales. XPCS has been successfully applied to study a wide range of systems. Recent developments in higher-frame-rate detectors, while aiding in the study of faster dynamical processes, creates large amounts of data that require parallel computational techniques to process in near real-time. Here, an implementation of the multi-tau and two-time autocorrelation algorithms using the Hadoop MapReduce framework for distributed computing is presented. In conclusion, the system scales well with regard to the increase in the data size, and has been serving the users of beamline 8-ID-I at the Advanced Photon Source for near real-time autocorrelations for the past five years.

Research Organization:
Argonne National Lab. (ANL), Argonne, IL (United States)
Sponsoring Organization:
USDOE Office of Science (SC)
Grant/Contract Number:
AC02-06CH11357
OSTI ID:
1466305
Journal Information:
Journal of Synchrotron Radiation (Online), Journal Name: Journal of Synchrotron Radiation (Online) Journal Issue: 4 Vol. 25; ISSN 1600-5775; ISSN JSYRES
Publisher:
International Union of CrystallographyCopyright Statement
Country of Publication:
United States
Language:
English

References (29)

MPICorrelator: A parallel code for performing time correlations
  • Tieman, Brian; Narayanan, Suresh; Sandy, Alec
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 649, Issue 1 https://doi.org/10.1016/j.nima.2010.11.062
journal September 2011
A graphical user interface for real-time analysis of XPCS using HPC
  • Sikorski, M.; Jiang, Z.; Sprung, M.
  • Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 649, Issue 1 https://doi.org/10.1016/j.nima.2010.12.173
journal September 2011
Observation of speckle by diffraction with coherent X-rays journal August 1991
Revealing the fast atomic motion of network glasses journal May 2014
Ultralow-angle dynamic light scattering with a charge coupled device camera based multispeckle, multitau correlator journal August 1999
Area detector based photon correlation in the regime of short data batches: Data reduction for dynamic x-ray scattering journal September 2000
Pixel array detectors for time resolved radiography (invited) journal March 2002
A fast, direct x-ray detection charge-coupled device journal August 2009
Noise on photon correlation data. I. Autocorrelation functions journal August 1990
Beyond simple exponential correlation functions and equilibrium dynamics in x-ray photon correlation spectroscopy journal May 2010
The LAMBDA photon-counting pixel detector journal March 2013
Development of LAMBDA: Large Area Medipix-Based Detector Array journal November 2011
Echoes in x-ray speckles track nanometer-scale plastic events in colloidal gels under shear journal December 2014
Depletion-Induced Structure and Dynamics in Bimodal Colloidal Suspensions journal May 2011
High Contrast X-ray Speckle from Atomic-Scale Order in Liquids and Glasses journal November 2012
Absence of Scaling for the Intermediate Scattering Function of a Hard-Sphere Suspension: Static and Dynamic X-Ray Scattering from Concentrated Polystyrene Latex Spheres journal January 2000
Smectic Membranes in Motion: Approaching the Fast Limits of X-Ray Photon Correlation Spectroscopy journal February 2002
Surface Dynamics of Polymer Films journal February 2003
Capillary Waves at the Transition from Propagating to Overdamped Behavior journal March 2004
Evolution of Particle-Scale Dynamics in an Aging Clay Suspension journal November 2004
X-Ray Intensity Fluctuation Spectroscopy Studies on Phase-Ordering Systems journal February 2005
Evidence for Viscoelastic Effects in Surface Capillary Waves of Molten Polymer Films journal May 2007
The PILATUS 1M detector journal February 2006
X-ray speckle visibility spectroscopy in the single-photon limit journal February 2013
Photon statistics and speckle visibility spectroscopy with partially coherent X-rays journal October 2014
First experimental feasibility study of VIPIC: a custom-made detector for X-ray speckle measurements journal February 2016
MapReduce: simplified data processing on large clusters journal January 2008
Using coherence to measure two-time correlation functions journal January 2003
High Contrast X-ray Speckle from Atomic-Scale Order in Liquids and Glasses text January 2012

Figures / Tables (11)

Fig. 1(p. 4)figure Fig. 1
Fig. 2(p. 5)figure Fig. 2
Fig. 3(p. 8)figure Fig. 3
Fig. 4(p. 10)figure Fig. 4
Fig. 5(p. 12)figure Fig. 5
Fig. 6(p. 15)figure Fig. 6
Fig. 7(p. 19)figure Fig. 7
Fig. 8(p. 20)figure Fig. 8
Fig. 9(p. 21)figure Fig. 9
Fig. 10(p. 22)figure Fig. 10
Fig. 11.(p. 24)figure Fig. 11.

Similar Records

20 µs-resolved high-throughput X-ray photon correlation spectroscopy on a 500k pixel detector enabled by data-management workflow
Journal Article · Thu Dec 31 19:00:00 EST 2020 · Journal of Synchrotron Radiation (Online) · OSTI ID:1844807
An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics
Journal Article · Mon Dec 20 23:00:00 EST 2010 · BMC Bioinformatics, 11(Suppl 12):S1 · OSTI ID:1019222

Related Subjects

73 NUCLEAR PHYSICS AND RADIATION PHYSICS
Advanced Photon Source
MapReduce
XPCS
multi-speckle X-ray photon correlation spectroscopy
parallel implementation