PvaPy streaming framework for real-time data processing

Veseli, Siniša; Hammonds, John; Henke, Steven; Parraga, Hannah; Frosik, Barbara; Schwarz, Nicholas

doi:10.1107/S1600577525002115

PvaPy streaming framework for real-time data processing

Journal Article · Fri Apr 25 00:00:00 EDT 2025 · Journal of Synchrotron Radiation (Online)

DOI:https://doi.org/10.1107/S1600577525002115· OSTI ID:2562725

; Hammonds, John; Henke, Steven; Parraga, Hannah; Frosik, Barbara; Schwarz, Nicholas

User facility upgrades, new measurement techniques, advances in data analysis algorithms as well as advances in detector capabilities result in an increasing amount of data collected at X-ray beamlines. Some of these data must be analyzed and reconstructed on demand to help execute experiments dynamically and modify them in real time. In turn, this requires a computing framework for real-time processing capable of moving data quickly from the detector to local or remote computing resources, processing data, and returning results to users. In this paper, we discuss the streaming framework built on top of PvaPy , a Python API for the EPICS pvAccess protocol. We describe the framework architecture and capabilities, and discuss scientific use cases and applications that benefit from streaming workflows implemented on top of this framework. We also illustrate the framework's performance in terms of achievable data-processing rates for various detector image sizes.

View Journal Article

Sponsoring Organization:: USDOE

Grant/Contract Number:: NONE; AC02-06CH11357

OSTI ID:: 2562725

Alternate ID(s):: OSTI ID: 2587354

Journal Information:: Journal of Synchrotron Radiation (Online), Journal Name: Journal of Synchrotron Radiation (Online) Journal Issue: 3 Vol. 32; ISSN 1600-5775; ISSN JSYRES

Publisher:: International Union of Crystallography (IUCr)Copyright Statement

Country of Publication:: Denmark

Language:: English

References (19)

Artifact Identification in X-ray Diffraction Data Using the Gradient Boosting Method Yanxon, Howard; Weng, James; Parraga, Hannah Intelligent Systems and Applications https://doi.org/10.1007/978-3-031-47718-8_34	book	January 2024
Coherent X-Ray Diffraction Imaging of Morphology and Strain in Nanomaterials Harder, Ross; Robinson, Ian K. JOM, Vol. 65, Issue 9 https://doi.org/10.1007/s11837-013-0682-4	journal	July 2013
Channel access: A software bus for the LAACS Hill, J. O. Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 293, Issue 1-2 https://doi.org/10.1016/0168-9002(90)91459-O	journal	August 1990
The experimental physics and industrial control system architecture: past, present, and future Dalesio, Leo R.; Hill, Jeffrey O.; Kraimer, Martin Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 352, Issue 1-2 https://doi.org/10.1016/0168-9002(94)91493-1	journal	December 1994
Coherent X-ray diffraction imaging of strain at the nanoscale Robinson, Ian; Harder, Ross Nature Materials, Vol. 8, Issue 4 https://doi.org/10.1038/nmat2400	journal	April 2009
Deep learning at the edge enables real-time streaming ptychographic imaging Babu, Anakha V.; Zhou, Tao; Kandel, Saugat Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-023-41496-z	journal	November 2023
AI-enabled high-resolution scanning coherent diffraction imaging Cherukara, Mathew J.; Zhou, Tao; Nashed, Youssef Applied Physics Letters, Vol. 117, Issue 4 https://doi.org/10.1063/5.0013065	journal	July 2020
A fast methodology to determine the characteristics of thousands of grains using three-dimensional X-ray diffraction. I. Overlapping diffraction peaks and parameters of the experimental setup Sharma, Hemant; Huizenga, Richard M.; Offerman, S. Erik Journal of Applied Crystallography, Vol. 45, Issue 4 https://doi.org/10.1107/S0021889812025563	journal	July 2012
A fast methodology to determine the characteristics of thousands of grains using three-dimensional X-ray diffraction. II. Volume, centre-of-mass position, crystallographic orientation and strain state of grains Sharma, Hemant; Huizenga, Richard M.; Offerman, S. Erik Journal of Applied Crystallography, Vol. 45, Issue 4 https://doi.org/10.1107/S0021889812025599	journal	July 2012
APS Data Management System Veseli, Siniša; Schwarz, Nicholas; Schmitz, Collin Journal of Synchrotron Radiation, Vol. 25, Issue 5 https://doi.org/10.1107/S1600577518010056	journal	August 2018
Real-time streaming tomographic reconstruction with on-demand data capturing and 3D zooming to regions of interest Nikitin, Viktor; Tekawade, Aniket; Duchkov, Anton Journal of Synchrotron Radiation, Vol. 29, Issue 3 https://doi.org/10.1107/S1600577522003095	journal	April 2022
Artifact identification in X-ray diffraction data using machine learning methods Yanxon, Howard; Weng, James; Parraga, Hannah Journal of Synchrotron Radiation, Vol. 30, Issue 1 https://doi.org/10.1107/S1600577522011274	journal	January 2023
BraggNN: fast X-ray Bragg peak analysis using deep learning Liu, Zhengchun; Sharma, Hemant; Park, Jun-Sang IUCrJ, Vol. 9, Issue 1 https://doi.org/10.1107/S2052252521011258	journal	December 2021
Globus Online: Accelerating and Democratizing Science through Cloud-Based Services Foster, Ian IEEE Internet Computing, Vol. 15, Issue 3 https://doi.org/10.1109/MIC.2011.64	journal	May 2011
Bridging Data Center AI Systems with Edge Computing for Actionable Information Retrieval Liu, Zhengchun; Ali, Ahsan; Kenesei, Peter 2021 3rd Annual Workshop on Extreme-scale Experiment-in-the-Loop Computing (XLOOP) https://doi.org/10.1109/XLOOP54565.2021.00008	conference	November 2021
SciStream Chung, Joaquin; Zacherek, Wojciech; Wisniewski, Aj Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3502181.3531475	conference	June 2022
Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing Veseli, Siniša; Hammonds, John; Henke, Steven Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis https://doi.org/10.1145/3624062.3624610	conference	November 2023
Empowering Scientific Discovery Through Computing at the Advanced Photon Source Parraga, Hannah; Hammonds, John; Henke, Steven Proceedings of the SC '23 Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis https://doi.org/10.1145/3624062.3624612	conference	November 2023
High-Energy X-Ray Diffraction Microscopy in Materials Science Bernier, Joel V.; Suter, Robert M.; Rollett, Anthony D. Annual Review of Materials Research, Vol. 50, Issue 1 https://doi.org/10.1146/annurev-matsci-070616-124125	journal	July 2020

Similar Records

PvaPy: Python API for EPICS PV Access

Conference · Thu Dec 31 23:00:00 EST 2015 · OSTI ID:1336186

Streaming Data from Experimental Facilities to Supercomputers for Real-Time Data Processing

Conference · Sat Nov 11 23:00:00 EST 2023 · OSTI ID:2246622

PvaPy streaming framework for real-time data processing

Citation Formats

References (19)

Similar Records

Related Subjects