Real-time data processing for serial crystallography experiments

White, Thomas; Schoof, Tim; Yakubov, Sergey; Tolstikova, Aleksandra; Middendorf, Philipp; Karnevskiy, Mikhail; Mariani, Valerio; Henkel, Alessandra; Klopprogge, Bjarne; Hannappel, Juergen; Oberthuer, Dominik; De Gennaro Aquino, Ivan; Egorov, Dmitry; Munke, Anna; Sprenger, Janina; Pompidor, Guillaume; Taberman, Helena; Gruzinov, Andrey; Meyer, Jan; Hakanpää, Johanna; Gasthuber, Martin

doi:10.1107/s2052252524011837

Real-time data processing for serial crystallography experiments

Journal Article · Wed Jan 01 00:00:00 EST 2025 · IUCrJ

DOI:https://doi.org/10.1107/s2052252524011837· OSTI ID:2533550

^[1]; Schoof, Tim ^[2]; ^[2]; ^[2]; ^[3]; Karnevskiy, Mikhail ^[2]; Mariani, Valerio ^[4]; Henkel, Alessandra ^[3]; Klopprogge, Bjarne ^[5]; ^[2]; ^[3]; De Gennaro Aquino, Ivan ^[3]; Egorov, Dmitry ^[3]; ^[3]; Sprenger, Janina ^[3]; ^[2]; ^[2]; Gruzinov, Andrey ^[2]; Meyer, Jan ^[2]; Hakanpää, Johanna ^[2] more »

Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Data and Computing in Natural Science (CDCS)
Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany)
Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL)
SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States). Linac Coherent Light Source (LCLS)
Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). Center for Free-Electron Laser Science (CFEL); Hamburg Centre for Ultrafast Imaging (Germany)

We report the use of streaming data interfaces to perform fully online data processing for serial crystallography experiments, without storing intermediate data on disk. The system produces Bragg reflection intensity measurements suitable for scaling and merging, with a latency of less than 1 s per frame. Our system uses the CrystFEL software in combination with the ASAP::O data framework. In a series of user experiments at PETRA III, frames from a 16 megapixel Dectris EIGER2 X detector were searched for peaks, indexed and integrated at the maximum full-frame readout speed of 133 frames per second. The computational resources required depend on various factors, most significantly the fraction of non-blank frames ('hits'). The average single-thread processing time per frame was 242 ms for blank frames and 455 ms for hits, meaning that a single 96-core computing node was sufficient to keep up with the data, with ample headroom for unexpected throughput reductions. Further significant improvements are expected, for example by binning pixel intensities together to reduce the pixel count. We discuss the implications of real-time data processing on the `data deluge' problem from recent and future photon-science experiments, in particular on calibration requirements, computing access patterns and the need for the preservation of raw data.

View Accepted Manuscript (DOE)

Research Organization:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-76SF00515

OSTI ID:: 2533550

Journal Information:: IUCrJ, Journal Name: IUCrJ Journal Issue: 1 Vol. 12; ISSN 2052-2525

Publisher:: International Union of CrystallographyCopyright Statement

Country of Publication:: United States

Language:: English

References (35)

CASS—CFEL-ASG software suite Foucar, Lutz; Barty, Anton; Coppola, Nicola Computer Physics Communications, Vol. 183, Issue 10 https://doi.org/10.1016/j.cpc.2012.04.023	journal	October 2012
Serial femtosecond crystallography datasets from G protein-coupled receptors White, Thomas A.; Barty, Anton; Liu, Wei Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.57	journal	August 2016
The trickle before the torrent—diffraction data from X-ray lasers Maia, Filipe R. N. C.; Hajdu, Janos Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.59	journal	August 2016
Double-flow focused liquid injector for efficient serial femtosecond crystallography Oberthuer, Dominik; Knoška, Juraj; Wiedorn, Max O. Scientific Reports, Vol. 7, Issue 1 https://doi.org/10.1038/srep44628	journal	March 2017
Experience with the CSPAD during dedicated detector runs at LCLS Carini, G. A.; Boutet, S.; Chollet, M. Journal of Physics: Conference Series, Vol. 493 https://doi.org/10.1088/1742-6596/493/1/012011	journal	March 2014
AGIPD, a high dynamic range fast detector for the European XFEL Allahgholi, A.; Becker, J.; Bianco, L. Journal of Instrumentation, Vol. 10, Issue 01 https://doi.org/10.1088/1748-0221/10/01/C01023	journal	January 2015
Characterisation of the high dynamic range Large Pixel Detector (LPD) and its use at X-ray free electron laser sources Veale, M. C.; Adkin, P.; Booker, P. Journal of Instrumentation, Vol. 12, Issue 12 https://doi.org/10.1088/1748-0221/12/12/P12003	journal	December 2017
CrystFEL : a software suite for snapshot serial crystallography White, Thomas A.; Kirian, Richard A.; Martin, Andrew V. Journal of Applied Crystallography, Vol. 45, Issue 2 https://doi.org/10.1107/S0021889812002312	journal	March 2012
Evaluation of single-crystal X-ray diffraction data from a position-sensitive detector Kabsch, W. Journal of Applied Crystallography, Vol. 21, Issue 6 https://doi.org/10.1107/S0021889888007903	journal	December 1988
Indexing in single-crystal diffractometry with an obstinate list of reflections Duisenberg, A. J. M. Journal of Applied Crystallography, Vol. 25, Issue 2 https://doi.org/10.1107/S0021889891010634	journal	April 1992
Lattices and reduced cells as points in 6-space and selection of Bravais lattice type by projections Andrews, L. C.; Bernstein, H. J. Acta Crystallographica Section A Foundations of Crystallography, Vol. 44, Issue 6 https://doi.org/10.1107/S0108767388006427	journal	November 1988
Features and development of Coot Emsley, P.; Lohkamp, B.; Scott, W. G. Acta Crystallographica Section D Biological Crystallography, Vol. 66, Issue 4 https://doi.org/10.1107/S0907444910007493	journal	March 2010
Towards automated crystallographic structure refinement with phenix.refine Afonine, Pavel V.; Grosse-Kunstleve, Ralf W.; Echols, Nathaniel Acta Crystallographica Section D Biological Crystallography, Vol. 68, Issue 4 https://doi.org/10.1107/S0907444912001308	journal	March 2012
The Rossmann Fourier autoindexing algorithm in MOSFLM Powell, Harold R. Acta Crystallographica Section D Biological Crystallography, Vol. 55, Issue 10 https://doi.org/10.1107/S0907444999009506	journal	October 1999
Indexing amyloid peptide diffraction from serial femtosecond crystallography: new algorithms for sparse patterns Brewster, Aaron S.; Sawaya, Michael R.; Rodriguez, Jose Acta Crystallographica Section D Biological Crystallography, Vol. 71, Issue 2 https://doi.org/10.1107/S1399004714026145	journal	January 2015
Cheetah : software for high-throughput reduction and analysis of serial femtosecond X-ray diffraction data Barty, Anton; Kirian, Richard A.; Maia, Filipe R. N. C. Journal of Applied Crystallography, Vol. 47, Issue 3 https://doi.org/10.1107/S1600576714007626	journal	May 2014
The NeXus data format Könnecke, Mark; Akeroyd, Frederick A.; Bernstein, Herbert J. Journal of Applied Crystallography, Vol. 48, Issue 1 https://doi.org/10.1107/S1600576714027575	journal	January 2015
Recent developments in CrystFEL White, Thomas A.; Mariani, Valerio; Brehm, Wolfgang Journal of Applied Crystallography, Vol. 49, Issue 2 https://doi.org/10.1107/S1600576716004751	journal	March 2016
OnDA : online data analysis and feedback for serial X-ray imaging Mariani, Valerio; Morgan, Andrew; Yoon, Chun Hong Journal of Applied Crystallography, Vol. 49, Issue 3 https://doi.org/10.1107/S1600576716007469	journal	May 2016
CFEL–ASG Software Suite ( CASS ): usage for free-electron laser experiments with biological focus Foucar, Lutz Journal of Applied Crystallography, Vol. 49, Issue 4 https://doi.org/10.1107/S1600576716009201	journal	June 2016
FELIX: an algorithm for indexing multiple crystallites in X-ray free-electron laser snapshot diffraction images Beyerlein, Kenneth R.; White, Thomas A.; Yefanov, Oleksandr Journal of Applied Crystallography, Vol. 50, Issue 4 https://doi.org/10.1107/S1600576717007506	journal	July 2017
TORO Indexer: a PyTorch-based indexing algorithm for kilohertz serial crystallography Gasparotto, Piero; Barba, Luis; Stadler, Hans-Christian Journal of Applied Crystallography, Vol. 57, Issue 4 https://doi.org/10.1107/S1600576724003182	journal	June 2024
The ePix10k 2-megapixel hard X-ray detector at LCLS van Driel, Tim Brandt; Nelson, Silke; Armenta, Rebecca Journal of Synchrotron Radiation, Vol. 27, Issue 3 https://doi.org/10.1107/S1600577520004257	journal	April 2020
The PDB_REDO server for macromolecular structure model optimization Joosten, Robbie P.; Long, Fei; Murshudov, Garib N. IUCrJ, Vol. 1, Issue 4 https://doi.org/10.1107/S2052252514009324	journal	May 2014
SPIND : a reference-based auto-indexing algorithm for sparse serial crystallography data Li, Chufeng; Li, Xuanxuan; Kirian, Richard IUCrJ, Vol. 6, Issue 1 https://doi.org/10.1107/S2052252518014951	journal	January 2019
Rapid and efficient room-temperature serial synchrotron crystallography using the CFEL TapeDrive Zielinski, Kara A.; Prester, Andreas; Andaleeb, Hina IUCrJ, Vol. 9, Issue 6 https://doi.org/10.1107/S2052252522010193	journal	October 2022
JINXED: just in time crystallization for easy structure determination of biological macromolecules Henkel, Alessandra; Galchenkova, Marina; Maracke, Julia IUCrJ, Vol. 10, Issue 3 https://doi.org/10.1107/S2052252523001653	journal	March 2023
Kilohertz serial crystallography with the JUNGFRAU detector at a fourth-generation synchrotron source Leonarski, Filip; Nan, Jie; Matej, Zdenek IUCrJ, Vol. 10, Issue 6 https://doi.org/10.1107/S2052252523008618	journal	October 2023
XGANDALF – extended gradient descent algorithm for lattice finding Gevorkov, Yaroslav; Yefanov, Oleksandr; Barty, Anton Acta Crystallographica Section A Foundations and Advances, Vol. 75, Issue 5 https://doi.org/10.1107/S2053273319010593	journal	August 2019
pinkIndexer – a universal indexer for pink-beam X-ray and electron diffraction snapshots Gevorkov, Yaroslav; Barty, Anton; Brehm, Wolfgang Acta Crystallographica Section A Foundations and Advances, Vol. 76, Issue 2 https://doi.org/10.1107/S2053273319015559	journal	January 2020
Approximating lattice similarity Andrews, Lawrence C.; Bernstein, Herbert J.; Sauter, Nicholas K. Acta Crystallographica Section A Foundations and Advances, Vol. 79, Issue 5 https://doi.org/10.1107/S2053273323003200	journal	July 2023
Measuring lattices Andrews, Lawrence C.; Bernstein, Herbert J. Acta Crystallographica Section A Foundations and Advances, Vol. 79, Issue 5 https://doi.org/10.1107/S2053273323004692	journal	August 2023
TakeTwo: an indexing algorithm suited to still images with known crystal parameters Ginn, Helen Mary; Roedig, Philip; Kuo, Anling Acta Crystallographica Section D Structural Biology, Vol. 72, Issue 8, p. 956-965 https://doi.org/10.1107/S2059798316010706	journal	July 2016
Fixed target combined with spectral mapping: approaching 100% hit rates for serial crystallography Oghbaey, Saeed; Sarracini, Antoine; Ginn, Helen M. Acta Crystallographica Section D Structural Biology, Vol. 72, Issue 8, p. 944-955 https://doi.org/10.1107/S2059798316010834	journal	July 2016
Serial Femtosecond Crystallography of G Protein-Coupled Receptors Liu, W.; Wacker, D.; Gati, C. Science, Vol. 342, Issue 6165 https://doi.org/10.1126/science.1244142	journal	December 2013

Similar Records

Pink-beam serial femtosecond crystallography for accurate structure factor determination at an X-ray free-electron laser

Dataset · Mon Feb 01 23:00:00 EST 2021 · OSTI ID:1814882

Raster-scanning serial protein crystallography using micro- and nano-focused synchrotron beams

Journal Article · Fri Apr 24 20:00:00 EDT 2015 · Acta Crystallographica. Section D: Biological Crystallography (Online) · OSTI ID:1213447

Related Subjects

47 OTHER INSTRUMENTATION
X-ray crystallography
macromolecular crystallography
real-time data processing
serial crystallography

Real-time data processing for serial crystallography experiments

Citation Formats

References (35)

Similar Records

Related Subjects