Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Welborn, Samuel S.; Harris, Chris; Ribet, Stephanie M.; Varnavides, Georgios; Ophus, Colin; Enders, Bjoern; Ercius, Peter

doi:10.1093/mam/ozae109

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Journal Article · Thu Nov 14 00:00:00 EST 2024 · Microscopy and Microanalysis

DOI:https://doi.org/10.1093/mam/ozae109· OSTI ID:2477477

^[1]; ^[1]; ^[1]; ^[1]; ^[1]; ^[1]; ^[1]

Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Data management is a critical component of modern experimental workflows. As data generation rates increase, transferring data from acquisition servers to processing servers via conventional file-based methods is becoming increasingly impractical. The 4D Camera at the National Center for Electron Microscopy generates data at a nominal rate of 480 Gbit s^-1 (87,000 frames s^-1⁠), producing a 700 GB dataset in 15 s. To address the challenges associated with storing and processing such quantities of data, we developed a streaming workflow that utilizes a high-speed network to connect the 4D Camera’s data acquisition system to supercomputing nodes at the National Energy Research Scientific Computing Center, bypassing intermediate file storage entirely. In this work, we demonstrate the effectiveness of our streaming pipeline in a production setting through an hour-long experiment that generated over 10 TB of raw data, yielding high-quality datasets suitable for advanced analyses. Additionally, we compare the efficacy of this streaming workflow against the conventional file-transfer workflow by conducting a postmortem analysis on historical data from experiments performed by real users. Our findings show that the streaming workflow significantly improves data turnaround time, enables real-time decision-making, and minimizes the potential for human error by eliminating manual user interactions.

View Journal Article

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States). National Energy Research Scientific Computing Center (NERSC)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 2477477

Alternate ID(s):: OSTI ID: 2481475
OSTI ID: 2480791

Journal Information:: Microscopy and Microanalysis, Journal Name: Microscopy and Microanalysis Journal Issue: 1 Vol. 31; ISSN 1431-9276

Publisher:: Microscopy Society of America (MSA)Copyright Statement

Country of Publication:: United States

Language:: English

References (28)

No authors listed Advanced Materials, Vol. 27, Issue 38, 5625-5844 https://doi.org/10.1002/adma.v27.38	journal	October 2015
Cluster imaging with a direct detection CMOS pixel sensor in Transmission Electron Microscopy Battaglia, Marco; Contarato, Devis; Denes, Peter Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, Vol. 608, Issue 2 https://doi.org/10.1016/j.nima.2009.07.017	journal	September 2009
Operation of TEAM I in a User Environment at NCEM Ercius, Peter; Boese, Markus; Duden, Thomas Microscopy and Microanalysis, Vol. 18, Issue 4 https://doi.org/10.1017/S1431927612001225	journal	July 2012
Development of STEM-Holography Yasin, Fehmi S.; Harvey, Tyler R.; Chess, Jordan J. Microscopy and Microanalysis, Vol. 22, Issue S3 https://doi.org/10.1017/S143192761600338X	journal	July 2016
Four-Dimensional Scanning Transmission Electron Microscopy (4D-STEM): From Scanning Nanodiffraction to Ptychography and Beyond Ophus, Colin Microscopy and Microanalysis, Vol. 25, Issue 3 https://doi.org/10.1017/S1431927619000497	journal	May 2019
py4DSTEM: A Software Package for Four-Dimensional Scanning Transmission Electron Microscopy Data Analysis Savitzky, Benjamin H.; Zeltmann, Steven E.; Hughes, Lauren A. Microscopy and Microanalysis https://doi.org/10.1017/S1431927621000477	journal	May 2021
Real-time interactive ptychography from electron event representation data Pelz, Philipp; Ercius, Peter; Ophus, Colin Microscopy and Microanalysis, Vol. 27, Issue S1 https://doi.org/10.1017/S1431927621001288	journal	July 2021
An Ultrafast Direct Electron Camera for 4D STEM Chatterjee, Debaditya; Wei, Jingrui; kvit, Alex Microscopy and Microanalysis, Vol. 27, Issue S1 https://doi.org/10.1017/S1431927621003809	journal	July 2021
Probing Light Atoms at Subnanometer Resolution: Realization of Scanning Transmission Electron Microscope Holography Yasin, Fehmi S.; Harvey, Tyler R.; Chess, Jordan J. Nano Letters, Vol. 18, Issue 11 https://doi.org/10.1021/acs.nanolett.8b03166	journal	September 2018
Resolution beyond the 'information limit' in transmission electron microscopy Nellist, P. D.; McCallum, B. C.; Rodenburg, J. M. Nature, Vol. 374, Issue 6523 https://doi.org/10.1038/374630a0	journal	April 1995
Differential phase-contrast microscopy at atomic resolution Shibata, Naoya; Findlay, Scott D.; Kohno, Yuji Nature Physics, Vol. 8, Issue 8 https://doi.org/10.1038/nphys2337	journal	June 2012
Solving complex nanostructures with ptychographic atomic electron tomography Pelz, Philipp M.; Griffin, Sinéad M.; Stonemeyer, Scott Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-023-43634-z	journal	November 2023
Towards data-driven next-generation transmission electron microscopy Spurgeon, Steven R.; Ophus, Colin; Jones, Lewys Nature Materials https://doi.org/10.1038/s41563-020-00833-z	journal	October 2020
Subsampled STEM-ptychography Stevens, Andrew; Yang, Hao; Hao, Weituo Applied Physics Letters, Vol. 113, Issue 3 https://doi.org/10.1063/1.5040496	journal	July 2018
Uncovering the three-dimensional structure of upconverting core–shell nanoparticles with multislice electron ptychography Ribet, Stephanie M.; Varnavides, Georgios; Pedroso, Cassio C. S. Applied Physics Letters, Vol. 124, Issue 24 https://doi.org/10.1063/5.0206814	journal	June 2024
Synchrotrons face a data deluge Rao, Rahul Physics Today, Vol. 2020, Issue 09 https://doi.org/10.1063/PT.6.2.20200925a	journal	September 2020
Advanced techniques in automated high-resolution scanning transmission electron microscopy Pattison, Alexander J.; Pedroso, Cassio C. S.; Cohen, Bruce E. Nanotechnology, Vol. 35, Issue 1 https://doi.org/10.1088/1361-6528/acf938	journal	October 2023
Direct detectors and their applications in electron microscopy for materials science Levin, Barnaby D. A. Journal of Physics: Materials, Vol. 4, Issue 4 https://doi.org/10.1088/2515-7639/ac0ff9	journal	July 2021
Theory and practice of electron diffraction from single atoms and extended objects using an EMPAD Cao, Michael C.; Han, Yimo; Chen, Zhen Microscopy, Vol. 67, Issue suppl_1 https://doi.org/10.1093/jmicro/dfx123	journal	December 2017
The 4D Camera: An 87 kHz Direct Electron Detector for Scanning/Transmission Electron Microscopy Ercius, Peter; Johnson, Ian J.; Pelz, Philipp Microscopy and Microanalysis https://doi.org/10.1093/mam/ozae086	journal	September 2024
A computational framework for ptychographic reconstructions Enders, B.; Thibault, P. Proceedings of the Royal Society A: Mathematical, Physical and Engineering Sciences, Vol. 472, Issue 2196 https://doi.org/10.1098/rspa.2016.0640	journal	December 2016
Intrinsic Instability of Aberration-Corrected Electron Microscopes Schramm, S. M.; van der Molen, S. J.; Tromp, R. M. Physical Review Letters, Vol. 109, Issue 16 https://doi.org/10.1103/PhysRevLett.109.163901	journal	October 2012
Phase-contrast imaging of multiply-scattering extended objects at atomic resolution by reconstruction of the scattering matrix Pelz, Philipp M.; Brown, Hamish G.; Stonemeyer, Scott Physical Review Research, Vol. 3, Issue 2 https://doi.org/10.1103/PhysRevResearch.3.023159	journal	May 2021
Interactive Supercomputing With Jupyter Thomas, Rollin; Cholia, Shreyas Computing in Science & Engineering, Vol. 23, Issue 2 https://doi.org/10.1109/MCSE.2021.3059037	journal	March 2021
Atomic electron tomography: 3D structures without crystals Miao, J.; Ercius, P.; Billinge, S. J. L. Science, Vol. 353, Issue 6306 https://doi.org/10.1126/science.aaf2157	journal	September 2016
The chain of chirality transfer in tellurium nanocrystals Ben-Moshe, Assaf; da Silva, Alessandra; Müller, Alexander Science, Vol. 372, Issue 6543 https://doi.org/10.1126/science.abf9645	journal	May 2021
High-frame rate and high-count rate hybrid pixel detector for 4D STEM applications Zambon, P.; Vávra, J.; Montemurro, G. Frontiers in Physics, Vol. 11 https://doi.org/10.3389/fphy.2023.1308321	journal	December 2023
Reference Array and Design Consideration for the Next-Generation Event Horizon Telescope Doeleman, Sheperd S.; Barrett, John; Blackburn, Lindy Galaxies, Vol. 11, Issue 5 https://doi.org/10.3390/galaxies11050107	journal	October 2023

Similar Records

The 4D Camera: An 87 kHz Direct Electron Detector for Scanning/Transmission Electron Microscopy

Journal Article · Wed Sep 18 20:00:00 EDT 2024 · Microscopy and Microanalysis · OSTI ID:2462736

Related Subjects

4D-STEM
97 MATHEMATICS AND COMPUTING
high-performance computing
real-time processing
streaming
zmq

Streaming Large-Scale Microscopy Data to a Supercomputing Facility

Citation Formats

References (28)

Similar Records

Related Subjects