Skip to main content
OSTI.GOVU.S. Department of Energy Office of Scientific and Technical Information
U.S. Department of Energy
Office of Scientific and Technical Information
 

ExaFEL: extreme-scale real-time data processing for X-ray free electron laser science

Journal Article · · Frontiers in High Performance Computing
 [1];  [1];  [1];  [1];  [2];  [3];  [1];  [2];  [4];  [1];  [2];  [2];  [1];  [2];  [2];  [1];  [5];  [1];  [6];  [2] more »;  [5];  [1];  [2];  [1];  [5];  [2];  [1] « less
  1. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
  2. SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)
  3. Virginia Polytechnic Inst. and State Univ. (Virginia Tech), Blacksburg, VA (United States)
  4. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)
  5. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
  6. Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Virginia Polytechnic Inst. and State Univ. (Virginia Tech), Blacksburg, VA (United States)
+ Show Author Affiliations
ExaFEL is an HPC-capable X-ray Free Electron Laser (XFEL) data analysis software suite for both Serial Femtosecond Crystallography (SFX) and Single Particle Imaging (SPI) developed in collaboration with the Linac Coherent Lightsource (LCLS), Lawrence Berkeley National Laboratory (LBNL) and Los Alamos National Laboratory. ExaFEL supports real-time data analysis via a cross-facility workflow spanning LCLS and HPC centers such as NERSC and OLCF. Our work therefore constitutes initial path-finding for the US Department of Energy's (DOE) Integrated Research Infrastructure (IRI) program. We present the ExaFEL team's 7 years of experience in developing real-time XFEL data analysis software for the DOE's exascale supercomputers. We present our experiences and lessons learned with the Perlmutter and Frontier supercomputers. Furthermore we outline essential data center services (and the implications for institutional policy) required for real-time data analysis. Finally we summarize our software and performance engineering approaches and our experiences with NERSC's Perlmutter and OLCF's Frontier systems. This work is intended to be a practical blueprint for similar efforts in integrating exascale compute resources into other cross-facility workflows.
Research Organization:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
National Energy Research Scientific Computing Center (NERSC); National Institutes of Health (NIH); USDOE National Nuclear Security Administration (NNSA); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF); USDOE Office of Science (SC), Biological and Environmental Research (BER)
Grant/Contract Number:
89233218CNA000001; AC02-05CH11231; AC02-76SF00515; AC05-00OR22725
OSTI ID:
2999940
Report Number(s):
LA-UR--24-30617; 10.3389/fhpcp.2024.1414569; 2813-7337
Journal Information:
Frontiers in High Performance Computing, Journal Name: Frontiers in High Performance Computing Vol. 2; ISSN 2813-7337
Publisher:
Frontiers Media S.A.Copyright Statement
Country of Publication:
United States
Language:
English

References (27)

Accelerating x‐ray tracing for exascale systems using Kokkos journal November 2023
Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis
  • Blaschke, Johannes P.; Brewster, Aaron S.; Paley, Daniel W.
  • Concurrency and Computation: Practice and Experience, Vol. 36, Issue 12 https://doi.org/10.1002/cpe.8019
journal February 2024
On the limited memory BFGS method for large scale optimization journal August 1989
A high-performance, portable implementation of the MPI message passing interface standard journal September 1996
Macromolecular diffractive imaging using imperfect crystals journal February 2016
Reconstruction from limited single-particle diffraction data via simultaneous determination of state, orientation, intensity, and phase journal June 2017
How a Lightsource Uses a Supercomputer for Live Interactive Analysis of Large Data Sets journal July 2023
The Computational Crystallography Toolbox : crystallographic algorithms in a reusable software framework journal January 2002
New Python-based methods for data processing journal June 2013
IOTA : integration optimization, triage and analysis tool for the processing of XFEL diffraction images journal May 2016
Skopi: a simulation package for diffractive imaging of noncrystalline biomolecules journal July 2022
Going around the Kok cycle of the water oxidation reaction with femtosecond X-ray crystallography journal October 2023
Towards the spatial resolution of metalloprotein charge states by detailed modeling of XFEL crystallographic diffraction journal February 2020
DIALS : implementation and evaluation of a new integration package journal February 2018
Enabling discovery data science through cross-facility workflows conference December 2021
cuFINUFFT: a load-balanced GPU library for general-purpose nonuniform FFTs conference June 2021
SPIRAL: Extreme Performance Portability journal November 2018
The Exascale Computing Project journal May 2017
Pygion: Flexible, Scalable Task-Based Parallelism with Python conference November 2019
Kokkos 3: Programming Model Extensions for the Exascale Era journal January 2021
Toeplitz-based iterative image reconstruction for MRI with correction for magnetic field inhomogeneity journal September 2005
Experiences with Cross-Facility Real-Time Light Source Data Analysis Workflows conference November 2021
Femtosecond structural dynamics drives the trans/cis isomerization in photoactive yellow protein journal May 2016
A survey of software implementations used by application codes in the Exascale Computing Project journal June 2021
Data systems for the Linac coherent light source journal January 2017
Integrated Research Infrastructure Architecture Blueprint Activity (Final Report 2023) report July 2023
Mapping the conformational landscape of a dynamic enzyme by multitemperature and XFEL crystallography journal September 2015

Similar Records

Real-Time XFEL Data Analysis at SLAC and NERSC: a Trial Run of Nascent Exascale Experimental Data Analysis
Journal Article · Mon Jun 21 00:00:00 EDT 2021 · OSTI ID:1827927
Real‐time XFEL data analysis at SLAC and NERSC: A trial run of nascent exascale experimental data analysis
Journal Article · Mon Feb 12 19:00:00 EST 2024 · Concurrency and Computation. Practice and Experience · OSTI ID:2322459

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
97 MATHEMATICS AND COMPUTING
Serial Femtosecond Crystallography
Single Particle Imaging
data-intensive
exascale
hardware acceleration
high-performance computing
interfacility
real-time processing