Toward an Autonomous Workflow for Single Crystal Neutron Diffraction

Yin, Junqi; Zhang, Guannan; Cao, Huibo; Dash, Sajal; Chakoumakos, Bryan C.; Wang, Feiyi

doi:10.1007/978-3-031-23606-8_15

Title: Toward an Autonomous Workflow for Single Crystal Neutron Diffraction

Full Record
Other Related Research

Abstract

The operation of the neutron facility relies heavily on beamline scientists. Some experiments can take one or two days with experts making decisions along the way. Leveraging the computing power of HPC platforms and AI advances in image analyses, here we demonstrate an autonomous workflow for the single-crystal neutron diffraction experiments. The workflow consists of three components: an inference service that provides real-time AI segmentation on the image stream from the experiments conducted at the neutron facility, a continuous integration service that launches distributed training jobs on Summit to update the AI model on newly collected images, and a frontend web service to display the AI tagged images to the expert. Ultimately, the feedback can be directly fed to the equipment at the edge in deciding the next-step experiment without requiring an expert in the loop. With the analyses of the requirements and benchmarks of the performance for each component, this effort serves as the first step toward an autonomous workflow for real-time experiment steering at ORNL neutron facilities.

Authors:

^[1];

^[1]; Dash, Sajal ^[1];

^[1];

^[1]

ORNL

Publication Date:: 2023-01-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

OSTI Identifier:: 1922312

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Journal Volume: 1690; Conference: Smoky Mountains Computational Sciences and Engineering Conference - Kingsport, Tennessee, United States of America - 8/23/2022 8:00:00 AM-8/25/2022 8:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Yin, Junqi, Zhang, Guannan, Cao, Huibo, Dash, Sajal, Chakoumakos, Bryan C., and Wang, Feiyi. Toward an Autonomous Workflow for Single Crystal Neutron Diffraction.  United States: N. p., 2023. 
        Web.  doi:10.1007/978-3-031-23606-8_15.

Copy to clipboard


                    Yin, Junqi, Zhang, Guannan, Cao, Huibo, Dash, Sajal, Chakoumakos, Bryan C., & Wang, Feiyi. Toward an Autonomous Workflow for Single Crystal Neutron Diffraction.  United States.  https://doi.org/10.1007/978-3-031-23606-8_15

Copy to clipboard


                    Yin, Junqi, Zhang, Guannan, Cao, Huibo, Dash, Sajal, Chakoumakos, Bryan C., and Wang, Feiyi. 2023.  
        "Toward an Autonomous Workflow for Single Crystal Neutron Diffraction".  United States.  https://doi.org/10.1007/978-3-031-23606-8_15.  https://www.osti.gov/servlets/purl/1922312.

Copy to clipboard


                    
@article{osti_1922312,

  title        = {Toward an Autonomous Workflow for Single Crystal Neutron Diffraction},

  author       = {Yin, Junqi and Zhang, Guannan and Cao, Huibo and Dash, Sajal and Chakoumakos, Bryan C. and Wang, Feiyi},

  abstractNote = {The operation of the neutron facility relies heavily on beamline scientists. Some experiments can take one or two days with experts making decisions along the way. Leveraging the computing power of HPC platforms and AI advances in image analyses, here we demonstrate an autonomous workflow for the single-crystal neutron diffraction experiments. The workflow consists of three components: an inference service that provides real-time AI segmentation on the image stream from the experiments conducted at the neutron facility, a continuous integration service that launches distributed training jobs on Summit to update the AI model on newly collected images, and a frontend web service to display the AI tagged images to the expert. Ultimately, the feedback can be directly fed to the equipment at the edge in deciding the next-step experiment without requiring an expert in the loop. With the analyses of the requirements and benchmarks of the performance for each component, this effort serves as the first step toward an autonomous workflow for real-time experiment steering at ORNL neutron facilities.},

  doi          = {10.1007/978-3-031-23606-8_15},

  url          = {https://www.osti.gov/biblio/1922312},
  journal      = {},

  issn         = {1865--0929},

  number       = ,

  volume       = 1690,

  place        = {United States},

  year         = {2023},

  month        = {1}

}

Copy to clipboard

Conference:

View Conference (4.80 MB)

https://doi.org/10.1007/978-3-031-23606-8_15

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Highly interactive, steered scientific workflows on HPC systems: optimizing design solutions

Conference Ossyra, John; Sedova, Ada; Baker, Matthew; ...

Scientific workflows are becoming increasingly important in high performance computing (HPC) settings, as the feasibility and appeal of many simultaneous heterogeneous tasks increases with increasing hardware capabilities. Currently no HPC-based workflow platform supports a dynamically adaptable workflow with interactive steering and analysis at run-time. Furthermore, for most workflow programs, compute resources are fixed for a given instance, resulting in a possible waste of expensive allocation resources when tasks are spawned and killed. Here we describe the design and testing of a run-time-interactive, adaptable, steered workflow tool capable of executing thousands of parallel tasks without an MPI programming model, using amore »« less
https://doi.org/10.1007/978-3-030-34356-9_39

Full Text Available
Toward designing effective exascale scientific computing workflows: experiences and best practices

Conference Coletti, Mark; Davidson, Russ; Sedova, Ada

Many fields within scientific computing have embraced advances in big-data analysis and machine learning, which often requires the deployment of large, distributed and complicated workflows that may combine training neural networks, performing simulations, running inference, and performing database queries and data analysis in asynchronous, parallel and pipelined execution frameworks. Such a shift has brought into focus the need for scalable, efficient workflow management solutions with reproducibility, error and provenance handling, traceability, and checkpoint-restart capabilities, among other needs. Here, we discuss challenges and best-practices for deploying exascale-generation computational science workflows on resources at the Oak Ridge Leadership Computing Facility (OLCF). Wemore »« less
Full Text Available
Three practical workflow schedulers for easy maximum parallelism

Journal Article Rogers, David - Software, Practice and Experience

Runtime scheduling and workflow systems are an increasingly popular algorithmic component in HPC because they allow full system utilization with relaxed synchronization requirements. There are so many special-purpose tools for task scheduling, one might wonder why more are needed. Use cases seen on the Summit supercomputer needed better integration with MPI and greater flexibility in job launch configurations. Preparation, execution, and analysis of computational chemistry simulations at the scale of tens of thousands of processors revealed three distinct workflow patterns. A separate job scheduler was implemented for each one using extremely simple and robust designs: file-based, task-list based, and bulk-synchronous.more »« less
https://doi.org/10.1002/spe.3047

Full Text Available
Workflow Submit Nodes as a Service on Leadership Class Systems

Conference Papadimitriou, George; Vahi, Karn; Kincl, Jason; ...

DOE scientists, today, have access to high performance computing (HPC) facilities with very powerful systems that enable them to execute their computations faster, more efficiently, and at greater scales than ever before. To further their knowledge and produce new discoveries, scientists rely on workflows - sometimes very complex - that provide them with an easy way to automate, reproduce and verify their computations. However, historically, creating workflow submission environments in large HPC facilities has been cumbersome, requires expertise and many man-hours of effort due to the peculiarities, policies, and the restrictions that these systems present. In this paper we discussmore »« less
Full Text Available
Porting Adaptive Ensemble Molecular Dynamics Workflows to the Summit Supercomputer

Conference Ossyra, John; Sedova, Ada; Tharrington, Arnold; ...

Molecular dynamics (MD) simulations must take very small (femtosecond) integration steps in simulation-time to avoid numerical errors. Efficient use of parallel programming models and accelerators in state-of-the art MD programs now is pushing Moore’s limit for time-per-MD step. As a result, directly simulating timescales beyond milliseconds will not be attainable directly, even at exascale. However, concepts from statistical physics can be used to combine many parallel simulations to provide information about longer timescales and to adequately sample the simulation space, while preserving details about the dynamics of the system. Implementing such an approach requires a workflow program that allows adaptablemore »« less
https://doi.org/10.1007/978-3-030-34356-9_30

Full Text Available

Similar Records