skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Software-Driven Network Architecture for Synchronous Data Acquisition

Abstract

Experiments at facilities such as the Facility for Rare Isotope Beams (FRIB) involve thousands of detector elements that produce raw experimental data at rates in excess of a GB/sec using new Flash ADC (FADC) based detectors. Data acquisition systems convert detector signals into a digital form in real-time. The data from each detector element is labeled with a precisely synchronized timestamp and transmitted to buffers. The total charge, the number of coincident elements, or other information summaries are used to determine whether something interesting has happened, which constitutes a trigger. If justified by the trigger, the data from these elements is assembled into a time-correlated event for later analysis, a process called Event Building. Event rates of 1 to 10 kevents/sec are anticipated in certain FRIB experiments. Crossfield Technology developed a software-driven system architecture for data acquisition and Event Building. The architecture features an Instrumentation Gateway (IG) that supports time-synchronous data acquisition, real-time digital signal processing, and data streaming across a low-latency, high-performance network to a High Performance Computer (HPC). The programmable IG can capture FADC data in real-time and implement a wide variety of algorithms for real-time Event Building. The Instrumentation Gateway is based on a highly integrated Fieldmore » Programmable Gate Array (FPGA) System on Chip (SoC) with built-in quad-core ARM processor. The IG architecture and design is based on industry standards (VITA, IEEE and IBTA) to enable interoperability with common network adapters, switches and HPCs. The IG uses Intel’s Stratix 10 SX FPGA SoC which supports GSPS FADCs and 100G Ethernet networking technologies. The Stratix 10 SX 2800 provides up to 9.2 TFLOPS of Digital Signal Processing (DSP) performance for real-time processing and 229 Mbits of embedded memory for data buffering. The goal of this program was to enable Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) for transfer of raw and/or processed FADC data. SoftRoCE is the term for the software based RoCE stack that runs in Linux and handles the InfiniBand transport layer inside the kernel and user space. SoftRoCE is comprised of kernel drivers and user space libraries. The drivers are located in the kernel, while the user space libraries are part of the RDMA Core software distribution. Crossfield adapted SoftRoCE to run on the embedded ARM processor in the Stratix 10 SX FPGA SoC as part of real-time Linux Board Support Package (BSP) for Crossfield’s 6U OpenVPX module. Crossfield’s approach is to run SoftRoCE on the HPS, with data transfers accelerated in the fabric after software establishes RDMA connections and initializes data transfers. An RDMA bypass offloads the actual data transfer in the fabric. The HPS is used to generate parameters for packet headers while the acquired data remains in the fabric for InfiniBand ICRC calculation and packet encapsulation. MATLAB/Simulink and DSP Builder can be used for design, simulation and code generation of real-time embedded algorithms, enabling researchers to rapidly implement new instrument functions and modalities. In addition, instruments can be implemented as Hardware Description Language (HDL) designs (Verilog or VHDL) or with C++ designs using High Level Synthesis.« less

Authors:
 [1];  [1];  [1]
  1. Crossfield Technology LLC
Publication Date:
Research Org.:
Crossfield Technology LLC
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1638083
Report Number(s):
DOE-CTL-15151
DOE Contract Number:  
SC0015151
Type / Phase:
SBIR (Phase II)
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
46 INSTRUMENTATION RELATED TO NUCLEAR SCIENCE AND TECHNOLOGY; Instrumentation, Remote Direct Memory Access, RDMA, RDMA over Converged Ethernet, RoCE, Field Programmable Gate Array, FPGA

Citation Formats

McMillian, Gary, McMillian, Brett, and DeWitt, Michael. Software-Driven Network Architecture for Synchronous Data Acquisition. United States: N. p., 2020. Web.
McMillian, Gary, McMillian, Brett, & DeWitt, Michael. Software-Driven Network Architecture for Synchronous Data Acquisition. United States.
McMillian, Gary, McMillian, Brett, and DeWitt, Michael. Fri . "Software-Driven Network Architecture for Synchronous Data Acquisition". United States.
@article{osti_1638083,
title = {Software-Driven Network Architecture for Synchronous Data Acquisition},
author = {McMillian, Gary and McMillian, Brett and DeWitt, Michael},
abstractNote = {Experiments at facilities such as the Facility for Rare Isotope Beams (FRIB) involve thousands of detector elements that produce raw experimental data at rates in excess of a GB/sec using new Flash ADC (FADC) based detectors. Data acquisition systems convert detector signals into a digital form in real-time. The data from each detector element is labeled with a precisely synchronized timestamp and transmitted to buffers. The total charge, the number of coincident elements, or other information summaries are used to determine whether something interesting has happened, which constitutes a trigger. If justified by the trigger, the data from these elements is assembled into a time-correlated event for later analysis, a process called Event Building. Event rates of 1 to 10 kevents/sec are anticipated in certain FRIB experiments. Crossfield Technology developed a software-driven system architecture for data acquisition and Event Building. The architecture features an Instrumentation Gateway (IG) that supports time-synchronous data acquisition, real-time digital signal processing, and data streaming across a low-latency, high-performance network to a High Performance Computer (HPC). The programmable IG can capture FADC data in real-time and implement a wide variety of algorithms for real-time Event Building. The Instrumentation Gateway is based on a highly integrated Field Programmable Gate Array (FPGA) System on Chip (SoC) with built-in quad-core ARM processor. The IG architecture and design is based on industry standards (VITA, IEEE and IBTA) to enable interoperability with common network adapters, switches and HPCs. The IG uses Intel’s Stratix 10 SX FPGA SoC which supports GSPS FADCs and 100G Ethernet networking technologies. The Stratix 10 SX 2800 provides up to 9.2 TFLOPS of Digital Signal Processing (DSP) performance for real-time processing and 229 Mbits of embedded memory for data buffering. The goal of this program was to enable Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) for transfer of raw and/or processed FADC data. SoftRoCE is the term for the software based RoCE stack that runs in Linux and handles the InfiniBand transport layer inside the kernel and user space. SoftRoCE is comprised of kernel drivers and user space libraries. The drivers are located in the kernel, while the user space libraries are part of the RDMA Core software distribution. Crossfield adapted SoftRoCE to run on the embedded ARM processor in the Stratix 10 SX FPGA SoC as part of real-time Linux Board Support Package (BSP) for Crossfield’s 6U OpenVPX module. Crossfield’s approach is to run SoftRoCE on the HPS, with data transfers accelerated in the fabric after software establishes RDMA connections and initializes data transfers. An RDMA bypass offloads the actual data transfer in the fabric. The HPS is used to generate parameters for packet headers while the acquired data remains in the fabric for InfiniBand ICRC calculation and packet encapsulation. MATLAB/Simulink and DSP Builder can be used for design, simulation and code generation of real-time embedded algorithms, enabling researchers to rapidly implement new instrument functions and modalities. In addition, instruments can be implemented as Hardware Description Language (HDL) designs (Verilog or VHDL) or with C++ designs using High Level Synthesis.},
doi = {},
url = {https://www.osti.gov/biblio/1638083}, journal = {},
number = ,
volume = ,
place = {United States},
year = {2020},
month = {7}
}

Technical Report:
This technical report may be released as soon as July 10, 2024
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that may hold this item. Keep in mind that many technical reports are not cataloged in WorldCat.

Save / Share: