Software-Driven Network Architecture for Synchronous Data Acquisition
- Crossfield Technology LLC
Experiments at facilities such as the Facility for Rare Isotope Beams (FRIB) involve thousands of detector elements that produce raw experimental data at rates in excess of a GB/sec using new Flash ADC (FADC) based detectors. Data acquisition systems convert detector signals into a digital form in real-time. The data from each detector element is labeled with a precisely synchronized timestamp and transmitted to buffers. The total charge, the number of coincident elements, or other information summaries are used to determine whether something interesting has happened, which constitutes a trigger. If justified by the trigger, the data from these elements is assembled into a time-correlated event for later analysis, a process called Event Building. Event rates of 1 to 10 kevents/sec are anticipated in certain FRIB experiments. Crossfield Technology developed a software-driven system architecture for data acquisition and Event Building. The architecture features an Instrumentation Gateway (IG) that supports time-synchronous data acquisition, real-time digital signal processing, and data streaming across a low-latency, high-performance network to a High Performance Computer (HPC). The programmable IG can capture FADC data in real-time and implement a wide variety of algorithms for real-time Event Building. The Instrumentation Gateway is based on a highly integrated Field Programmable Gate Array (FPGA) System on Chip (SoC) with built-in quad-core ARM processor. The IG architecture and design is based on industry standards (VITA, IEEE and IBTA) to enable interoperability with common network adapters, switches and HPCs. The IG uses Intel’s Stratix 10 SX FPGA SoC which supports GSPS FADCs and 100G Ethernet networking technologies. The Stratix 10 SX 2800 provides up to 9.2 TFLOPS of Digital Signal Processing (DSP) performance for real-time processing and 229 Mbits of embedded memory for data buffering. The goal of this program was to enable Remote Direct Memory Access (RDMA) over Converged Ethernet (RoCE) for transfer of raw and/or processed FADC data. SoftRoCE is the term for the software based RoCE stack that runs in Linux and handles the InfiniBand transport layer inside the kernel and user space. SoftRoCE is comprised of kernel drivers and user space libraries. The drivers are located in the kernel, while the user space libraries are part of the RDMA Core software distribution. Crossfield adapted SoftRoCE to run on the embedded ARM processor in the Stratix 10 SX FPGA SoC as part of real-time Linux Board Support Package (BSP) for Crossfield’s 6U OpenVPX module. Crossfield’s approach is to run SoftRoCE on the HPS, with data transfers accelerated in the fabric after software establishes RDMA connections and initializes data transfers. An RDMA bypass offloads the actual data transfer in the fabric. The HPS is used to generate parameters for packet headers while the acquired data remains in the fabric for InfiniBand ICRC calculation and packet encapsulation. MATLAB/Simulink and DSP Builder can be used for design, simulation and code generation of real-time embedded algorithms, enabling researchers to rapidly implement new instrument functions and modalities. In addition, instruments can be implemented as Hardware Description Language (HDL) designs (Verilog or VHDL) or with C++ designs using High Level Synthesis.
- Research Organization:
- Crossfield Technology LLC
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- SC0015151
- OSTI ID:
- 1638083
- Type / Phase:
- SBIR (Phase II)
- Report Number(s):
- DOE-CTL-15151
- Country of Publication:
- United States
- Language:
- English
Similar Records
Final Report for Project DE-FC02-06ER25755 [Pmodels2]
Evaluation of CHO Benchmarks on the Arria 10 FPGA using Intel FPGA SDK for OpenCL