NSTX-U advances in real-time deterministic PCIe-based internode communication

doi:https://doi.org/10.1016/j.fusengdes.2018.02.055

Title: NSTX-U advances in real-time deterministic PCIe-based internode communication

Abstract

Distributing control mechanisms across modern commercial off the shelf (COTS) GNU/Linux systems often introduces difficult to mitigate non-deterministic behavior. Existing methods to address this problem involve non-real-time technologies such as Remote Direct Memory Access (RDMA) over Infiniband or custom Ethernet solutions, such as RDMA over Converged Ethernet (RoCE) or userspace Ethernet drivers, that trade determinism for ease of use or lower cost. The National Spherical Torus Experiment Upgrade (NSTX-U) is pursuing a new design that allows direct communication between heterogeneous systems with scalable, microsecond latency with 1 μs of jitter on that latency, outside of the constant transmission delay at the physical layer. The future design of the NSTX-U Real-time Communication System will utilize direct PCIe-to-PCIe communication with kernel support tuned for low overhead, allowing two (or more, through a switch) real-time (RT) systems to communicate and share resources as one larger entity. As a result, this greatly increases the processing capability of the primary Plasma Control System (PCS), turning previously insurmountable computational challenges into a more manageable divide and conquer parallel task.

Authors:

^[1]; Boyer, M. Dan ^[1]; Higgins, David ^[2]

Princeton Univ. Plasma Physics Lab, Princeton, NJ (United States)
Concurrent Real-Time, Pompano Beach, FL (United States)

Publication Date:: 2018-06-07

Research Org.:: Princeton Plasma Physics Lab. (PPPL), Princeton, NJ (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Fusion Energy Sciences (FES) (SC-24)

OSTI Identifier:: 1460738

Alternate Identifier(s):: OSTI ID: 1582731

Grant/Contract Number:: AC02-09CH11466

Resource Type:: Journal Article: Accepted Manuscript

Journal Name:: Fusion Engineering and Design

Additional Journal Information:: Journal Volume: 133; Journal Issue: C; Journal ID: ISSN 0920-3796

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 70 PLASMA PHYSICS AND FUSION TECHNOLOGY; Real-time Linux; Reflective memory; RDMA; PCIe

Citation Formats


                    Erickson, Keith G., Boyer, M. Dan, and Higgins, David. NSTX-U advances in real-time deterministic PCIe-based internode communication.  United States: N. p., 2018. 
        Web.  doi:10.1016/j.fusengdes.2018.02.055.

Copy to clipboard


                    Erickson, Keith G., Boyer, M. Dan, & Higgins, David. NSTX-U advances in real-time deterministic PCIe-based internode communication.  United States.  https://doi.org/10.1016/j.fusengdes.2018.02.055

Copy to clipboard


                    Erickson, Keith G., Boyer, M. Dan, and Higgins, David. Thu .  
        "NSTX-U advances in real-time deterministic PCIe-based internode communication".  United States.  https://doi.org/10.1016/j.fusengdes.2018.02.055.  https://www.osti.gov/servlets/purl/1460738.

Copy to clipboard


                    
@article{osti_1460738,

  title        = {NSTX-U advances in real-time deterministic PCIe-based internode communication},

  author       = {Erickson, Keith G. and Boyer, M. Dan and Higgins, David},

  abstractNote = {Distributing control mechanisms across modern commercial off the shelf (COTS) GNU/Linux systems often introduces difficult to mitigate non-deterministic behavior. Existing methods to address this problem involve non-real-time technologies such as Remote Direct Memory Access (RDMA) over Infiniband or custom Ethernet solutions, such as RDMA over Converged Ethernet (RoCE) or userspace Ethernet drivers, that trade determinism for ease of use or lower cost. The National Spherical Torus Experiment Upgrade (NSTX-U) is pursuing a new design that allows direct communication between heterogeneous systems with scalable, microsecond latency with 1 μs of jitter on that latency, outside of the constant transmission delay at the physical layer. The future design of the NSTX-U Real-time Communication System will utilize direct PCIe-to-PCIe communication with kernel support tuned for low overhead, allowing two (or more, through a switch) real-time (RT) systems to communicate and share resources as one larger entity. As a result, this greatly increases the processing capability of the primary Plasma Control System (PCS), turning previously insurmountable computational challenges into a more manageable divide and conquer parallel task.},

  doi          = {10.1016/j.fusengdes.2018.02.055},

  url          = {https://www.osti.gov/biblio/1460738},
  journal      = {Fusion Engineering and Design},

  issn         = {0920-3796},

  number       = C,

  volume       = 133,

  place        = {United States},

  year         = {2018},

  month        = {6}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.fusengdes.2018.02.055

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Works referencing / citing this record:

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks
journal, March 2019

Boyer, M. D.; Kaye, S.; Erickson, K.
Nuclear Fusion, Vol. 59, Issue 5
https://doi.org/10.1088/1741-4326/ab0762

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks
journal, March 2019

Boyer, M. D.; Kaye, S.; Erickson, K.
Nuclear Fusion, Vol. 59, Issue 5
https://doi.org/10.1088/1741-4326/ab0762

Similar records in OSTI.GOV collections:

Software-Driven Network Architecture for Synchronous Data Acquisition

Technical Report McMillian, Gary ; McMillian, Brett ; DeWitt, Michael

Experiments at facilities such as the Facility for Rare Isotope Beams (FRIB) involve thousands of detector elements that produce raw experimental data at rates in excess of a GB/sec using new Flash ADC (FADC) based detectors. Data acquisition systems convert detector signals into a digital form in real-time. The data from each detector element is labeled with a precisely synchronized timestamp and transmitted to buffers. The total charge, the number of coincident elements, or other information summaries are used to determine whether something interesting has happened, which constitutes a trigger. If justified by the trigger, the data from these elementsmore »« less
Final Report for Project DE-FC02-06ER25755 [Pmodels2]

Technical Report Panda, Dhabaleswar ; Sadayappan, P.

In this report, we describe the research accomplished by the OSU team under the Pmodels2 project. The team has worked on various angles: designing high performance MPI implementations on modern networking technologies (Mellanox InfiniBand (including the new ConnectX2 architecture and Quad Data Rate), QLogic InfiniPath, the emerging 10GigE/iWARP and RDMA over Converged Enhanced Ethernet (RoCE) and Obsidian IB-WAN), studying MPI scalability issues for multi-thousand node clusters using XRC transport, scalable job start-up, dynamic process management support, efficient one-sided communication, protocol offloading and designing scalable collective communication libraries for emerging multi-core architectures. New designs conforming to the Argonne’s Nemesis interface havemore »« less
https://doi.org/10.2172/1122948

Full Text Available
FELIX: a PCIe based high-throughput approach for interfacing front-end and trigger electronics in the ATLAS Upgrade framework

Journal Article Anderson, J. ; Bauer, K. ; Borga, A. ; ... - Journal of Instrumentation

The ATLAS Phase-I upgrade (2019) requires a Trigger and Data Acquisition (TDAQ) system able to trigger and record data from up to three times the nominal LHC instantaneous luminosity. Furthermore, the Front-End LInk eXchange (FELIX) system provides an infrastructure to achieve this in a scalable, detector agnostic and easily upgradeable way. It is a PC-based gateway, interfacing custom radiation tolerant optical links from front-end electronics, via PCIe Gen3 cards, to a commodity switched Ethernet or InfiniBand network. FELIX enables reducing custom electronics in favour of software running on commercial servers. Here, the FELIX system, the design of the PCIe prototypemore »« less
Cited by 5
https://doi.org/10.1088/1748-0221/11/12/C12023

Full Text Available
Implementation of MP_Lite for the VI Architecture

Thesis/Dissertation Chen, Weiyi

MP_Lite is a light weight message-passing library designed to deliver the maximum performance to applications in a portable and user friendly manner. The Virtual Interface (VI) architecture is a user-level communication protocol that bypasses the operating system to provide much better performance than traditional network architectures. By combining the high efficiency of MP_Lite and high performance of the VI architecture, they are able to implement a high performance message-passing library that has much lower latency and better throughput. The design and implementation of MP_Lite for M-VIA, which is a modular implementation of the VI architecture on Linux, is discussed inmore »« less
https://doi.org/10.2172/803356

Full Text Available
Exploring the All-to-All Collective Optimization Space with ConnectX CORE-Direct

Conference Venkata, Manjunath Gorentla ; Graham, Richard L. ; Ladd, Joshua ; ... - 2012 41st International Conference on Parallel Processing; 10-13 Sept. 2012; Pittsburgh, PA, USA

The all-to-all collective communication operation is used by many scientific applications, and is one of the most time consuming and challenging collective operation to optimize. The algorithms for all-to-all operations typically fall into two classes, logarithmic and linear scaling algorithms, with Bruck's algorithm, a logarithmic scaling algorithm, used in many small-data all-to-all implementations. The recent addition of InfiniBand CORE-Direct support for network management of collective communications offers new opportunities for optimizing all-to-all operation as well as supporting truly asynchronous implementations of these operations. This paper presents several new enhancements to the Bruck small-data algorithm that leverage CORE-Direct and other InfiniBandmore »« less
https://doi.org/10.1109/ICPP.2012.28

Similar Records

Title: NSTX-U advances in real-time deterministic PCIe-based internode communication

Abstract

Citation Formats

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks journal, March 2019

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks journal, March 2019

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks
journal, March 2019

Real-time capable modeling of neutral beam injection on NSTX-U using neural networks
journal, March 2019