Achieving High Performance With TCP Over 40 GbE on NUMA Architectures for CMS Data Acquisition
TCP and the socket abstraction have barely changed over the last two decades, but at the network layer there has been a giant leap from a few megabits to 100 gigabits in bandwidth. At the same time, CPU architectures have evolved into the multicore era and applications are expected to make full use of all available resources. Applications in the data acquisition domain based on the standard socket library running in a Non-Uniform Memory Access (NUMA) architecture are unable to reach full efficiency and scalability without the software being adequately aware about the IRQ (Interrupt Request), CPU and memory affinities. During the first long shutdown of LHC, the CMS DAQ system is going to be upgraded for operation from 2015 onwards and a new software component has been designed and developed in the CMS online framework for transferring data with sockets. This software attempts to wrap the low-level socket library to ease higher-level programming with an API based on an asynchronous event driven model similar to the DAT uDAPL API. It is an event-based application with NUMA optimizations, that allows for a high throughput of data across a large distributed system. This paper describes the architecture, the technologies involved and the performance measurements of the software in the context of the CMS distributed event building.
- Research Organization:
- Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC), High Energy Physics (HEP) (SC-25)
- DOE Contract Number:
- AC02-07CH11359
- OSTI ID:
- 1346391
- Report Number(s):
- FERMILAB-CONF-14-594-CMS; 1367397
- Conference Information:
- Journal Name: IEEE Trans.Nucl.Sci. Journal Issue: 3 Journal Volume: 62
- Country of Publication:
- United States
- Language:
- English
Similar Records
NUMA-Aware Thread Scheduling for Big Data Transfers over Terabits Network Infrastructure
Critical Path-Based Thread Placement for NUMA Systems
Empirical Memory-Access Cost Models in Multicore NUMA Architectures
Journal Article
·
Sun May 06 20:00:00 EDT 2018
· Scientific Programming
·
OSTI ID:1565699
Critical Path-Based Thread Placement for NUMA Systems
Journal Article
·
Sat Dec 31 23:00:00 EST 2011
· Performance Evaluation Review
·
OSTI ID:1048161
Empirical Memory-Access Cost Models in Multicore NUMA Architectures
Conference
·
Fri Dec 31 23:00:00 EST 2010
·
OSTI ID:1011076