Scalable Memory Registration for High-Performance Networks Using Helper Threads
- ORNL
- Virginia Polytechnic Institute and State University (Virginia Tech)
- FORTH-ICS
- Lawrence Livermore National Laboratory (LLNL)
Remote DMA (RDMA) enables high performance networks to reduce data copying between an application and the operating system (OS). However RDMA operations in some high performance networks require communication memory explicitly registered with the network adapter and pinned by the OS. Memory registration and pinning limits the flexibility of the memory system and reduces the amount of memory that user processes can allocate. These issues become more significant on multicore platforms, since registered memory demand grows linearly with the number of processor cores. In this paper we propose a new memory registration/deregistration strategy to reduce registered memory on multicore architectures for HPC applications. We hide the cost of dynamic memory management by offloading all dynamic memory registration and deregistration requests to a dedicated memory management helper thread. We investigate design policies and performance implications of the helper thread approach. We evaluate our framework with the NAS parallel benchmarks, for which our registration scheme significantly reduces the registered memory (23.62% on average and up to 49.39%) and avoids memory registration/deregistration costs for reused communication memory. We show that our system enables the execution of problem sizes that could not complete under existing memory registration strategies.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- DE-AC05-00OR22725
- OSTI ID:
- 1036630
- Resource Relation:
- Conference: In Proceedings of ACM International Conference on Computer Frontier (CF), Unknown, Italy, 20110503, 20110505
- Country of Publication:
- United States
- Language:
- English
Similar Records
Critical Path-Based Thread Placement for NUMA Systems
Energy-aware Thread and Data Management in Heterogeneous Multi-core, Multi-memory Systems