Evaluation of Active Storage Strategies for the Lustre Parallel File System
Active Storage provides an opportunity for reducing the amount of data movement between storage and compute nodes of a parallel filesystem such as Lustre, PVFS, etc. It allows certain types of data processing operations to be performed directly on the storage nodes of modern parallel filesystems, near the data that they manage. This is possible by exploiting the underutilized processor and memory resources of storage nodes that are implemented using general purpose servers and operating systems. In this paper, we present a novel user-space implementation of Active Storage for Lustre, and compare it to the traditional kernel-based implementation. Based on microbenchmark and application level evaluation, we show that both approaches can reduce the network traffic, and take advantage of the extra computing capacity offered by the storage nodes at the same time. However, our user-space approach has proved to be faster, more flexible, portable, and readily deployable than the kernel-space version.
- Research Organization:
- Pacific Northwest National Laboratory (PNNL), Richland, WA (US)
- Sponsoring Organization:
- USDOE
- DOE Contract Number:
- AC05-76RL01830
- OSTI ID:
- 992022
- Report Number(s):
- PNNL-SA-56242; KJ0403000
- Country of Publication:
- United States
- Language:
- English
Similar Records
Certification of Completion of Level-2 Milestone 464: Complete Phase 1 Integration of Site-Wide Global Parallel File System (SWGPFS)
Parallelism in System Tools
Monitoring Extreme-scale Lustre Toolkit
Technical Report
·
Mon Jan 23 23:00:00 EST 2006
·
OSTI ID:898470
Parallelism in System Tools
Conference
·
Thu Dec 31 23:00:00 EST 2009
·
OSTI ID:988745
Monitoring Extreme-scale Lustre Toolkit
Conference
·
Wed Dec 31 23:00:00 EST 2014
·
OSTI ID:1185971