Parallelism in System Tools

Conference · Fri Jan 01 04:00:00 EST 2010

OSTI ID:988745

Shipman, Galen M ^[1]

---

1. ORNL

The Cray XT, when employed in conjunction with the Lustre filesystem, has provided the ability to generate huge amounts of data in the form of many files. Typically, this is accommodated by satisfying the requests of large numbers of Lustre clients in parallel. In contrast, a single service node (Lustre client) cannot adequately service such datasets. This means that the use of traditional UNIX tools like cp, tar, et alli (with have no parallel capability) can result in substantial impact to user productivity. For example, to copy a 10 TB dataset from the service node using cp would take about 24 hours, under more or less ideal conditions. During production operation, this could easily extend to 36 hours. In this paper, we introduce the Lustre User Toolkit for Cray XT, developed at the Oak Ridge Leadership Computing Facility (OLCF). We will show that Linux commands, implementing highly parallel I/O algorithms, provide orders of magnitude greater performance, greatly reducing impact to productivity.

• Research Organization: Oak Ridge National Laboratory (ORNL); Center for Computational Sciences
• Sponsoring Organization: SC USDOE - Office of Science (SC)
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 988745
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
ALGORITHMS
CRAY COMPUTERS
Cray XT
L CODES
PARALLEL PROCESSING
PERFORMANCE
PRODUCTIVITY
TIME DEPENDENCE
cp
parallel
pltar
spdcp
system tools
tar