Summary: Explicit Control in a Batch­Aware Distributed File System
John Bent, Douglas Thain,
Andrea C. Arpaci­Dusseau, Remzi H. Arpaci­Dusseau, and Miron Livny
Computer Sciences Department, University of Wisconsin, Madison
Abstract
We present the design, implementation, and evaluation
of the Batch­Aware Distributed File System (BAD­FS),
a system designed to orchestrate large, I/O­intensive
batch workloads on remote computing clusters distributed
across the wide area. BAD­FS consists of two novel com­
ponents: a storage layer that exposes control of tradi­
tionally fixed policies such as caching, consistency, and
replication; and a scheduler that exploits this control as
necessary for different workloads. By extracting control
from the storage layer and placing it within an external
scheduler, BAD­FS manages both storage and computa­
tion in a coordinated way while gracefully dealing with
cache consistency, fault­tolerance, and space manage­
ment issues in a workload­specific manner. Using both
microbenchmarks and real workloads, we demonstrate

Source: Arpaci-Dusseau, Andrea - Department of Computer Sciences, University of Wisconsin at Madison
Liblit, Ben - Department of Computer Sciences, University of Wisconsin at Madison
Wisconsin at Madison, University of - Department of Computer Sciences, Condor Project

Collections: Computer Technologies and Information Sciences