An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers

Technical Report · Thu May 20 00:00:00 UTC 2010

DOI:https://doi.org/10.2172/1050437· OSTI ID:1050437

Balman, Mehmet ^[1]; Kosar, Tevfik ^[1]

---

1. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Scientific applications and experimental facilities generate massive data sets that need to be transferred to remote collaborating sites for sharing, processing, and long term storage. In order to support increasingly data-intensive science, next generation research networks have been deployed to provide high-speed on-demand data access between collaborating institutions. In this paper, we present a practical model for online data scheduling in which data movement operations are scheduled in advance for end-to-end high performance transfers. In our model, data scheduler interacts with reservation managers and data transfer nodes in order to reserve available bandwidth to guarantee completion of jobs that are accepted and confirmed to satisfy preferred time constraint given by the user. Our methodology improves current systems by allowing researchers and higher level meta-schedulers to use data placement as a service where theycan plan ahead and reserve the scheduler time in advance for their data movement operations. We have implemented our algorithm and examined possible techniques for incorporation into current reservation frameworks. Performance measurements confirm that the proposed algorithm is efficient and scalable.

• Research Organization: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC02-05CH11231
• OSTI ID: 1050437
• Report Number(s): LBNL--5023E
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Advance Reservation
ALGORITHMS
DATA ACQUISITION
DATA BASE MANAGEMENT
Data Management
DATA PROCESSING
Distributed Computing
PERFORMANCE
PROCESSING
Resource Allocation
Scheduling
STORAGE