skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers

Abstract

Scientific applications and experimental facilities generate massive data sets that need to be transferred to remote collaborating sites for sharing, processing, and long term storage. In order to support increasingly data-intensive science, next generation research networks have been deployed to provide high-speed on-demand data access between collaborating institutions. In this paper, we present a practical model for online data scheduling in which data movement operations are scheduled in advance for end-to-end high performance transfers. In our model, data scheduler interacts with reservation managers and data transfer nodes in order to reserve available bandwidth to guarantee completion of jobs that are accepted and confirmed to satisfy preferred time constraint given by the user. Our methodology improves current systems by allowing researchers and higher level meta-schedulers to use data placement as a service where theycan plan ahead and reserve the scheduler time in advance for their data movement operations. We have implemented our algorithm and examined possible techniques for incorporation into current reservation frameworks. Performance measurements confirm that the proposed algorithm is efficient and scalable.

Authors:
;
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
Computational Research Division
OSTI Identifier:
1050437
Report Number(s):
LBNL-5023E
TRN: US201218%%1001
DOE Contract Number:  
DE-AC02-05CH11231
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICAL METHODS AND COMPUTING; ALGORITHMS; PERFORMANCE; PROCESSING; STORAGE; DATA ACQUISITION; DATA BASE MANAGEMENT; DATA PROCESSING; Advance Reservation, Resource Allocation, Scheduling, Data Management, Distributed Computing

Citation Formats

Balman, Mehmet, and Kosar, Tevfik. An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers. United States: N. p., 2010. Web. doi:10.2172/1050437.
Balman, Mehmet, & Kosar, Tevfik. An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers. United States. doi:10.2172/1050437.
Balman, Mehmet, and Kosar, Tevfik. Thu . "An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers". United States. doi:10.2172/1050437. https://www.osti.gov/servlets/purl/1050437.
@article{osti_1050437,
title = {An Online Scheduling Algorithm with Advance Reservation for Large-Scale Data Transfers},
author = {Balman, Mehmet and Kosar, Tevfik},
abstractNote = {Scientific applications and experimental facilities generate massive data sets that need to be transferred to remote collaborating sites for sharing, processing, and long term storage. In order to support increasingly data-intensive science, next generation research networks have been deployed to provide high-speed on-demand data access between collaborating institutions. In this paper, we present a practical model for online data scheduling in which data movement operations are scheduled in advance for end-to-end high performance transfers. In our model, data scheduler interacts with reservation managers and data transfer nodes in order to reserve available bandwidth to guarantee completion of jobs that are accepted and confirmed to satisfy preferred time constraint given by the user. Our methodology improves current systems by allowing researchers and higher level meta-schedulers to use data placement as a service where theycan plan ahead and reserve the scheduler time in advance for their data movement operations. We have implemented our algorithm and examined possible techniques for incorporation into current reservation frameworks. Performance measurements confirm that the proposed algorithm is efficient and scalable.},
doi = {10.2172/1050437},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2010},
month = {5}
}