Bridging the gap between peak and average loads on science networks
Backbone networks are typically overprovisioned in order to support peak loads. Research and education networks (RENs), for example, are often designed to operate at 20–30% of capacity. Thus, Internet2 upgrades its backbone interconnects when the weekly 95th-percentile load is reliably above 30% of link capacity, and analysis of ESnet traffic between major laboratories shows a substantial gap between peak and average utilization. As science data volumes increase exponentially, it is unclear whether this overprovisioning trend can continue into the future. Even if overprovisioning is possible, it may not be the most cost-effective (and desirable) approach going forward. Under the current mode of free access to RENs, traffic at peak load may include both flows that need to be transferred in near-real time–for example, for computation and instrument monitoring and steering–and flows that are less time-critical, for example, archival and storage replication operations. Thus, peak load does not necessarily indicate the capacity that is absolutely required at that moment. We thus examine how data transfers are impacted when the average network load is increased while the network capacity is kept at the current levels. We also classify data transfers into on-demand (time-critical) and best-effort (less time-critical) and study the impact onmore »
- Authors:
-
[1]
;
[2]
; [3]
; [4]
- Univ. of Chicago, Chicago, IL (United States)
- Hongik Univ. (South Korea)
- Argonne National Lab. (ANL), Lemont, IL (United States)
- Univ. of Chicago, Chicago, IL (United States); Argonne National Lab. (ANL), Lemont, IL (United States)
- Publication Date:
- Grant/Contract Number:
- AC02-06CH11357
- Type:
- Accepted Manuscript
- Journal Name:
- Future Generations Computer Systems
- Additional Journal Information:
- Journal Volume: 79; Journal Issue: P1; Journal ID: ISSN 0167-739X
- Publisher:
- Elsevier
- Research Org:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Org:
- USDOE Office of Science (SC)
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; Network utilization; Data transfer scheduling; Network planning
- OSTI Identifier:
- 1421948
Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, and Foster, Ian. Bridging the gap between peak and average loads on science networks. United States: N. p.,
Web. doi:10.1016/j.future.2017.05.012.
Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, & Foster, Ian. Bridging the gap between peak and average loads on science networks. United States. doi:10.1016/j.future.2017.05.012.
Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, and Foster, Ian. 2017.
"Bridging the gap between peak and average loads on science networks". United States.
doi:10.1016/j.future.2017.05.012. https://www.osti.gov/servlets/purl/1421948.
@article{osti_1421948,
title = {Bridging the gap between peak and average loads on science networks},
author = {Nickolay, Sam and Jung, Eun -Sung and Kettimuthu, Rajkumar and Foster, Ian},
abstractNote = {Backbone networks are typically overprovisioned in order to support peak loads. Research and education networks (RENs), for example, are often designed to operate at 20–30% of capacity. Thus, Internet2 upgrades its backbone interconnects when the weekly 95th-percentile load is reliably above 30% of link capacity, and analysis of ESnet traffic between major laboratories shows a substantial gap between peak and average utilization. As science data volumes increase exponentially, it is unclear whether this overprovisioning trend can continue into the future. Even if overprovisioning is possible, it may not be the most cost-effective (and desirable) approach going forward. Under the current mode of free access to RENs, traffic at peak load may include both flows that need to be transferred in near-real time–for example, for computation and instrument monitoring and steering–and flows that are less time-critical, for example, archival and storage replication operations. Thus, peak load does not necessarily indicate the capacity that is absolutely required at that moment. We thus examine how data transfers are impacted when the average network load is increased while the network capacity is kept at the current levels. We also classify data transfers into on-demand (time-critical) and best-effort (less time-critical) and study the impact on both classes for different proportions of both the number of on-demand transfers and amount of bandwidth allocated for on-demand transfers. For our study, we use real transfer logs from production GridFTP servers to do simulation-based experiments as well as real experiments on a testbed. We find that when the transfer load is doubled and the network capacity is fixed at the current level, the gap between peak and average throughput decreases by an average of 18% in the simulation experiments and 16% in the testbed experiments, and the average slowdown experienced by the data transfers is under 1.5×. Moreover, when transfers are classified as on-demand or best-effort, on-demand transfers experience almost no slowdown and the mean slowdown experienced by best-effort transfers is under 2× in the simulation experiments and under 1.2× in the testbed experiments.},
doi = {10.1016/j.future.2017.05.012},
journal = {Future Generations Computer Systems},
number = P1,
volume = 79,
place = {United States},
year = {2017},
month = {5}
}
- Free Publicly Accessible Full Text
-
Accepted Manuscript (DOE)
- Publisher's Version of Record
- https://doi.org/10.1016/j.future.2017.05.012