Bridging the gap between peak and average loads on science networks

Nickolay, Sam; Jung, Eun -Sung; Kettimuthu, Rajkumar; Foster, Ian

doi:10.1016/j.future.2017.05.012

Title: Bridging the gap between peak and average loads on science networks

Full Record
Other Related Research

Abstract

Backbone networks are typically overprovisioned in order to support peak loads. Research and education networks (RENs), for example, are often designed to operate at 20–30% of capacity. Thus, Internet2 upgrades its backbone interconnects when the weekly 95th-percentile load is reliably above 30% of link capacity, and analysis of ESnet traffic between major laboratories shows a substantial gap between peak and average utilization. As science data volumes increase exponentially, it is unclear whether this overprovisioning trend can continue into the future. Even if overprovisioning is possible, it may not be the most cost-effective (and desirable) approach going forward. Under the current mode of free access to RENs, traffic at peak load may include both flows that need to be transferred in near-real time–for example, for computation and instrument monitoring and steering–and flows that are less time-critical, for example, archival and storage replication operations. Thus, peak load does not necessarily indicate the capacity that is absolutely required at that moment. We thus examine how data transfers are impacted when the average network load is increased while the network capacity is kept at the current levels. We also classify data transfers into on-demand (time-critical) and best-effort (less time-critical) and study the impact onmore »« less

Authors:

Nickolay, Sam ^[1];

^[2]; Kettimuthu, Rajkumar ^[3]; Foster, Ian ^[4]

Univ. of Chicago, Chicago, IL (United States)
Hongik Univ. (South Korea)
Argonne National Lab. (ANL), Lemont, IL (United States)
Univ. of Chicago, Chicago, IL (United States); Argonne National Lab. (ANL), Lemont, IL (United States)

Publication Date:: 2017-05-12

Research Org.:: Argonne National Lab. (ANL), Argonne, IL (United States)

Sponsoring Org.:: USDOE Office of Science (SC)

OSTI Identifier:: 1421948

Alternate Identifier(s):: OSTI ID: 1550040

Grant/Contract Number:: AC02-06CH11357; DEAC02-06CH11357

Resource Type:: Accepted Manuscript

Journal Name:: Future Generations Computer Systems

Additional Journal Information:: Journal Volume: 79; Journal Issue: P1; Journal ID: ISSN 0167-739X

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; Network utilization; Data transfer scheduling; Network planning

Citation Formats


                    Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, and Foster, Ian. Bridging the gap between peak and average loads on science networks.  United States: N. p., 2017. 
Web.  doi:10.1016/j.future.2017.05.012.

Copy to clipboard


                    Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, & Foster, Ian. Bridging the gap between peak and average loads on science networks.  United States.  https://doi.org/10.1016/j.future.2017.05.012

Copy to clipboard


                    Nickolay, Sam, Jung, Eun -Sung, Kettimuthu, Rajkumar, and Foster, Ian. Fri .  
"Bridging the gap between peak and average loads on science networks".  United States.  https://doi.org/10.1016/j.future.2017.05.012.  https://www.osti.gov/servlets/purl/1421948.

Copy to clipboard


                    
@article{osti_1421948,

  title        = {Bridging the gap between peak and average loads on science networks},

  author       = {Nickolay, Sam and Jung, Eun -Sung and Kettimuthu, Rajkumar and Foster, Ian},

  abstractNote = {Backbone networks are typically overprovisioned in order to support peak loads. Research and education networks (RENs), for example, are often designed to operate at 20–30% of capacity. Thus, Internet2 upgrades its backbone interconnects when the weekly 95th-percentile load is reliably above 30% of link capacity, and analysis of ESnet traffic between major laboratories shows a substantial gap between peak and average utilization. As science data volumes increase exponentially, it is unclear whether this overprovisioning trend can continue into the future. Even if overprovisioning is possible, it may not be the most cost-effective (and desirable) approach going forward. Under the current mode of free access to RENs, traffic at peak load may include both flows that need to be transferred in near-real time–for example, for computation and instrument monitoring and steering–and flows that are less time-critical, for example, archival and storage replication operations. Thus, peak load does not necessarily indicate the capacity that is absolutely required at that moment. We thus examine how data transfers are impacted when the average network load is increased while the network capacity is kept at the current levels. We also classify data transfers into on-demand (time-critical) and best-effort (less time-critical) and study the impact on both classes for different proportions of both the number of on-demand transfers and amount of bandwidth allocated for on-demand transfers. For our study, we use real transfer logs from production GridFTP servers to do simulation-based experiments as well as real experiments on a testbed. We find that when the transfer load is doubled and the network capacity is fixed at the current level, the gap between peak and average throughput decreases by an average of 18% in the simulation experiments and 16% in the testbed experiments, and the average slowdown experienced by the data transfers is under 1.5×. Moreover, when transfers are classified as on-demand or best-effort, on-demand transfers experience almost no slowdown and the mean slowdown experienced by best-effort transfers is under 2× in the simulation experiments and under 1.2× in the testbed experiments.},

  doi          = {10.1016/j.future.2017.05.012},

  journal      = {Future Generations Computer Systems},

  number       = P1,

  volume       = 79,

  place        = {United States},

  year         = {2017},

  month        = {5}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.future.2017.05.012

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 2 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Similar Records in DOE PAGES and OSTI.GOV collections:

Distributed Logical Analytic Domains (DLADs) for Distributed HPC Security

Technical Report Ros-Giralt, Jordi

The DOE estimates that the costs of a cyber attack onto its main high performance computing (HPC) facilities could potentially be in the order of billions of US dollars. The same conclusion is reached in the commercial sector, with a recent study by Oxford Economics reporting a combined annualized loss of $12.5 billion in shareholder value for publicly traded companies (more than $50 billion aggregated during the past four years) due to cybersecurity breaches. On average, the cost of cybercrime for organizations is worsening, increasing at an annualized compound rate of 23 percent, according to Accenture’s 2017 report on themore »« less
Making the Traffic Operations Case for Congestion Pricing: Operational Impacts of Congestion Pricing

Technical Report Chin, Shih-Miao ; Hu, Patricia S ; Davidson, Diane

Congestion begins when an excess of vehicles on a segment of roadway at a given time, resulting in speeds that are significantly slower than normal or 'free flow' speeds. Congestion often means stop-and-go traffic. The transition occurs when vehicle density (the number of vehicles per mile in a lane) exceeds a critical level. Once traffic enters a state of congestion, recovery or time to return to a free-flow state is lengthy; and during the recovery process, delay continues to accumulate. The breakdown in speed and flow greatly impedes the efficient operation of the freeway system, resulting in economic, mobility, environmentalmore »« less
https://doi.org/10.2172/1048704

Full Text Available
Survey of U.S. Ancillary Services Markets

Technical Report Zhou, Zhi ; Levin, Todd ; Conzelmann, Guenter

In addition to providing energy to end-consumers, power system operators are also responsible for ensuring system reliability. To this end, power markets maintain an array of ancillary services to ensure that it is always possible to balance the supply and demand for energy in real-time. A subset of these ancillary services are commonly procured through market-based mechanisms: namely, Regulation, Spinning, and Non-spinning Reserves. Regulation Reserves are maintained to respond to supply/demand imbalances over short time frames, typically on the order of several seconds to one minute. Resources that provide Regulation Reserves adjust their generation or load levels in response tomore »« less
https://doi.org/10.2172/1327815

Full Text Available
An Elegant Sufficiency: Load-Aware Differentiated Scheduling of Data Transfers

Conference Kettimuthu, Rajkumar ; Vardoyan, Gayane ; Agrawal, Gagan ; ...

We investigate the file transfer scheduling problem, where transfers among different endpoints must be scheduled to maximize pertinent metrics. We propose two new algorithms that exploit the fact that the aggregate bandwidth obtained over a network or at a storage system tends to increase with the number of concurrent transfers—but only up to a certain limit. The first algorithm, SEAL, uses runtime information and data-driven models to approximate system load and adapt transfer schedules and concurrency so as to maximize performance while avoiding saturation. We implement this algorithm using GridFTP as the transfer protocol and evaluate it using real transfermore »« less
https://doi.org/10.1145/2807591.2807660
Real-Time Traffic Information for Emergency Evacuation Operations: Phase A Final Report

Technical Report Franzese, Oscar ; Zhang, Li ; Mahmoud, Anas M. ; ...

There are many instances in which it is possible to plan ahead for an emergency evacuation (e.g., an explosion at a chemical processing facility). For those cases, if an accident (or an attack) were to happen, then the best evacuation plan for the prevailing network and weather conditions would be deployed. In other cases (e.g., the derailment of a train transporting hazardous materials), there may not be any previously developed plan to be implemented and decisions must be made ad-hoc on how to proceed with an emergency evacuation. In both situations, the availability of real-time traffic information plays a criticalmore »« less
https://doi.org/10.2172/1004652

Full Text Available

Similar Records