skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Final Report: An Adaptive End-To-End Approach For Terabit Data Movement Optimization

Abstract

Final report for the Adaptive End-to-End Approach for Terabit Data Movement Optimization project (Terabit).

Authors:
 [1]
  1. Univ. of Tennessee, Knoxville, TN (United States)
Publication Date:
Research Org.:
Univ. of Tennessee, Knoxville, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1373561
Report Number(s):
DOE-UTK-ER26066
DOE Contract Number:
SC0007342
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING

Citation Formats

Bosilca, George. Final Report: An Adaptive End-To-End Approach For Terabit Data Movement Optimization. United States: N. p., 2017. Web. doi:10.2172/1373561.
Bosilca, George. Final Report: An Adaptive End-To-End Approach For Terabit Data Movement Optimization. United States. doi:10.2172/1373561.
Bosilca, George. 2017. "Final Report: An Adaptive End-To-End Approach For Terabit Data Movement Optimization". United States. doi:10.2172/1373561. https://www.osti.gov/servlets/purl/1373561.
@article{osti_1373561,
title = {Final Report: An Adaptive End-To-End Approach For Terabit Data Movement Optimization},
author = {Bosilca, George},
abstractNote = {Final report for the Adaptive End-to-End Approach for Terabit Data Movement Optimization project (Terabit).},
doi = {10.2172/1373561},
journal = {},
number = ,
volume = ,
place = {United States},
year = 2017,
month = 8
}

Technical Report:

Save / Share:
  • This final project report describes the accomplishments, products and publications from the award. It includes the overview of the project goals to devise a framework for managing resources in multi-domain, multi-layer networks, as well the details of the mathematical problem formulation and the description of the prototype built to prove the concept.
  • This report describes our accomplishments and activities for the project titled Terabit-Scale Hybrid Networking. The key accomplishment is that we developed, tested and deployed an Alpha Flow Characterization System (AFCS) in ESnet. It is being run in production mode since Sept. 2015. Also, a new QoS class was added to ESnet5 to support alpha flows.
  • A very general and robust approach to solving continuous-variable optimization problems involving uncertainty in the objective function is through the use of ordinal optimization. At each step in the optimization problem, improvement is based only on a relative ranking of the uncertainty effects on local design alternatives, rather than on precise quantification of the effects. One simply asks ''Is that alternative better or worse than this one?'' -not ''HOW MUCH better or worse is that alternative to this one?'' The answer to the latter question requires precise characterization of the uncertainty--with the corresponding sampling/integration expense for precise resolution. However, inmore » this report we demonstrate correct decision-making in a continuous-variable probabilistic optimization problem despite extreme vagueness in the statistical characterization of the design options. We present a new adaptive ordinal method for probabilistic optimization in which the trade-off between computational expense and vagueness in the uncertainty characterization can be conveniently managed in various phases of the optimization problem to make cost-effective stepping decisions in the design space. Spatial correlation of uncertainty in the continuous-variable design space is exploited to dramatically increase method efficiency. Under many circumstances the method appears to have favorable robustness and cost-scaling properties relative to other probabilistic optimization methods, and uniquely has mechanisms for quantifying and controlling error likelihood in design-space stepping decisions. The method is asymptotically convergent to the true probabilistic optimum, so could be useful as a reference standard against which the efficiency and robustness of other methods can be compared--analogous to the role that Monte Carlo simulation plays in uncertainty propagation.« less
  • Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant domain knowledge and in many cases application users are even not aware of their existence. By leveraging the functionalities of an existing Network-Aware Data Movement Advisor (NADMA) utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization formore » this DOE funded project. This WINDMA system integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database management. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support workflow system in several use cases based on its implementation and deployment in DOE wide-area networks.« less