A Workflow-based Intelligent Network Data Movement Advisor with End-to-end Performance Optimization
- Southern Illinois Univ., Carbondale, IL (United States)
- Univ. of Memphis, TN (United States)
Next-generation eScience applications often generate large amounts of simulation, experimental, or observational data that must be shared and managed by collaborative organizations. Advanced networking technologies and services have been rapidly developed and deployed to facilitate such massive data transfer. However, these technologies and services have not been fully utilized mainly because their use typically requires significant domain knowledge and in many cases application users are even not aware of their existence. By leveraging the functionalities of an existing Network-Aware Data Movement Advisor (NADMA) utility, we propose a new Workflow-based Intelligent Network Data Movement Advisor (WINDMA) with end-to-end performance optimization for this DOE funded project. This WINDMA system integrates three major components: resource discovery, data movement, and status monitoring, and supports the sharing of common data movement workflows through account and database management. This system provides a web interface and interacts with existing data/space management and discovery services such as Storage Resource Management, transport methods such as GridFTP and GlobusOnline, and network resource provisioning brokers such as ION and OSCARS. We demonstrate the efficacy of the proposed transport-support workflow system in several use cases based on its implementation and deployment in DOE wide-area networks.
- Research Organization:
- Southern Illinois Univ., Carbondale, IL (United States)
- Sponsoring Organization:
- USDOE SC Office of Advanced Scientific Computing Research (SC-21)
- DOE Contract Number:
- SC0002078
- OSTI ID:
- 1104532
- Report Number(s):
- DOE-SIUC-0002078-1
- Country of Publication:
- United States
- Language:
- English
Similar Records
Future SDN-HPON Control Plane Architecture and Protocol for On-Demand Terabit End-to-End Extreme-Scale Science Applications
Integrated End-to-end Performance Prediction and Diagnosis for Extreme Scientific Workflows