skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: On Performance Prediction of Big Data Transfer in High-performance Networks

Abstract

Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that ourmore » predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.« less

Authors:
 [1];  [2];  [1]; ORCiD logo [3];  [4];  [5]
  1. New Jersey Institute of Technology
  2. Harrisburg University
  3. ORNL
  4. Northwest University, China
  5. Zhejiang Sci-Tech University
Publication Date:
Research Org.:
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Sponsoring Org.:
USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
OSTI Identifier:
1648894
DOE Contract Number:  
AC05-00OR22725
Resource Type:
Conference
Resource Relation:
Conference: IEEE International Conference on Communication (ICC) - Virtual, , Ireland - 6/7/2020 8:00:00 AM-6/11/2020 8:00:00 AM
Country of Publication:
United States
Language:
English

Citation Formats

Liu, Wuji, Yun, Daqing, Wu, Chase Qishi, Rao, Nageswara S., Hou, Aiqin, and Shen, Wei. On Performance Prediction of Big Data Transfer in High-performance Networks. United States: N. p., 2020. Web.
Liu, Wuji, Yun, Daqing, Wu, Chase Qishi, Rao, Nageswara S., Hou, Aiqin, & Shen, Wei. On Performance Prediction of Big Data Transfer in High-performance Networks. United States.
Liu, Wuji, Yun, Daqing, Wu, Chase Qishi, Rao, Nageswara S., Hou, Aiqin, and Shen, Wei. Mon . "On Performance Prediction of Big Data Transfer in High-performance Networks". United States. https://www.osti.gov/servlets/purl/1648894.
@article{osti_1648894,
title = {On Performance Prediction of Big Data Transfer in High-performance Networks},
author = {Liu, Wuji and Yun, Daqing and Wu, Chase Qishi and Rao, Nageswara S. and Hou, Aiqin and Shen, Wei},
abstractNote = {Big data generated by large-scale scientific and industrial applications need to be transferred between different geographical locations for remote storage, processing, and analysis. High-speed dedicated connections provisioned in High-performance Networks (HPNs) are increasingly utilized to carry out such big data transfer. HPN management highly relies on an important capability of performance (mainly throughput) prediction to reserve sufficient bandwidth and meanwhile avoid over-provisioning that may result in unnecessary resource waste. This capability is critical to improving the resource (mainly bandwidth) utilization of dedicated connections and meeting various user requests for data transfer. Conventional methods conduct performance prediction by fitting prior observed transfer history with predefined loss functions, without considering unobservable latent factors such as competing loads on end hosts. Such latent factors also have a significant impact on the application-level data transfer performance, which may result in an inaccurate prediction model. In this paper, we first investigate the impact of latent factors and propose a clustering-based method to eliminate their negative impact on performance prediction. We then develop a robust machine learning-based performance predictor by: i) incorporating the proposed latent factor elimination method into data preprocessing, and ii) adopting a customized domain guided loss function. Extensive experimental results show that our predictor achieves significantly higher prediction accuracy than several other state-of-the-art methods.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2020},
month = {6}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: