Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Yun, Daqing; Liu, Wuji; Wu, Chase Q.; Rao, Nageswara S.V.; Kettimuthu, Rajkumar

doi:10.1016/j.engappai.2021.104285

Title: Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Journal Article · 04 May 2021 · Engineering Applications of Artificial Intelligence

DOI: https://doi.org/10.1016/j.engappai.2021.104285 · OSTI ID:1820703

Yun, Daqing ^[1];

^[2]; Wu, Chase Q. ^[2];

^[3]; Kettimuthu, Rajkumar ^[4]

Harrisburg Univ., Harrisburg, PA (United States)
New Jersey Institute of Technology, Newark, NJ (United States)
Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
Argonne National Lab. (ANL), Argonne, IL (United States)

Big data transfer in large-scale scientific and business applications is increasingly carried out over connections with guaranteed bandwidth provisioned in High-performance Networks (HPNs) via advance bandwidth reservation. Provisioning agents need to carefully schedule data transfer requests, compute network paths, and allocate appropriate bandwidths. Such reserved bandwidths, if not fully utilized, could be simply wasted due to the exclusive access during the approved time window, and cause extra overhead and complexity for resource management. This calls for accurate performance prediction to reserve bandwidths that match actual needs and avoid over-provisioning. We employ machine learning algorithms to predict big data transfer performance based on extensive performance measurements collected in the past several years from data transfer tests using different protocols and toolkits between various end sites on several real-life physical or emulated testbeds. We first analyze the performance patterns in response to a comprehensive list of parameters in end-host systems, network connections, and data transfer applications, which motivate the use of machine learning and also help us identify the effects of latent factors. We then propose threshold- and clustering-based methods to eliminate negative effects of latent factors in data preprocessing and build a robust performance predictor based on customized domain-oriented loss functions. The performance of the proposed methods is verified by extensive experiments using SVR and RFR as well as theoretical analysis of the general performance bound.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1820703

Journal Information:: Engineering Applications of Artificial Intelligence, Vol. 102; ISSN 0952-1976

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (21)

On the predictability of large transfer TCP throughput He, Qi; Dovrolis, Constantinos; Ammar, Mostafa Computer Networks, Vol. 51, Issue 14 https://doi.org/10.1016/j.comnet.2007.04.013	journal	October 2007
Control Plane for Advance Bandwidth Scheduling in Ultra High-Speed Networks Rao, Nageswara S. V.; Wu, Qishi; Ding, Song Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications https://doi.org/10.1109/INFOCOM.2006.35	conference	April 2006
Comparative analysis of big data transfer protocols in an international high-speed network Yu, Se-young; Brownlee, Nevil; Mahanti, Aniket 2015 IEEE 34th International Performance Computing and Communications Conference (IPCCC) https://doi.org/10.1109/PCCC.2015.7410300	conference	December 2015
Profiling Optimization for Big Data Transfer over Dedicated Channels Yun, Daqing; Wu, Chase Q.; Rao, Nageswara S. V. 2016 25th International Conference on Computer Communication and Networks (ICCCN) https://doi.org/10.1109/ICCCN.2016.7568562	conference	August 2016
A Machine Learning Approach to TCP Throughput Prediction Mirza, Mariyam; Sommers, Joel; Barford, Paul IEEE/ACM Transactions on Networking, Vol. 18, Issue 4 https://doi.org/10.1109/TNET.2009.2037812	journal	August 2010
B4: experience with a globally-deployed software defined wan Jain, Sushant; Zhu, Min; Zolla, Jon ACM SIGCOMM Computer Communication Review, Vol. 43, Issue 4 https://doi.org/10.1145/2534169.2486019	journal	August 2013
On Performance Prediction of Big Data Transfer in High-performance Networks Liu, Wuji; Yun, Daqing; Wu, Chase Q. ICC 2020 - 2020 IEEE International Conference on Communications (ICC) https://doi.org/10.1109/ICC40277.2020.9149386	conference	June 2020
Profiling transport performance for big data transfer over dedicated channels Yun, Daqing; Wu, Chase Q.; Rao, Nageswara S. V. 2015 International Conference on Computing, Networking and Communications (ICNC) https://doi.org/10.1109/ICCNC.2015.7069458	conference	February 2015
UDT: UDP-based data transfer for high-speed wide area networks Gu, Yunhong; Grossman, Robert L. Computer Networks, Vol. 51, Issue 7 https://doi.org/10.1016/j.comnet.2006.11.009	journal	May 2007
Advising Big Data Transfer Over Dedicated Connections Based on Profiling Optimization Yun, Daqing; Wu, Chase Q.; Rao, Nageswara S. V. IEEE/ACM Transactions on Networking, Vol. 27, Issue 6 https://doi.org/10.1109/TNET.2019.2943884	journal	December 2019
Modeling TCP Reno performance: a simple model and its empirical validation Padhye, J.; Firoiu, V.; Towsley, D. F. IEEE/ACM Transactions on Networking, Vol. 8, Issue 2 https://doi.org/10.1109/90.842137	journal	April 2000
End system optimizations for high-speed TCP Chase, J. S.; Gallatin, A. J.; Yocum, K. G. IEEE Communications Magazine, Vol. 39, Issue 4 https://doi.org/10.1109/35.917506	journal	April 2001
Explaining Wide Area Data Transfer Performance Liu, Zhengchun; Balaprakash, Prasanna; Kettimuthu, Rajkumar HPDC '17: The 26th International Symposium on High-Performance Parallel and Distributed Computing, Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing https://doi.org/10.1145/3078597.3078605	conference	June 2017
Bandwidth Reservation Strategies for Scheduling Maximization in Dedicated Networks Zuo, Liudong; Zhu, Michelle M.; Wu, Chase Q. IEEE Transactions on Network and Service Management, Vol. 15, Issue 2 https://doi.org/10.1109/TNSM.2018.2794300	journal	June 2018
On Source-Based Route Computation for Quickest Paths under Dynamic Bandwidth Constraints Grimmell, William C.; Rao, Nageswara S. V. International Journal of Foundations of Computer Science, Vol. 14, Issue 03 https://doi.org/10.1142/S0129054103001868	journal	June 2003
Complexity Analysis and Algorithm Design for Advance Bandwidth Scheduling in Dedicated Networks Lin, Yunyue; Wu, Qishi IEEE/ACM Transactions on Networking, Vol. 21, Issue 1 https://doi.org/10.1109/TNET.2012.2189127	journal	February 2013
Time Series Analysis for Efficient Sample Transfers Sapkota, Hemanta; Pehlivan, Bahadir A.; Arslan, Engin Proceedings of the ACM Workshop on Systems and Network Telemetry and Analytics - SNTA '19 https://doi.org/10.1145/3322798.3329256	conference	January 2019
Simple sample bound for feedforward sigmoid networks with bounded weights Rao, Nageswara S. V. Neurocomputing, Vol. 29, Issue 1-3 https://doi.org/10.1016/S0925-2312(99)00113-7	journal	November 1999
Intra and Interdomain Circuit Provisioning Using the OSCARS Reservation System Guok, Chin; Robertson, David; Thompson, Mary 2006 3rd International Conference on Broadband Communications, Networks and Systems https://doi.org/10.1109/BROADNETS.2006.4374316	conference	October 2006
Improving network performance on multicore systems: Impact of core affinities on high throughput flows Hanford, Nathan; Ahuja, Vishal; Farrens, Matthew Future Generation Computer Systems, Vol. 56 https://doi.org/10.1016/j.future.2015.09.012	journal	March 2016
TCP Throughput Profiles Using Measurements over Dedicated Connections Rao, Nageswara S. V.; Liu, Qiang; Sen, Satyabrata Proceedings of the 26th International Symposium on High-Performance Parallel and Distributed Computing - HPDC '17 https://doi.org/10.1145/3078597.3078615	conference	January 2017

Similar Records

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Conference · 2020 · OSTI ID:1648995

Yun, Daqing; Liu, Wuji; Wu, Qishi; +2 more

Performance Prediction of Big Data Transfer Through Experimental Analysis and Machine Learning

Conference · 2020 · OSTI ID:1668342

Yun, Daqing; Liu, Wuji; Wu, Chase Q.; +2 more

On Performance Prediction of Big Data Transfer in High-performance Networks

Conference · 2020 · OSTI ID:1648894

Liu, Wuji; Yun, Daqing; Wu, Chase Qishi; +3 more

Related Subjects

97 MATHEMATICS AND COMPUTING
Performance prediction
Latent effect
Machine learning
Big data transfer

Title: Exploratory analysis and performance prediction of big data transfer in High-performance Networks

Citation Formats

References (21)

Similar Records

Related Subjects