A Network Management System for Handling Scientific Data Flows

Yan, Zhenzhen; Tracy, Chris; Veeraraghavan, Malathi; Jin, Tian; Liu, Zhengyang

doi:10.1007/s10922-014-9336-2

A Network Management System for Handling Scientific Data Flows

Journal Article · Sat Oct 11 00:00:00 EDT 2014 · Journal of Network and Systems Management

DOI:https://doi.org/10.1007/s10922-014-9336-2· OSTI ID:1532178

Yan, Zhenzhen ^[1]; Tracy, Chris ^[2]; Veeraraghavan, Malathi ^[1]; Jin, Tian ^[1]; Liu, Zhengyang ^[1]

Univ. of Virginia, Charlottesville, VA (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Large scientific data transfers often occur at high rates causing increased burstiness in Internet traffic. To limit the adverse effects of these high-rate large-sized flows, which are referred to as $$\alpha $$ flows, on delay-sensitive audio/video flows, a network management system called Alpha Flow Traffic Engineering System (AFTES) is proposed for intra-domain traffic engineering. An offline approach is used in which AFTES analyzes NetFlow records collected by routers, extracts source–destination address prefixes of $$\alpha $$ flows, and uses these prefixes to configure firewall filters at ingress routers of a provider’s network to redirect future $$\alpha $$ flows to traffic-engineered paths and isolated queues. The effectiveness of this scheme was evaluated through an analysis of 7 months of NetFlow data obtained from an ESnet router. For this data set, 91 % of bytes generated by $$\alpha $$ flows during high-rate intervals would have been directed had AFTES been deployed. The negative aspect of using address prefixes in firewall filters, i.e., the redirection of $$\beta $$ flows to $$\alpha $$-flow paths/queues, was also quantified.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1532178

Journal Information:: Journal of Network and Systems Management, Journal Name: Journal of Network and Systems Management Journal Issue: 1 Vol. 24; ISSN 1064-7570

Country of Publication:: United States

Language:: English

References (33)

Providing and verifying advanced IP services in hierarchical DiffServ networks-the case of GEANT Liakopoulos, Athanassios; Maglaris, Basil; Bouras, Christos International Journal of Communication Systems, Vol. 17, Issue 4 https://doi.org/10.1002/dac.645	journal	April 2004
Internet traffic engineering using multi-protocol label switching (MPLS) Awduche, Daniel O.; Jabbari, Bijan Computer Networks, Vol. 40, Issue 1 https://doi.org/10.1016/S1389-1286(02)00269-4	journal	September 2002
A measurement study of correlations of Internet flow characteristics Lan, Kun-chan; Heidemann, John Computer Networks, Vol. 50, Issue 1 https://doi.org/10.1016/j.comnet.2005.02.008	journal	January 2006
Estimating the Heavy Tail Index from Scaling Properties Crovella, Mark E.; Taqqu, Murad S. Methodology And Computing In Applied Probability, Vol. 1, Issue 1, p. 55-79 https://doi.org/10.1023/A:1010012224103	journal	January 1999
Wide-area Internet traffic patterns and characteristics Thompson, K.; Miller, G. J.; Wilder, R. IEEE Network, Vol. 11, Issue 6 https://doi.org/10.1109/65.642356	journal	January 1997
Traffic engineering with MPLS in the Internet No authors listed IEEE Network, Vol. 14, Issue 2 https://doi.org/10.1109/65.826369	journal	January 2000
Characterization of high-rate large-sized flows Jin, Tian; Tracy, Chris; Veeraraghavan, Malathi 2014 IEEE International Black Sea Conference on Communications and Networking (BlackSeaCom) https://doi.org/10.1109/BlackSeaCom.2014.6849008	conference	May 2014
An overview of routing optimization for internet traffic engineering Wang, Ning; Ho, Kin; Pavlou, George IEEE Communications Surveys & Tutorials, Vol. 10, Issue 1 https://doi.org/10.1109/COMST.2008.4483669	journal	January 2008
Tracking elephant flows in internet backbone traffic with an FPGA-based cache Zadnik, Martin; Canini, Marco; Moore, Andrew W. 2009 International Conference on Field Programmable Logic and Applications (FPL) https://doi.org/10.1109/FPL.2009.5272387	conference	August 2009
ElephantTrap: A low cost device for identifying large flows Lu, Yi; Prabhakar, Balaji; Bonomi, Flavio 15th Annual IEEE Symposium on High-Performance Interconnects (HOTI 2007) https://doi.org/10.1109/HOTI.2007.13	conference	August 2007
Identifying high-rate flows based on Bayesian single sampling Zhang, Yu; Fang, Binxing; Zhang, Yongzheng 2010 2nd International Conference on Computer Engineering and Technology https://doi.org/10.1109/ICCET.2010.5486097	conference	April 2010
Beyond MLU: An application-centric comparison of traffic engineering schemes Sharma, Abhigyan; Mishra, Aditya; Kumar, Vikas IEEE INFOCOM 2011 - IEEE Conference on Computer Communications, 2011 Proceedings IEEE INFOCOM https://doi.org/10.1109/INFCOM.2011.5935255	conference	April 2011
Simple and Accurate Identification of High-Rate Flows by Packet Sampling Kamiyama, N.; Mori, T. Proceedings IEEE INFOCOM 2006. 25TH IEEE International Conference on Computer Communications https://doi.org/10.1109/INFOCOM.2006.324	conference	April 2006
Self-management of hybrid optical and packet switching networks Fioreze, Tiago; Pras, Aiko 2011 IFIP/IEEE International Symposium on Integrated Network Management (IM 2011), 12th IFIP/IEEE International Symposium on Integrated Network Management (IM 2011) and Workshops https://doi.org/10.1109/INM.2011.5990527	conference	May 2011
Understanding Internet traffic streams: dragonflies and tortoises Brownlee, N.; Claffy, K. C. IEEE Communications Magazine, Vol. 40, Issue 10 https://doi.org/10.1109/MCOM.2002.1039865	journal	October 2002
Real-time Detection of Grid Bulk Transfer Traffic Paisley, J.; Sventek, J. 2006 IEEE/IFIP Network Operations and Management Symposium NOMS 2006 https://doi.org/10.1109/NOMS.2006.1687539	conference	January 2006
On using virtual circuits for GridFTP transfers Liu, Z.; Veeraraghavan, M.; Yan, Z. 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.84	conference	November 2012
A survey of techniques for internet traffic classification using machine learning Nguyen, Thuy T. T.; Armitage, Grenville IEEE Communications Surveys & Tutorials, Vol. 10, Issue 4 https://doi.org/10.1109/SURV.2008.080406	journal	January 2008
A Survey on Internet Traffic Identification Callado, Arthur; Kamienski, Carlos; Szabo, Geza IEEE Communications Surveys & Tutorials, Vol. 11, Issue 3 https://doi.org/10.1109/SURV.2009.090304	journal	January 2009
A Survey on the Path Computation Element (PCE) Architecture Paolucci, Francesco; Cugini, Filippo; Giorgetti, Alessio IEEE Communications Surveys & Tutorials, Vol. 15, Issue 4 https://doi.org/10.1109/SURV.2013.011413.00087	journal	January 2013
Estimating flow distributions from sampled flow statistics Duffield, N.; Lund, C.; Thorup, M. IEEE/ACM Transactions on Networking, Vol. 13, Issue 5 https://doi.org/10.1109/TNET.2005.852874	journal	October 2005
Inverting sampled traffic Hohn, N.; Veitch, D. IEEE/ACM Transactions on Networking, Vol. 14, Issue 1 https://doi.org/10.1109/TNET.2005.863456	journal	February 2006
Identifying elephant flows through periodically sampled packets Mori, Tatsuya; Uchida, Masato; Kawahara, Ryoichi Proceedings of the 4th ACM SIGCOMM conference on Internet measurement - IMC '04 https://doi.org/10.1145/1028788.1028803	conference	January 2004
A methodology for studying persistency aspects of internet flows Wallerich, Jörg; Dreger, Holger; Feldmann, Anja ACM SIGCOMM Computer Communication Review, Vol. 35, Issue 2 https://doi.org/10.1145/1064413.1064417	journal	April 2005
c-Through: part-time optics in data centers Wang, Guohui; Andersen, David G.; Kaminsky, Michael ACM SIGCOMM Computer Communication Review, Vol. 40, Issue 4 https://doi.org/10.1145/1851275.1851222	journal	August 2010
Helios: a hybrid electrical/optical switch architecture for modular data centers Farrington, Nathan; Porter, George; Radhakrishnan, Sivasankar ACM SIGCOMM Computer Communication Review, Vol. 40, Issue 4 https://doi.org/10.1145/1851275.1851223	journal	August 2010
Experiences with 100Gbps network applications Balman, Mehmet; Pouyoul, Eric; Yao, Yushu Proceedings of the fifth international workshop on Data-Intensive Distributed Computing Date - DIDC '12 https://doi.org/10.1145/2286996.2287004	conference	January 2012
Programming your network at run-time for big data applications Wang, Guohui; Ng, T. S. Eugene; Shaikh, Anees Proceedings of the first workshop on Hot topics in software defined networks - HotSDN '12 https://doi.org/10.1145/2342441.2342462	conference	January 2012
Application-awareness in SDN Qazi, Zafar Ayyub; Lee, Jeongkeun; Jin, Tao Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM - SIGCOMM '13 https://doi.org/10.1145/2486001.2491700	conference	January 2013
Connection-level analysis and modeling of network traffic Sarvotham, Shriram; Riedi, Rudolf; Baraniuk, Richard Proceedings of the First ACM SIGCOMM Workshop on Internet Measurement Workshop - IMW '01 https://doi.org/10.1145/505213.505215	conference	January 2001
A pragmatic definition of elephants in internet backbone traffic Papagiannaki, K.; Taft, N.; Bhattacharyya, S. Proceedings of the second ACM SIGCOMM Workshop on Internet measurment - IMW '02 https://doi.org/10.1145/637201.637227	conference	January 2002
An Architecture for Differentiated Services Blake, S.; Black, D.; Carlson, M. https://doi.org/10.17487/rfc2475	report	December 1998
Cisco Systems NetFlow Services Export Version 9 Claise, B. https://doi.org/10.17487/rfc3954	report	September 2004

Similar Records

Self-Configuring Network Monitor

Software · Fri Apr 30 20:00:00 EDT 2004 · OSTI ID:code-56950

Effects of inter-packet spacing on the delivery of multimedia content

Conference · Sun Dec 31 23:00:00 EST 2000 · OSTI ID:975139

Netostat: analyzing dynamic flow patterns in high-speed networks

Journal Article · Thu Mar 24 20:00:00 EDT 2022 · Cluster Computing · OSTI ID:1969509

Related Subjects

97 MATHEMATICS AND COMPUTING
Elephant flows
MPLS
NetFlow traffic analysis
Research and education networks (RENs)
Scientific computing
Virtual circuits

A Network Management System for Handling Scientific Data Flows

Citation Formats

References (33)

Similar Records

Related Subjects