Flow Ordering and Hierarchical Bottleneck Identification for High Speed Data Networks - Phase I Final Report
Driven by new bandwidth-intensive applications such as big data, artificial intelligence, cloud applications and the Internet of Things, data growth is exploding worldwide as it continues to expand into all areas of society including business, science and leisure. Key to ensuring the productivity of our economic and social systems is the transportation of these ever increasing large datasets in a timely and cost-effective manner. For instance, backbone Research and Education Network (REN) providers such as the DOE’s Energy Sciences Network (ESnet) [ESN19a] or Internet2 [IN218] provide high speed, reliable network services to scientists located at national laboratories and universities across the nation and abroad. These networks are crucial to science as they provide the transportation means to move the large scientific datasets to offsite storage pools or to other supercomputing sites for further processing. The same data deluge is faced by the large cloud and application providers such as Google, Amazon, Netflix, Facebook and Microsoft, which have had to develop their own high performance network architectures [GOO17, GOO18, FACE17] to enable the degree of flexibility, availability, and cost effectiveness required to connect their global large scale data centers. Reservoir Labs has developed GradientGraph (G2) Analytics, a technology based on a newly discovered network optimization framework designed to tackle the above challenge. G2 allows network operators to gain high-precision knowledge of the bottleneck structure of their networks and their impact on the performance of congestion-controlled flows. Among other applications, G2 can be used to perform optimal traffic engineering and capacity planning operations. This report describes the work performed to implement the GradientGraph technology under DOE SBIR Phase I contract DE-SC0019523. The key objectives achieved during the Phase I are: Developed the Theory of Bottleneck Ordering (TBO), a new mathematical framework to optimize network performance. Among others, this theory provides key insights to the problems of optimal traffic engineering and capacity planning. Implemented an initial prototype of GradientGraph Analytics, a new network optimization platform based on TBO that allows operators to gain key bottleneck and flow performance insights to help optimize their networks. Implemented a prototype of G2 Dashboards, a graphical user interface (GUI) that allows network operators to visually interact with GradientGraph Analytics to help optimize network performance. Implemented G2 plugins, a southbound API that allows GradientGraph Analytics to integrate with production networks using protocols such as NetFlow, sFlow, SNMP, or BGP-LS among others. Submitted a new patent application to protect the GradientGraph invention ("Systems And Methods For Quality Of Service Based Management Of Bottlenecks And Flows In Networks", USPTO Application Number 16/580,718). Published a theoretical article describing the mathematics of GradientGraph at ACM SIGMETRICS 2020, the ACM flagship conference in network performance analysis and modeling. (With one of the lowest acceptance rates, it is also considered the leading conference in the world in this field.) Published two additional engineering articles, one at Supercomputing/INDIS 2019 describing the technical architecture of GradientGraph and another one at SIAM/Network Science 2019 describing applications of the G2 mathematics in the field of linear programming. Implemented G2-Mininet, a new network testing environment based on the Mininet network emulator designed to help evaluate and quantify the performance benefits derived from GradientGraph Analytics. Performed extensive simulations using the G2-mininet environment through our subcontracted team at the Yale Institute of Network Science (YINS). Successfully demonstrated GradientGraph as part of the SCinet Network at Supercomputing 2019. As an outcome of this effort, Reservoir Labs received five (5) letters of endorsement from various companies and organizations stating the relevance of the GradientGraph technology, including: Nokia Bell Labs, IBM Watson Research Center, Pacific Research Platform / San Diego Supercomputing Center, ESnet (Department of Energy) and Columbia University. (These letters of endorsement will be included in the Phase II proposal.)
- Research Organization:
- Reservoir Labs
- Sponsoring Organization:
- USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR)
- Contributing Organization:
- Yale Institute of Network Science
- DOE Contract Number:
- SC0019523
- OSTI ID:
- 1576796
- Type / Phase:
- SBIR (Phase I)
- Report Number(s):
- DOE-RESERVOIR-SC0019523PI; 2127800527110
- Resource Relation:
- Related Information: - Jordi Ros-Giralt, Atul Bohara, Sruthi Yellamraju, Harper Langston, Richard Lethin, Yuang Jiang, Leandros Tassiulas, Josie Li, Ying Lin, Yuanlong Tan, Malathi Veeraraghavan, "On the Bottleneck Structure of Congestion-Controlled Networks," accepted to be presented at ACM SIGMETRICS, Boston, June 2020.- Jordi Ros-Giralt, Sruthi Yellamraju, Atul Bohara, Richard Lethin, Josie Li, Ying Lin, Yuanlong Tan, Malathi Veeraraghavan, Yuang Jiang, Leandros Tassiulas, "G2: A Network Optimization Framework for High-Precision Analysis of Bottleneck and Flow Performance," International Workshop on Innovating the Network for Data Intensive Science (INDIS), Supercomputing, Denver, Nov 2019.- Jordi Ros-Giralt, Harper Langston, Aditya Gudibanda, Richard Lethin, "On the Bottleneck Structure of Positive Linear Programming", SIAM Workshop on Network Science (NS19), Utah May 2019.- Aditya Gudibanda, Jordi Ros-Giralt, Alan Commike, Richard Lethin, “Fast Detection of Elephant Flows with Dirichlet-Categorical Inference,” Supercomputing/INDIS 2018, November 11, 2018, Dallas, TX,USA.
- Country of Publication:
- United States
- Language:
- English
Similar Records
Flow Ordering and Hierarchical Bottleneck Identification for High-Speed Data Networks - Phase II Final Report
SDN for End-to-end Networked Science at the Exascale (SENSE) - Final Technical Report