Performance trade-offs in reconfigurable networks for HPC

Teh, Min Yee; Wu, Zhenguo; Glick, Madeleine; Rumley, Sebastien; Ghobadi, Manya; Bergman, Keren

doi:10.1364/JOCN.451760

Performance trade-offs in reconfigurable networks for HPC

Journal Article · Wed May 11 00:00:00 EDT 2022 · Journal of Optical Communications and Networking

DOI:https://doi.org/10.1364/JOCN.451760· OSTI ID:1874993

Teh, Min Yee; Wu, Zhenguo; ; ; Ghobadi, Manya; Bergman, Keren

Designing efficient interconnects to support high-bandwidth and low-latency communication is critical toward realizing high performance computing (HPC) and data center (DC) systems in the exascale era. At extreme computing scales, providing the requisite bandwidth through overprovisioning becomes impractical. These challenges have motivated studies exploring reconfigurable network architectures that can adapt to traffic patterns at runtime using optical circuit switching. Despite the plethora of proposed architectures, surprisingly little is known about the relative performances and trade-offs among different reconfigurable network designs. We aim to bridge this gap by tackling two key issues in reconfigurable network design. First, we study how cost, power consumption, network performance, and scalability vary based on optical circuit switch (OCS) placement in the physical topology. Specifically, we consider two classes of reconfigurable architectures: one that places OCSs between top-of-rack (ToR) switches—ToR-reconfigurable networks (TRNs)—and one that places OCSs between pods of racks—pod-reconfigurable networks (PRNs). Second, we tackle the effects of reconfiguration frequency on network performance. Our results, based on network simulations driven by real HPC and DC workloads, show that while TRNs are optimized for low fan-out communication patterns, they are less suited for carrying high fan-out workloads. PRNs exhibit better overall trade-off, capable of performing comparably to a fully non-blocking fat tree for low fan-out workloads, and significantly outperform TRNs for high fan-out communication patterns.

View Journal Article

Sponsoring Organization:: USDOE Advanced Research Projects Agency - Energy (ARPA-E)

OSTI ID:: 1874993

Alternate ID(s):: OSTI ID: 1867045

Journal Information:: Journal of Optical Communications and Networking, Journal Name: Journal of Optical Communications and Networking Journal Issue: 6 Vol. 14; ISSN JOCNBB; ISSN 1943-0620

Publisher:: Optical Society of AmericaCopyright Statement

Country of Publication:: United States

Language:: English

References (42)

On the impossibility of Directed Moore Graphs Bridges, W. G.; Toueg, Sam Journal of Combinatorial Theory, Series B, Vol. 29, Issue 3 https://doi.org/10.1016/0095-8956(80)90091-X	journal	December 1980
Synchronous subnanosecond clock and data recovery for optically switched data centres using clock phase caching Clark, Kari A.; Cletheroe, Daniel; Gerard, Thomas Nature Electronics, Vol. 3, Issue 7 https://doi.org/10.1038/s41928-020-0423-y	journal	June 2020
Baldur: A Power-Efficient and Scalable Network Using All-Optical Switches Jokar, Mohammad Reza; Qiu, Junyi; Chong, Frederic T. 2020 IEEE International Symposium on High Performance Computer Architecture (HPCA) https://doi.org/10.1109/HPCA47549.2020.00022	conference	February 2020
Technology-Driven, Highly-Scalable Dragonfly Topology Kim, John; Dally, Wiliam J.; Scott, Steve 2008 35th International Symposium on Computer Architecture (ISCA), 2008 International Symposium on Computer Architecture https://doi.org/10.1109/ISCA.2008.19	conference	June 2008
PULSE: Optical Circuit Switched Data Center Architecture Operating at Nanosecond Timescales Benjamin, Joshua L.; Gerard, Thomas; Lavery, Domanic Journal of Lightwave Technology, Vol. 38, Issue 18 https://doi.org/10.1109/JLT.2020.2997664	journal	September 2020
A Multiport Microsecond Optical Circuit Switch for Data Center Networking Farrington, Nathan; Forencich, Alex; Porter, George IEEE Photonics Technology Letters, Vol. 25, Issue 16 https://doi.org/10.1109/LPT.2013.2270462	journal	August 2013
NEPHELE: An End-to-End Scalable and Dynamically Reconfigurable Optical Architecture for Application-Aware SDN Cloud Data Centers Bakopoulos, Paraskevas; Christodoulopoulos, Konstantinos; Landi, Giada IEEE Communications Magazine, Vol. 56, Issue 2 https://doi.org/10.1109/MCOM.2018.1600804	journal	February 2018
Cray Cascade: A scalable HPC system based on a Dragonfly network Faanes, Greg; Bataineh, Abdulla; Roweth, Duncan 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.39	conference	November 2012
Flexfly: Enabling a Reconfigurable Dragonfly through Silicon Photonics Wen, Ke; Samadi, Payman; Rumley, Sebastien SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.14	conference	November 2016
Architecture and Performance Studies of 3D-Hyper-FleX-LION for Reconfigurable All-to-All HPC Networks Liu, Gengchen; Proietti, Roberto; Fariborz, Marjan SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00030	conference	November 2020
FatPaths: Routing in Supercomputers and Data Centers when Shortest Paths Fall Short Besta, Maciej; Schneider, Marcel; Konieczny, Marek SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00031	conference	November 2020
Fat-trees: Universal networks for hardware-efficient supercomputing Leiserson, Charles E. IEEE Transactions on Computers, Vol. C-34, Issue 10 https://doi.org/10.1109/TC.1985.6312192	journal	October 1985
Latest standardization status and its future directions for high speed optical transceivers Isono, Hideki Metro and Data Center Optical Networks and Short-Reach Links II https://doi.org/10.1117/12.2507321	conference	March 2019
Cope Wang, Hao; Xie, Haiyong; Qiu, Lili Proceedings of the 2006 conference on Applications, technologies, architectures, and protocols for computer communications https://doi.org/10.1145/1159913.1159926	conference	August 2006
A scalable, commodity data center network architecture Al-Fares, Mohammad; Loukissas, Alexander; Vahdat, Amin Proceedings of the ACM SIGCOMM 2008 conference on Data communication - SIGCOMM '08 https://doi.org/10.1145/1402958.1402967	conference	January 2008
VL2: a scalable and flexible data center network Greenberg, Albert; Hamilton, James R.; Jain, Navendu Proceedings of the ACM SIGCOMM 2009 conference on Data communication - SIGCOMM '09 https://doi.org/10.1145/1592568.1592576	conference	January 2009
Understanding data center traffic characteristics Benson, Theophilus; Anand, Ashok; Akella, Aditya Proceedings of the 1st ACM workshop on Research on enterprise networking - WREN '09 https://doi.org/10.1145/1592681.1592692	conference	January 2009
HyperX: topology, routing, and packaging of efficient large-scale networks Ahn, Jung Ho; Binkert, Nathan; Davis, Al Proceedings of the Conference on High Performance Computing Networking, Storage and Analysis - SC '09 https://doi.org/10.1145/1654059.1654101	conference	January 2009
Network traffic characteristics of data centers in the wild Benson, Theophilus; Akella, Aditya; Maltz, David A. IMC '10 Proceedings of the 10th ACM SIGCOMM conference on Internet measurement https://doi.org/10.1145/1879141.1879175	conference	January 2010
Mirror mirror on the ceiling: flexible wireless links for data centers Zhou, Xia; Zhang, Zengbin; Zhu, Yibo Proceedings of the ACM SIGCOMM 2012 conference on Applications, technologies, architectures, and protocols for computer communication - SIGCOMM '12 https://doi.org/10.1145/2342356.2342440	conference	January 2012
Integrating microsecond circuit switching into the data center Porter, George; Strong, Richard; Farrington, Nathan SIGCOMM'13: ACM SIGCOMM 2013 Conference, Proceedings of the ACM SIGCOMM 2013 conference on SIGCOMM https://doi.org/10.1145/2486001.2486007	conference	August 2013
Scheduling techniques for hybrid circuit/packet networks Liu, He; Mukerjee, Matthew K.; Li, Conglong CoNEXT '15: Conference on emerging Networking Experiments and Technologies, Proceedings of the 11th ACM Conference on Emerging Networking Experiments and Technologies https://doi.org/10.1145/2716281.2836126	conference	December 2015
Inside the Social Network's (Datacenter) Network Roy, Arjun; Zeng, Hongyi; Bagga, Jasmeet SIGCOMM '15: ACM SIGCOMM 2015 Conference, Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication https://doi.org/10.1145/2785956.2787472	conference	August 2015
Jupiter Rising: A Decade of Clos Topologies and Centralized Control in Google's Datacenter Network Singh, Arjun; Ong, Joon; Agarwal, Amit SIGCOMM '15: ACM SIGCOMM 2015 Conference, Proceedings of the 2015 ACM Conference on Special Interest Group on Data Communication https://doi.org/10.1145/2785956.2787508	conference	August 2015
ProjecToR: Agile Reconfigurable Data Center Interconnect Ghobadi, Monia; Mahajan, Ratul; Phanishayee, Amar SIGCOMM '16: ACM SIGCOMM 2016 Conference, Proceedings of the 2016 ACM SIGCOMM Conference https://doi.org/10.1145/2934872.2934911	conference	August 2016
Xpander: Towards Optimal-Performance Datacenters Valadarsky, Asaf; Shahaf, Gal; Dinitz, Michael Proceedings of the 12th International on Conference on emerging Networking EXperiments and Technologies - CoNEXT '16 https://doi.org/10.1145/2999572.2999580	conference	January 2016
A Tale of Two Topologies: Exploring Convertible Data Center Network Architectures with Flat-tree Xia, Yiting; Sun, Xiaoye Steven; Dzinamarira, Simbarashe SIGCOMM '17: ACM SIGCOMM 2017 Conference, Proceedings of the Conference of the ACM Special Interest Group on Data Communication https://doi.org/10.1145/3098822.3098837	conference	August 2017
RotorNet: A Scalable, Low-complexity, Optical Datacenter Network Mellette, William M.; McGuinness, Rob; Roy, Arjun SIGCOMM '17: ACM SIGCOMM 2017 Conference, Proceedings of the Conference of the ACM Special Interest Group on Data Communication https://doi.org/10.1145/3098822.3098838	conference	August 2017
Theoretical Improvements in Algorithmic Efficiency for Network Flow Problems Edmonds, Jack; Karp, Richard M. Journal of the ACM, Vol. 19, Issue 2 https://doi.org/10.1145/321694.321699	journal	April 1972
Bandwidth steering in HPC using silicon nanophotonics Michelogiannakis, George; Shen, Yiwen; Teh, Min Yee SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356145	conference	November 2019
Topology-custom UGAL routing on dragonfly Rahman, Md Shafayat; Bhowmik, Saptarshi; Ryasnianskiy, Yevgeniy SC '19: The International Conference for High Performance Computing, Networking, Storage, and Analysis, Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3295500.3356208	conference	November 2019
Sirius: A Flat Datacenter Network with Nanosecond Optical Switching Ballani, Hitesh; Costa, Paolo; Behrendt, Raphael SIGCOMM '20: Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication, Proceedings of the Annual conference of the ACM Special Interest Group on Data Communication on the applications, technologies, architectures, and protocols for computer communication https://doi.org/10.1145/3387514.3406221	conference	July 2020
Abstractions for Reconfigurable Hybrid Network Update and A Consistent Update Approach Wang, Weitao; Das, Sushovan; Ng, T. S. Eugene SIGCOMM '21: ACM SIGCOMM 2021 Conference, Proceedings of the ACM SIGCOMM 2021 Workshop on Optical Systems https://doi.org/10.1145/3473938.3474506	conference	August 2021
A Case For Intra-rack Resource Disaggregation in HPC Michelogiannakis, George; Klenk, Benjamin; Cook, Brandon ACM Transactions on Architecture and Code Optimization, Vol. 19, Issue 2 https://doi.org/10.1145/3514245	journal	June 2022
Finding the K Shortest Loopless Paths in a Network Yen, Jin Y. Management Science, Vol. 17, Issue 11 https://doi.org/10.1287/mnsc.17.11.712	journal	July 1971
Flexspander: augmenting expander networks in high-performance systems with optical bandwidth steering Teh, Min Yee; Wu, Zhenguo; Bergman, Keren Journal of Optical Communications and Networking, Vol. 12, Issue 4 https://doi.org/10.1364/JOCN.379487	journal	January 2020
Spatial Division Multiplexing for High Capacity Optical Interconnects in Modular Data Centers Fiorani, Matteo; Tornatore, Massimo; Chen, Jiajia Journal of Optical Communications and Networking, Vol. 9, Issue 2 https://doi.org/10.1364/JOCN.9.00A143	journal	January 2017
Novel flat datacenter network architecture based on scalable and flow-controlled optical switch system Miao, Wang; Luo, Jun; Di Lucente, Stefano Optics Express, Vol. 22, Issue 3 https://doi.org/10.1364/OE.22.002465	journal	January 2014
Fast, High-radix Silicon Photonic Switches Chu, Tao; Qiao, Lei; Tang, Weijie Optical Fiber Communication Conference https://doi.org/10.1364/OFC.2018.Th1J.4	conference	January 2018
Large-scale broadband digital silicon photonic switches with vertical adiabatic couplers Seok, Tae Joon; Quack, Niels; Han, Sangyoon Optica, Vol. 3, Issue 1 https://doi.org/10.1364/OPTICA.3.000064	journal	January 2016
Moore Graphs and Beyond: A survey of the Degree/Diameter Problem Miller, Mirka; Sirán, Jozef The Electronic Journal of Combinatorics, Vol. 1000 https://doi.org/10.37236/35	journal	January 2012
A Simulator for Large-Scale Parallel Computer Architectures Janssen, Curtis L.; Adalsteinsson, Helgi; Cranford, Scott International Journal of Distributed Systems and Technologies, Vol. 1, Issue 2 https://doi.org/10.4018/jdst.2010040104	journal	April 2010

Similar Records

Flexspander: augmenting expander networks in high-performance systems with optical bandwidth steering

Journal Article · Thu Feb 27 23:00:00 EST 2020 · Journal of Optical Communications and Networking · OSTI ID:1601693

Distributed deep learning training using silicon photonic switched architectures

Journal Article · Mon Feb 28 23:00:00 EST 2022 · APL Photonics · OSTI ID:1978979

Optics Enabled Networks and Architectures for Data Center Cost and Power Efficiency

Journal Article · Fri Oct 15 00:00:00 EDT 2021 · Journal of Optical Communications and Networking · OSTI ID:1828354

Performance trade-offs in reconfigurable networks for HPC

Citation Formats

References (42)

Similar Records

Related Subjects