Distributed deep learning training using silicon photonic switched architectures

Zhu, Ziyi; Teh, Min Yee; Wu, Zhenguo; Glick, Madeleine Strom; Yan, Shijia; Hattink, Maarten; Bergman, Keren

doi:10.1063/5.0070711

Distributed deep learning training using silicon photonic switched architectures

Journal Article · Tue Mar 01 00:00:00 EST 2022 · APL Photonics

DOI:https://doi.org/10.1063/5.0070711· OSTI ID:1978979

^[1]; Teh, Min Yee ^[2]; Wu, Zhenguo ^[2]; Glick, Madeleine Strom ^[2]; Yan, Shijia ^[2]; Hattink, Maarten ^[2]; Bergman, Keren ^[2]

Columbia University, New York, NY (United States); Columbia University, New York, New York 10027, USA
Columbia University, New York, NY (United States)

The scaling trends of deep learning models and distributed training workloads are challenging network capacities in today’s datacenters and high-performance computing (HPC) systems. We propose a system architecture that leverages silicon photonic (SiP) switch-enabled server regrouping using bandwidth steering to tackle the challenges and accelerate distributed deep learning training. In addition, our proposed system architecture utilizes a highly integrated operating system-based SiP switch control scheme to reduce implementation complexity. To demonstrate the feasibility of our proposal, we built an experimental testbed with a SiP switch-enabled reconfigurable fat tree topology and evaluated the network performance of distributed ring all-reduce and parameter server workloads. The experimental results show up to 3.6× improvements over the static non-reconfigurable fat tree. Our large-scale simulation results show that server regrouping can deliver up to 2.3× flow throughput improvement for a 2× tapered fat tree and a further 11% improvement when higher-layer bandwidth steering is employed. The collective results show the potential of integrating SiP switches into datacenters and HPC systems to accelerate distributed deep learning training.

View Accepted Manuscript (DOE)

Research Organization:: Columbia University, New York, NY (United States)

Sponsoring Organization:: National Security Agency (NSA); USDOE Advanced Research Projects Agency - Energy (ARPA-E); USDOE Office of Science (SC), Office of SBIR/STTR Programs (SBIR/STTR)

Grant/Contract Number:: AR0000843

OSTI ID:: 1978979

Journal Information:: APL Photonics, Journal Name: APL Photonics Journal Issue: 3 Vol. 7; ISSN 2378-0967

Publisher:: American Institute of Physics (AIP)Copyright Statement

Country of Publication:: United States

Language:: English

References (37)

Silicon Photonics Circuit Design: Methods, Tools and Challenges Bogaerts, Wim; Chrostowski, Lukas Laser & Photonics Reviews, Vol. 12, Issue 4 https://doi.org/10.1002/lpor.201700237	journal	March 2018
32 × 32 silicon electro-optic switch with built-in monitors and balanced-status units Qiao, Lei; Tang, Weijie; Chu, Tao Scientific Reports, Vol. 7, Issue 1 https://doi.org/10.1038/srep42306	journal	February 2017
APHiD: Hierarchical Task Placement to Enable a Tapered Fat Tree Topology for Lower Power and Cost in HPC Networks Michelogiannakis, George; Ibrahim, Khaled Z.; Shalf, John 2017 17th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGRID) https://doi.org/10.1109/CCGRID.2017.33	conference	May 2017
Demonstration of Multi-Casting in a 1 × 9 LCOS Wavelength Selective Switch Robertson, Brian; Yang, Haining; Redmond, Maura M. Journal of Lightwave Technology, Vol. 32, Issue 3 https://doi.org/10.1109/JLT.2013.2293919	journal	February 2014
Wavelength Locking and Thermally Stabilizing Microring Resonators Using Dithering Signals Padmaraju, Kishore; Logan, Dylan F.; Shiraishi, Takashi Journal of Lightwave Technology, Vol. 32, Issue 3 https://doi.org/10.1109/JLT.2013.2294564	journal	February 2014
Low-Insertion-Loss and Power-Efficient 32 × 32 Silicon Photonics Switch With Extremely High-Δ Silica PLC Connector Suzuki, Keijiro; Konoike, Ryotaro; Hasegawa, Junichi Journal of Lightwave Technology, Vol. 37, Issue 1 https://doi.org/10.1109/JLT.2018.2867575	journal	January 2019
Multi-Stage 8 × 8 Silicon Photonic Switch Based on Dual-Microring Switching Elements Huang, Yishen; Cheng, Qixiang; Hung, Yu-Han Journal of Lightwave Technology, Vol. 38, Issue 2 https://doi.org/10.1109/JLT.2019.2945941	journal	January 2020
Flexfly: Enabling a Reconfigurable Dragonfly through Silicon Photonics Wen, Ke; Samadi, Payman; Rumley, Sebastien SC16: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2016.14	conference	November 2016
A Scalable AWGR-Based Optical Switch No authors listed Journal of Lightwave Technology, Vol. 33, Issue 22 https://doi.org/10.1109/jlt.2015.2479296	journal	November 2015
Large-Scale Polarization-Insensitive Silicon Photonic MEMS Switches Han, Sangyoon; Seok, Tae Joon; Yu, Kyoungsik Journal of Lightwave Technology, Vol. 36, Issue 10 https://doi.org/10.1109/jlt.2018.2791502	journal	May 2018
Nonduplicate Polarization-Diversity 32 × 32 Silicon Photonics Switch Based on a SiN/Si Double-Layer Platform Suzuki, Keijiro; Namiki, Shu; Kawashima, Hitoshi Journal of Lightwave Technology, Vol. 38, Issue 2 https://doi.org/10.1109/jlt.2019.2934763	journal	January 2020
Low-Loss, Low-Crosstalk, and Large-Scale Optical Switch Based on Silicon Photonics Suzuki, Keijiro; Konoike, Ryotaro; Suda, Satoshi Journal of Lightwave Technology, Vol. 38, Issue 2 https://doi.org/10.1109/jlt.2019.2934768	journal	January 2020
Photonic Switched Optically Connected Memory: An Approach to Address Memory Challenges in Deep Learning Zhu, Ziyi; Guglielmo, Giuseppe Di; Cheng, Qixiang Journal of Lightwave Technology, Vol. 38, Issue 10 https://doi.org/10.1109/jlt.2020.2975976	journal	May 2020
X-NEST: A Scalable, Flexible, and High-Performance Network Architecture for Distributed Machine Learning Lu, Yunfeng; Gu, Huaxi; Yu, Xiaoshan Journal of Lightwave Technology, Vol. 39, Issue 13 https://doi.org/10.1109/jlt.2021.3073277	journal	July 2021
Optical Crosspoint Matrix Using Broadband Resonant Switches DasMahapatra, Prometheus; Stabile, Ripalta; Rohit, Abhinav IEEE Journal of Selected Topics in Quantum Electronics, Vol. 20, Issue 4 https://doi.org/10.1109/jstqe.2013.2296746	journal	July 2014
1100 x 1100 port MEMS-based optical crossconnect with 4-dB maximum loss Kim, J.; Nuzman, C. J.; Kumar, B. IEEE Photonics Technology Letters, Vol. 15, Issue 11 https://doi.org/10.1109/lpt.2003.818653	journal	November 2003
32 × 32 silicon photonic MEMS switch with gap-adjustable directional couplers fabricated in commercial CMOS foundry Han, Sangyoon; Beguelin, Jeremy; Ochikubo, Lane Journal of Optical Microsystems, Vol. 1, Issue 02 https://doi.org/10.1117/1.jom.1.2.024003	journal	March 2021
Reconfigurable hybrid interconnection for static and dynamic scientific applications Kamil, Shoaib; Pinar, Ali; Gunter, Daniel Proceedings of the 4th international conference on Computing frontiers - CF '07 https://doi.org/10.1145/1242531.1242559	conference	January 2007
Helios: a hybrid electrical/optical switch architecture for modular data centers Farrington, Nathan; Porter, George; Radhakrishnan, Sivasankar Proceedings of the ACM SIGCOMM 2010 conference on SIGCOMM - SIGCOMM '10 https://doi.org/10.1145/1851182.1851223	conference	January 2010
FireFly: a reconfigurable wireless data center fabric using free-space optics Hamedazimi, Navid; Qazi, Zafar; Gupta, Himanshu SIGCOMM'14: ACM SIGCOMM 2014 Conference, Proceedings of the 2014 ACM conference on SIGCOMM https://doi.org/10.1145/2619239.2626328	conference	August 2014
ProjecToR: Agile Reconfigurable Data Center Interconnect Ghobadi, Monia; Mahajan, Ratul; Phanishayee, Amar SIGCOMM '16: ACM SIGCOMM 2016 Conference, Proceedings of the 2016 ACM SIGCOMM Conference https://doi.org/10.1145/2934872.2934911	conference	August 2016
Deep Neural Networks for YouTube Recommendations Covington, Paul; Adams, Jay; Sargin, Emre Proceedings of the 10th ACM Conference on Recommender Systems https://doi.org/10.1145/2959100.2959190	conference	September 2016
RotorNet: A Scalable, Low-complexity, Optical Datacenter Network Mellette, William M.; McGuinness, Rob; Roy, Arjun SIGCOMM '17: ACM SIGCOMM 2017 Conference, Proceedings of the Conference of the ACM Special Interest Group on Data Communication https://doi.org/10.1145/3098822.3098838	conference	August 2017
Characterizing the algorithmic complexity of reconfigurable data center architectures Foerster, Klaus-Tycho; Ghobadi, Manya; Schmid, Stefan Proceedings of the 2018 Symposium on Architectures for Networking and Communications Systems https://doi.org/10.1145/3230718.3230722	conference	July 2018
SiP-ML Khani, Mehrdad; Ghobadi, Manya; Alizadeh, Mohammad Proceedings of the 2021 ACM SIGCOMM 2021 Conference https://doi.org/10.1145/3452296.3472900	conference	August 2021
Flexspander: augmenting expander networks in high-performance systems with optical bandwidth steering Teh, Min Yee; Wu, Zhenguo; Bergman, Keren Journal of Optical Communications and Networking, Vol. 12, Issue 4 https://doi.org/10.1364/jocn.379487	journal	January 2020
Control of integrated micro-resonator wavelength via balanced homodyne locking Cox, Jonathan A.; Lentine, Anthony L.; Trotter, Douglas C. Optics Express, Vol. 22, Issue 9, p. 11279-11289 https://doi.org/10.1364/oe.22.011279	journal	January 2014
Recent advances in silicon-based passive and active optical interconnects Subbaraman, Harish; Xu, Xiaochuan; Hosseini, Amir Optics Express, Vol. 23, Issue 3 https://doi.org/10.1364/oe.23.002487	journal	January 2015
16 × 16 non-blocking silicon optical switch based on electro-optic Mach-Zehnder interferometers Lu, Liangjun; Zhao, Shuoyi; Zhou, Linjie Optics Express, Vol. 24, Issue 9 https://doi.org/10.1364/oe.24.009295	journal	January 2016
Software-defined control-plane for wavelength selective unicast and multicast of optical data in a silicon photonic platform Gazman, Alexander; Browning, Colm; Bahadori, Meisam Optics Express, Vol. 25, Issue 1 https://doi.org/10.1364/oe.25.000232	journal	January 2017
Software-defined networking control plane for seamless integration of multiple silicon photonic switches in Datacom networks Shen, Yiwen; Hattink, Maarten H. N.; Samadi, Payman Optics Express, Vol. 26, Issue 8 https://doi.org/10.1364/oe.26.010914	journal	January 2018
Photonic switching in high performance datacenters [Invited] Cheng, Qixiang; Rumley, Sébastien; Bahadori, Meisam Optics Express, Vol. 26, Issue 12 https://doi.org/10.1364/oe.26.016022	journal	January 2018
Demonstration of the feasibility of large-port-count optical switching using a hybrid Mach–Zehnder interferometer–semiconductor optical amplifier switch module in a recirculating loop Cheng, Q.; Wonfor, A.; Wei, J. L. Optics Letters, Vol. 39, Issue 18 https://doi.org/10.1364/ol.39.005244	journal	January 2014
Recent advances in optical technologies for data centers: a review Cheng, Qixiang; Bahadori, Meisam; Glick, Madeleine Optica, Vol. 5, Issue 11 https://doi.org/10.1364/optica.5.001354	journal	January 2018
Wafer-scale silicon photonic switches beyond die size limit Seok, Tae Joon; Kwon, Kyungmok; Henriksson, Johannes Optica, Vol. 6, Issue 4 https://doi.org/10.1364/optica.6.000490	journal	January 2019
Ultralow-crosstalk, strictly non-blocking microring-based optical switch Cheng, Qixiang; Dai, Liang Yuan; Abrams, Nathan C. Photonics Research, Vol. 7, Issue 2 https://doi.org/10.1364/prj.7.000155	journal	January 2019
Hybrid Electrical/Optical Switch Architectures for Training Distributed Deep Learning in Large-Scale Truong, Thao-Nguyen; Takano, Ryousei IEICE Transactions on Information and Systems, Vol. E104.D, Issue 8 https://doi.org/10.1587/transinf.2020edp7201	journal	August 2021

Similar Records

Flexible silicon photonic architecture for accelerating distributed deep learning

Journal Article · Mon Jan 08 19:00:00 EST 2024 · Journal of Optical Communications and Networking · OSTI ID:2280467

Performance trade-offs in reconfigurable networks for HPC

Journal Article · Tue May 10 20:00:00 EDT 2022 · Journal of Optical Communications and Networking · OSTI ID:1874993

Optics Enabled Networks and Architectures for Data Center Cost and Power Efficiency

Journal Article · Thu Oct 14 20:00:00 EDT 2021 · Journal of Optical Communications and Networking · OSTI ID:1828354

Related Subjects

71 CLASSICAL AND QUANTUM MECHANICS
GENERAL PHYSICS
high performance computing
machine learning
optical communication systems
silicon photonics

Distributed deep learning training using silicon photonic switched architectures

Citation Formats

References (37)

Similar Records

Related Subjects