Deep learning (DL) has become a key technique for solving complex problems in scientific research and discovery. DL training for science is substantially challenging because it has to deal with massive quantities of multi-dimensional data. High- performance computing (HPC) supercomputers are increasingly being employed for meeting the exponentially growing demand for DL. Multiple GPUs and high-speed interconnect network are needed for supporting DL on HPC systems. However, the excessive use of GPUs without considering effective benefits leads to inefficient resource utilization of these expensive setups. In this paper, we conduct a quantitative analysis to gauge the efficacy of DL workloads on the latest HPC system and identify viability of next-generation DL-optimized heterogeneous supercomputers for enabling researchers to develop more efficient resource management and distributed DL middleware. We evaluate well- known DL models with large-scale datasets using the popular TensorFlow framework, and provide a thorough evaluation in- cluding scalability, accuracy, variability, storage resource, GPU- GPU/GPU-CPU data transfer, and GPU utilization. Our analysis reveals that the latest heterogeneous supercomputing cluster shows varying performance trend as compared to the existing literature for single- and multi-node training. To the best of our knowledge, this is the first work to conduct such a quantitative and comprehensive study of DL training on a supercomputing system with multiple GPUs.
Han, Jingoo, et al. "A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers." , Sep. 2019. https://doi.org/10.1109/CLUSTER.2019.8890993
Han, Jingoo, Xu, Luna, Rafique, Mustafa, Butt, Ali R., & Lim, Seung-Hwan (2019). A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers. https://doi.org/10.1109/CLUSTER.2019.8890993
Han, Jingoo, Xu, Luna, Rafique, Mustafa, et al., "A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers," (2019), https://doi.org/10.1109/CLUSTER.2019.8890993
@conference{osti_1569375,
author = {Han, Jingoo and Xu, Luna and Rafique, Mustafa and Butt, Ali R. and Lim, Seung-Hwan},
title = {A Quantitative Study of Deep Learning Training on Heterogeneous Supercomputers},
annote = {Deep learning (DL) has become a key technique for solving complex problems in scientific research and discovery. DL training for science is substantially challenging because it has to deal with massive quantities of multi-dimensional data. High- performance computing (HPC) supercomputers are increasingly being employed for meeting the exponentially growing demand for DL. Multiple GPUs and high-speed interconnect network are needed for supporting DL on HPC systems. However, the excessive use of GPUs without considering effective benefits leads to inefficient resource utilization of these expensive setups. In this paper, we conduct a quantitative analysis to gauge the efficacy of DL workloads on the latest HPC system and identify viability of next-generation DL-optimized heterogeneous supercomputers for enabling researchers to develop more efficient resource management and distributed DL middleware. We evaluate well- known DL models with large-scale datasets using the popular TensorFlow framework, and provide a thorough evaluation in- cluding scalability, accuracy, variability, storage resource, GPU- GPU/GPU-CPU data transfer, and GPU utilization. Our analysis reveals that the latest heterogeneous supercomputing cluster shows varying performance trend as compared to the existing literature for single- and multi-node training. To the best of our knowledge, this is the first work to conduct such a quantitative and comprehensive study of DL training on a supercomputing system with multiple GPUs.},
doi = {10.1109/CLUSTER.2019.8890993},
url = {https://www.osti.gov/biblio/1569375},
place = {United States},
organization = {Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)},
year = {2019},
month = {09}}
2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), 2009 IEEE Conference on Computer Vision and Pattern Recognitionhttps://doi.org/10.1109/CVPR.2009.5206848
Kurth, Thorsten; Smorkalov, Mikhail; Deslippe, Jack
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17https://doi.org/10.1145/3126908.3126916
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17https://doi.org/10.1145/3126908.3126932