skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Fine-Grained Exploitation of Mixed Precision for Faster CNN Training

Conference ·

As deep convolutional neural networks (CNNs) have become increasingly popular and successful at an ever-widening number of machine learning tasks specialized hardware has become increasingly available for training and deploying them. NVIDIA's recent Volta architecture includes tensor cores which perform a fused operation reduced and mixed precision (16-bit multiply, 32-bit accumulate). Recent research indicates that, typically, very little is lost (in terms of training accuracy) when half precision is used in place of single precision, and performance gains can be made by doing arithmetic in reduced precision. In this work we demonstrate that making layer-by-layer choices as to the arithmetic/data precision can lead to further performance improvement. In our study of 25,200 CNNs we demonstrate an average speedup (over purely half precision) of 1.27x and speedups as high as 3.64x by appropriately combining single and half precision arithmetic and data types on a layer-by-layer basis.c

Research Organization:
Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
Sponsoring Organization:
DOE Contract Number:
Resource Relation:
Conference: IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) - Denver, Colorado, United States of America - 11/17/2019 9:00:00 AM-11/18/2019 9:00:00 AM
Country of Publication:
United States

Similar Records

Fine-Grained Exploitation of Mixed Precision for Faster CNN Training
Conference · 2019 · OSTI ID:1608214

Mixed-precision iterative refinement using tensor cores on GPUs to accelerate solution of linear systems
Journal Article · 2020 · Proceedings of the Royal Society. A. Mathematical, Physical and Engineering Sciences · OSTI ID:1608214

Improving scalability of parallel CNN training by adaptively adjusting parameter update frequency
Journal Article · 2021 · Journal of Parallel and Distributed Computing · OSTI ID:1608214

Related Subjects