TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML

doi:10.1002/cpe.4989

Title: TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML

Authors:

^[1]; Smorkalov, Mikhail ^[2]; Mendygral, Peter ^[3]; Sridharan, Srinivas ^[4]; Mathuriya, Amrita ^[5]

National Energy Research Scientific Computing CenterLawrence Berkeley National Laboratory Berkeley California
Software and Services GroupIntel Corporation Moscow Russia
Cray Programming Environments Performance EngineeringCray Inc Bloomington Minnesota
Parallel Computing LabsIntel Corporation Karnataka India
Data Center GroupIntel Corporation Hillsboro Oregon

Publication Date:: 2018-10-21

Sponsoring Org.:: USDOE

OSTI Identifier:: 1479562

Grant/Contract Number:: AC02-05CH11231

Resource Type:: Publisher's Accepted Manuscript

Journal Name:: Concurrency and Computation. Practice and Experience

Additional Journal Information:: Journal Name: Concurrency and Computation. Practice and Experience; Journal ID: ISSN 1532-0626

Publisher:: Wiley Blackwell (John Wiley & Sons)

Country of Publication:: United Kingdom

Language:: English

Citation Formats


                    Kurth, Thorsten, Smorkalov, Mikhail, Mendygral, Peter, Sridharan, Srinivas, and Mathuriya, Amrita. TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML.  United Kingdom: N. p., 2018. 
        Web.  doi:10.1002/cpe.4989.

Copy to clipboard


                    Kurth, Thorsten, Smorkalov, Mikhail, Mendygral, Peter, Sridharan, Srinivas, & Mathuriya, Amrita. TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML.  United Kingdom.  doi:10.1002/cpe.4989.

Copy to clipboard


                    Kurth, Thorsten, Smorkalov, Mikhail, Mendygral, Peter, Sridharan, Srinivas, and Mathuriya, Amrita. Sun .  
        "TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML".  United Kingdom.  doi:10.1002/cpe.4989.

Copy to clipboard


                    
@article{osti_1479562,

  title        = {TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML},

  author       = {Kurth, Thorsten and Smorkalov, Mikhail and Mendygral, Peter and Sridharan, Srinivas and Mathuriya, Amrita},

  abstractNote = {},

  doi          = {10.1002/cpe.4989},

  journal      = {Concurrency and Computation. Practice and Experience},

  number       = ,

  volume       = ,

  place        = {United Kingdom},

  year         = {2018},

  month        = {10}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Publisher's Version of Record
DOI: 10.1002/cpe.4989

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 1 work

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Backpropagation and stochastic gradient descent method
journal, June 1993

Amari, Shun-ichi
Neurocomputing, Vol. 5, Issue 4-5
DOI: 10.1016/0925-2312(93)90006-O

Deep learning at 15PF: supervised and semi-supervised classification for scientific data
conference, January 2017

Kurth, Thorsten; Smorkalov, Mikhail; Deslippe, Jack
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17
DOI: 10.1145/3126908.3126916

DELPHES 3: a modular framework for fast simulation of a generic collider experiment
journal, February 2014

de Favereau, J.; Delaere, C.; Demin, P.
Journal of High Energy Physics, Vol. 2014, Issue 2
DOI: 10.1007/JHEP02(2014)057

Similar Records in DOE PAGES and OSTI.GOV collections:

Performance analysis of Darshan 2.2.3 on the Cray XE6 platform.

Technical Report Carns, P ; Harms, K ; Latham, R ; ...
DOI: 10.2172/1055420

Full Text Available
EDA and ML - A Perfect Pair for Large-Scale Data Analysis

Conference Hafen, Ryan P. ; Critchlow, Terence J.

This short position paper presents our view of how exploratory data analysis techiniques and machine learning methods can be applied together for large-scale data analysis.
DOI: 10.1109/IPDPSW.2013.118
Comand: Context-Aware ML Middleware for Real-Time Distributed Streaming Big Data Analysis

Technical Report Xie, Bin

DOE and many applications needs big data streaming capability to process massive energy or relevant data sets using Machine Learning (ML) algorithms in a context-aware and real-time fashion. Such a capability is able to quickly process and analyze the big data in a real-time fashion to extract useful information in support of decision making. Previous approaches are very limited in such a sophisticate capability. In this project, InfoBeyond advocates the development of COMAND technology to enable context-aware and real-time ML streaming capability. Context means that the big data exhibit different statistic properties along the time and other environmental factors. Meanwhile,more »« less
Implementation and performance analysis of numerical algorithms on the MPP, Flex/32,and Cray/2

Thesis/Dissertation Fatoohi, R A

Results are presented of the implementation of a number of numerical algorithms on three parallel/vector computers. The object of this research is to determine how well, or poorly, a number of numerical algorithms would map onto three different architectures and to analyze the performance of these architectures using these algorithms. These algorithms are: a relaxation scheme for the solution of the Cauchy-Riemann equations, an ADI method for the solution of the diffusion equations, and a compact difference scheme for the solution of two-dimensional Navier-Stokes equations. The computers were chosen so as to encompass a variety of architectures. They are: themore »« less
Experiences in the Performance Analysis and Optimization of a Deterministic Radiation Transport Code on the Cray SV1

Conference Cebull, Peter

The Attila radiation transport code, which solves the Boltzmann neutron transport equation on three-dimensional unstructured tetrahedral meshes, was ported to a Cray SV1. Cray's performance analysis tools pointed to two subroutines that together accounted for 80%-90% of the total CPU time. Source code modifications were performed to enable vectorization of the most significant loops, to correct unfavorable strides through memory, and to replace a conjugate gradient solver subroutine with a call to the Cray Scientific Library. These optimizations resulted in a speedup of 7.79 for the INEEL's largest ATR model. Parallel scalability of the OpenMP version of the code ismore »« less
Full Text Available

Similar Records

Title: TensorFlow at Scale: Performance and productivity analysis of distributed training with Horovod, MLSL, and Cray PE ML

Citation Formats

Backpropagation and stochastic gradient descent method journal, June 1993

Deep learning at 15PF: supervised and semi-supervised classification for scientific data conference, January 2017

DELPHES 3: a modular framework for fast simulation of a generic collider experiment journal, February 2014

Backpropagation and stochastic gradient descent method
journal, June 1993

Deep learning at 15PF: supervised and semi-supervised classification for scientific data
conference, January 2017

DELPHES 3: a modular framework for fast simulation of a generic collider experiment
journal, February 2014