CosmoFlow: Using Deep Learning to Learn the Universe at Scale

Mathuriya, Amrita; Bard, Deborah; Mendygral, Peter; Meadows, Lawrence; Arnemann, James; Shao, Lei; He, Siyu; Karna, Tuomas; Moise, Diana; Pennycook, Simon J.; Maschhoff, Kristyn; Sewall, Jason; Kumar, Nalini; Ho, Shirley; Ringenburg, Michael F.; Prabhat, Prabhat; Lee, Victor

doi:10.1109/sc.2018.00068

CosmoFlow: Using Deep Learning to Learn the Universe at Scale

Journal Article · Thu Mar 14 00:00:00 EDT 2019 · International Conference for High Performance Computing, Networking, Storage and Analysis

DOI:https://doi.org/10.1109/sc.2018.00068· OSTI ID:1510756

Mathuriya, Amrita ^[1]; Bard, Deborah ^[2]; Mendygral, Peter ^[3]; Meadows, Lawrence ^[1]; Arnemann, James ^[4]; Shao, Lei ^[5]; He, Siyu ^[6]; Karna, Tuomas ^[1]; Moise, Diana ^[3]; Pennycook, Simon J. ^[5]; Maschhoff, Kristyn ^[3]; Sewall, Jason ^[5]; Kumar, Nalini ^[5]; Ho, Shirley ^[6]; Ringenburg, Michael F. ^[3]; Prabhat, Prabhat ^[2]; Lee, Victor ^[5]

Intel Corp., Hillsboro, OR (United States)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Cray Inc., Seattle, WA (United States)
Univ. of California, Berkeley, CA (United States)
Intel Corp., Santa Clara, CA (United States)
Flatiron Inst., New York, NY (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Carnegie Mellon Univ., Pittsburgh, PA (United States)

Deep learning is a promising tool to determine the physical model that describes our universe. To handle the considerable computational cost of this problem, we present CosmoFlow: a highly scalable deep learning application built on top of the TensorFlow framework. CosmoFlow uses efficient implementations of 3D convolution and pooling primitives, together with improvements in threading for many element-wise operations, to improve training performance on Intel^® Xeon Phi™ processors. We also utilize the Cray PE Machine Learning Plugin for efficient scaling to multiple nodes. We demonstrate fully synchronous data-parallel training on 8192 nodes of Cori with 77% parallel efficiency, achieving 3.5 Pflop/s sustained performance. To our knowledge, this is the first large-scale science application of the TensorFlow framework at supercomputer scale with fully-synchronous training. Here, these enhancements enable us to process large 3D dark matter distribution and predict the cosmological parameters Ω_M, σ₈ and n_s with unprecedented accuracy.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1510756

Journal Information:: International Conference for High Performance Computing, Networking, Storage and Analysis, Journal Name: International Conference for High Performance Computing, Networking, Storage and Analysis Vol. 2018; ISSN 2167-4329

Publisher:: IEEECopyright Statement

Country of Publication:: United States

Language:: English

References (19)

Solving large scale structure in ten easy steps with COLA Tassev, Svetlin; Zaldarriaga, Matias; Eisenstein, Daniel J. Journal of Cosmology and Astroparticle Physics, Vol. 2013, Issue 06 https://doi.org/10.1088/1475-7516/2013/06/036	journal	June 2013
FireCaffe: Near-Linear Acceleration of Deep Neural Network Training on Compute Clusters Iandola, Forrest N.; Moskewicz, Matthew W.; Ashraf, Khalid 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/cvpr.2016.284	conference	June 2016
Hsp90 is important for fecundity, longevity, and buffering of cryptic deleterious variation in wild fly populations Chen, Bing; Wagner, Andreas BMC Evolutionary Biology, Vol. 12, Issue 1 https://doi.org/10.1186/1471-2148-12-25	journal	January 2012
cuDNN: Efficient Primitives for Deep Learning Chetlur, Sharan; Woolley, Cliff; Vandermersch, Philippe arXiv https://doi.org/10.48550/arxiv.1410.0759	preprint	January 2014
sCOLA: The N-body COLA Method Extended to the Spatial Domain Tassev, Svetlin; Eisenstein, Daniel J.; Wandelt, Benjamin D. arXiv https://doi.org/10.48550/arxiv.1502.07751	preprint	January 2015
Distributed Deep Learning Using Synchronous Stochastic Gradient Descent Das, Dipankar; Avancha, Sasikanth; Mudigere, Dheevatsa arXiv https://doi.org/10.48550/arxiv.1602.06709	preprint	January 2016
Enabling Dark Energy Science with Deep Generative Models of Galaxy Images Ravanbakhsh, Siamak; Lanusse, Francois; Mandelbaum, Rachel arXiv https://doi.org/10.48550/arxiv.1609.05796	preprint	January 2016
WOMBAT: A Scalable and High Performance Astrophysical MHD Code Mendygral, Peter; Radcliffe, Nick; Kandalla, Krishna arXiv https://doi.org/10.48550/arxiv.1701.07452	text	January 2017
Cosmological model discrimination with Deep Learning Schmelzle, Jorit; Lucchi, Aurelien; Kacprzak, Tomasz arXiv https://doi.org/10.48550/arxiv.1707.05167	preprint	January 2017
Evaluating the networking characteristics of the Cray XC-40 Intel Knights Landing-based Cori supercomputer at NERSC: Evaluating the Networking Characteristics of the Cray XC-40 Intel Knights Landing Based Cori Supercomputer at NERSC Doerfler, Douglas; Austin, Brian; Cook, Brandon Concurrency and Computation: Practice and Experience, Vol. 30, Issue 1 https://doi.org/10.1002/cpe.4297	journal	September 2017
Planck 2015 results : XIII. Cosmological parameters Ade, P. A. R.; Aghanim, N.; Arnaud, M. Astronomy & Astrophysics, Vol. 594 https://doi.org/10.1051/0004-6361/201525830	journal	September 2016
Rotation-invariant convolutional neural networks for galaxy morphology prediction Dieleman, Sander; Willett, Kyle W.; Dambre, Joni Monthly Notices of the Royal Astronomical Society, Vol. 450, Issue 2 https://doi.org/10.1093/mnras/stv632	journal	April 2015
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Distributed asynchronous deterministic and stochastic gradient optimization algorithms Tsitsiklis, J.; Bertsekas, D.; Athans, M. IEEE Transactions on Automatic Control, Vol. 31, Issue 9 https://doi.org/10.1109/TAC.1986.1104412	journal	September 1986
Multi-scale initial conditions for cosmological simulations: Multi-scale initial conditions Hahn, Oliver; Abel, Tom Monthly Notices of the Royal Astronomical Society, Vol. 415, Issue 3 https://doi.org/10.1111/j.1365-2966.2011.18820.x	journal	July 2011
In-Datacenter Performance Analysis of a Tensor Processing Unit Jouppi, Norman P.; Borchers, Al; Boyle, Rick Proceedings of the 44th Annual International Symposium on Computer Architecture - ISCA '17 https://doi.org/10.1145/3079856.3080246	conference	January 2017
Deep learning at 15PF: supervised and semi-supervised classification for scientific data Kurth, Thorsten; Smorkalov, Mikhail; Deslippe, Jack Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '17 https://doi.org/10.1145/3126908.3126916	conference	January 2017
Evolving Deep Networks Using HPC Young, Steven R.; Rose, Derek C.; Johnston, Travis Proceedings of the Machine Learning on HPC Environments - MLHPC'17 https://doi.org/10.1145/3146347.3146355	conference	January 2017
WOMBAT: A Scalable and High-performance Astrophysical Magnetohydrodynamics Code Mendygral, P. J.; Radcliffe, N.; Kandalla, K. The Astrophysical Journal Supplement Series, Vol. 228, Issue 2 https://doi.org/10.3847/1538-4365/aa5b9c	journal	February 2017

Cited By (12)

A computational-graph partitioning method for training memory-constrained DNNs Qararyah, Fareed; Wahib, Mohamed; Dikbayır, Doğa Parallel Computing, Vol. 104-105 https://doi.org/10.1016/j.parco.2021.102792	journal	July 2021
DisCo: Physics-Based Unsupervised Discovery of Coherent Structures in Spatiotemporal Systems Rupe, Adam; Prabhat, Mr; Crutchfield, James P. 2019 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments (MLHPC) https://doi.org/10.1109/mlhpc49564.2019.00013	conference	November 2019
Exascale Deep Learning for Climate Analytics Kurth, Thorsten; Treichler, Sean; Romero, Joshua SC18: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/sc.2018.00054	conference	November 2018
Clairvoyant prefetching for distributed machine learning I/O Dryden, Nikoli; Böhringer, Roman; Ben-Nun, Tal Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3458817.3476181	conference	November 2021
HPC AI500: A Benchmark Suite for HPC AI Systems Jiang, Zihan; Gao, Wanling; Wang, Lei arXiv https://doi.org/10.48550/arxiv.1908.02607	preprint	January 2019
Parallelizing Training of Deep Generative Models on Massive Scientific Datasets Jacobs, Sam Ade; Van Essen, Brian; Hysom, David arXiv https://doi.org/10.48550/arxiv.1910.02270	preprint	January 2019
Derivation and Analysis of Fast Bilinear Algorithms for Convolution Ju, Caleb; Solomonik, Edgar arXiv https://doi.org/10.48550/arxiv.1910.13367	preprint	January 2019
Response to NITRD, NCO, NSF Request for Information on "Update to the 2016 National Artificial Intelligence Research and Development Strategic Plan" Amundson, J.; Annis, J.; Avestruz, C. arXiv https://doi.org/10.48550/arxiv.1911.05796	preprint	January 2019
The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism Oyama, Yosuke; Maruyama, Naoya; Dryden, Nikoli arXiv https://doi.org/10.48550/arxiv.2007.12856	preprint	January 2020
Quasar Detection using Linear Support Vector Machine with Learning From Mistakes Methodology Herle, Aniruddh; Channegowda, Janamejaya; Prabhu, Dinakar arXiv https://doi.org/10.48550/arxiv.2010.00401	text	January 2020
Learning to predict the cosmological structure formation He, Siyu; Li, Yin; Feng, Yu Proceedings of the National Academy of Sciences, Vol. 116, Issue 28 https://doi.org/10.1073/pnas.1821458116	journal	June 2019
Learning to Predict the Cosmological Structure Formation He, Siyu; Li, Yin; Feng, Yu arXiv https://doi.org/10.48550/arxiv.1811.06533	text	January 2018

Similar Records

Preparing NERSC users for Cori, a Cray XC40 system with Intel many integrated cores

Journal Article · Thu Aug 24 20:00:00 EDT 2017 · Concurrency and Computation. Practice and Experience · OSTI ID:1459400

The Case for Strong Scaling in Deep Learning: Training Large 3D CNNs with Hybrid Parallelism

Journal Article · Tue Dec 29 19:00:00 EST 2020 · IEEE Transactions on Parallel and Distributed Systems · OSTI ID:1959404

Cataloging the visible universe through Bayesian inference in Julia at petascale

Journal Article · Wed May 01 00:00:00 EDT 2019 · Journal of Parallel and Distributed Computing · OSTI ID:1527362

Related Subjects

98 NUCLEAR DISARMAMENT, SAFEGUARDS, AND PHYSICAL PROTECTION
Cosmology
Deep Learning
High Performance Computing
Machine Learning
TensorFlow

CosmoFlow: Using Deep Learning to Learn the Universe at Scale

Citation Formats

References (19)

Cited By (12)

Similar Records

Related Subjects