Evaluation of pre-training large language models on leadership-class supercomputers

Yin, Junqi; Dash, Sajal; Gounley, John; Wang, Feiyi; Tourassi, Georgia

doi:10.1007/s11227-023-05479-7

Evaluation of pre-training large language models on leadership-class supercomputers

Journal Article · Fri Jun 16 00:00:00 EDT 2023 · Journal of Supercomputing

DOI:https://doi.org/10.1007/s11227-023-05479-7· OSTI ID:1994640

^[1]; Dash, Sajal ^[1]; Gounley, John ^[1]; ^[1]; ^[1]

Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Large language models (LLMs) have arisen rapidly to the center stage of artificial intelligence as the foundation models applicable to many downstream learning tasks. However, how to effectively build, train, and serve such models for many high-stake and first-principle-based scientific use cases are both of great interests and of great challenges. Moreover, pre-training LLMs with billions or even trillions of parameters can be prohibitively expensive not just for academic institutions, but also for well-funded industrial and government labs. Furthermore, the energy cost and the environmental impact of developing LLMs must be kept in mind. Here, in this work, we conduct a first-of-its-kind performance analysis to understand the time and energy cost of pre-training LLMs on the Department of Energy (DOE)’s leadership-class supercomputers. Employing state-of-the-art distributed training techniques, we evaluate the computational performance of various parallelization approaches at scale for a range of model sizes, and establish a projection model for the cost of full training. Our findings provide baseline results, best practices, and heuristics for pre-training such large models that should be valuable to HPC community at large. We also offer insights and optimization strategies for using the first exascale computing system, Frontier, to train models of the size of GPT-3 and beyond.

Research Organization:: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Basic Energy Sciences (BES). Scientific User Facilities (SUF)

Grant/Contract Number:: AC05-00OR22725

OSTI ID:: 1994640

Journal Information:: Journal of Supercomputing, Journal Name: Journal of Supercomputing Journal Issue: 18 Vol. 79; ISSN 0920-8542

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (14)

Quantifying the advantage of domain-specific pre-training on named entity recognition tasks in materials science Trewartha, Amalie; Walker, Nicholas; Huo, Haoyan Patterns, Vol. 3, Issue 4 https://doi.org/10.1016/j.patter.2022.100488	journal	April 2022
Comparative evaluation of deep learning workloads for leadership-class systems Yin, Junqi; Tsaris, Aristeidis; Dash, Sajal BenchCouncil Transactions on Benchmarks, Standards and Evaluations, Vol. 1, Issue 1 https://doi.org/10.1016/j.tbench.2021.100005	journal	October 2021
CORE: Three Access Levels to Underpin Open Access Knoth, Petr; Zdrahal, Zdenek D-Lib Magazine, Vol. 18, Issue 11/12 https://doi.org/10.1045/november2012-knoth	journal	November 2012
Accelerating Distributed Deep Learning Training with Compression Assisted Allgather and Reduce-Scatter Communication Zhou, Qinghua; Anthony, Quentin; Xu, Lang 2023 IEEE International Parallel and Distributed Processing Symposium (IPDPS) https://doi.org/10.1109/IPDPS54959.2023.00023	conference	May 2023
ZeRO: Memory optimizations Toward Training Trillion Parameter Models Rajbhandari, Samyam; Rasley, Jeff; Ruwase, Olatunji SC20: International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC41405.2020.00024	conference	November 2020
DeepSpeed Rasley, Jeff; Rajbhandari, Samyam; Ruwase, Olatunji Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3394486.3406703	conference	August 2020
Efficient large-scale language model training on GPU clusters using megatron-LM Narayanan, Deepak; Shoeybi, Mohammad; Casper, Jared Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1145/3458817.3476209	conference	November 2021
PyTorch distributed Li, Shen; Zhao, Yanli; Varma, Rohan Proceedings of the VLDB Endowment, Vol. 13, Issue 12 https://doi.org/10.14778/3415478.3415530	journal	August 2020
Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned Horawalavithana, Sameera; Ayton, Ellyn; Sharma, Shivam Proceedings of BigScience Episode #5 -- Workshop on Challenges & Perspectives in Creating Large Language Models https://doi.org/10.18653/v1/2022.bigscience-1.12	conference	January 2022
Zero-Shot Text-to-Image Generation Ramesh, Aditya; Pavlov, Mikhail; Goh, Gabriel arXiv https://doi.org/10.48550/ARXIV.2102.12092	preprint	January 2021
Training Compute-Optimal Large Language Models Hoffmann, Jordan; Borgeaud, Sebastian; Mensch, Arthur arXiv https://doi.org/10.48550/ARXIV.2203.15556	preprint	January 2022
OPT: Open Pre-trained Transformer Language Models Zhang, Susan; Roller, Stephen; Goyal, Naman arXiv https://doi.org/10.48550/ARXIV.2205.01068	preprint	January 2022
The Diminishing Returns of Masked Language Models to Science Hong, Zhi; Ajith, Aswathy; Pauloski, Gregory arXiv https://doi.org/10.48550/ARXIV.2205.11342	preprint	January 2022
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model Workshop, BigScience; Scao, Teven Le; Fan, Angela arXiv https://doi.org/10.48550/ARXIV.2211.05100	preprint	January 2022

Similar Records

Comparative Study of Large Language Model Architectures on Frontier

Conference · Wed May 01 00:00:00 EDT 2024 · OSTI ID:2406796

Optimizing Distributed Training on Frontier for Large Language Models

Conference · Wed May 01 00:00:00 EDT 2024 · OSTI ID:2438819

Revealing power, energy and thermal dynamics of a 200PF pre-exascale supercomputer

Conference · Mon Nov 01 00:00:00 EDT 2021 · OSTI ID:1833956

Related Subjects

97 MATHEMATICS AND COMPUTING
AI foundation model
distributed training
performance analysis

Evaluation of pre-training large language models on leadership-class supercomputers

Citation Formats

References (14)

Similar Records

Related Subjects