DeepFreeze: Towards Scalable Asynchronous Checkpointing of Deep Learning Models

Conference · Tue Dec 31 23:00:00 EST 2019

DOI:https://doi.org/10.1109/CCGrid49817.2020.00-76· OSTI ID:1770321

Nicolae, Bogdan; Li, Jiali; Wozniak, Justin M.; Bosilca, George; Dorier, Matthieu; Cappello, Franck

In the age of big data, deep learning has emerged as a powerful tool to extract insight and exploit its value, both in industry and scientific applications. One common pattern emerging in such applications is frequent checkpointing of the state of the learning model during training, needed in a variety of scenarios: analysis of intermediate states to explain features and correlations with training data, exploration strategies involving alternative models that share a common ancestor, knowledge transfer, resilience, etc. However, with increasing size of the learning models and popularity of distributed data-parallel training approaches, simple checkpointing techniques used so far face several limitations: low serialization performance, blocking I/O, stragglers due to the fact that only a single process is involved in checkpointing. This paper proposes a checkpointing technique specifically designed to address the aforementioned limitations, introducing efficient asynchronous techniques to hide the overhead of serialization and I/O, and distribute the load over all participating processes. Experiments with two deep learning applications (CANDLE and ResNet) on a pre-Exascale HPC platform (Theta) shows significant improvement over state-of-art, both in terms of checkpointing duration and runtime overhead.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: USDOE Exascale Computing Project (ECP)
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 1770321
• Country of Publication: United States
• Language: English

Related Subjects

checkpointing
deep learning
fine-grain asynchronous I/O
multi-level data persistence
state preservation