Large Scale Caching and Streaming of Training Data for Online Deep Learning

Conference · 01 July 2022

DOI:https://doi.org/10.1145/3526058.3535453· OSTI ID:1887185

Liu, Jie; Nicolae, Bogdan; Li, Dong; Wozniak, Justin M.; Bicer, Tekin; Liu, Zhengchun; Foster, Ian

The training of deep neural network models on large data remains a difficult problem, despite progress towards scalable techniques. In particular, there is a mismatch between the random but predetermined order in which AI flows select training samples and the streaming I/O patterns for which traditional HPC data storage (e.g., parallel file systems) are designed. In addition, as more data are obtained, it is feasible neither simply to train learning models incrementally, due to catastrophic forgetting (i.e., bias towards new samples), nor to train frequently from scratch, due to prohibitive time and/or resource constraints. In this paper, we study data management techniques that combine caching and streaming with rehearsal support in order to enable efficient access to training samples in both offline training and continual learning. We revisit state-of-art streaming approaches based on data pipelines that transparently handle prefetching, caching, shuffling, and data augmentation, and discuss the challenges and opportunities that arise when combining these methods with data-parallel training techniques. We also report on preliminary experiments that evaluate the I/O overheads involved in accessing the training samples from a parallel file system (PFS) under several concurrency scenarios, highlighting the impact of the PFS on the design of the data pipelines.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: USDOE Office of Science - Office of Advanced Scientific Computing Research (ASCR)
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 1887185
• Country of Publication: United States
• Language: English

Related Subjects

data pipelines
deep learning
distributed caching
reuse of training data