A model for optimizing file access patterns using spatio-temporal parallelism
- Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
- Kitware Inc., Clifton Park, NY (United States)
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)
- Lawrence Livermore National Lab. (LLNL), Livermore, CA (United States)
For many years now, I/O read time has been recognized as the primary bottleneck for parallel visualization and analysis of large-scale data. In this paper, we introduce a model that can estimate the read time for a file stored in a parallel filesystem when given the file access pattern. Read times ultimately depend on how the file is stored and the access pattern used to read the file. The file access pattern will be dictated by the type of parallel decomposition used. We employ spatio-temporal parallelism, which combines both spatial and temporal parallelism, to provide greater flexibility to possible file access patterns. Using our model, we were able to configure the spatio-temporal parallelism to design optimized read access patterns that resulted in a speedup factor of approximately 400 over traditional file access patterns.
- Research Organization:
- Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- AC05-00OR22725
- OSTI ID:
- 1154873
- Resource Relation:
- Conference: Ultrascale Visualization Workshop, 20131117, 20131117, Denver, CO, USA
- Country of Publication:
- United States
- Language:
- English
Similar Records
Optimizing tertiary storage organization and access for spatio-temporal datasets
...And Eat it Too: High Read Performance in Write-Optimized HPC I/O Middleware File Formats