Usage Pattern-Driven Dynamic Data Layout Reorganization

Conference · Sun May 01 04:00:00 EDT 2016

DOI:https://doi.org/10.1109/CCGrid.2016.15· OSTI ID:1567419

Tang, Houjun; Byna, Suren; Harenberg, Steve; Zou, Xiaocheng; Zhang, Wenzhao; Wu, Kesheng; Dong, Bin; Rubel, Oliver; Bouchard, Kristofer; Klasky, Scott; Samatova, Nagiza F.

As scientific simulations and experiments move toward extremely large scales and generate massive amounts of data, the data access performance of analytic applications becomes crucial. A mismatch often happens between write and read patterns of data accesses, typically resulting in poor read performance. Data layout reorganization has been used to improve the locality of data accesses. However, current data reorganizations are static and focus on generating a single (or set of) optimized layouts that rely on prior knowledge of exact future access patterns. We propose a framework that dynamically recognizes the data usage patterns, replicates the data of interest in multiple reorganized layouts that would benefit common read patterns, and makes runtime decisions on selecting a favorable layout for a given read pattern. This framework supports reading individual elements and chunks of a multi-dimensional array of variables. Our pattern-driven layout selection strategy achieves multi-fold speedups compared to reading from the original dataset.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
• Sponsoring Organization: USDOE Office of Science (SC)
• OSTI ID: 1567419
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Computer Science
Engineering