FlexIO: I/O Middleware for Location-Flexible Scientific Data Analytics

Conference · Tue Jul 30 04:00:00 EDT 2013 · IEEE 27TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS 2013)

DOI:https://doi.org/10.1109/IPDPS.2013.46· OSTI ID:1567340

Zheng, Fang; Zou, Hongbo; Eisenhauer, Greg; Schwan, Karsten; Wolf, Matthew; Dayal, Jai; Nguyen, Tuan-Anh; Cao, Jianting; Abbasi, Hasan; Klasky, Scott; Podhorszki, Norbert; Yu, Hongfeng

Increasingly severe I/O bottlenecks on High-End Computing machines are prompting scientists to process simulation output data online while simulations are running and before storing data on disk. There are several options to place data analytics along the I/O path: on compute nodes, on separate nodes dedicated to analytics, or after data is stored on persistent storage. Since different placements have different impact on performance and cost, there is a consequent need for flexibility in the location of data analytics. The FlexIO middleware described in this paper makes it easy for scientists to obtain such flexibility, by offering simple abstractions and diverse data movement methods to couple simulation with analytics. Various placement policies can be built on top of FlexIO to exploit the trade-offs in performing analytics at different levels of the I/O hierarchy. Experimental results demonstrate that FlexIO can support a variety of simulation and analytics workloads at large scale through flexible placement options, efficient data movement, and dynamic deployment of data manipulation functionalities.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC)
• OSTI ID: 1567340
• Conference Information: Journal Name: IEEE 27TH INTERNATIONAL PARALLEL AND DISTRIBUTED PROCESSING SYMPOSIUM (IPDPS 2013)
• Country of Publication: United States
• Language: English

Related Subjects

Computer Science