Combining In-situ and In-transit Processing to Enable Extreme-Scale Scientific Analysis

Conference · Thu Nov 01 04:00:00 EDT 2012

DOI:https://doi.org/10.1109/SC.2012.31· OSTI ID:1096981

Bennett, Janine C. ^[1]; Abbasi, Hasan ^[2]; Bremer, Peer-Timo ^[3]; Grout, Ray ^[4]; Gyulassy, Attila ^[5]; Jin, Tong ^[6]; Klasky, Scott A. ^[2]; Kolla, Hemanth ^[1]; Parashar, Manish ^[6]; Pascucci, Valerio ^[5]; Pebay, Philippe ^[7]; Thompson, David ^[7]; Yu, Hongfeng ^[1]; Zhang, Fan ^[6]; Chen, Jacqueline H. ^[1]

---

1. Sandia National Laboratories (SNL)
2. ORNL
3. Lawrence Livermore National Laboratory (LLNL)
4. National Renewable Energy Laboratory (NREL)
5. University of Utah
6. Rutgers University
7. Kitware

With the onset of extreme-scale computing, I/O constraints make it increasingly difficult for scientists to save a sufficient amount of raw simulation data to persistent storage. One potential solution is to change the data analysis pipeline from a post-process centric to a concurrent approach based on either in-situ or in-transit processing. In this context computations are considered in-situ if they utilize the primary compute resources, while in-transit processing refers to offloading computations toa set of secondary resources using asynchronous data transfers. In this paper we explore the design and implementation of three common analysis techniques typically performed on large-scale scientific simulations: topological analysis, descriptive statistics, and visualization. We summarize algorithmic developments, describe a resource scheduling system to coordinate the execution ofvarious analysis workflows, and discuss our implementation using the DataSpaces and ADIOS frameworks that support efficient data movement between in-situ and in-transit computations. Wedemonstrate the efficiency of our lightweight, flexible framework by deploying it on the Jaguar XK6 to analyze data generated by S3D, a massively parallel turbulent combustion code. Our framework allows scientists dealing with the data deluge at extreme scale to perform analyses at increased temporal resolutions, mitigate I/O costs, and significantly improve the time to insight.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 1096981
• Country of Publication: United States
• Language: English