DataSpaces: an interaction and coordination framework for coupled simulation workflows
Emerging high-performance distributed computing environments are enabling new end-to-end formulations in science and engineering that involve multiple interacting processes and data-intensive application workflows. For example, current fusion simulation efforts are exploring coupled models and codes that simultaneously simulate separate application processes, such as the core and the edge turbulence. These components run on different high performance computing resources, need to interact at runtime with each other and with services for data monitoring, data analysis and visualization, and data archiving. As a result, they require efficient and scalable support for dynamic and flexible couplings and interactions, which remains a challenge. This paper presents DataSpaces a flexible interaction and coordination substrate that addresses this challenge. DataSpaces essentially implements a semantically specialized virtual shared space abstraction that can be associatively accessed by all components and services in the application workflow. It enables live data to be extracted from running simulation components, indexes this data online, and then allows it to be monitored, queried and accessed by other components and services via the space using semantically meaningful operators. The underlying data transport is asynchronous, low-overhead and largely memory-to-memory. The design, implementation, and experimental evaluation of DataSpaces using a coupled fusion simulation workflow is presented.
- Research Organization:
- Rutgers Univ., Piscataway, NJ (United States); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF)
- Sponsoring Organization:
- USDOE Office of Science (SC)
- DOE Contract Number:
- FC02-06ER54857
- OSTI ID:
- 1564811
- Journal Information:
- Cluster Computing, Vol. 15, Issue 2; ISSN 1386-7857
- Publisher:
- Springer
- Country of Publication:
- United States
- Language:
- English
Similar Records
Enabling In-situ Execution of Coupled Scientific Workflow on Multi-core Platform
CoREC: Scalable and Resilient In-memory Data Staging for In-situ Workflows