Swift : fast, reliable, loosely coupled parallel computation.
A common pattern in scientific computing involves the execution of many tasks that are coupled only in the sense that the output of one may be passed as input to one or more others - for example, as a file, or via a Web Services invocation. While such 'loosely coupled' computations can involve large amounts of computation and communication, the concerns of the programmer tend to be different than in traditional high performance computing, being focused on management issues relating to the large numbers of datasets and tasks (and often, the complexities inherent in 'messy' data organizations) rather than the optimization of interprocessor communication. To address these concerns, we have developed Swift, a system that combines a novel scripting language called SwiftScript with a powerful runtime system based on CoG Karajan and Falkon to allow for the concise specification, and reliable and efficient execution, of large loosely coupled computations. Swift adopts and adapts ideas first explored in the GriPhyN virtual data system, improving on that system in many regards. We describe the SwiftScript language and its use of XDTM to describe the logical structure of complex file system structures. We also present the Swift system and its use of CoG Karajan, Falkon, and Globus services to dispatch and manage the execution of many tasks in different execution environments. We summarize application experiences and detail performance experiments that quantify the cost of Swift operations.
- Research Organization:
- Argonne National Lab. (ANL), Argonne, IL (United States)
- Sponsoring Organization:
- USDOE Office of Science (SC); National Science Foundation (NSF); National Institutes of Health (NIH)
- DOE Contract Number:
- DE-AC02-06CH11357
- OSTI ID:
- 971148
- Report Number(s):
- ANL/MCS/CP-59297; TRN: US201003%%596
- Resource Relation:
- Conference: IEEE 2007 International Conference on Web Services (ICWS); Jul. 7, 2007 - Jul. 13, 2007; Salt Lake City, UT
- Country of Publication:
- United States
- Language:
- ENGLISH
Similar Records
Multiprocessor execution of functional programs
Unified and Federated Storage Middleware for Unstructured Big Data Analytics and Management