Methods and Experiences for Developing Abstractions for Data-intensive, Scientific Applications

Conference · Tue Jul 28 04:00:00 EDT 2020 · 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)

DOI:https://doi.org/10.1109/ipdpsw50202.2020.00106· OSTI ID:1777424

Luckow, Andre ^[1]; Jha, Shantenu ^[2]

---

1. Ludwig Maximilian Univ. of Munich, Munich (Germany)
2. Rutgers Univ., Piscataway, NJ (United States)

Developing software for scientific applications that require the integration of diverse types of computing, instruments, and data present challenges that are distinct from commercial software. These applications require scale, and the need to integrate various programming and computational models with evolving and heterogeneous infrastructure. Pervasive and effective abstractions for distributed infrastructures are thus critical; however, the process of developing abstractions for scientific applications and infrastructures is not well understood. While theory-based approaches for system development are suited for well-defined, closed environments, they have severe limitations for designing abstractions for scientific systems and applications. The design science research (DSR) method provides the basis for designing practical systems that can handle real-world complexities at all levels. In contrast to theory-centric approaches, DSR emphasizes both practical relevance and knowledge creation by building and rigorously evaluating all artifacts. In this work, we show how DSR provides a well-defined framework for developing abstractions and middleware systems for distributed systems. Specifically, we address the critical problem of distributed resource management on heterogeneous infrastructure over a dynamic range of scales, a challenge that currently limits many scientific applications. We use the pilot-abstraction, a widely used resource management abstraction for high-performance, high throughput, big data, and streaming applications, as a case study for evaluating the DSR activities. For this purpose, we analyze the research process and artifacts produced during the design and evaluation of the pilot-abstraction. We find DSR provides a concise framework for iteratively designing and evaluating systems. Finally, we capture our experiences and formulate different lessons learned.

• Research Organization: Brookhaven National Laboratory (BNL), Upton, NY (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research
• DOE Contract Number: SC0012704
• OSTI ID: 1777424
• Report Number(s): BNL--221188-2021-JAAM; ISBN: 978-1-7281-7445-7
• Conference Information: Journal Name: 2020 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW) Journal Volume: 2020
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Benchmark testing
Complexity theory
Computational modeling
Computer architecture
Distributed databases
Resource management
Software