iDDS: intelligent distributed dispatch and scheduling for workflow orchestration

Journal Article · Sat Jan 24 00:00:00 EST 2026 · European Physical Journal. C, Particles and Fields (Online)

DOI:https://doi.org/10.1140/epjc/s10052-025-15275-7· OSTI ID:3017617

Guan, Wen ^[1]; Maeno, Tadashi ^[1]; Alekseev, Aleksandr ^[2]; Megino, Fernando Harald Barreiro ^[2]; De, Kaushik ^[2]; Karavakis, Edward ^[1]; Klimentov, Alexei ^[1]; Korchuganova, Tatiana ^[3]; Lin, FaHui ^[2]; Nilsson, Paul ^[1]; Wenaus, Torre ^[1]; Yang, Zhaoyu ^[1]; Zhao, Xin ^[1]

---

1. Brookhaven National Laboratory (BNL), Upton, NY (United States)
2. Univ. of Texas, Arlington, TX (United States)
3. Univ. of Pittsburgh, PA (United States)

The intelligent distributed dispatch and scheduling (iDDS) service is a versatile workflow orchestration system designed for large-scale, distributed scientific computing. iDDS extends traditional workload and data management by integrating data-aware execution, conditional logic, and programmable workflows, enabling automation of complex and dynamic processing pipelines. Originally developed for the ATLAS experiment at the large hadron collider, iDDS has evolved into an experiment-agnostic platform that supports both template-driven workflows and a Function-as-a-Task model for Python-based orchestration. This paper presents the architecture and core components of iDDS, highlighting its scalability, modular message-driven design, and integration with systems such as PanDA and Rucio. We demonstrate its versatility through real-world use cases: fine-grained tape resource optimization for ATLAS, orchestration of large Directed Acyclic Graph (DAG) workflows for the Rubin Observatory, distributed hyperparameter optimization for machine learning applications, active learning for physics analyses, and AI-assisted detector design at the electron–ion collider. By unifying workload scheduling, data movement, and adaptive decision-making, iDDS reduces operational overhead and enables reproducible, high-throughput workflows across heterogeneous infrastructures. We conclude with current challenges and future directions, including interactive, cloud-native, and serverless workflow support.

• Research Organization: Brookhaven National Laboratory (BNL), Upton, NY (United States)
• Sponsoring Organization: National Science Foundation (NSF); USDOE Office of Science (SC), Nuclear Physics (NP)
• Grant/Contract Number: SC0012704
• Other Award/Contract Number: OAC-1836650
• OSTI ID: 3017617
• Report Number(s): BNL--229406-2026-JAAM
• Journal Information: European Physical Journal. C, Particles and Fields (Online), Journal Name: European Physical Journal. C, Particles and Fields (Online) Journal Issue: 1 Vol. 86; ISSN 1434-6052
• Publisher: Springer Nature
• Country of Publication: United States
• Language: English

Related Subjects

97 MATHEMATICS AND COMPUTING
Automation
Big Data
Cloud Computing
Data integration
Data-driven Science
Modeling and Theory Building
Intelligence Infrastructure