A Job Sizing Strategy for High-Throughput Scientific Workflows

Journal Article · Thu Feb 01 04:00:00 EST 2018 · IEEE Transactions on Parallel and Distributed Systems

DOI:https://doi.org/10.1109/TPDS.2017.2762310· OSTI ID:1472078

Tovar, Benjamin; da Silva, Rafael Ferreira; Juve, Gideon; Deelman, Ewa; Allcock, William; Thain, Douglas; Livny, Miron

The user of a computing facility must make a critical decision when submitting jobs for execution: how many resources (such as cores, memory, and disk) should be requested for each job? If the request is too small, the job may fail due to resource exhaustion; if the request is too big, the job may succeed, but resources will be wasted. This decision is especially important when running hundreds of thousands of jobs in a high throughput workflow, which may exhibit complex, long tailed distributions of resource consumption. In this paper, we present a strategy for solving the job sizing problem: (1) applications are monitored and measured in user-space as they run; (2) the resource usage is collected into an online archive; and (3) jobs are automatically sized according to historical data in order to maximize throughput or minimize waste. We evaluate the solution analytically, and present case studies of applying the technique to high throughput physics and bioinformatics workflows consisting of hundreds of thousands of jobs, demonstrating an increase in throughput of 10-400 percent compared to naive approaches.

• Research Organization: Argonne National Laboratory (ANL)
• Sponsoring Organization: USDOE Office of Science - Office of Basic Energy Sciences
• DOE Contract Number: AC02-06CH11357
• OSTI ID: 1472078
• Journal Information: IEEE Transactions on Parallel and Distributed Systems, Journal Name: IEEE Transactions on Parallel and Distributed Systems Journal Issue: 2 Vol. 29; ISSN 1045-9219
• Publisher: IEEE
• Country of Publication: United States
• Language: English