Flexible Pilot Jobs Framework for Distributed High Throughput Computing

Technical Report · Thu Oct 02 04:00:00 EDT 2025

DOI:https://doi.org/10.2172/2998402· OSTI ID:2998402

Terranova, Franco ^[1]; Mambelli, Marco ^[1]

---

1. Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)

Experimental particle physics has been at the forefront of analyzing the world’s largest datasets for decades. The high-energy physics (HEP) community was among the first to develop suitable software and computing tools for this purpose. GlideinWMS is a Glidein-based workload management system whose purpose is to provide experiments like CMS at CERN, DUNE at Fermilab, and others, a way to access and efficiently use vast amounts of computing resources. This system wants to provide a simple way to submit jobs to a set of computing resources, that will be provided to users behind the scenes. Glideins are the pilot jobs executed on the worker nodes at the grid sites, performing operations such as hardware detection, environment setup, and error handling. After all these operations, they will launch the actual user job. Many grid sites are supported, such as shared clusters, Google CE, and AWS. My internship aimed to design and code a flexible pilot jobs framework that will replace the one used by GlideinWMS, developing a modular and flexible skeleton of the Glidein and adding further functionalities. My project also focused on the application of machine learning techniques as support to this management system.

• Research Organization: Fermi National Accelerator Laboratory (FNAL), Batavia, IL (United States)
• Sponsoring Organization: USDOE Office of Science (SC), High Energy Physics (HEP)
• DOE Contract Number: 89243024CSC000002
• OSTI ID: 2998402
• Report Number(s): FERMILAB-TM--2895-STUDENT; oai:inspirehep.net:3033775
• Country of Publication: United States
• Language: English

Related Subjects

72 PHYSICS OF ELEMENTARY PARTICLES AND FIELDS
96 KNOWLEDGE MANAGEMENT AND PRESERVATION
97 MATHEMATICS AND COMPUTING