Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems

Yildirim, Esma

doi:10.5281/zenodo.15545096

Title: Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems

Dataset
Other Related Research

Abstract

This is a time series data set of resource utilizations and runtime for jobs run on both HPC systems (IC2 at Brookhaven Naional Lab, Polaris at Argonne National Lab and Amazon Web Services. The data set can be used for machine learning models to predict runtime and resource utilization of jobs on a variety of systems.

Authors:

Queensborough Community College, CUNY; Rutgers University - Busch Campus

Publication Date:: Wed Jan 01 04:00:00 UTC 2025

DOE Contract Number:: SC0012704; AC02-06CH11357

Research Org.:: Brookhaven National Laboratory (BNL), Upton, NY (United States)

Sponsoring Org.:: U.S. National Science Foundation; USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR); USDOE Office of Science (SC), Office of Workforce Development for Teachers & Scientists (WDTS)

Subject:: cloud computing; high performance computing

OSTI Identifier:: 3005909

DOI:: https://doi.org/10.5281/zenodo.15545096

Citation Formats


                    Yildirim, Esma. Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems.  United States: N. p., 2025. 
        Web.  doi:10.5281/zenodo.15545096.

Copy to clipboard


                    Yildirim, Esma. Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems.  United States.  doi:https://doi.org/10.5281/zenodo.15545096

Copy to clipboard


                    Yildirim, Esma. 2025.  
"Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems".  United States.  doi:https://doi.org/10.5281/zenodo.15545096.  https://www.osti.gov/servlets/purl/3005909. Pub date:Wed Jan 01 04:00:00 UTC 2025

Copy to clipboard


                    
@article{osti_3005909,

  title        = {Predicting Runtime and Resource Utilization of Jobs on Integrated Cloud and HPC Systems},

  author       = {Yildirim, Esma},

  abstractNote = {This is a time series data set of resource utilizations and runtime for jobs run on both HPC systems (IC2 at Brookhaven Naional Lab,  Polaris at Argonne National Lab and Amazon Web Services. The data set can be used for machine learning models to predict runtime and resource utilization of jobs on a variety of systems.},

  doi          = {10.5281/zenodo.15545096},

  journal      = {},

  number       = ,

  volume       = ,

  place        = {United States},

  year         = {Wed Jan 01 04:00:00 UTC 2025},

  month        = {Wed Jan 01 04:00:00 UTC 2025}

}

Copy to clipboard

Dataset:

View Dataset

DOI: https://doi.org/10.5281/zenodo.15545096

Save / Share:

Export Metadata

Save to My Library

Similar records in DOE Data Explorer and OSTI.GOV collections:

NREL HPC Eagle Jobs Data

Dataset Eash, Matthew ; Egan, Hilary

3 months of Eagle jobs data including Slurm metadata and derived power use characterization.
HPC ontology dataset

Dataset Liao, Chunhua ; Lin, Pei-Hung
Modeling the Integration of Marine Energy into Microgrids - Wave Resource Assessment

Dataset Mankle, Hannah ; Robertson, Bryson

This submission has wave resource assessments which were conducted for six locations based on IEC requirements using the DOE WPTO Hindcast data and MHKiT. The locations are chosen to provide varying wave climates and include PacWave South, OR; Wave Energy Testing Site (WETS), HI; Molokai, HI; St. Paul, AK; Yakutat, Ak; and Sebastion, FL. It includes the data gathered and the resulting report. This submission also includes a link to Hindcast dataset and some relevant software.
SMC 2021 : Analyzing Resource Utilization and User Behavior on Titan Supercomputer

Dataset Dash, Sajal

Resource utilization statistics of submitted jobs on a supercomputer can help us understand how users from various scientific domains use HPC platforms and better design a job scheduler. We explore to generate insight regarding workload distribution and usage pattern domains from job scheduler trace, GPU failure information, and project-specific information collected from Titan supercomputer. Furthermore, we want to know how the scheduler performance varies over time and how the usersâ scheduling behavior changes following a system failure. These observations have the potential to provide valuable insight, which is helpful to prepare for system failures. These practices will help us developmore »« less
Life Cycle Water Consumption and Water Resource Assessment for Utility-Scale Geothermal Systems: An In-Depth Analysis of Historical and Forthcoming EGS Projects

Dataset Schroeder, Jenna N.

This report is the third in a series of reports sponsored by the U.S. Department of Energy Geothermal Technologies Program in which a range of water-related issues surrounding geothermal power production are evaluated. The first report made an initial attempt at quantifying the life cycle fresh water requirements of geothermal power-generating systems and explored operational and environmental concerns related to the geochemical composition of geothermal fluids. The initial analysis of life cycle fresh water consumption of geothermal power-generating systems identified that operational water requirements consumed the vast majority of water across the life cycle. However, it relied upon limited operationalmore »« less

Similar Records