skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: GPU age-aware scheduling to improve the reliability of leadership jobs on Titan

Abstract

In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ~50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.


Citation Formats

Zimmer, Christopher J., Maxwell, Don E., Mcnally, Stephen T., Atchley, Scott, and Vazhkudai, Sudharshan S. GPU age-aware scheduling to improve the reliability of leadership jobs on Titan. United States: N. p., 2018. Web.
Zimmer, Christopher J., Maxwell, Don E., Mcnally, Stephen T., Atchley, Scott, & Vazhkudai, Sudharshan S. GPU age-aware scheduling to improve the reliability of leadership jobs on Titan. United States.
Zimmer, Christopher J., Maxwell, Don E., Mcnally, Stephen T., Atchley, Scott, and Vazhkudai, Sudharshan S. Thu . "GPU age-aware scheduling to improve the reliability of leadership jobs on Titan". United States. https://www.osti.gov/servlets/purl/1489583.
@article{osti_1489583,
title = {GPU age-aware scheduling to improve the reliability of leadership jobs on Titan},
author = {Zimmer, Christopher J. and Maxwell, Don E. and Mcnally, Stephen T. and Atchley, Scott and Vazhkudai, Sudharshan S.},
abstractNote = {In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ~50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2018},
month = {11}
}

Conference:
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share: