A Digital Twin Framework for Liquid-cooled Supercomputers as Demonstrated at Exascale

Conference · Fri Nov 01 04:00:00 EDT 2024

DOI:https://doi.org/10.1109/SC41406.2024.00029· OSTI ID:2479037

Brewer, Wes ^[1]; Maiterth, Matthias ^[1]; Kumar, Vineet ^[1]; Wojda, Rafal ^[1]; Bouknight, Sedrick ^[1]; Hines, Jesse ^[1]; Shin, Woong ^[1]; Greenwood, Scott ^[1]; Grant, David ^[1]; Williams, Wesley ^[1]; Wang, Feiyi ^[1]

---

1. ORNL

We present ExaDigiT, an open-source framework for developing comprehensive digital twins of liquid-cooled supercomputers. It integrates three main modules: (1) a resource allocator and power simulator, (2) a transient thermo-fluidic cooling model, and (3) an augmented reality model of the supercomputer and central energy plant. The framework enables the study of "what-if" scenarios, system optimizations, and virtual prototyping of future systems. Using Frontier as a case study, we demonstrate the framework's capabilities by replaying six months of system telemetry for systematic verification and validation. Such a comprehensive analysis of a liquid-cooled exascale supercomputer is the first of its kind. ExaDigiT elucidates complex transient cooling system dynamics, runs synthetic or real workloads, and predicts energy losses due to rectification and voltage conversion. Throughout our paper, we present lessons learned to benefit HPC practitioners developing similar digital twins. We envision the digital twin will be a key enabler for sustainable, energy-efficient supercomputing.

• Research Organization: Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
• Sponsoring Organization: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21); USDOE
• DOE Contract Number: AC05-00OR22725
• OSTI ID: 2479037
• Country of Publication: United States
• Language: English