NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning

Conference · Tue Sep 27 04:00:00 EDT 2022

OSTI ID:1890739

Andersen, Aaron; Rager, David

Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.

• Research Organization: National Renewable Energy Laboratory (NREL), Golden, CO (United States)
• Sponsoring Organization: USDOE National Renewable Energy Laboratory (NREL)
• DOE Contract Number: AC36-08GO28308
• OSTI ID: 1890739
• Report Number(s): NREL/PR-2C00-81102; MainId:79878; UUID:a1a3d769-7afb-483a-a6b1-1f676aec7816; MainAdminID:67596
• Country of Publication: United States
• Language: English

Related Subjects

AI
HPC
IoT
MATHEMATICS AND COMPUTING
cloud computing
cloud vision
compute workflow
data modeling
edge computing
edge inference
high performance computing
machine learning
modsim