28 NREL Stratus - Enabling Workflows to Fuse Data Streams, Modeling, Simulation, and Machine Learning: Preprint
Integrating cloud services into advanced computing facilities provides significant new capabilities over focusing solely on traditional high performance computing (HPC) workloads. This brings complementary capabilities as well as enabling new focused roles for HPC. They are especially potent for workflows that fuse data streams, modeling and simulation ('modsim') and machine learning. A key challenge to adopting a hybrid edge-cloud-HPC model is to align optimal capability, data, and user intent on the right resources for each step in a workflow.?The NREL Stratus service provides a basis for this: Stratus layers capabilities needed to make?cloud services accessible to a lab-based scientific community on commercial offerings, and; currently supports upwards of 200 projects ranging from IOT integration to traditional modeling and simulation. This provides a real-world inventory of scientific workflow elements. A growing knowledge base enables placing these elements appropriately between the edge, cloud, and traditional HPC. This paper outlines a vision via reference architecture and the application of that architecture in a typical workflow highlighting multiple components: sensor data intake, cleaning and transforming (edge/cloud suitable); generation of synthetic data through modsim, computationally heavy ML training and hyperparameter optimization (HPC suitable), and; inference and deployment (cloud ideal). Every step in such a workflow involves a cost-benefit analysis regarding the data movement, computational efficiency, availability, latency, and resource capabilities. The reference architecture and examples outlined allow for understanding new opportunities in the context of emerging workflows that combine IOT, cloud, and HPC to bolster scientific productivity.
- Research Organization:
- National Renewable Energy Lab. (NREL), Golden, CO (United States)
- Sponsoring Organization:
- USDOE National Renewable Energy Laboratory (NREL)
- DOE Contract Number:
- AC36-08GO28308
- OSTI ID:
- 1899966
- Report Number(s):
- NREL/CP-2C00-80214; MainId:42417; UUID:152cd422-5d6f-4184-b476-4db7218385ea; MainAdminID:67630
- Resource Relation:
- Conference: Presented at the Smokey Mountains Computational Sciences and Engineering Conference, 24-26 August 2021, Kingsport, Tennessee
- Country of Publication:
- United States
- Language:
- English
Similar Records
ProvLight: Efficient Workflow Provenance Capture on the Edge-to-Cloud Continuum
Enabling discovery data science through cross-facility workflows