Navigating the Noise: Bringing Clarity to ML Parameterization Design With O $$\boldsymbol{\mathcal{O}}$$(100) Ensembles
[1];
[2];
[1];
[3];
[5];
[6];
[8]
- Department of Earth System Sciences University of California at Irvine Irvine CA USA
- Multimodal Cognitive AI Intel Labs Santa Clara CA USA
- Faculty of Geosciences and Environment University of Lausanne Lausanne Switzerland, Expertise Center for Climate Extremes University of Lausanne Lausanne Switzerland
- Department of Statistics University of California at Irvine Irvine CA USA
- Department of Earth and Planetary Sciences Harvard University Cambridge MA USA, NVIDIA Research Santa Clara CA USA
- LEAP Science and Technology Center School of Engineering and Applied Sciences Climate School Columbia University New York NY USA
- Department of Electrical Engineering and Computer Science Berkeley AI Research (BAIR) University of California at Berkeley Berkeley CA USA, Department of Biomedical Data Science Stanford University School of Medicine Palo Alto CA USA
- Department of Earth System Sciences University of California at Irvine Irvine CA USA, NVIDIA Research Santa Clara CA USA
Abstract Machine‐learning (ML) parameterizations of subgrid processes (here of turbulence, convection, and radiation) may one day replace conventional parameterizations by emulating high‐resolution physics without the cost of explicit simulation. However, uncertainty about the relationship between offline and online performance (i.e., when integrated with a large‐scale general circulation model) hinders their development. Much of this uncertainty stems from limited sampling of the noisy, emergent effects of upstream ML design decisions on downstream online hybrid simulation. Our work rectifies the sampling issue via the construction of a semi‐automated, end‐to‐end pipeline for size ensembles of hybrid simulations, revealing important nuances in how systematic reductions in offline error manifest in changes to online error and online stability. For example, removing dropout and switching from a Mean Squared Error to a Mean Absolute Error loss both reduce offline error, but they have opposite effects on online error and online stability. Other design decisions, like incorporating memory, converting moisture input from specific humidity to relative humidity, using batch normalization, and training on multiple climates do not come with any such compromises. Finally, we show that ensemble sizes of may be necessary to reliably detect causally relevant differences online. By enabling rapid online experimentation at scale, we can empirically settle debates regarding subgrid ML parameterization design that would have otherwise remained unresolved in the noise.
- Sponsoring Organization:
- USDOE
- Grant/Contract Number:
- SC0022255; SC0023368
- OSTI ID:
- 2555905
- Journal Information:
- Journal of Advances in Modeling Earth Systems, Journal Name: Journal of Advances in Modeling Earth Systems Journal Issue: 4 Vol. 17; ISSN 1942-2466
- Publisher:
- American Geophysical Union (AGU)Copyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
Stable Simulation of the Community Atmosphere Model Using Machine‐Learning Physical Parameterization Trained With Experience Replay
Uncertainty guided online ensemble for non-stationary data streams in fusion science