Recommendations for Comprehensive and Independent Evaluation of Machine Learning-Based Earth System Models

Journal Article · Fri Mar 14 00:00:00 UTC 2025 · Journal of Geophysical Research. Machine Learning and Computation (Online)

DOI: https://doi.org/10.1029/2024jh000496 · OSTI ID:2543152

Ullrich, Paul A.  ^[1]; Barnes, Elizabeth A.  ^[2]; Collins, William D.  ^[3]; Dagon, Katherine  ^[4]; Duan, Shiheng  ^[5]; Elms, Joshua ^[6]; Lee, Jiwoo  ^[5]; Leung, L. Ruby  ^[7]; Lu, Dan  ^[8]; Molina, Maria J.  ^[9]; O’Brien, T. A.  ^[10]; Rebassoo, F. O. ^[5]

---

1. Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); University of California, Davis, CA (United States)
2. Colorado State University, Fort Collins, CO (United States)
3. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); University of California, Berkeley, CA (United States)
4. National Center for Atmospheric Research (NCAR), Boulder, CO (United States)
5. Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States)
6. Indiana University, Bloomington, IN (United States)
7. Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
8. Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States)
9. University of Maryland, College Park, MD (United States)
10. Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States); National Center for Atmospheric Research (NCAR), Boulder, CO (United States)

Machine learning (ML) is a revolutionary technology with demonstrable applications across multiple disciplines. Within the Earth science community, ML has been most visible for weather forecasting, producing forecasts that rival modern physics-based models. Given the importance of deepening our understanding and improving predictions of the Earth system on all time scales, efforts are now underway to develop Earth-system models (ESMs) capable of representing all components of the coupled Earth system (or their aggregated behavior) and their response to external changes over long timescales. Building trust in ESMs is a much more difficult problem than for weather forecast models, not least because the model must represent the alternate (e.g., future or paleoclimatic) coupled states of the system for which there are no direct observations. Given that the physical principles that enable predictions about the response of the Earth system are often not explicitly coded in these ML-based models, demonstrating the credibility of ML-based ESMs thus requires us to build evidence of their consistency with the physical system. To this end, this paper puts forward five recommendations to enhance comprehensive, standardized, and independent evaluation of ML-based ESMs to strengthen their credibility and promote their wider use.

• Research Organization: Lawrence Livermore National Laboratory (LLNL), Livermore, CA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States); Univ. of Maryland, College Park, MD (United States)
• Sponsoring Organization: National Science Foundation (NSF); USDOE; USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Science (SC), Biological and Environmental Research (BER); USDOE Office of Science (SC), Biological and Environmental Research (BER). Earth & Environmental Systems Science (EESS)
• Grant/Contract Number: AC02-05CH11231; AC05-00OR22725; AC05-76RL01830; AC52-07NA27344; SC0022070; SC0023519; SC0024093
• OSTI ID: 2543152
• Report Number(s): LLNL--JRNL-870866; PNNL-SA--208755
• Journal Information: Journal of Geophysical Research. Machine Learning and Computation (Online), Journal Name: Journal of Geophysical Research. Machine Learning and Computation (Online) Journal Issue: 1 Vol. 2; ISSN 2993-5210
• Publisher: WileyCopyright Statement
• Country of Publication: United States
• Language: English