MatFold: systematic insights into materials discovery models' performance through standardized cross-validation protocols
Machine learning (ML) models in the materials sciences that are validated by overly simplistic cross-validation (CV) protocols can yield biased performance estimates for downstream modeling or materials screening tasks. This can be particularly counterproductive for applications where the time and cost of failed validation efforts (experimental synthesis, characterization, and testing) are consequential. We propose a set of standardized and increasingly difficult splitting protocols for chemically and structurally motivated CV that can be followed to validate any ML model for materials discovery. Among several benefits, this enables systematic insights into model generalizability, improvability, and uncertainty, provides benchmarks for fair comparison between competing models with access to differing quantities of data, and systematically reduces possible data leakage through increasingly strict splitting protocols. Performing thorough CV investigations across increasingly strict chemical/structural splitting criteria, local vs. global property prediction tasks, small vs. large datasets, and structure vs. compositional model architectures, some common threads are observed; however, several marked differences exist across these exemplars, indicating the need for comprehensive analysis to fully understand each model's generalization accuracy and potential for materials discovery. For this we provide a general-purpose, featurization-agnostic toolkit, MatFold, to automate reproducible construction of these CV splits and encourage further community use in model benchmarking.
- Research Organization:
- Sandia National Laboratories (SNL-CA), Livermore, CA (United States)
- Sponsoring Organization:
- USDOE; USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE Office of Energy Efficiency and Renewable Energy (EERE), Office of Sustainable Transportation and Fuels. Hydrogen and Fuel Cell Technologies Office (HFTO)
- Grant/Contract Number:
- NA0003525
- OSTI ID:
- 2483691
- Report Number(s):
- SAND--2025-03037J
- Journal Information:
- Digital Discovery, Journal Name: Digital Discovery Journal Issue: 3 Vol. 4; ISSN 2635-098X
- Publisher:
- Royal Society of ChemistryCopyright Statement
- Country of Publication:
- United States
- Language:
- English
Similar Records
2018 Water Splitting Technologies Benchmarking and Protocols Workshop
wa-hls4ml: A Benchmark and Surrogate Models for hls4ml Resource and Latency Estimation