A critical examination of compound stability predictions from machine-learned formation energies

Bartel, Christopher J.; Trewartha, Amalie; Wang, Qi; Dunn, Alexander; Jain, Anubhav; Ceder, Gerbrand

doi:10.1038/s41524-020-00362-y

A critical examination of compound stability predictions from machine-learned formation energies

Journal Article · Fri Jul 10 00:00:00 UTC 2020 · npj Computational Materials

DOI:https://doi.org/10.1038/s41524-020-00362-y· OSTI ID:1638103

; Trewartha, Amalie; Wang, Qi; ; ;

Machine learning has emerged as a novel tool for the efficient prediction of material properties, and claims have been made that machine-learned models for the formation energy of compounds can approach the accuracy of Density Functional Theory (DFT). The models tested in this work include five recently published compositional models, a baseline model using stoichiometry alone, and a structural model. By testing seven machine learning models for formation energy on stability predictions using the Materials Project database of DFT calculations for 85,014 unique chemical compositions, we show that while formation energies can indeed be predicted well, all compositional models perform poorly on predicting the stability of compounds, making them considerably less useful than DFT for the discovery and design of new solids. Most critically, in sparse chemical spaces where few stoichiometries have stable compounds, only the structural model is capable of efficiently detecting which materials are stable. The nonincremental improvement of structural models compared with compositional models is noteworthy and encourages the use of structural models for materials discovery, with the constraint that for any new composition, the ground-state structure is not known a priori. This work demonstrates that accurate predictions of formation energy do not imply accurate predictions of stability, emphasizing the importance of assessing model performance on stability predictions, for which we provide a set of publicly available tests.

View Journal Article

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES). Materials Sciences & Engineering Division

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1638103

Alternate ID(s):: OSTI ID: 1633275

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 6; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (39)

Data‐Driven Materials Science: Status, Challenges, and Perspectives Himanen, Lauri; Geurts, Amber; Foster, Adam Stuart Advanced Science, Vol. 6, Issue 21 https://doi.org/10.1002/advs.201900808	journal	September 2019
Machine learning for heterogeneous catalyst design and discovery Goldsmith, Bryan R.; Esterhuizen, Jacques; Liu, Jin-Xun AIChE Journal, Vol. 64, Issue 7 https://doi.org/10.1002/aic.16198	journal	May 2018
Lithium insertion into manganese spinels Thackeray, M. M.; David, W. I. F.; Bruce, P. G. Materials Research Bulletin, Vol. 18, Issue 4, p. 461-472 https://doi.org/10.1016/0025-5408(83)90138-1	journal	April 1983
AFLOW: An automatic framework for high-throughput materials discovery Curtarolo, Stefano; Setyawan, Wahyu; Hart, Gus L. W. Computational Materials Science, Vol. 58 https://doi.org/10.1016/j.commatsci.2012.02.005	journal	June 2012
The Materials Application Programming Interface (API): A simple, flexible and efficient API for materials data based on REpresentational State Transfer (REST) principles Ong, Shyue Ping; Cholia, Shreyas; Jain, Anubhav Computational Materials Science, Vol. 97 https://doi.org/10.1016/j.commatsci.2014.10.037	journal	February 2015
AiiDA: automated interactive infrastructure and database for computational science Pizzi, Giovanni; Cepellotti, Andrea; Sabatini, Riccardo Computational Materials Science, Vol. 111 https://doi.org/10.1016/j.commatsci.2015.09.013	journal	January 2016
Matminer: An open source toolkit for materials data mining Ward, Logan; Dunn, Alexander; Faghaninia, Alireza Computational Materials Science, Vol. 152 https://doi.org/10.1016/j.commatsci.2018.05.018	journal	September 2018
Inverse Design of Solid-State Materials via a Continuous Representation Noh, Juhwan; Kim, Jaehoon; Stein, Helge S. Matter, Vol. 1, Issue 5 https://doi.org/10.1016/j.matt.2019.08.017	journal	November 2019
Graph Networks as a Universal Machine Learning Framework for Molecules and Crystals Chen, Chi; Ye, Weike; Zuo, Yunxing Chemistry of Materials, Vol. 31, Issue 9 https://doi.org/10.1021/acs.chemmater.9b01294	journal	April 2019
Li−Fe−P−O ₂ Phase Diagram from First Principles Calculations Ong, Shyue Ping; Wang, Lei; Kang, Byoungwoo Chemistry of Materials, Vol. 20, Issue 5 https://doi.org/10.1021/cm702327g	journal	February 2008
Synthesis of layered LiMnO2 as an electrode for rechargeable lithium batteries Armstrong, A. Robert; Bruce, Peter G. Nature, Vol. 381, Issue 6582 https://doi.org/10.1038/381499a0	journal	June 1996
Universal fragment descriptors for predicting properties of inorganic crystals Isayev, Olexandr; Oses, Corey; Toher, Cormac Nature Communications, Vol. 8, Issue 1 https://doi.org/10.1038/ncomms15679	journal	June 2017
The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies Kirklin, Scott; Saal, James E.; Meredig, Bryce npj Computational Materials, Vol. 1, Issue 1 https://doi.org/10.1038/npjcompumats.2015.10	journal	December 2015
A general-purpose machine learning framework for predicting properties of inorganic materials Ward, Logan; Agrawal, Ankit; Choudhary, Alok npj Computational Materials, Vol. 2, Issue 1 https://doi.org/10.1038/npjcompumats.2016.28	journal	August 2016
Physical descriptor for the Gibbs energy of inorganic crystalline solids and temperature-dependent materials chemistry Bartel, Christopher J.; Millican, Samantha L.; Deml, Ann M. Nature Communications, Vol. 9, Issue 1 https://doi.org/10.1038/s41467-018-06682-4	journal	October 2018
Machine learning in materials informatics: recent applications and prospects Ramprasad, Rampi; Batra, Rohit; Pilania, Ghanshyam npj Computational Materials, Vol. 3, Issue 1 https://doi.org/10.1038/s41524-017-0056-5	journal	December 2017
The role of decomposition reactions in assessing first-principles predictions of solid stability Bartel, Christopher J.; Weimer, Alan W.; Lany, Stephan npj Computational Materials, Vol. 5, Issue 1 https://doi.org/10.1038/s41524-018-0143-2	journal	January 2019
Recent advances and applications of machine learning in solid-state materials science Schmidt, Jonathan; Marques, Mário R. G.; Botti, Silvana npj Computational Materials, Vol. 5, Issue 1 https://doi.org/10.1038/s41524-019-0221-0	journal	August 2019
Inverse design in search of materials with target functionalities Zunger, Alex Nature Reviews Chemistry, Vol. 2, Issue 4 https://doi.org/10.1038/s41570-018-0121	journal	March 2018
Reversible Mn2+/Mn4+ double redox in lithium-excess cathode materials Lee, Jinhyuk; Kitchaev, Daniil A.; Kwon, Deok-Hwang Nature, Vol. 556, Issue 7700 https://doi.org/10.1038/s41586-018-0015-4	journal	April 2018
Machine learning for molecular and materials science Butler, Keith T.; Davies, Daniel W.; Cartwright, Hugh Nature, Vol. 559, Issue 7715 https://doi.org/10.1038/s41586-018-0337-2	journal	July 2018
Li₂MnO₃-stabilized LiMO₂ (M = Mn, Ni, Co) electrodes for lithium-ion batteries Thackeray, Michael M.; Kang, Sun-Ho; Johnson, Christopher S. Journal of Materials Chemistry, Vol. 17, Issue 30, p. 3112-3125 https://doi.org/10.1039/b702425h	journal	January 2007
Commentary: The Materials Project: A materials genome approach to accelerating materials innovation Jain, Anubhav; Ong, Shyue Ping; Hautier, Geoffroy APL Materials, Vol. 1, Issue 1 https://doi.org/10.1063/1.4812323	journal	July 2013
The Inorganic Crystal Structure Database (ICSD)—Present and Future Hellenbrandt, Mariette Crystallography Reviews, Vol. 10, Issue 1 https://doi.org/10.1080/08893110410001664882	journal	January 2004
Oxidation energies of transition metal oxides within the GGA + U framework Wang, Lei; Maxisch, Thomas; Ceder, Gerbrand Physical Review B, Vol. 73, Issue 19 https://doi.org/10.1103/PhysRevB.73.195107	journal	May 2006
Formation enthalpies by mixing GGA and GGA + U calculations Jain, Anubhav; Hautier, Geoffroy; Ong, Shyue Ping Physical Review B, Vol. 84, Issue 4 https://doi.org/10.1103/PhysRevB.84.045115	journal	July 2011
Correcting density functional theory for accurate predictions of compound enthalpies of formation: Fitted elemental-phase reference energies Stevanović, Vladan; Lany, Stephan; Zhang, Xiuwen Physical Review B, Vol. 85, Issue 11 https://doi.org/10.1103/PhysRevB.85.115104	journal	March 2012
Accuracy of density functional theory in predicting formation energies of ternary oxides from binary oxides and its implication on phase stability Hautier, Geoffroy; Ong, Shyue Ping; Jain, Anubhav Physical Review B, Vol. 85, Issue 15 https://doi.org/10.1103/PhysRevB.85.155208	journal	April 2012
On representing chemical environments Bartók, Albert P.; Kondor, Risi; Csányi, Gábor Physical Review B, Vol. 87, Issue 18 https://doi.org/10.1103/PhysRevB.87.184115	journal	May 2013
Combinatorial screening for new materials in unconstrained composition space with machine learning Meredig, B.; Agrawal, A.; Kirklin, S. Physical Review B, Vol. 89, Issue 9 https://doi.org/10.1103/PhysRevB.89.094104	journal	March 2014
Heats of formation of solids with error estimation: The mBEEF functional with and without fitted reference energies Pandey, Mohnish; Jacobsen, Karsten W. Physical Review B, Vol. 91, Issue 23 https://doi.org/10.1103/PhysRevB.91.235201	journal	June 2015
Predicting density functional theory total energies and enthalpies of formation of metal-nonmetal compounds by linear regression Deml, Ann M.; O’Hayre, Ryan; Wolverton, Chris Physical Review B, Vol. 93, Issue 8 https://doi.org/10.1103/PhysRevB.93.085142	journal	February 2016
Including crystal structure attributes in machine learning models of formation energies via Voronoi tessellations Ward, Logan; Liu, Ruoqian; Krishna, Amar Physical Review B, Vol. 96, Issue 2 https://doi.org/10.1103/PhysRevB.96.024104	journal	July 2017
Machine Learning Energies of 2 Million Elpasolite ( A B C 2 D 6 ) Crystals Faber, Felix A.; Lindmaa, Alexander; von Lilienfeld, O. Anatole Physical Review Letters, Vol. 117, Issue 13 https://doi.org/10.1103/PhysRevLett.117.135502	journal	September 2016
Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties Xie, Tian; Grossman, Jeffrey C. Physical Review Letters, Vol. 120, Issue 14 https://doi.org/10.1103/PhysRevLett.120.145301	journal	April 2018
Performance of the strongly constrained and appropriately normed density functional for solid-state materials Isaacs, Eric B.; Wolverton, Chris Physical Review Materials, Vol. 2, Issue 6 https://doi.org/10.1103/PhysRevMaterials.2.063801	journal	June 2018
Thermodynamic limit for synthesis of metastable inorganic materials Aykol, Muratahan; Dwaraknath, Shyam S.; Sun, Wenhao Science Advances, Vol. 4, Issue 4 https://doi.org/10.1126/sciadv.aaq0148	journal	April 2018
XGBoost: A Scalable Tree Boosting System Chen, Tianqi; Guestrin, Carlos Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining - KDD '16 https://doi.org/10.1145/2939672.2939785	conference	January 2016
NOMAD: The FAIR concept for big data-driven materials science Draxl, Claudia; Scheffler, Matthias MRS Bulletin, Vol. 43, Issue 9 https://doi.org/10.1557/mrs.2018.208	journal	September 2018

Similar Records

Machine-learning enabled thermodynamic model for the design of new rare-earth compounds

Journal Article · Wed Feb 23 00:00:00 UTC 2022 · Acta Materialia · OSTI ID:1856819

Related Subjects

36 MATERIALS SCIENCE

A critical examination of compound stability predictions from machine-learned formation energies

Citation Formats

References (39)

Similar Records

Related Subjects