JARVIS-Leaderboard: a large scale benchmark of materials design methods

Choudhary, Kamal; Wines, Daniel; Li, Kangming; Garrity, Kevin F.; Gupta, Vishu; Romero, Aldo H.; Krogel, Jaron T.; Saritas, Kayahan; Fuhr, Addis; Ganesh, Panchapakesan; Kent, Paul R. C.; Yan, Keqiang; Lin, Yuchao; Ji, Shuiwang; Blaiszik, Ben; Reiser, Patrick; Friederich, Pascal; Agrawal, Ankit; Tiwary, Pratyush; Beyerle, Eric; Minch, Peter; Rhone, Trevor David; Takeuchi, Ichiro; Wexler, Robert B.; Mannodi-Kanakkithodi, Arun; Ertekin, Elif; Mishra, Avanish; Mathew, Nithin; Wood, Mitchell; Rohskopf, Andrew Dale; Hattrick-Simpers, Jason; Wang, Shih-Han; Achenie, Luke E. K.; Xin, Hongliang; Williams, Maureen; Biacchi, Adam J.; Tavazza, Francesca

doi:10.1038/s41524-024-01259-w

Title: JARVIS-Leaderboard: a large scale benchmark of materials design methods

Journal Article · 06 May 2024 · npj Computational Materials

DOI: https://doi.org/10.1038/s41524-024-01259-w · OSTI ID:2346228

;

; Beyerle, Eric more »

Abstract Lack of rigorous reproducibility and validation are significant hurdles for scientific development across many fields. Materials science, in particular, encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC), and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard/

View Journal Article

Cite

Export

Save

Sponsoring Organization:: USDOE

OSTI ID:: 2346228

Alternate ID(s):: OSTI ID: 2351089; OSTI ID: 2367504

Journal Information:: npj Computational Materials, Journal Name: npj Computational Materials Journal Issue: 1 Vol. 10; ISSN 2057-3960

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United Kingdom

Language:: English

References (159)

Absence of Superconductivity in LK-99 at Ambient Conditions Kumar, Kapil; Kumar Karn, Navneet; Kumar, Yogesh ACS Omega, Vol. 8, Issue 44 https://doi.org/10.1021/acsomega.3c06096	journal	October 2023
Enabling deeper learning on big data for materials informatics applications Jha, Dipendra; Gupta, Vishu; Ward, Logan Scientific Reports, Vol. 11, Issue 1 https://doi.org/10.1038/s41598-021-83193-1	journal	February 2021
Accurate and Numerically Efficient r ² SCAN Meta-Generalized Gradient Approximation Furness, James W.; Kaplan, Aaron D.; Ning, Jinliang The Journal of Physical Chemistry Letters, Vol. 11, Issue 19 https://doi.org/10.1021/acs.jpclett.0c02405	journal	September 2020
A general-purpose machine learning framework for predicting properties of inorganic materials Ward, Logan; Agrawal, Ankit; Choudhary, Alok npj Computational Materials, Vol. 2, Issue 1 https://doi.org/10.1038/npjcompumats.2016.28	journal	August 2016
Variational Quantum Computation of Excited States Higgott, Oscar; Wang, Daochen; Brierley, Stephen Quantum, Vol. 3 https://doi.org/10.22331/q-2019-07-01-156	journal	July 2019
New empirical approach for the structure and energy of covalent systems Tersoff, J. Physical Review B, Vol. 37, Issue 12 https://doi.org/10.1103/PhysRevB.37.6991	journal	April 1988
Named Entity Recognition and Normalization Applied to Large-Scale Information Extraction from the Materials Science Literature Weston, L.; Tshitoyan, V.; Dagdelen, J. Journal of Chemical Information and Modeling, Vol. 59, Issue 9 https://doi.org/10.1021/acs.jcim.9b00470	journal	July 2019
Cross-property deep transfer learning framework for enhanced predictive analytics on small materials data Gupta, Vishu; Choudhary, Kamal; Tavazza, Francesca Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-021-26921-5	journal	November 2021
Benchmark database of accurate (MP2 and CCSD(T) complete basis set limit) interaction energies of small model complexes, DNA base pairs, and amino acid pairs Jurečka, Petr; Šponer, Jiří; Černý, Jiří Physical Chemistry Chemical Physics, Vol. 8, Issue 17, p. 1985-1993 https://doi.org/10.1039/B600027D	journal	January 2006
Interpretable Machine Learning for Discovery: Statistical Challenges and Opportunities Allen, Genevera I.; Gan, Luqin; Zheng, Lili Annual Review of Statistics and Its Application, Vol. 11, Issue 1 https://doi.org/10.1146/annurev-statistics-040120-030919	journal	November 2023
Deep materials informatics: Applications of deep learning in materials science Agrawal, Ankit; Choudhary, Alok MRS Communications, Vol. 9, Issue 3 https://doi.org/10.1557/mrc.2019.73	journal	June 2019
Adaptive machine learning framework to accelerate ab initio molecular dynamics Botu, Venkatesh; Ramprasad, Rampi International Journal of Quantum Chemistry, Vol. 115, Issue 16 https://doi.org/10.1002/qua.24836	journal	December 2014
Large-Scale Benchmark of Exchange–Correlation Functionals for the Determination of Electronic Band Gaps of Solids Borlido, Pedro; Aull, Thorsten; Huran, Ahmad W. Journal of Chemical Theory and Computation, Vol. 15, Issue 9 https://doi.org/10.1021/acs.jctc.9b00322	journal	July 2019
Improved side-chain torsion potentials for the Amber ff99SB protein force field Lindorff-Larsen, Kresten; Piana, Stefano; Palmo, Kim Proteins: Structure, Function, and Bioinformatics https://doi.org/10.1002/prot.22711	journal	January 2010
CHARMM36m: an improved force field for folded and intrinsically disordered proteins Huang, Jing; Rauscher, Sarah; Nawrocki, Grzegorz Nature Methods, Vol. 14, Issue 1 https://doi.org/10.1038/nmeth.4067	journal	November 2016
Blind test of density-functional-based methods on intermolecular interaction energies Taylor, DeCarlos E.; Ángyán, János G.; Galli, Giulia The Journal of Chemical Physics, Vol. 145, Issue 12 https://doi.org/10.1063/1.4961095	journal	September 2016
SchNet – A deep learning architecture for molecules and materials Schütt, K. T.; Sauceda, H. E.; Kindermans, P. -J. The Journal of Chemical Physics, Vol. 148, Issue 24 https://doi.org/10.1063/1.5019779	journal	June 2018
Moment tensor potentials as a promising tool to study diffusion processes Novoselov, I. I.; Yanilkin, A. V.; Shapeev, A. V. Computational Materials Science, Vol. 164 https://doi.org/10.1016/j.commatsci.2019.03.049	journal	June 2019
Quantum chemistry structures and properties of 134 kilo molecules Ramakrishnan, Raghunathan; Dral, Pavlo O.; Rupp, Matthias Scientific Data, Vol. 1, Issue 1 https://doi.org/10.1038/sdata.2014.22	journal	August 2014
The Open Catalyst 2022 (OC22) Dataset and Challenges for Oxide Electrocatalysts Tran, Richard; Lan, Janice; Shuaibi, Muhammed ACS Catalysis, Vol. 13, Issue 5 https://doi.org/10.1021/acscatal.2c05426	journal	February 2023
ChemNLP: A Natural Language-Processing-Based Library for Materials Chemistry Text Data Choudhary, Kamal; Kelley, Mathew L. The Journal of Physical Chemistry C, Vol. 127, Issue 35 https://doi.org/10.1021/acs.jpcc.3c03106	journal	August 2023
On representing chemical environments Bartók, Albert P.; Kondor, Risi; Csányi, Gábor Physical Review B, Vol. 87, Issue 18 https://doi.org/10.1103/PhysRevB.87.184115	journal	May 2013
sGDML: Constructing accurate and data efficient molecular force fields using machine learning Chmiela, Stefan; Sauceda, Huziel E.; Poltavsky, Igor Computer Physics Communications, Vol. 240 https://doi.org/10.1016/j.cpc.2019.02.007	journal	July 2019
An Inter-Laboratory Study of Zn–Sn–Ti–O Thin Films using High-Throughput Experimental Methods Hattrick-Simpers, Jason R.; Zakutayev, Andriy; Barron, Sara C. ACS Combinatorial Science, Vol. 21, Issue 5 https://doi.org/10.1021/acscombsci.8b00158	journal	March 2019
Can machine learning identify the next high-temperature superconductor? Examining extrapolation performance for materials discovery Meredig, Bryce; Antono, Erin; Church, Carena Molecular Systems Design & Engineering, Vol. 3, Issue 5 https://doi.org/10.1039/C8ME00012C	journal	January 2018
Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set Kresse, G.; Furthmüller, J. Computational Materials Science, Vol. 6, Issue 1, p. 15-50 https://doi.org/10.1016/0927-0256(96)00008-0	journal	July 1996
Commentary: The Materials Project: A materials genome approach to accelerating materials innovation Jain, Anubhav; Ong, Shyue Ping; Hautier, Geoffroy APL Materials, Vol. 1, Issue 1 https://doi.org/10.1063/1.4812323	journal	July 2013
Evolution of artificial intelligence for application in contemporary materials science Gupta, Vishu; Liao, Wei-keng; Choudhary, Alok MRS Communications, Vol. 13, Issue 5 https://doi.org/10.1557/s43579-023-00433-3	journal	August 2023
Atomic cluster expansion for accurate and transferable interatomic potentials Drautz, Ralf Physical Review B, Vol. 99, Issue 1 https://doi.org/10.1103/PhysRevB.99.014104	journal	January 2019
The Amber biomolecular simulation programs Case, David A.; Cheatham, Thomas E.; Darden, Tom Journal of Computational Chemistry, Vol. 26, Issue 16 https://doi.org/10.1002/jcc.20290	journal	January 2005
GuacaMol: Benchmarking Models for de Novo Molecular Design Brown, Nathan; Fiscato, Marco; Segler, Marwin H. S. Journal of Chemical Information and Modeling, Vol. 59, Issue 3 https://doi.org/10.1021/acs.jcim.8b00839	journal	October 2018
Interlaboratory Study for Nickel Alloy 625 Made by Laser Powder Bed Fusion to Quantify Mechanical Property Variability Brown, Christopher U.; Jacob, Gregor; Stoudt, Mark Journal of Materials Engineering and Performance, Vol. 25, Issue 8 https://doi.org/10.1007/s11665-016-2169-2	journal	June 2016
Moving closer to experimental level materials property prediction using AI Jha, Dipendra; Gupta, Vishu; Liao, Wei-keng Scientific Reports, Vol. 12, Issue 1 https://doi.org/10.1038/s41598-022-15816-0	journal	July 2022
Reproducibility in density functional theory calculations of solids Lejaeghere, K.; Bihlmayer, G.; Bjorkman, T. Science, Vol. 351, Issue 6280 https://doi.org/10.1126/science.aad3000	journal	March 2016
Inhomogeneous Electron Gas Hohenberg, P.; Kohn, W. Physical Review, Vol. 136, Issue 3B, p. B864-B871 https://doi.org/10.1103/PhysRev.136.B864	journal	November 1964
Benchmarking Accuracy and Generalizability of Four Graph Neural Networks Using Large In Vitro ADME Datasets from Different Chemical Spaces Broccatelli, Fabio; Trager, Richard; Reutlinger, Michael Molecular Informatics, Vol. 41, Issue 8 https://doi.org/10.1002/minf.202100321	journal	February 2022
Open Catalyst 2020 (OC20) Dataset and Community Challenges Chanussot, Lowik; Das, Abhishek; Goyal, Siddharth ACS Catalysis, Vol. 11, Issue 10 https://doi.org/10.1021/acscatal.0c04525	journal	May 2021
The OpenKIM processing pipeline: A cloud-based automatic material property computation engine Karls, D. S.; Bierbaum, M.; Alemi, A. A. The Journal of Chemical Physics, Vol. 153, Issue 6 https://doi.org/10.1063/5.0014267	journal	August 2020
Recent advances and applications of deep learning methods in materials science Choudhary, Kamal; DeCost, Brian; Chen, Chi npj Computational Materials, Vol. 8, Issue 1 https://doi.org/10.1038/s41524-022-00734-6	journal	April 2022
Common workflows for computing material properties using different quantum engines Huber, Sebastiaan P.; Bosoni, Emanuele; Bercx, Marnik npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00594-6	journal	August 2021
Deep Potential Molecular Dynamics: A Scalable Model with the Accuracy of Quantum Mechanics Zhang, Linfeng; Han, Jiequn; Wang, Han Physical Review Letters, Vol. 120, Issue 14 https://doi.org/10.1103/PhysRevLett.120.143001	journal	April 2018
Examining graph neural networks for crystal structures: limitations and opportunities for capturing periodicity Gong, Sheng; Xie, Tian; Shao-Horn, Yang https://doi.org/10.21203/rs.3.rs-2042719/v1	preprint	October 2022
Benchmarking materials property prediction methods: the Matbench test set and Automatminer reference algorithm Dunn, Alexander; Wang, Qi; Ganose, Alex npj Computational Materials, Vol. 6, Issue 1 https://doi.org/10.1038/s41524-020-00406-3	journal	September 2020
A critical examination of compound stability predictions from machine-learned formation energies Bartel, Christopher J.; Trewartha, Amalie; Wang, Qi npj Computational Materials, Vol. 6, Issue 1 https://doi.org/10.1038/s41524-020-00362-y	journal	July 2020
Golem: an algorithm for robust experiment and process optimization Aldeghi, Matteo; Häse, Florian; Hickman, Riley J. Chemical Science, Vol. 12, Issue 44 https://doi.org/10.1039/D1SC01545A	journal	January 2021
2.0 - MOOSE: Enabling massively parallel multiphysics simulation Lindsay, Alexander D.; Gaston, Derek R.; Permann, Cody J. SoftwareX, Vol. 20 https://doi.org/10.1016/j.softx.2022.101202	journal	December 2022
SupermarQ: A Scalable Quantum Benchmark Suite Tomesh, Teague; Gokhale, Pranav; Omole, Victory 2022 IEEE International Symposium on High-Performance Computer Architecture (HPCA) https://doi.org/10.1109/HPCA53966.2022.00050	conference	April 2022
Materials Image Informatics Using Deep Learning Agrawal, Ankit; Gopalakrishnan, Kasthurirangan; Choudhary, Alok Handbook on Big Data and Machine Learning in the Physical Sciences https://doi.org/10.1142/9789811204555_0006	book	March 2020
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 Camerer, Colin F.; Dreber, Anna; Holzmeister, Felix Nature Human Behaviour, Vol. 2, Issue 9 https://doi.org/10.1038/s41562-018-0399-z	journal	August 2018
TEQUILA: a platform for rapid development of quantum algorithms Kottmann, Jakob S.; Alperin-Lea, Sumner; Tamayo-Mendoza, Teresa Quantum Science and Technology, Vol. 6, Issue 2 https://doi.org/10.1088/2058-9565/abe567	journal	March 2021
Performance of various density-functional approximations for cohesive properties of 64 bulk solids Zhang, Guo-Xu; Reilly, Anthony M.; Tkatchenko, Alexandre New Journal of Physics, Vol. 20, Issue 6 https://doi.org/10.1088/1367-2630/aac7f0	journal	June 2018
Materials property prediction with uncertainty quantification: A benchmark study Varivoda, Daniel; Dong, Rongzhi; Omee, Sadman Sadeed Applied Physics Reviews, Vol. 10, Issue 2 https://doi.org/10.1063/5.0133528	journal	May 2023
PFHub: The Phase-Field Community Hub Wheeler, Daniel; Keller, Trevor; DeWitt, Stephen J. Journal of Open Research Software, Vol. 7 https://doi.org/10.5334/jors.276	journal	January 2019
Automated discovery of a robust interatomic potential for aluminum Smith, Justin S.; Nebgen, Benjamin; Mathew, Nithin Nature Communications, Vol. 12, Issue 1 https://doi.org/10.1038/s41467-021-21376-0	journal	February 2021
Can a deep-learning model make fast predictions of vacancy formation in diverse materials? Choudhary, Kamal; Sumpter, Bobby G. AIP Advances, Vol. 13, Issue 9 https://doi.org/10.1063/5.0135382	journal	September 2023
Electronic structure calculations with dynamical mean-field theory Kotliar, G.; Savrasov, S. Y.; Haule, K. Reviews of Modern Physics, Vol. 78, Issue 3 https://doi.org/10.1103/RevModPhys.78.865	journal	August 2006
An updated version of wannier90: A tool for obtaining maximally-localised Wannier functions Mostofi, Arash A.; Yates, Jonathan R.; Pizzi, Giovanni Computer Physics Communications, Vol. 185, Issue 8 https://doi.org/10.1016/j.cpc.2014.05.003	journal	August 2014
Materials science optimization benchmark dataset for multi-objective, multi-fidelity optimization of hard-sphere packing simulations Baird, Sterling G.; Issa, Ramsey; Sparks, Taylor D. Data in Brief, Vol. 50 https://doi.org/10.1016/j.dib.2023.109487	journal	October 2023
Hybrid functionals based on a screened Coulomb potential Heyd, Jochen; Scuseria, Gustavo E.; Ernzerhof, Matthias The Journal of Chemical Physics, Vol. 118, Issue 18 https://doi.org/10.1063/1.1564060	journal	May 2003
Recommended Protocol for Round-Robin Studies in Additive Manufacturing Moylan, Shawn; Brown, Christopher U.; Slotwinski, John Journal of Testing and Evaluation, Vol. 44, Issue 2 https://doi.org/10.1520/JTE20150317	journal	January 2016
The FAIR Guiding Principles for scientific data management and stewardship Wilkinson, Mark D.; Dumontier, Michel; Aalbersberg, IJsbrand Jan Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.18	journal	March 2016
A variational eigenvalue solver on a photonic quantum processor Peruzzo, Alberto; McClean, Jarrod; Shadbolt, Peter Nature Communications, Vol. 5, Issue 1 https://doi.org/10.1038/ncomms5213	journal	July 2014
Pb9Cu(PO4)6(OH)2 : Phonon bands, localized flat-band magnetism, models, and chemical analysis Jiang, Yi; Lee, Scott B.; Herzog-Arbeitman, Jonah Physical Review B, Vol. 108, Issue 23 https://doi.org/10.1103/PhysRevB.108.235127	journal	December 2023
Materials Design and Discovery with High-Throughput Density Functional Theory: The Open Quantum Materials Database (OQMD) Saal, James E.; Kirklin, Scott; Aykol, Muratahan JOM, Vol. 65, Issue 11 https://doi.org/10.1007/s11837-013-0755-4	journal	September 2013
Machine learning of accurate energy-conserving molecular force fields Chmiela, Stefan; Tkatchenko, Alexandre; Sauceda, Huziel E. Science Advances, Vol. 3, Issue 5 https://doi.org/10.1126/sciadv.1603015	journal	May 2017
Mechanical MNIST: A benchmark dataset for mechanical metamodels Lejeune, Emma Extreme Mechanics Letters, Vol. 36 https://doi.org/10.1016/j.eml.2020.100659	journal	April 2020
Guidelines for Interlaboratory Testing Programs Pierson, Raymond H.; Fay, Edward A. Analytical Chemistry, Vol. 31, Issue 12 https://doi.org/10.1021/ac60156a708	journal	December 1959
Validating quantum computers using randomized model circuits Cross, Andrew W.; Bishop, Lev S.; Sheldon, Sarah Physical Review A, Vol. 100, Issue 3 https://doi.org/10.1103/PhysRevA.100.032328	journal	September 2019
Multi-View Photometric Stereo: A Robust Solution and Benchmark Dataset for Spatially Varying Isotropic Materials Li, Min; Zhou, Zhenglong; Wu, Zhe IEEE Transactions on Image Processing, Vol. 29 https://doi.org/10.1109/TIP.2020.2968818	journal	January 2020
SGOOP-d: Estimating Kinetic Distances and Reaction Coordinate Dimensionality for Rare Event Systems from Biased/Unbiased Simulations Tsai, Sun-Ting; Smith, Zachary; Tiwary, Pratyush Journal of Chemical Theory and Computation, Vol. 17, Issue 11 https://doi.org/10.1021/acs.jctc.1c00431	journal	October 2021
GROMACS: Fast, flexible, and free Van Der Spoel, David; Lindahl, Erik; Hess, Berk Journal of Computational Chemistry, Vol. 26, Issue 16 https://doi.org/10.1002/jcc.20291	journal	January 2005
Electronic excitations: density-functional versus many-body Green’s-function approaches Onida, Giovanni; Reining, Lucia; Rubio, Angel Reviews of Modern Physics, Vol. 74, Issue 2 https://doi.org/10.1103/RevModPhys.74.601	journal	June 2002
Materials informatics: From the atomic-level to the continuum Rickman, J. M.; Lookman, T.; Kalinin, S. V. Acta Materialia, Vol. 168 https://doi.org/10.1016/j.actamat.2019.01.051	journal	April 2019
Artificial Intelligence for Materials Audus, Debra J.; Choudhary, Kamal; DeCost, Brian L. Artificial Intelligence for Science https://doi.org/10.1142/9789811265679_0023	book	April 2023
State predictive information bottleneck Wang, Dedi; Tiwary, Pratyush The Journal of Chemical Physics, Vol. 154, Issue 13 https://doi.org/10.1063/5.0038198	journal	April 2021
ImageNet Large Scale Visual Recognition Challenge Russakovsky, Olga; Deng, Jia; Su, Hao International Journal of Computer Vision, Vol. 115, Issue 3 https://doi.org/10.1007/s11263-015-0816-y	journal	April 2015
Ferromagnetic half levitation of LK-99-like synthetic samples Guo, Kaizhen; Li, Yuan; Jia, Shuang Science China Physics, Mechanics & Astronomy, Vol. 66, Issue 10 https://doi.org/10.1007/s11433-023-2201-9	journal	August 2023
Atomistic Line Graph Neural Network for improved materials property predictions Choudhary, Kamal; DeCost, Brian npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00650-1	journal	November 2021
Van der Waals density functionals applied to solids Klimeš, Jiří; Bowler, David R.; Michaelides, Angelos Physical Review B, Vol. 83, Issue 19 https://doi.org/10.1103/PhysRevB.83.195131	journal	May 2011
Benchmark Tests of Atom Segmentation Deep Learning Models with a Consistent Dataset Wei, Jingrui; Blaiszik, Ben; Scourtas, Aristana Microscopy and Microanalysis, Vol. 29, Issue 2 https://doi.org/10.1093/micmic/ozac043	journal	December 2022
Importance of the Kinetic Energy Density for Band Gap Calculations in Solids with Density Functional Theory Tran, Fabien; Blaha, Peter The Journal of Physical Chemistry A, Vol. 121, Issue 17 https://doi.org/10.1021/acs.jpca.7b02882	journal	April 2017
Including crystal structure attributes in machine learning models of formation energies via Voronoi tessellations Ward, Logan; Liu, Ruoqian; Krishna, Amar Physical Review B, Vol. 96, Issue 2 https://doi.org/10.1103/PhysRevB.96.024104	journal	July 2017
IRNet: A General Purpose Deep Residual Regression Framework for Materials Discovery Jha, Dipendra; Ward, Logan; Yang, Zijiang KDD '19: The 25th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining https://doi.org/10.1145/3292500.3330703	conference	July 2019
Metadata Stewardship in Nanosafety Research: Community-Driven Organisation of Metadata Schemas to Support FAIR Nanoscience Data Papadiamantis, Anastasios G.; Klaessig, Frederick C.; Exner, Thomas E. Nanomaterials, Vol. 10, Issue 10 https://doi.org/10.3390/nano10102033	journal	October 2020
Improving deep learning model performance under parametric constraints for materials informatics applications Gupta, Vishu; Peltekian, Alec; Liao, Wei-keng Scientific Reports, Vol. 13, Issue 1 https://doi.org/10.1038/s41598-023-36336-5	journal	June 2023
Quantum Monte Carlo simulations of solids Foulkes, W. M. C.; Mitas, L.; Needs, R. J. Reviews of Modern Physics, Vol. 73, Issue 1 https://doi.org/10.1103/RevModPhys.73.33	journal	January 2001
The Open Quantum Materials Database (OQMD): assessing the accuracy of DFT formation energies Kirklin, Scott; Saal, James E.; Meredig, Bryce npj Computational Materials, Vol. 1, Issue 1 https://doi.org/10.1038/npjcompumats.2015.10	journal	December 2015
The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design Choudhary, Kamal; Garrity, Kevin F.; Reid, Andrew C. E. npj Computational Materials, Vol. 6, Issue 1 https://doi.org/10.1038/s41524-020-00440-1	journal	November 2020
Factors Governing Oxygen Vacancy Formation in Oxide Perovskites Wexler, Robert B.; Gautam, Gopalakrishnan Sai; Stechel, Ellen B. Journal of the American Chemical Society, Vol. 143, Issue 33 https://doi.org/10.1021/jacs.1c05570	journal	August 2021
Evaluating variability with atomistic simulations: the effect of potential and calculation methodology on the modeling of lattice and elastic constants Hale, Lucas M.; Trautt, Zachary T.; Becker, Chandler A. Modelling and Simulation in Materials Science and Engineering, Vol. 26, Issue 5 https://doi.org/10.1088/1361-651X/aabc05	journal	May 2018
Exploiting redundancy in large materials datasets for efficient machine learning with less data Li, Kangming; Persaud, Daniel; Choudhary, Kamal Nature Communications, Vol. 14, Issue 1 https://doi.org/10.1038/s41467-023-42992-y	journal	November 2023
Benchmark datasets incorporating diverse tasks, sample sizes, material systems, and data heterogeneity for materials informatics Henderson, Ashley N.; Kauwe, Steven K.; Sparks, Taylor D. Data in Brief, Vol. 37 https://doi.org/10.1016/j.dib.2021.107262	journal	August 2021
Matminer: An open source toolkit for materials data mining Ward, Logan; Dunn, Alexander; Faghaninia, Alireza Computational Materials Science, Vol. 152 https://doi.org/10.1016/j.commatsci.2018.05.018	journal	September 2018
A critical examination of robustness and generalizability of machine learning prediction of materials properties Li, Kangming; DeCost, Brian; Choudhary, Kamal npj Computational Materials, Vol. 9, Issue 1 https://doi.org/10.1038/s41524-023-01012-9	journal	April 2023
Transformers: State-of-the-Art Natural Language Processing Wolf, Thomas; Debut, Lysandre; Sanh, Victor Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations https://doi.org/10.18653/v1/2020.emnlp-demos.6	conference	January 2020
ElemNet: Deep Learning the Chemistry of Materials From Only Elemental Composition Jha, Dipendra; Ward, Logan; Paul, Arindam Scientific Reports, Vol. 8, Issue 1 https://doi.org/10.1038/s41598-018-35934-y	journal	December 2018
Effects of temperature and humidity on high-strength p-aramid fibers used in body armor Engelbrecht-Wiggans, A.; Burni, F.; Guigues, E. Textile Research Journal, Vol. 90, Issue 21-22 https://doi.org/10.1177/0040517520918232	journal	April 2020
Evaluation and comparison of classical interatomic potentials through a user-friendly interactive web-interface Choudhary, Kamal; Congo, Faical Yannick P.; Liang, Tao Scientific Data, Vol. 4, Issue 1 https://doi.org/10.1038/sdata.2016.125	journal	January 2017
Round-robin studies of two potential seebeck coefficient standard reference materials Lowhorn, Nathan D.; Wong-Ng, W.; Zhang, W. 2007 26th International Conference on Thermoelectrics https://doi.org/10.1109/ICT.2007.4569495	conference	June 2007
Phase-Field Models for Microstructure Evolution Chen, Long-Qing Annual Review of Materials Research, Vol. 32, Issue 1 https://doi.org/10.1146/annurev.matsci.32.112001.132041	journal	August 2002
Benchmarking Quantum Chemical Methods: Are We Heading in the Right Direction? Mata, Ricardo A.; Suhm, Martin A. Angewandte Chemie International Edition, Vol. 56, Issue 37 https://doi.org/10.1002/anie.201611308	journal	April 2017
MoleculeNet: a benchmark for molecular machine learning Wu, Zhenqin; Ramsundar, Bharath; Feinberg, Evan N. Chemical Science, Vol. 9, Issue 2 https://doi.org/10.1039/C7SC02664A	journal	January 2018
Precision and efficiency in solid-state pseudopotential calculations Prandini, Gianluca; Marrazzo, Antimo; Castelli, Ivano E. npj Computational Materials, Vol. 4, Issue 1 https://doi.org/10.1038/s41524-018-0127-2	journal	December 2018
Enhancing materials property prediction by leveraging computational and experimental data using deep transfer learning Jha, Dipendra; Choudhary, Kamal; Tavazza, Francesca Nature Communications, Vol. 10, Issue 1 https://doi.org/10.1038/s41467-019-13297-w	journal	November 2019
Quartet protein reference materials and datasets for multi-platform assessment of label-free proteomics Tian, Sha; Zhan, Dongdong; Yu, Ying Genome Biology, Vol. 24, Issue 1 https://doi.org/10.1186/s13059-023-03048-y	journal	September 2023
Recent progress in the JARVIS infrastructure for next-generation data-driven materials design Wines, Daniel; Gurunathan, Ramya; Garrity, Kevin F. Applied Physics Reviews, Vol. 10, Issue 4 https://doi.org/10.1063/5.0159299	journal	October 2023
Are We Evaluating Rigorously? Benchmarking Recommendation for Reproducible Evaluation and Fair Comparison Sun, Zhu; Yu, Di; Fang, Hui Fourteenth ACM Conference on Recommender Systems https://doi.org/10.1145/3383313.3412489	conference	September 2020
Benchmarking graph neural networks for materials chemistry Fung, Victor; Zhang, Jiaxin; Juarez, Eric npj Computational Materials, Vol. 7, Issue 1 https://doi.org/10.1038/s41524-021-00554-0	journal	June 2021
Benchmark AFLOW Data Sets for Machine Learning Clement, Conrad L.; Kauwe, Steven K.; Sparks, Taylor D. Integrating Materials and Manufacturing Innovation, Vol. 9, Issue 2 https://doi.org/10.1007/s40192-020-00174-4	journal	May 2020
Recent developments in the ABINIT software package Gonze, X.; Jollet, F.; Abreu Araujo, F. Computer Physics Communications, Vol. 205 https://doi.org/10.1016/j.cpc.2016.04.003	journal	August 2016
Performance and Cost Assessment of Machine Learning Interatomic Potentials Zuo, Yunxing; Chen, Chi; Li, Xiangguo The Journal of Physical Chemistry A, Vol. 124, Issue 4 https://doi.org/10.1021/acs.jpca.9b08723	journal	October 2019
Heterogeneous parallelization and acceleration of molecular dynamics simulations in GROMACS Páll, Szilárd; Zhmurov, Artem; Bauer, Paul The Journal of Chemical Physics, Vol. 153, Issue 13 https://doi.org/10.1063/5.0018516	journal	October 2020
Machine learning models for the lattice thermal conductivity prediction of inorganic materials Chen, Lihua; Tran, Huan; Batra, Rohit Computational Materials Science, Vol. 170 https://doi.org/10.1016/j.commatsci.2019.109155	journal	December 2019
A reference high-pressure CO2 adsorption isotherm for ammonium ZSM-5 zeolite: results of an interlaboratory study Nguyen, H. G. T.; Espinal, L.; van Zee, R. D. Adsorption, Vol. 24, Issue 6 https://doi.org/10.1007/s10450-018-9958-x	journal	July 2018
Efficient iterative schemes for ab initio total-energy calculations using a plane-wave basis set Kresse, G.; Furthmüller, J. Physical Review B, Vol. 54, Issue 16, p. 11169-11186 https://doi.org/10.1103/PhysRevB.54.11169	journal	October 1996
Reproducibility of density functional approximations: How new functionals should be reported Lehtola, Susi; Marques, Miguel A. L. The Journal of Chemical Physics, Vol. 159, Issue 11 https://doi.org/10.1063/5.0167763	journal	September 2023
Kohn-Sham potential with discontinuity for band gap materials Kuisma, M.; Ojanen, J.; Enkovaara, J. Physical Review B, Vol. 82, Issue 11 https://doi.org/10.1103/PhysRevB.82.115106	journal	September 2010
Graph neural networks in TensorFlow-Keras with RaggedTensor representation (kgcnn) Reiser, Patrick; Eberhard, André; Friederich, Pascal Software Impacts, Vol. 9 https://doi.org/10.1016/j.simpa.2021.100095	journal	August 2021
Restoring the Density-Gradient Expansion for Exchange in Solids and Surfaces Perdew, John P.; Ruzsinszky, Adrienn; Csonka, Gábor I. Physical Review Letters, Vol. 100, Issue 13 https://doi.org/10.1103/PhysRevLett.100.136406	journal	April 2008
A universal graph deep learning interatomic potential for the periodic table Chen, Chi; Ong, Shyue Ping Nature Computational Science, Vol. 2, Issue 11 https://doi.org/10.1038/s43588-022-00349-3	journal	November 2022
Perspective: Materials informatics and big data: Realization of the “fourth paradigm” of science in materials science Agrawal, Ankit; Choudhary, Alok APL Materials, Vol. 4, Issue 5 https://doi.org/10.1063/1.4946894	journal	April 2016
Strongly Constrained and Appropriately Normed Semilocal Density Functional Sun, Jianwei; Ruzsinszky, Adrienn; Perdew, John P. Physical Review Letters, Vol. 115, Issue 3 https://doi.org/10.1103/PhysRevLett.115.036402	journal	July 2015
The Abinitproject: Impact, environment and recent developments Gonze, Xavier; Amadon, Bernard; Antonius, Gabriel Computer Physics Communications, Vol. 248 https://doi.org/10.1016/j.cpc.2019.107042	journal	March 2020
The earth is flat (p > 0.05): significance thresholds and the crisis of unreplicable research Amrhein, Valentin; Korner-Nievergelt, Fränzi; Roth, Tobias PeerJ, Vol. 5 https://doi.org/10.7717/peerj.3544	journal	July 2017
Highly accurate protein structure prediction with AlphaFold Jumper, John; Evans, Richard; Pritzel, Alexander Nature https://doi.org/10.1038/s41586-021-03819-2	journal	July 2021
Accelerating All-Atom Simulations and Gaining Mechanistic Understanding of Biophysical Systems through State Predictive Information Bottleneck Mehdi, Shams; Wang, Dedi; Pant, Shashank Journal of Chemical Theory and Computation, Vol. 18, Issue 5 https://doi.org/10.1021/acs.jctc.2c00058	journal	April 2022
Invited Article: A round robin test of the uncertainty on the measurement of the thermoelectric dimensionless figure of merit of Co0.97Ni0.03Sb3 Alleno, E.; Bérardan, D.; Byl, C. Review of Scientific Instruments, Vol. 86, Issue 1 https://doi.org/10.1063/1.4905250	journal	January 2015
Graph neural network predictions of metal organic framework CO2 adsorption properties Choudhary, Kamal; Yildirim, Taner; Siderius, Daniel W. Computational Materials Science, Vol. 210 https://doi.org/10.1016/j.commatsci.2022.111388	journal	July 2022
AFLOW: An automatic framework for high-throughput materials discovery Curtarolo, Stefano; Setyawan, Wahyu; Hart, Gus L. W. Computational Materials Science, Vol. 58 https://doi.org/10.1016/j.commatsci.2012.02.005	journal	June 2012
Cirq https://doi.org/10.5281/zenodo.7465577	software	December 2022
Opinion: Is science really facing a reproducibility crisis, and do we need it to? Fanelli, Daniele Proceedings of the National Academy of Sciences, Vol. 115, Issue 11 https://doi.org/10.1073/pnas.1708272114	journal	March 2018
How Reproducible Are Isotherm Measurements in Metal–Organic Frameworks? Park, Jongwoo; Howe, Joshua D.; Sholl, David S. Chemistry of Materials, Vol. 29, Issue 24 https://doi.org/10.1021/acs.chemmater.7b04287	journal	November 2017
Pre-Activation based Representation Learning to Enhance Predictive Analytics on Small Materials Data Gupta, Vishu; Liao, Wei-keng; Choudhary, Alok 2023 International Joint Conference on Neural Networks (IJCNN) https://doi.org/10.1109/IJCNN54540.2023.10191086	conference	June 2023
Artificial intelligence faces reproducibility crisis Hutson, Matthew Science, Vol. 359, Issue 6377 https://doi.org/10.1126/science.359.6377.725	journal	February 2018
Embedded-atom method: Derivation and application to impurities, surfaces, and other defects in metals Daw, Murray S.; Baskes, M. I. Physical Review B, Vol. 29, Issue 12 https://doi.org/10.1103/PhysRevB.29.6443	journal	June 1984
DiLiGenT10²: A Photometric Stereo Benchmark Dataset with Controlled Shape and Material Variation Ren, Jieji; Wang, Feishi; Zhang, Jiahao 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR52688.2022.01225	conference	June 2022
Recent developments in libxc — A comprehensive library of functionals for density functional theory Lehtola, Susi; Steigemann, Conrad; Oliveira, Micael J. T. SoftwareX, Vol. 7 https://doi.org/10.1016/j.softx.2017.11.002	journal	January 2018
Convergence and machine learning predictions of Monkhorst-Pack k-points and plane-wave cut-off in high-throughput DFT calculations Choudhary, Kamal; Tavazza, Francesca Computational Materials Science, Vol. 161 https://doi.org/10.1016/j.commatsci.2019.02.006	journal	April 2019
AtomVision: A Machine Vision Library for Atomistic Images Choudhary, Kamal; Gurunathan, Ramya; DeCost, Brian Journal of Chemical Information and Modeling, Vol. 63, Issue 6 https://doi.org/10.1021/acs.jcim.2c01533	journal	March 2023
Generalized Gradient Approximation Made Simple Perdew, John P.; Burke, Kieron; Ernzerhof, Matthias Physical Review Letters, Vol. 77, Issue 18, p. 3865-3868 https://doi.org/10.1103/PhysRevLett.77.3865	journal	October 1996
ABINIT: Overview and focus on selected capabilities Romero, Aldo H.; Allan, Douglas C.; Amadon, Bernard The Journal of Chemical Physics, Vol. 152, Issue 12 https://doi.org/10.1063/1.5144261	journal	March 2020
How to verify the precision of density-functional-theory implementations via reproducible and universal workflows Bosoni, Emanuele; Beal, Louis; Bercx, Marnik Nature Reviews Physics, Vol. 6, Issue 1 https://doi.org/10.1038/s42254-023-00655-3	journal	November 2023
Crystal Graph Convolutional Neural Networks for an Accurate and Interpretable Prediction of Material Properties Xie, Tian; Grossman, Jeffrey C. Physical Review Letters, Vol. 120, Issue 14 https://doi.org/10.1103/PhysRevLett.120.145301	journal	April 2018
The S66x8 benchmark for noncovalent interactions revisited: explicitly correlated ab initio methods and density functional theory Brauer, Brina; Kesharwani, Manoj K.; Kozuch, Sebastian Physical Chemistry Chemical Physics, Vol. 18, Issue 31 https://doi.org/10.1039/C6CP00688D	journal	January 2016
AtomAI framework for deep learning analysis of image and spectroscopy data in electron and scanning probe microscopy Ziatdinov, Maxim; Ghosh, Ayana; Wong, Chun Yin Nature Machine Intelligence, Vol. 4, Issue 12 https://doi.org/10.1038/s42256-022-00555-8	journal	December 2022
RASPA: molecular simulation software for adsorption and diffusion in flexible nanoporous materials Dubbeldam, David; Calero, Sofía; Ellis, Donald E. Molecular Simulation, Vol. 42, Issue 2 https://doi.org/10.1080/08927022.2015.1010082	journal	February 2015
Structure-aware graph neural network based deep transfer learning framework for enhanced predictive analytics on diverse materials datasets Gupta, Vishu; Choudhary, Kamal; DeCost, Brian npj Computational Materials, Vol. 10, Issue 1 https://doi.org/10.1038/s41524-023-01185-3	journal	January 2024
BRNet: Branched Residual Network for Fast and Accurate Predictive Modeling of Materials Properties Gupta, Vishu; Liao, Wei-keng; Choudhary, Alok Proceedings of the 2022 SIAM International Conference on Data Mining (SDM) https://doi.org/10.1137/1.9781611977172.39	book	January 2022
Quantum Computation and Quantum Information James, Daniel F. V. Physics Today, Vol. 54, Issue 11 https://doi.org/10.1063/1.1428442	journal	November 2001
LAMMPS - a flexible simulation tool for particle-based materials modeling at the atomic, meso, and continuum scales Thompson, Aidan P.; Aktulga, H. Metin; Berger, Richard Computer Physics Communications, Vol. 271 https://doi.org/10.1016/j.cpc.2021.108171	journal	February 2022
Electronic Structure Martin, Richard M. https://doi.org/10.1017/9781108555586	book	September 2020
Improving transparency and scientific rigor in academic publishing Prager, Eric M.; Chambers, Karen E.; Plotkin, Joshua L. Journal of Neuroscience Research, Vol. 97, Issue 4 https://doi.org/10.1002/jnr.24340	journal	December 2018
Modelling science trustworthiness under publish or perish pressure Grimes, David Robert; Bauch, Chris T.; Ioannidis, John P. A. Royal Society Open Science, Vol. 5, Issue 1 https://doi.org/10.1098/rsos.171511	journal	January 2018
Enumeration of 166 Billion Organic Small Molecules in the Chemical Universe Database GDB-17 Ruddigkeit, Lars; van Deursen, Ruud; Blum, Lorenz C. Journal of Chemical Information and Modeling, Vol. 52, Issue 11 https://doi.org/10.1021/ci300415d	journal	November 2012
CHGNet as a pretrained universal neural network potential for charge-informed atomistic modelling Deng, Bowen; Zhong, Peichen; Jun, KyuJung Nature Machine Intelligence, Vol. 5, Issue 9 https://doi.org/10.1038/s42256-023-00716-3	journal	September 2023
Material structure-property linkages using three-dimensional convolutional neural networks Cecen, Ahmet; Dai, Hanjun; Yabansu, Yuksel C. Acta Materialia, Vol. 146 https://doi.org/10.1016/j.actamat.2017.11.053	journal	March 2018
Database of Wannier tight-binding Hamiltonians using high-throughput density functional theory Garrity, Kevin F.; Choudhary, Kamal Scientific Data, Vol. 8, Issue 1 https://doi.org/10.1038/s41597-021-00885-z	journal	April 2021
1,500 scientists lift the lid on reproducibility Baker, Monya Nature, Vol. 533, Issue 7604 https://doi.org/10.1038/533452a	journal	May 2016
A DFT study of BeX (X = S, Se, Te) semiconductor: Modified Becke Johnson (mBJ) potential Rai, D. P.; Ghimire, M. P.; Thapa, R. K. Semiconductors, Vol. 48, Issue 11 https://doi.org/10.1134/S1063782614110244	journal	November 2014

Similar Records

The joint automated repository for various integrated simulations (JARVIS) for data-driven materials design

Journal Article · 2020 · npj Computational Materials · OSTI ID:1720234

Choudhary, Kamal; Garrity, Kevin F.; Reid, Andrew C. E.; +25 more

Applications of physics informed neural operators

Journal Article · 2023 · Machine Learning: Science and Technology · OSTI ID:1974276

Rosofsky, Shawn G.; Al Majed, Hani; Huerta, E. A.

Computational scanning tunneling microscope image database

Journal Article · 2021 · Scientific Data · OSTI ID:1785180

Choudhary, Kamal; Garrity, Kevin F.; Camp, Charles; +4 more

Title: JARVIS-Leaderboard: a large scale benchmark of materials design methods

Citation Formats

References (159)

Similar Records

Related Subjects