Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities

Bligaard, Thomas; Bullock, R. Morris; Campbell, Charles T.; Chen, Jingguang G.; Gates, Bruce C.; Gorte, Raymond J.; Jones, Christopher W.; Jones, William D.; Kitchin, John R.; Scott, Susannah L.

doi:10.1021/acscatal.6b00183

Title: Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities

Journal Article · Fri Apr 01 00:00:00 EDT 2016 · ACS Catalysis

DOI:https://doi.org/10.1021/acscatal.6b00183· OSTI ID:1243133

Bligaard, Thomas; Bullock, R. Morris; Campbell, Charles T.; Chen, Jingguang G.; Gates, Bruce C.; Gorte, Raymond J.; Jones, Christopher W.; Jones, William D.; Kitchin, John R.; Scott, Susannah L.

Benchmarking is a community-based and (preferably) community-driven activity involving consensus-based decisions on how to make reproducible, fair, and relevant assessments. In catalysis science, important catalyst performance metrics include activity, selectivity, and the deactivation profile, which enable comparisons between new and standard catalysts. Benchmarking also requires careful documentation, archiving, and sharing of methods and measurements, to ensure that the full value of research data can be realized. Beyond these goals, benchmarking presents unique opportunities to advance and accelerate understanding of complex reaction systems by combining and comparing experimental information from multiple, in situ and operando techniques with theoretical insights derived from calculations characterizing model systems. This Perspective describes the origins and uses of benchmarking and its applications in computational catalysis, heterogeneous catalysis, molecular catalysis, and electrocatalysis. As a result, it also discusses opportunities and challenges for future developments in these fields.

View Journal Article

Cite

Export

Save

Research Organization:: SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Brookhaven National Laboratory (BNL), Upton, NY (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Basic Energy Sciences (BES)

Grant/Contract Number:: AC02-76SF00515; AC05-76RL01830; SC0012704

OSTI ID:: 1243133

Alternate ID(s):: OSTI ID: 1245000; OSTI ID: 1257959; OSTI ID: 1337213; OSTI ID: 1351981; OSTI ID: 1773108

Report Number(s):: BNL-112274-2016-JA; PNNL-SA-116548

Journal Information:: ACS Catalysis, Journal Name: ACS Catalysis Vol. 6; ISSN 2155-5435

Publisher:: American Chemical SocietyCopyright Statement

Country of Publication:: United States

Language:: English

Citation Metrics:

Cited by: 174 works

Citation information provided by
Web of Science

References (74)

Benchmarking Hydrogen Evolving Reaction and Oxygen Evolving Reaction Electrocatalysts for Solar Water Splitting Devices McCrory, Charles C. L.; Jung, Suho; Ferrer, Ivonne M. Journal of the American Chemical Society, Vol. 137, Issue 13 https://doi.org/10.1021/ja510442p	journal	March 2015
Origin of highly active metal–organic framework catalysts: defects? Defects! Canivet, J.; Vandichel, M.; Farrusseng, D. Dalton Transactions, Vol. 45, Issue 10 https://doi.org/10.1039/C5DT03522H	journal	January 2016
Electron Microscopy of Solid Catalysts—Transforming from a Challenge to a Toolbox Su, Dang Sheng; Zhang, Bingsen; Schlögl, Robert Chemical Reviews, Vol. 115, Issue 8 https://doi.org/10.1021/cr500084c	journal	March 2015
Sharing Detailed Research Data Is Associated with Increased Citation Rate Piwowar, Heather A.; Day, Roger S.; Fridsma, Douglas B. PLoS ONE, Vol. 2, Issue 3 https://doi.org/10.1371/journal.pone.0000308	journal	March 2007
Promoting an open research culture Nosek, B. A.; Alter, G.; Banks, G. C. Science, Vol. 348, Issue 6242 https://doi.org/10.1126/science.aab2374	journal	June 2015
Comprehensive Benchmarking of a Density-Dependent Dispersion Correction Steinmann, Stephan N.; Corminboeuf, Clemence Journal of Chemical Theory and Computation, Vol. 7, Issue 11 https://doi.org/10.1021/ct200602x	journal	October 2011
Priming Intelligent Behavior: An Elusive Phenomenon Shanks, David R.; Newell, Ben R.; Lee, Eun Hee PLoS ONE, Vol. 8, Issue 4 https://doi.org/10.1371/journal.pone.0056515	journal	April 2013
Stability and Metastability of Clusters in a Reactive Atmosphere: Theoretical Evidence for Unexpected Stoichiometries of ${Mg}_{M} O_{x}$ Bhattacharya, Saswata; Levchenko, Sergey V.; Ghiringhelli, Luca M. Physical Review Letters, Vol. 111, Issue 13 https://doi.org/10.1103/PhysRevLett.111.135501	journal	September 2013
Activity of CeOx and TiOx Nanoparticles Grown on Au(111) in the Water-Gas Shift Reaction Rodriguez, J. A.; Ma, S.; Liu, P. Science, Vol. 318, Issue 5857 https://doi.org/10.1126/science.1150038	journal	December 2007
Catalysts under Controlled Atmospheres in the Transmission Electron Microscope Hansen, Thomas W.; Wagner, Jakob B. ACS Catalysis, Vol. 4, Issue 6 https://doi.org/10.1021/cs401148d	journal	April 2014
Activity benchmarks and requirements for Pt, Pt-alloy, and non-Pt oxygen reduction catalysts for PEMFCs Gasteiger, Hubert A.; Kocha, Shyam S.; Sompalli, Bhaskar Applied Catalysis B: Environmental, Vol. 56, Issue 1-2, p. 9-35 https://doi.org/10.1016/j.apcatb.2004.06.021	journal	March 2005
Experimental Aspects in Benchmarking of the Electrocatalytic Activity Čolić, Viktor; Tymoczko, Jakub; Maljusch, Artjom ChemElectroChem, Vol. 2, Issue 1 https://doi.org/10.1002/celc.201402295	journal	November 2014
Raise standards for preclinical cancer research Begley, C. Glenn; Ellis, Lee M. Nature, Vol. 483, Issue 7391 https://doi.org/10.1038/483531a	journal	March 2012
What Drives Academic Data Sharing? Fecher, Benedikt; Friesike, Sascha; Hebing, Marcel PLOS ONE, Vol. 10, Issue 2 https://doi.org/10.1371/journal.pone.0118053	journal	February 2015
A benchmark database for adsorption bond energies to transition metal surfaces and comparison to selected DFT functionals Wellendorff, Jess; Silbaugh, Trent L.; Garcia-Pintos, Delfina Surface Science, Vol. 640 https://doi.org/10.1016/j.susc.2015.03.023	journal	October 2015
Believe it or not: how much can we rely on published data on potential drug targets? Prinz, Florian; Schlange, Thomas; Asadullah, Khusru Nature Reviews Drug Discovery, Vol. 10, Issue 9 https://doi.org/10.1038/nrd3439-c1	journal	August 2011
Why Most Published Research Findings Are False Ioannidis, John P. A. PLoS Medicine, Vol. 2, Issue 8 https://doi.org/10.1371/journal.pmed.0020124	journal	August 2005
Predicting accurate absolute binding energies in aqueous solution: thermodynamic considerations for electronic structure methods Jensen, Jan H. Physical Chemistry Chemical Physics, Vol. 17, Issue 19 https://doi.org/10.1039/C5CP00628G	journal	January 2015
Analysis of Catalyst Surface Structure by Physical Sorption Hammond, Karl D.; Conner, Wm. Curtis Advances in Catalysis https://doi.org/10.1016/B978-0-12-420173-6.00001-2	book	January 2013
Continuous flow organometallic catalysis: new wind in old sails Hintermair, Ulrich; Franciò, Giancarlo; Leitner, Walter Chemical Communications, Vol. 47, Issue 13 https://doi.org/10.1039/c0cc04958a	journal	January 2011
Benchmark Energetic Data in a Model System for Grubbs II Metathesis Catalysis and Their Use for the Development, Assessment, and Validation of Electronic Structure Methods Zhao, Yan; Truhlar, Donald G. Journal of Chemical Theory and Computation, Vol. 5, Issue 2 https://doi.org/10.1021/ct800386d	journal	January 2009
“Turning Over” Definitions in Catalytic Cycles Kozuch, Sebastian; Martin, Jan M. L. ACS Catalysis, Vol. 2, Issue 12 https://doi.org/10.1021/cs3005264	journal	November 2012
A Standard Set of Pericyclic Reactions of Hydrocarbons for the Benchmarking of Computational Methods: The Performance of ab Initio, Density Functional, CASSCF, CASPT2, and CBS-QB3 Methods for the Prediction of Activation Barriers, Reaction Energetics, and Transition State Geometries Guner, Vildan; Khuong, Kelli S.; Leach, Andrew G. The Journal of Physical Chemistry A, Vol. 107, Issue 51 https://doi.org/10.1021/jp035501w	journal	December 2003
Quantum Chemical Benchmarking, Validation, and Prediction of Acidity Constants for Substituted Pyridinium Ions and Pyridinyl Radicals Keith, John A.; Carter, Emily A. Journal of Chemical Theory and Computation, Vol. 8, Issue 9 https://doi.org/10.1021/ct300295g	journal	August 2012
Density functional theory in surface chemistry and catalysis Norskov, J. K.; Abild-Pedersen, F.; Studt, F. Proceedings of the National Academy of Sciences, Vol. 108, Issue 3 https://doi.org/10.1073/pnas.1006652108	journal	January 2011
Tests of the RPBE, revPBE, τ-HCTHhyb, ωB97X-D, and MOHLYP density functional approximations and 29 others against representative databases for diverse bond energies and barrier heights in catalysis Yang, Ke; Zheng, Jingjing; Zhao, Yan The Journal of Chemical Physics, Vol. 132, Issue 16 https://doi.org/10.1063/1.3382342	journal	April 2010
Control of Metal Nanocrystal Size Reveals Metal-Support Interface Role for Ceria Catalysts Cargnello, M.; Doan-Nguyen, V. V. T.; Gordon, T. R. Science, Vol. 341, Issue 6147 https://doi.org/10.1126/science.1240148	journal	July 2013
Experimental criterion for the absence of artifacts in the measurement of rates of heterogeneous catalytic reactions Madon, Rostam J.; Boudart, Michel Industrial & Engineering Chemistry Fundamentals, Vol. 21, Issue 4, p. 438-447 https://doi.org/10.1021/i100008a022	journal	November 1982
Not so loosely bound rare gas atoms: finite-temperature vibrational fingerprints of neutral gold-cluster complexes Ghiringhelli, Luca M.; Gruene, Philipp; Lyon, Jonathan T. New Journal of Physics, Vol. 15, Issue 8 https://doi.org/10.1088/1367-2630/15/8/083003	journal	August 2013
Rate-Enhancing Roles of Water Molecules in Methyltrioxorhenium-Catalyzed Olefin Epoxidation by Hydrogen Peroxide Goldsmith, Bryan R.; Hwang, Taeho; Seritan, Stefan Journal of the American Chemical Society, Vol. 137, Issue 30 https://doi.org/10.1021/jacs.5b03750	journal	July 2015
Using scaling relations to understand trends in the catalytic activity of transition metals Jones, G.; Bligaard, T.; Abild-Pedersen, F. Journal of Physics: Condensed Matter, Vol. 20, Issue 6 https://doi.org/10.1088/0953-8984/20/6/064239	journal	January 2008
Finding the Rate-Determining Step in a Mechanism Campbell, Charles T. Journal of Catalysis, Vol. 204, Issue 2 https://doi.org/10.1006/jcat.2001.3396	journal	December 2001
Examples of Effective Data Sharing in Scientific Publishing Kitchin, John R. ACS Catalysis, Vol. 5, Issue 6 https://doi.org/10.1021/acscatal.5b00538	journal	May 2015
Bringing Digital Science Deep Inside the Scientific Article: the Elsevier Article of the Future Project Aalbersberg, IJsbrand Jan; Atzeni, Sophia; Koers, Hylke LIBER Quarterly, Vol. 23, Issue 4 https://doi.org/10.18352/lq.8446	journal	April 2014
The Open Knowledge Foundation: Open Data Means Better Science Molloy, Jennifer C. PLoS Biology, Vol. 9, Issue 12 https://doi.org/10.1371/journal.pbio.1001195	journal	December 2011
Dispersion Interactions with Density-Functional Theory: Benchmarking Semiempirical and Interatomic Pairwise Corrected Density Functionals Marom, Noa; Tkatchenko, Alexandre; Rossi, Mariana Journal of Chemical Theory and Computation, Vol. 7, Issue 12 https://doi.org/10.1021/ct2005616	journal	October 2011
The Economics of Reproducibility in Preclinical Research Freedman, Leonard P.; Cockburn, Iain M.; Simcoe, Timothy S. PLOS Biology, Vol. 13, Issue 6 https://doi.org/10.1371/journal.pbio.1002165	journal	June 2015
A balanced procedure for the treatment of cluster-ligand interactions on gold phosphine systems in catalysis Mollenhauer, Doreen; Gaston, Nicola Journal of Computational Chemistry, Vol. 35, Issue 13 https://doi.org/10.1002/jcc.23578	journal	March 2014
Temperature-Programmed Desorption of Hydrogen from Platinum Particles on γ-Al2O3: Evidence of Platinum-Catalyzed Dehydroxylation of γ-Al2O3 Alexeev, Oleg; Kim, Do-Woan; Graham, George W. Journal of Catalysis, Vol. 185, Issue 1 https://doi.org/10.1006/jcat.1999.2489	journal	July 1999
Highly active metallocene catalysts for olefin polymerization Kaminsky, Walter Journal of the Chemical Society, Dalton Transactions, Issue 9 https://doi.org/10.1039/a800056e	journal	January 1998
A Case Study of the Mechanism of Alcohol-Mediated Morita Baylis–Hillman Reactions. The Importance of Experimental Observations Plata, R. Erik; Singleton, Daniel A. Journal of the American Chemical Society, Vol. 137, Issue 11 https://doi.org/10.1021/ja5111392	journal	March 2015
Chemical accuracy in ab initio thermochemistry and spectroscopy: current strategies and future challenges Peterson, Kirk A.; Feller, David; Dixon, David A. Theoretical Chemistry Accounts, Vol. 131, Issue 1 https://doi.org/10.1007/s00214-011-1079-5	journal	January 2012
Introducing structural sensitivity into adsorption–energy scaling relations by means of coordination numbers Calle-Vallejo, Federico; Loffreda, David; Koper, Marc T. M. Nature Chemistry, Vol. 7, Issue 5 https://doi.org/10.1038/nchem.2226	journal	April 2015
Self-correction in science at work Alberts, B.; Cicerone, R. J.; Fienberg, S. E. Science, Vol. 348, Issue 6242 https://doi.org/10.1126/science.aab3847	journal	June 2015
Gold reference catalysts Hashmi, A. Stephen K. Gold Bulletin, Vol. 36, Issue 1, p. 24-24 https://doi.org/10.1007/BF03214864	journal	March 2003
Benchmarking of Homogeneous Electrocatalysts: Overpotential, Turnover Frequency, Limiting Turnover Number Costentin, Cyrille; Passard, Guillaume; Savéant, Jean-Michel Journal of the American Chemical Society, Vol. 137, Issue 16 https://doi.org/10.1021/jacs.5b00914	journal	April 2015
On the Stability and Recyclability of Supported Metal–Ligand Complex Catalysts: Myths, Misconceptions and Critical Research Needs Jones, Christopher W. Topics in Catalysis, Vol. 53, Issue 13-14 https://doi.org/10.1007/s11244-010-9513-9	journal	May 2010
“Mindless” DFT Benchmarking Korth, Martin; Grimme, Stefan Journal of Chemical Theory and Computation, Vol. 5, Issue 4 https://doi.org/10.1021/ct800511q	journal	March 2009
Water-Catalyzed Activation of H ₂ O ₂ by Methyltrioxorhenium: A Combined Computational–Experimental Study Hwang, Taeho; Goldsmith, Bryan R.; Peters, Baron Inorganic Chemistry, Vol. 52, Issue 24 https://doi.org/10.1021/ic401343m	journal	November 2013
Toward a Database of Chemically Accurate Barrier Heights for Reactions of Molecules with Metal Surfaces Kroes, Geert-Jan The Journal of Physical Chemistry Letters, Vol. 6, Issue 20 https://doi.org/10.1021/acs.jpclett.5b01344	journal	September 2015
Quantitative Assay for the Direct Comparison of Platinum Catalysts in Benzene H/D Exchange Hickman, Amanda J.; Villalobos, Janette M.; Sanford, Melanie S. Organometallics, Vol. 28, Issue 18 https://doi.org/10.1021/om900495n	journal	September 2009
Selectivity in Propene Polymerization with Metallocene Catalysts Resconi, Luigi; Cavallo, Luigi; Fait, Anna Chemical Reviews, Vol. 100, Issue 4 https://doi.org/10.1021/cr9804691	journal	April 2000
Can Contemporary Density Functional Theory Predict Energy Spans in Molecular Catalysis Accurately Enough To Be Applicable for in Silico Catalyst Design? A Computational/Experimental Case Study for the Ruthenium-Catalyzed Hydrogenation of Olefins Rohmann, Kai; Hölscher, Markus; Leitner, Walter Journal of the American Chemical Society, Vol. 138, Issue 1 https://doi.org/10.1021/jacs.5b11997	journal	December 2015
Preparing Your Manuscript for Submission to ACS Catalysis Jones, Christopher W.; Smith, David J.; Gunnoe, T. Brent ACS Catalysis, Vol. 4, Issue 8 https://doi.org/10.1021/cs501043q	journal	July 2014
Benchmarking Heterogeneous Electrocatalysts for the Oxygen Evolution Reaction McCrory, Charles C. L.; Jung, Suho; Peters, Jonas C. Journal of the American Chemical Society, Vol. 135, Issue 45 https://doi.org/10.1021/ja407115p	journal	October 2013
On the accuracy of density-functional theory exchange-correlation functionals for H bonds in small water clusters: Benchmarks approaching the complete basis set limit Santra, Biswajit; Michaelides, Angelos; Scheffler, Matthias The Journal of Chemical Physics, Vol. 127, Issue 18 https://doi.org/10.1063/1.2790009	journal	November 2007
Using benchmarking to advance research: a challenge to software engineering Sim, S. E.; Easterbrook, S.; Holt, R. C. 25th International Conference on Software Engineering, 2003. Proceedings. https://doi.org/10.1109/ICSE.2003.1201189	conference	January 2003
Toward the rational benchmarking of homogeneous H ₂ -evolving catalysts Artero, Vincent; Saveant, Jean-Michel Energy Environ. Sci., Vol. 7, Issue 11 https://doi.org/10.1039/C4EE01709A	journal	January 2014
Multielectron, Multistep Molecular Catalysis of Electrochemical Reactions: Benchmarking of Homogeneous Catalysts Costentin, Cyrille; Savéant, Jean-Michel ChemElectroChem, Vol. 1, Issue 7 https://doi.org/10.1002/celc.201300263	journal	June 2014
Do Practical Standard Coupled Cluster Calculations Agree Better than Kohn–Sham Calculations with Currently Available Functionals When Compared to the Best Available Experimental Data for Dissociation Energies of Bonds to 3 d Transition Metals? Xu, Xuefei; Zhang, Wenjing; Tang, Mingsheng Journal of Chemical Theory and Computation, Vol. 11, Issue 5 https://doi.org/10.1021/acs.jctc.5b00081	journal	April 2015
Enhancing reproducibility Doerr, Allison; Vogt, Nina Nature Methods, Vol. 10, Issue 5, p. 367-367 https://doi.org/10.1038/nmeth.2471	journal	April 2013
Recent Advances in Asymmetric Catalytic Metal Carbene Transformations Doyle, Michael P.; Forbes, David C. Chemical Reviews, Vol. 98, Issue 2 https://doi.org/10.1021/cr940066a	journal	April 1998
Making scientific computations reproducible Schwab, M.; Karrenbach, N.; Claerbout, J. Computing in Science & Engineering, Vol. 2, Issue 6 https://doi.org/10.1109/5992.881708	journal	January 2000
How Benchmarking in Science Can Lead to a Reversal of Priorities Farlin, Julien; Majewsky, Marius Environmental Science & Technology, Vol. 49, Issue 5 https://doi.org/10.1021/acs.est.5b00389	journal	February 2015
Benchmarking the Stability of Oxygen Evolution Reaction Catalysts: The Importance of Monitoring Mass Losses Frydendal, Rasmus; Paoli, Elisa A.; Knudsen, Brian P. ChemElectroChem, Vol. 1, Issue 12 https://doi.org/10.1002/celc.201402262	journal	October 2014
Towards the computational design of solid catalysts Nørskov, J.; Bligaard, T.; Rossmeisl, J. Nature Chemistry, Vol. 1, Issue 1, p. 37-46 https://doi.org/10.1038/nchem.121	journal	April 2009
Striving for Reproducible Science Sweedler, Jonathan V. Analytical Chemistry, Vol. 87, Issue 23 https://doi.org/10.1021/acs.analchem.5b04300	journal	November 2015
Solving reproducibility Buck, S. Science, Vol. 348, Issue 6242 https://doi.org/10.1126/science.aac8041	journal	June 2015
An invitation to reproducible computational research Donoho, D. L. Biostatistics, Vol. 11, Issue 3 https://doi.org/10.1093/biostatistics/kxq028	journal	June 2010
Comment on “‘Turning Over’ Definitions in Catalytic Cycles” Lente, Gábor ACS Catalysis, Vol. 3, Issue 3 https://doi.org/10.1021/cs300846b	journal	February 2013
Reaction Progress Kinetic Analysis: A Powerful Methodology for Mechanistic Studies of Complex Catalytic Reactions Blackmond, Donna G. Angewandte Chemie International Edition, Vol. 44, Issue 28 https://doi.org/10.1002/anie.200462544	journal	July 2005
Editorial: Introducing Tutorials Chirik, Paul J. Organometallics, Vol. 34, Issue 20 https://doi.org/10.1021/acs.organomet.5b00815	journal	July 2015
Degree of Rate Control: How Much the Energies of Intermediates and Transition States Control Rates Stegelmann, Carsten; Andreasen, Anders; Campbell, Charles T. Journal of the American Chemical Society, Vol. 131, Issue 23 https://doi.org/10.1021/ja9000097	journal	June 2009
Homogeneous Ni Catalysts for H ₂ Oxidation and Production: An Assessment of Theoretical Methods, from Density Functional Theory to Post Hartree−Fock Correlated Wave-Function Theory Chen, Shentan; Raugei, Simone; Rousseau, Roger The Journal of Physical Chemistry A, Vol. 114, Issue 48 https://doi.org/10.1021/jp106800n	journal	December 2010

Similar Records

Challenges at the Frontiers of Matter and Energy: Transformative Opportunities for Discovery Science

Program Document · Sun Nov 01 00:00:00 EDT 2015 · OSTI ID:1243133

Hemminger, John C.; Sarrao, John; Crabtree, George; +2 more

Best practices in catalysis: A perspective

Journal Article · Mon Dec 20 00:00:00 EST 2021 · Journal of Catalysis · OSTI ID:1243133

Bhan, Aditya; Delgass, W. Nicholas

Catalysis Research of Relevance to Carbon Management: Progress, Challenges, and Opportunities

Journal Article · Wed Apr 11 00:00:00 EDT 2001 · Chemical Reviews · OSTI ID:1243133

Arakawa, Hironori; Aresta, Michele; Armor, John; +32 more

Related Subjects

37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY
benchmarking
catalytic performance
computational catalysis
heterogeneous catalysis
molecular catalysis
electrocatalysis

Title: Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities

Citation Formats

References (74)

Similar Records

Related Subjects