Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities
Abstract
Benchmarking is a community-based and (preferably) community-driven activity involving consensus-based decisions on how to make reproducible, fair, and relevant assessments. In catalysis science, important catalyst performance metrics include activity, selectivity, and the deactivation profile, which enable comparisons between new and standard catalysts. Benchmarking also requires careful documentation, archiving, and sharing of methods and measurements, to ensure that the full value of research data can be realized. Beyond these goals, benchmarking presents unique opportunities to advance and accelerate understanding of complex reaction systems by combining and comparing experimental information from multiple, in situ and operando techniques with theoretical insights derived from calculations characterizing model systems. This Perspective describes the origins and uses of benchmarking and its applications in computational catalysis, heterogeneous catalysis, molecular catalysis, and electrocatalysis. As a result, it also discusses opportunities and challenges for future developments in these fields.
- Authors:
- Publication Date:
- Research Org.:
- SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Brookhaven National Laboratory (BNL), Upton, NY (United States); Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC), Basic Energy Sciences (BES)
- OSTI Identifier:
- 1243133
- Alternate Identifier(s):
- OSTI ID: 1245000; OSTI ID: 1257959; OSTI ID: 1337213; OSTI ID: 1351981; OSTI ID: 1773108
- Report Number(s):
- BNL-112274-2016-JA; PNNL-SA-116548
Journal ID: ISSN 2155-5435
- Grant/Contract Number:
- AC02-76SF00515; AC05-76RL01830; SC0012704
- Resource Type:
- Published Article
- Journal Name:
- ACS Catalysis
- Additional Journal Information:
- Journal Name: ACS Catalysis Journal Volume: 6; Journal ID: ISSN 2155-5435
- Publisher:
- American Chemical Society
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 37 INORGANIC, ORGANIC, PHYSICAL, AND ANALYTICAL CHEMISTRY; benchmarking; catalytic performance; computational catalysis; heterogeneous catalysis; molecular catalysis; electrocatalysis
Citation Formats
Bligaard, Thomas, Bullock, R. Morris, Campbell, Charles T., Chen, Jingguang G., Gates, Bruce C., Gorte, Raymond J., Jones, Christopher W., Jones, William D., Kitchin, John R., and Scott, Susannah L. Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities. United States: N. p., 2016.
Web. doi:10.1021/acscatal.6b00183.
Bligaard, Thomas, Bullock, R. Morris, Campbell, Charles T., Chen, Jingguang G., Gates, Bruce C., Gorte, Raymond J., Jones, Christopher W., Jones, William D., Kitchin, John R., & Scott, Susannah L. Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities. United States. https://doi.org/10.1021/acscatal.6b00183
Bligaard, Thomas, Bullock, R. Morris, Campbell, Charles T., Chen, Jingguang G., Gates, Bruce C., Gorte, Raymond J., Jones, Christopher W., Jones, William D., Kitchin, John R., and Scott, Susannah L. Fri .
"Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities". United States. https://doi.org/10.1021/acscatal.6b00183.
@article{osti_1243133,
title = {Toward Benchmarking in Catalysis Science: Best Practices, Challenges, and Opportunities},
author = {Bligaard, Thomas and Bullock, R. Morris and Campbell, Charles T. and Chen, Jingguang G. and Gates, Bruce C. and Gorte, Raymond J. and Jones, Christopher W. and Jones, William D. and Kitchin, John R. and Scott, Susannah L.},
abstractNote = {Benchmarking is a community-based and (preferably) community-driven activity involving consensus-based decisions on how to make reproducible, fair, and relevant assessments. In catalysis science, important catalyst performance metrics include activity, selectivity, and the deactivation profile, which enable comparisons between new and standard catalysts. Benchmarking also requires careful documentation, archiving, and sharing of methods and measurements, to ensure that the full value of research data can be realized. Beyond these goals, benchmarking presents unique opportunities to advance and accelerate understanding of complex reaction systems by combining and comparing experimental information from multiple, in situ and operando techniques with theoretical insights derived from calculations characterizing model systems. This Perspective describes the origins and uses of benchmarking and its applications in computational catalysis, heterogeneous catalysis, molecular catalysis, and electrocatalysis. As a result, it also discusses opportunities and challenges for future developments in these fields.},
doi = {10.1021/acscatal.6b00183},
journal = {ACS Catalysis},
number = ,
volume = 6,
place = {United States},
year = {Fri Apr 01 00:00:00 EDT 2016},
month = {Fri Apr 01 00:00:00 EDT 2016}
}
https://doi.org/10.1021/acscatal.6b00183
Web of Science
Works referenced in this record:
Benchmarking Hydrogen Evolving Reaction and Oxygen Evolving Reaction Electrocatalysts for Solar Water Splitting Devices
journal, March 2015
- McCrory, Charles C. L.; Jung, Suho; Ferrer, Ivonne M.
- Journal of the American Chemical Society, Vol. 137, Issue 13
Origin of highly active metal–organic framework catalysts: defects? Defects!
journal, January 2016
- Canivet, J.; Vandichel, M.; Farrusseng, D.
- Dalton Transactions, Vol. 45, Issue 10
Electron Microscopy of Solid Catalysts—Transforming from a Challenge to a Toolbox
journal, March 2015
- Su, Dang Sheng; Zhang, Bingsen; Schlögl, Robert
- Chemical Reviews, Vol. 115, Issue 8
Sharing Detailed Research Data Is Associated with Increased Citation Rate
journal, March 2007
- Piwowar, Heather A.; Day, Roger S.; Fridsma, Douglas B.
- PLoS ONE, Vol. 2, Issue 3
Promoting an open research culture
journal, June 2015
- Nosek, B. A.; Alter, G.; Banks, G. C.
- Science, Vol. 348, Issue 6242
Comprehensive Benchmarking of a Density-Dependent Dispersion Correction
journal, October 2011
- Steinmann, Stephan N.; Corminboeuf, Clemence
- Journal of Chemical Theory and Computation, Vol. 7, Issue 11
Priming Intelligent Behavior: An Elusive Phenomenon
journal, April 2013
- Shanks, David R.; Newell, Ben R.; Lee, Eun Hee
- PLoS ONE, Vol. 8, Issue 4
Stability and Metastability of Clusters in a Reactive Atmosphere: Theoretical Evidence for Unexpected Stoichiometries of
journal, September 2013
- Bhattacharya, Saswata; Levchenko, Sergey V.; Ghiringhelli, Luca M.
- Physical Review Letters, Vol. 111, Issue 13
Activity of CeOx and TiOx Nanoparticles Grown on Au(111) in the Water-Gas Shift Reaction
journal, December 2007
- Rodriguez, J. A.; Ma, S.; Liu, P.
- Science, Vol. 318, Issue 5857
Catalysts under Controlled Atmospheres in the Transmission Electron Microscope
journal, April 2014
- Hansen, Thomas W.; Wagner, Jakob B.
- ACS Catalysis, Vol. 4, Issue 6
Activity benchmarks and requirements for Pt, Pt-alloy, and non-Pt oxygen reduction catalysts for PEMFCs
journal, March 2005
- Gasteiger, Hubert A.; Kocha, Shyam S.; Sompalli, Bhaskar
- Applied Catalysis B: Environmental, Vol. 56, Issue 1-2, p. 9-35
Experimental Aspects in Benchmarking of the Electrocatalytic Activity
journal, November 2014
- Čolić, Viktor; Tymoczko, Jakub; Maljusch, Artjom
- ChemElectroChem, Vol. 2, Issue 1
Raise standards for preclinical cancer research
journal, March 2012
- Begley, C. Glenn; Ellis, Lee M.
- Nature, Vol. 483, Issue 7391
What Drives Academic Data Sharing?
journal, February 2015
- Fecher, Benedikt; Friesike, Sascha; Hebing, Marcel
- PLOS ONE, Vol. 10, Issue 2
A benchmark database for adsorption bond energies to transition metal surfaces and comparison to selected DFT functionals
journal, October 2015
- Wellendorff, Jess; Silbaugh, Trent L.; Garcia-Pintos, Delfina
- Surface Science, Vol. 640
Believe it or not: how much can we rely on published data on potential drug targets?
journal, August 2011
- Prinz, Florian; Schlange, Thomas; Asadullah, Khusru
- Nature Reviews Drug Discovery, Vol. 10, Issue 9
Why Most Published Research Findings Are False
journal, August 2005
- Ioannidis, John P. A.
- PLoS Medicine, Vol. 2, Issue 8
Predicting accurate absolute binding energies in aqueous solution: thermodynamic considerations for electronic structure methods
journal, January 2015
- Jensen, Jan H.
- Physical Chemistry Chemical Physics, Vol. 17, Issue 19
Analysis of Catalyst Surface Structure by Physical Sorption
book, January 2013
- Hammond, Karl D.; Conner, Wm. Curtis
- Advances in Catalysis
Continuous flow organometallic catalysis: new wind in old sails
journal, January 2011
- Hintermair, Ulrich; Franciò, Giancarlo; Leitner, Walter
- Chemical Communications, Vol. 47, Issue 13
Benchmark Energetic Data in a Model System for Grubbs II Metathesis Catalysis and Their Use for the Development, Assessment, and Validation of Electronic Structure Methods
journal, January 2009
- Zhao, Yan; Truhlar, Donald G.
- Journal of Chemical Theory and Computation, Vol. 5, Issue 2
“Turning Over” Definitions in Catalytic Cycles
journal, November 2012
- Kozuch, Sebastian; Martin, Jan M. L.
- ACS Catalysis, Vol. 2, Issue 12
A Standard Set of Pericyclic Reactions of Hydrocarbons for the Benchmarking of Computational Methods: The Performance of ab Initio, Density Functional, CASSCF, CASPT2, and CBS-QB3 Methods for the Prediction of Activation Barriers, Reaction Energetics, and Transition State Geometries
journal, December 2003
- Guner, Vildan; Khuong, Kelli S.; Leach, Andrew G.
- The Journal of Physical Chemistry A, Vol. 107, Issue 51
Quantum Chemical Benchmarking, Validation, and Prediction of Acidity Constants for Substituted Pyridinium Ions and Pyridinyl Radicals
journal, August 2012
- Keith, John A.; Carter, Emily A.
- Journal of Chemical Theory and Computation, Vol. 8, Issue 9
Density functional theory in surface chemistry and catalysis
journal, January 2011
- Norskov, J. K.; Abild-Pedersen, F.; Studt, F.
- Proceedings of the National Academy of Sciences, Vol. 108, Issue 3
Tests of the RPBE, revPBE, τ-HCTHhyb, ωB97X-D, and MOHLYP density functional approximations and 29 others against representative databases for diverse bond energies and barrier heights in catalysis
journal, April 2010
- Yang, Ke; Zheng, Jingjing; Zhao, Yan
- The Journal of Chemical Physics, Vol. 132, Issue 16
Control of Metal Nanocrystal Size Reveals Metal-Support Interface Role for Ceria Catalysts
journal, July 2013
- Cargnello, M.; Doan-Nguyen, V. V. T.; Gordon, T. R.
- Science, Vol. 341, Issue 6147
Experimental criterion for the absence of artifacts in the measurement of rates of heterogeneous catalytic reactions
journal, November 1982
- Madon, Rostam J.; Boudart, Michel
- Industrial & Engineering Chemistry Fundamentals, Vol. 21, Issue 4, p. 438-447
Not so loosely bound rare gas atoms: finite-temperature vibrational fingerprints of neutral gold-cluster complexes
journal, August 2013
- Ghiringhelli, Luca M.; Gruene, Philipp; Lyon, Jonathan T.
- New Journal of Physics, Vol. 15, Issue 8
Rate-Enhancing Roles of Water Molecules in Methyltrioxorhenium-Catalyzed Olefin Epoxidation by Hydrogen Peroxide
journal, July 2015
- Goldsmith, Bryan R.; Hwang, Taeho; Seritan, Stefan
- Journal of the American Chemical Society, Vol. 137, Issue 30
Using scaling relations to understand trends in the catalytic activity of transition metals
journal, January 2008
- Jones, G.; Bligaard, T.; Abild-Pedersen, F.
- Journal of Physics: Condensed Matter, Vol. 20, Issue 6
Finding the Rate-Determining Step in a Mechanism
journal, December 2001
- Campbell, Charles T.
- Journal of Catalysis, Vol. 204, Issue 2
Examples of Effective Data Sharing in Scientific Publishing
journal, May 2015
- Kitchin, John R.
- ACS Catalysis, Vol. 5, Issue 6
Bringing Digital Science Deep Inside the Scientific Article: the Elsevier Article of the Future Project
journal, April 2014
- Aalbersberg, IJsbrand Jan; Atzeni, Sophia; Koers, Hylke
- LIBER Quarterly, Vol. 23, Issue 4
The Open Knowledge Foundation: Open Data Means Better Science
journal, December 2011
- Molloy, Jennifer C.
- PLoS Biology, Vol. 9, Issue 12
Dispersion Interactions with Density-Functional Theory: Benchmarking Semiempirical and Interatomic Pairwise Corrected Density Functionals
journal, October 2011
- Marom, Noa; Tkatchenko, Alexandre; Rossi, Mariana
- Journal of Chemical Theory and Computation, Vol. 7, Issue 12
The Economics of Reproducibility in Preclinical Research
journal, June 2015
- Freedman, Leonard P.; Cockburn, Iain M.; Simcoe, Timothy S.
- PLOS Biology, Vol. 13, Issue 6
A balanced procedure for the treatment of cluster-ligand interactions on gold phosphine systems in catalysis
journal, March 2014
- Mollenhauer, Doreen; Gaston, Nicola
- Journal of Computational Chemistry, Vol. 35, Issue 13
Temperature-Programmed Desorption of Hydrogen from Platinum Particles on γ-Al2O3: Evidence of Platinum-Catalyzed Dehydroxylation of γ-Al2O3
journal, July 1999
- Alexeev, Oleg; Kim, Do-Woan; Graham, George W.
- Journal of Catalysis, Vol. 185, Issue 1
Highly active metallocene catalysts for olefin polymerization
journal, January 1998
- Kaminsky, Walter
- Journal of the Chemical Society, Dalton Transactions, Issue 9
A Case Study of the Mechanism of Alcohol-Mediated Morita Baylis–Hillman Reactions. The Importance of Experimental Observations
journal, March 2015
- Plata, R. Erik; Singleton, Daniel A.
- Journal of the American Chemical Society, Vol. 137, Issue 11
Chemical accuracy in ab initio thermochemistry and spectroscopy: current strategies and future challenges
journal, January 2012
- Peterson, Kirk A.; Feller, David; Dixon, David A.
- Theoretical Chemistry Accounts, Vol. 131, Issue 1
Introducing structural sensitivity into adsorption–energy scaling relations by means of coordination numbers
journal, April 2015
- Calle-Vallejo, Federico; Loffreda, David; Koper, Marc T. M.
- Nature Chemistry, Vol. 7, Issue 5
Self-correction in science at work
journal, June 2015
- Alberts, B.; Cicerone, R. J.; Fienberg, S. E.
- Science, Vol. 348, Issue 6242
Gold reference catalysts
journal, March 2003
- Hashmi, A. Stephen K.
- Gold Bulletin, Vol. 36, Issue 1, p. 24-24
Benchmarking of Homogeneous Electrocatalysts: Overpotential, Turnover Frequency, Limiting Turnover Number
journal, April 2015
- Costentin, Cyrille; Passard, Guillaume; Savéant, Jean-Michel
- Journal of the American Chemical Society, Vol. 137, Issue 16
On the Stability and Recyclability of Supported Metal–Ligand Complex Catalysts: Myths, Misconceptions and Critical Research Needs
journal, May 2010
- Jones, Christopher W.
- Topics in Catalysis, Vol. 53, Issue 13-14
“Mindless” DFT Benchmarking
journal, March 2009
- Korth, Martin; Grimme, Stefan
- Journal of Chemical Theory and Computation, Vol. 5, Issue 4
Water-Catalyzed Activation of H 2 O 2 by Methyltrioxorhenium: A Combined Computational–Experimental Study
journal, November 2013
- Hwang, Taeho; Goldsmith, Bryan R.; Peters, Baron
- Inorganic Chemistry, Vol. 52, Issue 24
Toward a Database of Chemically Accurate Barrier Heights for Reactions of Molecules with Metal Surfaces
journal, September 2015
- Kroes, Geert-Jan
- The Journal of Physical Chemistry Letters, Vol. 6, Issue 20
Quantitative Assay for the Direct Comparison of Platinum Catalysts in Benzene H/D Exchange
journal, September 2009
- Hickman, Amanda J.; Villalobos, Janette M.; Sanford, Melanie S.
- Organometallics, Vol. 28, Issue 18
Selectivity in Propene Polymerization with Metallocene Catalysts
journal, April 2000
- Resconi, Luigi; Cavallo, Luigi; Fait, Anna
- Chemical Reviews, Vol. 100, Issue 4
Can Contemporary Density Functional Theory Predict Energy Spans in Molecular Catalysis Accurately Enough To Be Applicable for in Silico Catalyst Design? A Computational/Experimental Case Study for the Ruthenium-Catalyzed Hydrogenation of Olefins
journal, December 2015
- Rohmann, Kai; Hölscher, Markus; Leitner, Walter
- Journal of the American Chemical Society, Vol. 138, Issue 1
Preparing Your Manuscript for Submission to ACS Catalysis
journal, July 2014
- Jones, Christopher W.; Smith, David J.; Gunnoe, T. Brent
- ACS Catalysis, Vol. 4, Issue 8
Benchmarking Heterogeneous Electrocatalysts for the Oxygen Evolution Reaction
journal, October 2013
- McCrory, Charles C. L.; Jung, Suho; Peters, Jonas C.
- Journal of the American Chemical Society, Vol. 135, Issue 45
On the accuracy of density-functional theory exchange-correlation functionals for H bonds in small water clusters: Benchmarks approaching the complete basis set limit
journal, November 2007
- Santra, Biswajit; Michaelides, Angelos; Scheffler, Matthias
- The Journal of Chemical Physics, Vol. 127, Issue 18
Using benchmarking to advance research: a challenge to software engineering
conference, January 2003
- Sim, S. E.; Easterbrook, S.; Holt, R. C.
- 25th International Conference on Software Engineering, 2003. Proceedings.
Toward the rational benchmarking of homogeneous H 2 -evolving catalysts
journal, January 2014
- Artero, Vincent; Saveant, Jean-Michel
- Energy Environ. Sci., Vol. 7, Issue 11
Multielectron, Multistep Molecular Catalysis of Electrochemical Reactions: Benchmarking of Homogeneous Catalysts
journal, June 2014
- Costentin, Cyrille; Savéant, Jean-Michel
- ChemElectroChem, Vol. 1, Issue 7
Do Practical Standard Coupled Cluster Calculations Agree Better than Kohn–Sham Calculations with Currently Available Functionals When Compared to the Best Available Experimental Data for Dissociation Energies of Bonds to 3 d Transition Metals?
journal, April 2015
- Xu, Xuefei; Zhang, Wenjing; Tang, Mingsheng
- Journal of Chemical Theory and Computation, Vol. 11, Issue 5
Enhancing reproducibility
journal, April 2013
- Doerr, Allison; Vogt, Nina
- Nature Methods, Vol. 10, Issue 5, p. 367-367
Recent Advances in Asymmetric Catalytic Metal Carbene Transformations
journal, April 1998
- Doyle, Michael P.; Forbes, David C.
- Chemical Reviews, Vol. 98, Issue 2
Making scientific computations reproducible
journal, January 2000
- Schwab, M.; Karrenbach, N.; Claerbout, J.
- Computing in Science & Engineering, Vol. 2, Issue 6
How Benchmarking in Science Can Lead to a Reversal of Priorities
journal, February 2015
- Farlin, Julien; Majewsky, Marius
- Environmental Science & Technology, Vol. 49, Issue 5
Benchmarking the Stability of Oxygen Evolution Reaction Catalysts: The Importance of Monitoring Mass Losses
journal, October 2014
- Frydendal, Rasmus; Paoli, Elisa A.; Knudsen, Brian P.
- ChemElectroChem, Vol. 1, Issue 12
Towards the computational design of solid catalysts
journal, April 2009
- Nørskov, J.; Bligaard, T.; Rossmeisl, J.
- Nature Chemistry, Vol. 1, Issue 1, p. 37-46
Striving for Reproducible Science
journal, November 2015
- Sweedler, Jonathan V.
- Analytical Chemistry, Vol. 87, Issue 23
An invitation to reproducible computational research
journal, June 2010
- Donoho, D. L.
- Biostatistics, Vol. 11, Issue 3
Comment on “‘Turning Over’ Definitions in Catalytic Cycles”
journal, February 2013
- Lente, Gábor
- ACS Catalysis, Vol. 3, Issue 3
Reaction Progress Kinetic Analysis: A Powerful Methodology for Mechanistic Studies of Complex Catalytic Reactions
journal, July 2005
- Blackmond, Donna G.
- Angewandte Chemie International Edition, Vol. 44, Issue 28
Editorial: Introducing Tutorials
journal, July 2015
- Chirik, Paul J.
- Organometallics, Vol. 34, Issue 20
Degree of Rate Control: How Much the Energies of Intermediates and Transition States Control Rates
journal, June 2009
- Stegelmann, Carsten; Andreasen, Anders; Campbell, Charles T.
- Journal of the American Chemical Society, Vol. 131, Issue 23
Homogeneous Ni Catalysts for H 2 Oxidation and Production: An Assessment of Theoretical Methods, from Density Functional Theory to Post Hartree−Fock Correlated Wave-Function Theory
journal, December 2010
- Chen, Shentan; Raugei, Simone; Rousseau, Roger
- The Journal of Physical Chemistry A, Vol. 114, Issue 48