DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Journal Article · · Nature Methods
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3];  [4]; ORCiD logo [5]; ORCiD logo [6]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [9];  [10]; ORCiD logo [11]; ORCiD logo [7]; ORCiD logo [12];  [13]; ORCiD logo [13]; ORCiD logo [14]; ORCiD logo [15]; ORCiD logo [7]; ORCiD logo [16]; ORCiD logo [17] more »; ORCiD logo [18]; ORCiD logo [8]; ORCiD logo [19]; ORCiD logo [20]; ORCiD logo [21]; ORCiD logo [2]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [22]; ORCiD logo [23]; ORCiD logo [24]; ORCiD logo [25]; ORCiD logo [26]; ORCiD logo [27]; ORCiD logo [28]; ORCiD logo [29]; ORCiD logo [30]; ORCiD logo [31];  [32]; ORCiD logo [24]; ORCiD logo [33]; ORCiD logo [18]; ORCiD logo [34]; ORCiD logo [18];  [13]; ORCiD logo [22];  [14];  [26]; ORCiD logo [8];  [10]; ORCiD logo [35]; ORCiD logo [25]; ORCiD logo [36]; ORCiD logo [37] « less
  1. Rutgers Univ., Piscataway, NJ (United States)
  2. Univ. of California, Davis, CA (United States)
  3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
  4. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  5. The University of Texas Health Science Center at Houston, TX (United States)
  6. The Scripps Research Inst., La Jolla, CA (United States)
  7. Univ. of York (United Kingdom)
  8. UKRI Science and Technology Facilities Council, Didcot (United Kingdom)
  9. Pacific Lutheran University, Tacoma, WA (United States)
  10. Univ. of Missouri, Columbia, MO (United States)
  11. Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). European Molecular Biology Laboratory
  12. Stony Brook Univ., NY (United States)
  13. Univ. of Washington, Seattle, WA (United States)
  14. Univ. of California, San Francisco, CA (United States)
  15. Univ. of California, San Diego, CA (United States)
  16. Saint Louis University, MO (United States)
  17. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  18. Max Planck Institute for Biophysical Chemistry, Göttingen (Germany)
  19. Purdue Univ., West Lafayette, IN (United States)
  20. Baylor College of Medicine, Houston, TX (United States)
  21. Arizona State Univ., Tempe, AZ (United States); VIT Bhopal University (India)
  22. The European Bioinformatics Institute (EMBL-EBI), Hinxton (United States)
  23. Univ. of Florida, Gainesville, FL (United States)
  24. Univ. of Washington, Bothell, WA (United States)
  25. Stanford Univ., CA (United States)
  26. Duke Univ., Durham, NC (United States)
  27. Francis Crick Institute, London (United Kingdom)
  28. Purdue Univ., West Lafayette, IN (United States); Arizona State Univ., Tempe, AZ (United States)
  29. Forschungszentrum Juelich (Germany)
  30. SLAC National Accelerator Lab., Menlo Park, CA (United States). Stanford Synchrotron Radiation Lightsource (SSRL); Stanford Univ., CA (United States)
  31. Forschungszentrum Juelich (Germany); Heinrich Heine University Düsseldorf (Germany)
  32. Arizona State Univ., Tempe, AZ (United States); Broad Institute of MIT and Harvard, Cambridge, MA (United States)
  33. Arizona State Univ., Tempe, AZ (United States)
  34. New Mexico Consortium, Los Alamos, NM (United States)
  35. Janssen Research and Development, Spring House, PA (United States)
  36. Rutgers Univ., Piscataway, NJ (United States); Univ. of Southern California, Los Angeles, CA (United States)
  37. Stanford Univ., CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United States). Stanford Synchrotron Radiation Lightsource (SSRL)

This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Research Organization:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States)
Sponsoring Organization:
Biotechnology and Biological Sciences Research Council; Max Planck Society German Research Foundation; Medical Research Council; National Institute of General Medical Sciences (NIGMS); National Institutes of Health (NIH); National Science Foundation (NSF); Purdue Institute of Drug Discovery; USDOE Office of Science (SC), Basic Energy Sciences (BES); Welch Foundation; Wellcome Trust
Grant/Contract Number:
89233218CNA000001; AC02-76SF00515; AC05-00OR22725
OSTI ID:
1777461
Report Number(s):
LA-UR--20-25444
Journal Information:
Nature Methods, Journal Name: Nature Methods Journal Issue: 2 Vol. 18; ISSN 1548-7091
Publisher:
Nature Publishing GroupCopyright Statement
Country of Publication:
United States
Language:
English

References (57)

2019 EMDataResource Model Metrics Challenge Dataset dataset January 2020
Using VMD: An Introductory Tutorial journal December 2008
Real-space refinement with DireX: From global fitting to side-chain improvements journal June 2012
UCSF Chimera?A visualization system for exploratory research and analysis journal January 2004
Cryo-EM model validation using independent map reconstructions: Cryo-EM Model Validation journal May 2013
KING (Kinemage, Next Generation): A versatile interactive molecular and scientific visualization program journal November 2009
MolProbity: More and better reference data for improved all-atom structure validation: PROTEIN SCIENCE.ORG journal November 2017
New tools in MolProbity validation: CaBLAM for CryoEM backbone, UnDowser to rethink “waters,” and NGL Viewer to recapture online 3D graphics journal November 2019
CAD-score: A new contact area difference-based function for evaluation of protein structural models
  • Olechnovič, Kliment; Kulberkytė, Eleonora; Venclovas, Česlovas
  • Proteins: Structure, Function, and Bioinformatics, Vol. 81, Issue 1 https://doi.org/10.1002/prot.24172
journal September 2012
CASP prediction center infrastructure and evaluation measures in CASP10 and CASP ROLL: CASP Prediction Center journal October 2013
Cryo‐electron microscopy targets in CASP13: Overview and evaluation of results journal August 2019
Satisfying Hydrogen Bonding Potential in Proteins journal May 1994
The universal mechanism for iron translocation to the ferroxidase site in ferritin, which is mediated by the well conserved transit site journal September 2010
Resolution and Probabilistic Models of Components in CryoEM Maps of Mature P22 Bacteriophage journal February 2016
Optimal Determination of Particle Orientation, Absolute Hand, and Contrast Loss in Single-particle Electron Cryomicroscopy journal October 2003
Scoring functions for cryoEM density fitting journal May 2011
Web-based visualisation and analysis of 3D electron-microscopy data from EMDB and PDB journal November 2013
Improved metrics for comparing structures of macromolecular assemblies determined by 3D electron-microscopy journal July 2017
Evaluation system and web infrastructure for the second cryo-EM model challenge journal October 2018
Automation and assessment of de novo modeling with Pathwalking in near atomic resolution cryoEM density maps journal December 2018
Comparing cryo-EM structures journal December 2018
Flexible Fitting of Atomic Structures into Electron Microscopy Maps Using Molecular Dynamics journal May 2008
A New Generation of Crystallographic Validation Tools for the Protein Data Bank journal October 2011
Outcome of the First Electron Microscopy Validation Task Force Meeting journal February 2012
Recommendations of the wwPDB NMR Validation Task Force journal September 2013
Validation of Structures in the Protein Data Bank journal December 2017
EMRinger: side chain–directed model and map validation for 3D cryo-electron microscopy journal August 2015
RosettaES: a sampling strategy enabling automated interpretation of difficult cryo-EM maps journal June 2017
Version 1.2 of the Crystallography and NMR system journal October 2007
De novo main-chain modeling for EM maps using MAINMAST journal April 2018
High-resolution structure determination of sub-100 kDa complexes using conventional cryo-EM journal March 2019
Measurement of atom resolvability in cryo-EM maps with Q-scores journal February 2020
Modeling protein quaternary structure of homo- and hetero-oligomers beyond binary interactions by homology journal September 2017
Deep Learning to Predict Protein Backbone Structure from High-Resolution Cryo-EM Density Maps journal March 2020
Evolving data standards for cryo-EM structures journal January 2020
Determining protein structures by combining semireliable data with atomistic physical models by Bayesian inference journal May 2015
Accurate model annotation of a near-atomic resolution cryo-EM map journal March 2017
lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests journal August 2013
LGA: a method for finding 3D similarities in protein structures journal July 2003
Protein Data Bank: the single global archive for 3D macromolecular structure data journal October 2018
Automated and accurate deposition of structures solved by X-ray diffraction to the Protein Data Bank journal September 2004
MolProbity : all-atom structure validation for macromolecular crystallography journal December 2009
Features and development of Coot journal March 2010
Use of knowledge-based restraints in phenix.refine to improve macromolecular refinement at low resolution journal March 2012
Tools for macromolecular model building and refinement into electron cryo-microscopy reconstructions journal January 2015
TEMPy : a Python library for assessment of three-dimensional electron microscopy density fits journal June 2015
Visualization of biological macromolecules at near-atomic resolution: cryo-electron microscopy comes of age journal January 2019
Recent developments in the CCP-EM software suite journal May 2017
Automated map sharpening by maximization of detail and connectivity journal May 2018
New tools for the analysis and validation of cryo-EM maps and atomic models journal September 2018
Sequence assignment for low-resolution modelling of protein crystal structures journal July 2019
Macromolecular structure determination using X-rays, neutrons and electrons: recent developments in Phenix journal October 2019
Current approaches for automated model building into cryo-EM maps using Buccaneer with CCP-EM journal May 2020
2019 EMDataResource Model Metrics Challenge Dataset dataset January 2020
2019 EMDataResource Model Metrics Challenge Dataset dataset January 2020
Molecular dynamics-based refinement and validation for sub-5 Å cryo-electron microscopy maps journal July 2016
Automated cryo-EM structure refinement using correlation-driven molecular dynamics journal March 2019

Cited By (38)

MolProbity: More and better reference data for improved all-atom structure validation: PROTEIN SCIENCE.ORG journal November 2017
New tools in MolProbity validation: CaBLAM for CryoEM backbone, UnDowser to rethink “waters,” and NGL Viewer to recapture online 3D graphics journal November 2019
Distribution and Interaction Patterns of Bacterial Communities in an Ornithogenic Soil of Seymour Island, Antarctica journal October 2014
Tartrate-resistant acid phosphatase deficiency causes a bone dysplasia with autoimmunity and a type I interferon expression signature journal January 2011
Correcting pervasive errors in RNA crystallography through enumerative structure prediction journal December 2012
Modeling and docking of antibody structures with Rosetta journal January 2017
Allosteric control of the ribosome by small-molecule antibiotics journal August 2012
Dynamic changes of the luminal and mucosa-associated gut microbiota during and after antibiotic therapy with paromomycin journal June 2015
Rhizosphere microbiomes of potato cultivated in the High Andes show stable and dynamic core microbiomes with different responses to plant development journal December 2016
Novel mutations in malonyl-CoA-acyl carrier protein transacylase provoke autosomal recessive optic neuropathy journal January 2020
Streptozotocin-induced type-1-diabetes disease onset in Sprague–Dawley rats is associated with an altered intestinal microbiota composition and decreased diversity journal January 2015
Improved chemistry restraints for crystallographic refinement by integrating the Amber force field into Phenix journal January 2020
MolProbity : all-atom structure validation for macromolecular crystallography journal December 2009
Handling ligands with Coot journal March 2012
Real-space refinement in PHENIX for cryo-EM and crystallography journal May 2018
Journey to the center of the protein: allostery from multitemperature multiconformer X-ray crystallography journal January 2019
Salinity affects compositional traits of epibacterial communities on the brown macroalga Fucus vesiculosus journal March 2014
Resilience of the Natural Phyllosphere Microbiota of the Grapevine to Chemical and Biological Pesticides journal March 2014
The individual-specific and diverse nature of the preterm infant microbiota journal January 2013
Spatial variation of the colonic microbiota in patients with ulcerative colitis and control volunteers journal January 2015
The influence of rosuvastatin on the gastrointestinal microbiota and host gene expression profiles journal May 2017
EMPeror: a tool for visualizing high-throughput microbial community data journal November 2013
Bacterial DNA patterns identified using paired-end Illumina sequencing of 16S rRNA genes from whole blood samples of septic patients in the emergency room and intensive care unit journal July 2018
The Earth Microbiome project: successes and aspirations journal August 2014
The role of small in-frame insertions/deletions in inherited eye disorders and how structural modelling can help estimate their pathogenicity journal September 2016
Intestinal microbial communities associated with acute enteric infections and disease recovery journal September 2015
Structural evidence for product stabilization by the ribosomal mRNA helicase journal December 2018
Effects of Diet on Resource Utilization by a Model Human Gut Microbiota Containing Bacteroides cellulosilyticus WH2, a Symbiont with an Extensive Glycobiome journal August 2013
Targeting the Microbiota to Address Diet-Induced Obesity: A Time Dependent Challenge journal June 2013
Sequencing-Based Analysis of the Bacterial and Fungal Composition of Kefir Grains and Milks from Multiple Sources journal July 2013
Soil-Borne Bacterial Structure and Diversity Does Not Reflect Community Activity in Pampa Biome journal October 2013
In Silico Characterization of Meloidogyne Genus Nematode Cellulose Binding Proteins journal January 2019
ISOLDE: a physically realistic environment for model building into low-resolution electron-density maps. text January 2018
Macromolecular structure determination using X-rays, neutrons and electrons: recent developments in Phenix. text January 2019
Restructuring of Epibacterial Communities on Fucus vesiculosus forma mytili in Response to Elevated pCO2 and Increased Temperature Levels journal March 2016
Mutations in ADAR1 cause Aicardi-Goutières syndrome associated with a type I interferon signature text January 2012
Hidden biosphere in an oxygen-deficient Atlantic open-ocean eddy: future implications of ocean deoxygenation on primary production in the eastern tropical North Atlantic journal January 2015
Effect of different heterotrophic plate count methods on the estimation of the composition of the culturable microbial community journal January 2015