skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Cryo-EM model validation recommendations based on outcomes of the 2019 EMDataResource challenge

Journal Article · · Nature Methods
ORCiD logo [1]; ORCiD logo [2]; ORCiD logo [3];  [4]; ORCiD logo [5]; ORCiD logo [6]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [9];  [10]; ORCiD logo [11]; ORCiD logo [7]; ORCiD logo [12];  [13]; ORCiD logo [13]; ORCiD logo [14]; ORCiD logo [15]; ORCiD logo [7]; ORCiD logo [16]; ORCiD logo [17] more »; ORCiD logo [18]; ORCiD logo [8]; ORCiD logo [19]; ORCiD logo [20]; ORCiD logo [21]; ORCiD logo [2]; ORCiD logo [7]; ORCiD logo [8]; ORCiD logo [22]; ORCiD logo [23]; ORCiD logo [24]; ORCiD logo [25]; ORCiD logo [26]; ORCiD logo [27]; ORCiD logo [28]; ORCiD logo [29]; ORCiD logo [30]; ORCiD logo [31];  [32]; ORCiD logo [24]; ORCiD logo [33]; ORCiD logo [18]; ORCiD logo [34]; ORCiD logo [18];  [13]; ORCiD logo [22];  [14];  [26]; ORCiD logo [8];  [10]; ORCiD logo [35]; ORCiD logo [25]; ORCiD logo [36]; ORCiD logo [37] « less
  1. Rutgers Univ., Piscataway, NJ (United States)
  2. Univ. of California, Davis, CA (United States)
  3. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
  4. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  5. The University of Texas Health Science Center at Houston, TX (United States)
  6. The Scripps Research Inst., La Jolla, CA (United States)
  7. Univ. of York (United Kingdom)
  8. UKRI Science and Technology Facilities Council, Didcot (United Kingdom)
  9. Pacific Lutheran University, Tacoma, WA (United States)
  10. Univ. of Missouri, Columbia, MO (United States)
  11. Deutsches Elektronen-Synchrotron (DESY), Hamburg (Germany). European Molecular Biology Laboratory
  12. Stony Brook Univ., NY (United States)
  13. Univ. of Washington, Seattle, WA (United States)
  14. Univ. of California, San Francisco, CA (United States)
  15. Univ. of California, San Diego, CA (United States)
  16. Saint Louis University, MO (United States)
  17. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
  18. Max Planck Institute for Biophysical Chemistry, Göttingen (Germany)
  19. Purdue Univ., West Lafayette, IN (United States)
  20. Baylor College of Medicine, Houston, TX (United States)
  21. Arizona State Univ., Tempe, AZ (United States); VIT Bhopal University (India)
  22. The European Bioinformatics Institute (EMBL-EBI), Hinxton (United States)
  23. Univ. of Florida, Gainesville, FL (United States)
  24. Univ. of Washington, Bothell, WA (United States)
  25. Stanford Univ., CA (United States)
  26. Duke Univ., Durham, NC (United States)
  27. Francis Crick Institute, London (United Kingdom)
  28. Purdue Univ., West Lafayette, IN (United States); Arizona State Univ., Tempe, AZ (United States)
  29. Forschungszentrum Juelich (Germany)
  30. SLAC National Accelerator Lab., Menlo Park, CA (United States). Stanford Synchrotron Radiation Lightsource (SSRL); Stanford Univ., CA (United States)
  31. Forschungszentrum Juelich (Germany); Heinrich Heine University Düsseldorf (Germany)
  32. Arizona State Univ., Tempe, AZ (United States); Broad Institute of MIT and Harvard, Cambridge, MA (United States)
  33. Arizona State Univ., Tempe, AZ (United States)
  34. New Mexico Consortium, Los Alamos, NM (United States)
  35. Janssen Research and Development, Spring House, PA (United States)
  36. Rutgers Univ., Piscataway, NJ (United States); Univ. of Southern California, Los Angeles, CA (United States)
  37. Stanford Univ., CA (United States); SLAC National Accelerator Lab., Menlo Park, CA (United States). Stanford Synchrotron Radiation Lightsource (SSRL)
+ Show Author Affiliations

This paper describes outcomes of the 2019 Cryo-EM Model Challenge. The goals were to (1) assess the quality of models that can be produced from cryogenic electron microscopy (cryo-EM) maps using current modeling software, (2) evaluate reproducibility of modeling results from different software developers and users and (3) compare performance of current metrics used for model evaluation, particularly Fit-to-Map metrics, with focus on near-atomic resolution. Our findings demonstrate the relatively high accuracy and reproducibility of cryo-EM models derived by 13 participating teams from four benchmark maps, including three forming a resolution series (1.8 to 3.1 Å). The results permit specific recommendations to be made about validating near-atomic cryo-EM structures both in the context of individual experiments and structure data archives such as the Protein Data Bank. We recommend the adoption of multiple scoring parameters to provide full and objective annotation and assessment of the model, reflective of the observed cryo-EM map density.

Research Organization:
SLAC National Accelerator Laboratory (SLAC), Menlo Park, CA (United States); Oak Ridge National Laboratory (ORNL), Oak Ridge, TN (United States). Oak Ridge Leadership Computing Facility (OLCF); Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Organization:
USDOE Office of Science (SC), Basic Energy Sciences (BES); National Institutes of Health (NIH); National Institute of General Medical Sciences (NIGMS); National Science Foundation (NSF); Purdue Institute of Drug Discovery; Max Planck Society German Research Foundation; Welch Foundation; Medical Research Council; Wellcome Trust; Biotechnology and Biological Sciences Research Council
Grant/Contract Number:
AC02-76SF00515; R01GM079429; AC05-00OR22725; R35GM131883; P01GM063210; MCB-1942763 (CAREER); R01GM095583; R01GM123055; R01GM133840; DMS1614777; CMMI1825941; MCB1925643; DBI2003635; R01GM123159; IG 109/1-1; FOR-1805; R37AI36040; Q1279; DBI2030381; MR/N009614/1; 208398/Z/17/Z; BB/P000517/1; BB/P000975/1; 89233218CNA000001
OSTI ID:
1777461
Alternate ID(s):
OSTI ID: 1784701
Report Number(s):
LA-UR-20-25444; TRN: US2209463
Journal Information:
Nature Methods, Vol. 18, Issue 2; ISSN 1548-7091
Publisher:
Nature Publishing GroupCopyright Statement
Country of Publication:
United States
Language:
English

References (56)

Molecular dynamics-based refinement and validation for sub-5 Å cryo-electron microscopy maps journal July 2016
Sequence assignment for low-resolution modelling of protein crystal structures journal July 2019
Satisfying Hydrogen Bonding Potential in Proteins journal May 1994
Visualization of biological macromolecules at near-atomic resolution: cryo-electron microscopy comes of age journal January 2019
RosettaES: a sampling strategy enabling automated interpretation of difficult cryo-EM maps journal June 2017
Improved metrics for comparing structures of macromolecular assemblies determined by 3D electron-microscopy journal July 2017
A New Generation of Crystallographic Validation Tools for the Protein Data Bank journal October 2011
Evolving data standards for cryo-EM structures journal January 2020
MolProbity: More and better reference data for improved all-atom structure validation: PROTEIN SCIENCE.ORG journal November 2017
Accurate model annotation of a near-atomic resolution cryo-EM map journal March 2017
TEMPy : a Python library for assessment of three-dimensional electron microscopy density fits journal June 2015
Automated and accurate deposition of structures solved by X-ray diffraction to the Protein Data Bank journal September 2004
Web-based visualisation and analysis of 3D electron-microscopy data from EMDB and PDB journal November 2013
Macromolecular structure determination using X-rays, neutrons and electrons: recent developments in Phenix journal October 2019
Recent developments in the CCP-EM software suite journal May 2017
Measurement of atom resolvability in cryo-EM maps with Q-scores journal February 2020
Automated map sharpening by maximization of detail and connectivity journal May 2018
CASP prediction center infrastructure and evaluation measures in CASP10 and CASP ROLL: CASP Prediction Center journal October 2013
Outcome of the First Electron Microscopy Validation Task Force Meeting journal February 2012
Resolution and Probabilistic Models of Components in CryoEM Maps of Mature P22 Bacteriophage journal February 2016
UCSF Chimera?A visualization system for exploratory research and analysis journal January 2004
Flexible Fitting of Atomic Structures into Electron Microscopy Maps Using Molecular Dynamics journal May 2008
Recommendations of the wwPDB NMR Validation Task Force journal September 2013
New tools in MolProbity validation: CaBLAM for CryoEM backbone, UnDowser to rethink “waters,” and NGL Viewer to recapture online 3D graphics journal November 2019
Modeling protein quaternary structure of homo- and hetero-oligomers beyond binary interactions by homology journal September 2017
Use of knowledge-based restraints in phenix.refine to improve macromolecular refinement at low resolution journal March 2012
Optimal Determination of Particle Orientation, Absolute Hand, and Contrast Loss in Single-particle Electron Cryomicroscopy journal October 2003
Automation and assessment of de novo modeling with Pathwalking in near atomic resolution cryoEM density maps journal December 2018
Automated cryo-EM structure refinement using correlation-driven molecular dynamics journal March 2019
MolProbity : all-atom structure validation for macromolecular crystallography journal December 2009
Features and development of Coot journal March 2010
EMRinger: side chain–directed model and map validation for 3D cryo-electron microscopy journal August 2015
Using VMD: An Introductory Tutorial journal December 2008
2019 EMDataResource Model Metrics Challenge Dataset dataset January 2020
2019 EMDataResource Model Metrics Challenge Dataset dataset January 2020
Cryo-EM model validation using independent map reconstructions: Cryo-EM Model Validation journal May 2013
Current approaches for automated model building into cryo-EM maps using Buccaneer with CCP-EM journal May 2020
Real-space refinement with DireX: From global fitting to side-chain improvements journal June 2012
Validation of Structures in the Protein Data Bank journal December 2017
Determining protein structures by combining semireliable data with atomistic physical models by Bayesian inference journal May 2015
CAD-score: A new contact area difference-based function for evaluation of protein structural models
  • Olechnovič, Kliment; Kulberkytė, Eleonora; Venclovas, Česlovas
  • Proteins: Structure, Function, and Bioinformatics, Vol. 81, Issue 1 https://doi.org/10.1002/prot.24172
journal September 2012
Version 1.2 of the Crystallography and NMR system journal October 2007
Evaluation system and web infrastructure for the second cryo-EM model challenge journal October 2018
lDDT: a local superposition-free score for comparing protein structures and models using distance difference tests journal August 2013
Scoring functions for cryoEM density fitting journal May 2011
Comparing cryo-EM structures journal December 2018
Tools for macromolecular model building and refinement into electron cryo-microscopy reconstructions journal January 2015
KING (Kinemage, Next Generation): A versatile interactive molecular and scientific visualization program journal November 2009
LGA: a method for finding 3D similarities in protein structures journal July 2003
New tools for the analysis and validation of cryo-EM maps and atomic models journal September 2018
Protein Data Bank: the single global archive for 3D macromolecular structure data journal October 2018
Cryo‐electron microscopy targets in CASP13: Overview and evaluation of results journal August 2019
De novo main-chain modeling for EM maps using MAINMAST journal April 2018
The universal mechanism for iron translocation to the ferroxidase site in ferritin, which is mediated by the well conserved transit site journal September 2010
High-resolution structure determination of sub-100 kDa complexes using conventional cryo-EM journal March 2019
Deep Learning to Predict Protein Backbone Structure from High-Resolution Cryo-EM Density Maps journal March 2020

Cited By (37)

Modeling and docking of antibody structures with Rosetta journal January 2017
MolProbity: More and better reference data for improved all-atom structure validation: PROTEIN SCIENCE.ORG journal November 2017
Macromolecular structure determination using X-rays, neutrons and electrons: recent developments in Phenix. text January 2019
Dynamic changes of the luminal and mucosa-associated gut microbiota during and after antibiotic therapy with paromomycin journal June 2015
Allosteric control of the ribosome by small-molecule antibiotics journal August 2012
In Silico Characterization of Meloidogyne Genus Nematode Cellulose Binding Proteins journal January 2019
Intestinal microbial communities associated with acute enteric infections and disease recovery journal September 2015
Resilience of the Natural Phyllosphere Microbiota of the Grapevine to Chemical and Biological Pesticides journal March 2014
The individual-specific and diverse nature of the preterm infant microbiota journal January 2013
Distribution and Interaction Patterns of Bacterial Communities in an Ornithogenic Soil of Seymour Island, Antarctica journal October 2014
Journey to the center of the protein: allostery from multitemperature multiconformer X-ray crystallography journal January 2019
EMPeror: a tool for visualizing high-throughput microbial community data journal November 2013
Mutations in ADAR1 cause Aicardi-Goutières syndrome associated with a type I interferon signature text January 2012
Tartrate-resistant acid phosphatase deficiency causes a bone dysplasia with autoimmunity and a type I interferon expression signature journal January 2011
Rhizosphere microbiomes of potato cultivated in the High Andes show stable and dynamic core microbiomes with different responses to plant development journal December 2016
The role of small in-frame insertions/deletions in inherited eye disorders and how structural modelling can help estimate their pathogenicity journal September 2016
Targeting the Microbiota to Address Diet-Induced Obesity: A Time Dependent Challenge journal June 2013
Correcting pervasive errors in RNA crystallography through enumerative structure prediction journal December 2012
Spatial variation of the colonic microbiota in patients with ulcerative colitis and control volunteers journal January 2015
Soil-Borne Bacterial Structure and Diversity Does Not Reflect Community Activity in Pampa Biome journal October 2013
Improved chemistry restraints for crystallographic refinement by integrating the Amber force field into Phenix journal January 2020
The influence of rosuvastatin on the gastrointestinal microbiota and host gene expression profiles journal May 2017
Effects of Diet on Resource Utilization by a Model Human Gut Microbiota Containing Bacteroides cellulosilyticus WH2, a Symbiont with an Extensive Glycobiome journal August 2013
Handling ligands with Coot journal March 2012
Novel mutations in malonyl-CoA-acyl carrier protein transacylase provoke autosomal recessive optic neuropathy journal January 2020
Restructuring of Epibacterial Communities on Fucus vesiculosus forma mytili in Response to Elevated pCO2 and Increased Temperature Levels journal March 2016
New tools in MolProbity validation: CaBLAM for CryoEM backbone, UnDowser to rethink “waters,” and NGL Viewer to recapture online 3D graphics journal November 2019
The Earth Microbiome project: successes and aspirations journal August 2014
Effect of different heterotrophic plate count methods on the estimation of the composition of the culturable microbial community journal January 2015
Hidden biosphere in an oxygen-deficient Atlantic open-ocean eddy: future implications of ocean deoxygenation on primary production in the eastern tropical North Atlantic journal January 2015
Salinity affects compositional traits of epibacterial communities on the brown macroalga Fucus vesiculosus journal March 2014
Bacterial DNA patterns identified using paired-end Illumina sequencing of 16S rRNA genes from whole blood samples of septic patients in the emergency room and intensive care unit journal July 2018
Structural evidence for product stabilization by the ribosomal mRNA helicase journal December 2018
Real-space refinement in PHENIX for cryo-EM and crystallography journal May 2018
Streptozotocin-induced type-1-diabetes disease onset in Sprague–Dawley rats is associated with an altered intestinal microbiota composition and decreased diversity journal January 2015
Sequencing-Based Analysis of the Bacterial and Fungal Composition of Kefir Grains and Milks from Multiple Sources journal July 2013
ISOLDE: a physically realistic environment for model building into low-resolution electron-density maps. text January 2018

Similar Records

Cryo-EM and MD infer water-mediated proton transport and autoinhibition mechanisms of Vo complex
Journal Article · Wed Oct 07 00:00:00 EDT 2020 · Science Advances · OSTI ID:1777461
+4 more
Data-guided Multi-Map variables for ensemble refinement of molecular movies
Journal Article · Tue Dec 01 00:00:00 EST 2020 · Journal of Chemical Physics · OSTI ID:1777461
+4 more
Measurement of atom resolvability in cryo-EM maps with Q-scores
Journal Article · Mon Feb 10 00:00:00 EST 2020 · Nature Methods · OSTI ID:1777461
+3 more

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
97 MATHEMATICS AND COMPUTING
Cryoelectron microscopy
Protein databases
Proteins
Statistical methods
biological science
structural biology
cryo-EM
computational biology