skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12

Abstract

Every two years groups worldwide participate in the Critical Assessment of Protein Structure Prediction (CASP) experiment to blindly test the strengths and weaknesses of their computational methods. CASP has significantly advanced the field but many hurdles still remain, which may require new ideas and collaborations. In 2012 a web-based effort called WeFold, was initiated to promote collaboration within the CASP community and attract researchers from other fields to contribute new ideas to CASP. Members of the WeFold coopetition (cooperation and competition) participated in CASP as individual teams, but also shared components of their methods to create hybrid pipelines and actively contributed to this effort. We assert that the scale and diversity of integrative prediction pipelines could not have been achieved by any individual lab or even by any collaboration among a few partners. The models contributed by the participating groups and generated by the pipelines are publicly available at the WeFold website providing a wealth of data that remains to be tapped. Here, we analyze the results of the 2014 and 2016 pipelines showing improvements according to the CASP assessment as well as areas that require further adjustments and research.

Authors:
 [1];  [2];  [3];  [4];  [5];  [6];  [7];  [8];  [5];  [9];  [7];  [8];  [10];  [11];  [9]; ORCiD logo [9];  [11];  [11];  [12]; ORCiD logo [13] more »;  [5];  [14];  [14];  [15];  [16];  [17];  [11];  [13];  [18];  [5];  [11];  [11];  [2];  [19];  [20];  [11];  [16];  [17];  [14]; ORCiD logo [16];  [1];  [11];  [11]; ORCiD logo [11];  [15];  [16];  [14];  [11];  [21];  [22];  [23];  [23];  [17];  [24];  [8];  [17];  [17];  [17];  [11];  [17];  [5];  [11]; ORCiD logo [8];  [16];  [11]; ORCiD logo [23];  [25];  [21];  [13]; ORCiD logo [19];  [9]; ORCiD logo [5];  [26] « less
  1. Ben Gurion Univ. of the Negev, Be'er sheva (Israel)
  2. Univ. of Reading (United Kingdom)
  3. Linköping Univ. (Sweden)
  4. Purdue Univ., West Lafayette, IN (United States)
  5. Univ. of Missouri, Columbia, MO (United States)
  6. Univ. of Missouri, Columbia, MO (United States); Auburn Univ., AL (United States)
  7. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
  8. Univ. of Sao Paulo (Brazil)
  9. Indian Inst. of Science, Bangalore (India)
  10. Research and Information Systems, LLC, Carmel, IN (United States);; IU School of Medicine, Indianapolis, IN (United States); The Research Inst. at Nationwide Children's Hospital, Columbus, OH (United States)
  11. Univ. of Gdansk (Poland)
  12. Univ. of California, Merced, CA (United States)
  13. Seoul National Univ. (Korea, Republic of)
  14. Univ. of Massachusetts, Dartmouth, MA (United States)
  15. Princeton Univ., NJ (United States)
  16. Texas A & M Univ., College Station, TX (United States)
  17. Univ. of Washington, Seattle, WA (United States)
  18. Univ. of Missouri, Columbia, MO (United States); NorthEast Normal Univ., Changchun (China)
  19. Max Planck Inst. for Biophysical Chemistry, Gottingen (Germany)
  20. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); California Polytechnic State Univ. (CalPoly), Paloma, CA (United States)
  21. Cornell Univ., Ithaca, NY (United States)
  22. Univ. of Michigan, Ann Arbor, MI (United States)
  23. Univ. of Gdansk (Poland); Medical Univ. of Gdansk (Poland)
  24. Northeastern Univ., Boston, MA (United States)
  25. Stanford Univ., CA (United States)
  26. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Davis, CA (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
Contributing Org.:
Foldit Players consortium
OSTI Identifier:
1478361
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
Scientific Reports
Additional Journal Information:
Journal Volume: 8; Journal Issue: 1; Journal ID: ISSN 2045-2322
Publisher:
Nature Publishing Group
Country of Publication:
United States
Language:
English
Subject:
59 BASIC BIOLOGICAL SCIENCES

Citation Formats

Keasar, Chen, McGuffin, Liam J., Wallner, Björn, Chopra, Gaurav, Adhikari, Badri, Bhattacharya, Debswapna, Blake, Lauren, Bortot, Leandro Oliveira, Cao, Renzhi, Dhanasekaran, B. K., Dimas, Itzhel, Faccioli, Rodrigo Antonio, Faraggi, Eshel, Ganzynkowicz, Robert, Ghosh, Sambit, Ghosh, Soma, Giełdoń, Artur, Golon, Lukasz, He, Yi, Heo, Lim, Hou, Jie, Khan, Main, Khatib, Firas, Khoury, George A., Kieslich, Chris, Kim, David E., Krupa, Pawel, Lee, Gyu Rie, Li, Hongbo, Li, Jilong, Lipska, Agnieszka, Liwo, Adam, Maghrabi, Ali Hassan A., Mirdita, Milot, Mirzaei, Shokoufeh, Mozolewska, Magdalena A., Onel, Melis, Ovchinnikov, Sergey, Shah, Anand, Shah, Utkarsh, Sidi, Tomer, Sieradzan, Adam K., Ślusarz, Magdalena, Ślusarz, Rafal, Smadbeck, James, Tamamis, Phanourios, Trieber, Nicholas, Wirecki, Tomasz, Yin, Yanping, Zhang, Yang, Bacardit, Jaume, Baranowski, Maciej, Chapman, Nicholas, Cooper, Seth, Defelicibus, Alexandre, Flatten, Jeff, Koepnick, Brian, Popović, Zoran, Zaborowski, Bartlomiej, Baker, David, Cheng, Jianlin, Czaplewski, Cezary, Delbem, Alexandre Cláudio Botazzo, Floudas, Christodoulos, Kloczkowski, Andrzej, Ołdziej, Stanislaw, Levitt, Michael, Scheraga, Harold, Seok, Chaok, Söding, Johannes, Vishveshwara, Saraswathi, Xu, Dong, and Crivelli, Silvia N. An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12. United States: N. p., 2018. Web. doi:10.1038/s41598-018-26812-8.
Keasar, Chen, McGuffin, Liam J., Wallner, Björn, Chopra, Gaurav, Adhikari, Badri, Bhattacharya, Debswapna, Blake, Lauren, Bortot, Leandro Oliveira, Cao, Renzhi, Dhanasekaran, B. K., Dimas, Itzhel, Faccioli, Rodrigo Antonio, Faraggi, Eshel, Ganzynkowicz, Robert, Ghosh, Sambit, Ghosh, Soma, Giełdoń, Artur, Golon, Lukasz, He, Yi, Heo, Lim, Hou, Jie, Khan, Main, Khatib, Firas, Khoury, George A., Kieslich, Chris, Kim, David E., Krupa, Pawel, Lee, Gyu Rie, Li, Hongbo, Li, Jilong, Lipska, Agnieszka, Liwo, Adam, Maghrabi, Ali Hassan A., Mirdita, Milot, Mirzaei, Shokoufeh, Mozolewska, Magdalena A., Onel, Melis, Ovchinnikov, Sergey, Shah, Anand, Shah, Utkarsh, Sidi, Tomer, Sieradzan, Adam K., Ślusarz, Magdalena, Ślusarz, Rafal, Smadbeck, James, Tamamis, Phanourios, Trieber, Nicholas, Wirecki, Tomasz, Yin, Yanping, Zhang, Yang, Bacardit, Jaume, Baranowski, Maciej, Chapman, Nicholas, Cooper, Seth, Defelicibus, Alexandre, Flatten, Jeff, Koepnick, Brian, Popović, Zoran, Zaborowski, Bartlomiej, Baker, David, Cheng, Jianlin, Czaplewski, Cezary, Delbem, Alexandre Cláudio Botazzo, Floudas, Christodoulos, Kloczkowski, Andrzej, Ołdziej, Stanislaw, Levitt, Michael, Scheraga, Harold, Seok, Chaok, Söding, Johannes, Vishveshwara, Saraswathi, Xu, Dong, & Crivelli, Silvia N. An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12. United States. doi:10.1038/s41598-018-26812-8.
Keasar, Chen, McGuffin, Liam J., Wallner, Björn, Chopra, Gaurav, Adhikari, Badri, Bhattacharya, Debswapna, Blake, Lauren, Bortot, Leandro Oliveira, Cao, Renzhi, Dhanasekaran, B. K., Dimas, Itzhel, Faccioli, Rodrigo Antonio, Faraggi, Eshel, Ganzynkowicz, Robert, Ghosh, Sambit, Ghosh, Soma, Giełdoń, Artur, Golon, Lukasz, He, Yi, Heo, Lim, Hou, Jie, Khan, Main, Khatib, Firas, Khoury, George A., Kieslich, Chris, Kim, David E., Krupa, Pawel, Lee, Gyu Rie, Li, Hongbo, Li, Jilong, Lipska, Agnieszka, Liwo, Adam, Maghrabi, Ali Hassan A., Mirdita, Milot, Mirzaei, Shokoufeh, Mozolewska, Magdalena A., Onel, Melis, Ovchinnikov, Sergey, Shah, Anand, Shah, Utkarsh, Sidi, Tomer, Sieradzan, Adam K., Ślusarz, Magdalena, Ślusarz, Rafal, Smadbeck, James, Tamamis, Phanourios, Trieber, Nicholas, Wirecki, Tomasz, Yin, Yanping, Zhang, Yang, Bacardit, Jaume, Baranowski, Maciej, Chapman, Nicholas, Cooper, Seth, Defelicibus, Alexandre, Flatten, Jeff, Koepnick, Brian, Popović, Zoran, Zaborowski, Bartlomiej, Baker, David, Cheng, Jianlin, Czaplewski, Cezary, Delbem, Alexandre Cláudio Botazzo, Floudas, Christodoulos, Kloczkowski, Andrzej, Ołdziej, Stanislaw, Levitt, Michael, Scheraga, Harold, Seok, Chaok, Söding, Johannes, Vishveshwara, Saraswathi, Xu, Dong, and Crivelli, Silvia N. Mon . "An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12". United States. doi:10.1038/s41598-018-26812-8. https://www.osti.gov/servlets/purl/1478361.
@article{osti_1478361,
title = {An analysis and evaluation of the WeFold collaborative for protein structure prediction and its pipelines in CASP11 and CASP12},
author = {Keasar, Chen and McGuffin, Liam J. and Wallner, Björn and Chopra, Gaurav and Adhikari, Badri and Bhattacharya, Debswapna and Blake, Lauren and Bortot, Leandro Oliveira and Cao, Renzhi and Dhanasekaran, B. K. and Dimas, Itzhel and Faccioli, Rodrigo Antonio and Faraggi, Eshel and Ganzynkowicz, Robert and Ghosh, Sambit and Ghosh, Soma and Giełdoń, Artur and Golon, Lukasz and He, Yi and Heo, Lim and Hou, Jie and Khan, Main and Khatib, Firas and Khoury, George A. and Kieslich, Chris and Kim, David E. and Krupa, Pawel and Lee, Gyu Rie and Li, Hongbo and Li, Jilong and Lipska, Agnieszka and Liwo, Adam and Maghrabi, Ali Hassan A. and Mirdita, Milot and Mirzaei, Shokoufeh and Mozolewska, Magdalena A. and Onel, Melis and Ovchinnikov, Sergey and Shah, Anand and Shah, Utkarsh and Sidi, Tomer and Sieradzan, Adam K. and Ślusarz, Magdalena and Ślusarz, Rafal and Smadbeck, James and Tamamis, Phanourios and Trieber, Nicholas and Wirecki, Tomasz and Yin, Yanping and Zhang, Yang and Bacardit, Jaume and Baranowski, Maciej and Chapman, Nicholas and Cooper, Seth and Defelicibus, Alexandre and Flatten, Jeff and Koepnick, Brian and Popović, Zoran and Zaborowski, Bartlomiej and Baker, David and Cheng, Jianlin and Czaplewski, Cezary and Delbem, Alexandre Cláudio Botazzo and Floudas, Christodoulos and Kloczkowski, Andrzej and Ołdziej, Stanislaw and Levitt, Michael and Scheraga, Harold and Seok, Chaok and Söding, Johannes and Vishveshwara, Saraswathi and Xu, Dong and Crivelli, Silvia N.},
abstractNote = {Every two years groups worldwide participate in the Critical Assessment of Protein Structure Prediction (CASP) experiment to blindly test the strengths and weaknesses of their computational methods. CASP has significantly advanced the field but many hurdles still remain, which may require new ideas and collaborations. In 2012 a web-based effort called WeFold, was initiated to promote collaboration within the CASP community and attract researchers from other fields to contribute new ideas to CASP. Members of the WeFold coopetition (cooperation and competition) participated in CASP as individual teams, but also shared components of their methods to create hybrid pipelines and actively contributed to this effort. We assert that the scale and diversity of integrative prediction pipelines could not have been achieved by any individual lab or even by any collaboration among a few partners. The models contributed by the participating groups and generated by the pipelines are publicly available at the WeFold website providing a wealth of data that remains to be tapped. Here, we analyze the results of the 2014 and 2016 pipelines showing improvements according to the CASP assessment as well as areas that require further adjustments and research.},
doi = {10.1038/s41598-018-26812-8},
journal = {Scientific Reports},
number = 1,
volume = 8,
place = {United States},
year = {2018},
month = {7}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 8 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Protein Structure and Function: Looking through the Network of Side-Chain Interactions
journal, November 2015


Contact map prediction using a large-scale ensemble of rule sets and the fusion of multiple predicted structural features
journal, July 2012


Purely Structural Protein Scoring Functions Using Support Vector Machine and Ensemble Learning
journal, January 2019

  • Mirzaei, Shokoufeh; Sidi, Tomer; Keasar, Chen
  • IEEE/ACM Transactions on Computational Biology and Bioinformatics
  • DOI: 10.1109/TCBB.2016.2602269

APOLLO: a quality assessment service for single and multiple protein models
journal, May 2011


3Drefine: Consistent protein structure refinement by optimizing hydrogen bonding network and atomic-level energy minimization
journal, September 2012

  • Bhattacharya, Debswapna; Cheng, Jianlin
  • Proteins: Structure, Function, and Bioinformatics, Vol. 81, Issue 1
  • DOI: 10.1002/prot.24167

GalaxyRefine: protein structure refinement driven by side-chain repacking
journal, May 2013

  • Heo, Lim; Park, Hahnbeom; Seok, Chaok
  • Nucleic Acids Research, Vol. 41, Issue W1
  • DOI: 10.1093/nar/gkt458

High-Resolution Comparative Modeling with RosettaCM
journal, October 2013


A decade of CASP: progress, bottlenecks and prognosis in protein structure prediction
journal, June 2005


Effective protein model structure refinement by loop modeling and overall relaxation: Refinement with Loop Modeling and MD Relaxation
journal, July 2015

  • Lee, Gyu Rie; Heo, Lim; Seok, Chaok
  • Proteins: Structure, Function, and Bioinformatics, Vol. 84
  • DOI: 10.1002/prot.24858

WeFold: A coopetition for protein structure prediction: WeFold Protein Structure Prediction Coopetition
journal, July 2014

  • Khoury, George A.; Liwo, Adam; Khatib, Firas
  • Proteins: Structure, Function, and Bioinformatics, Vol. 82, Issue 9
  • DOI: 10.1002/prot.24538

Princeton_TIGRESS: Protein geometry refinement using simulations and support vector machines: Princeton_TIGRESS
journal, November 2013

  • Khoury, George A.; Tamamis, Phanourios; Pinnaduwage, Neesha
  • Proteins: Structure, Function, and Bioinformatics, Vol. 82, Issue 5
  • DOI: 10.1002/prot.24459

Assessing the utility of coevolution-based residue-residue contact predictions in a sequence- and structure-rich era
journal, September 2013

  • Kamisetty, H.; Ovchinnikov, S.; Baker, D.
  • Proceedings of the National Academy of Sciences, Vol. 110, Issue 39
  • DOI: 10.1073/pnas.1314045110

Critical assessment of methods of protein structure prediction: Progress and new directions in round XI: Progress in CASP XI
journal, June 2016

  • Moult, John; Fidelis, Krzysztof; Kryshtafovych, Andriy
  • Proteins: Structure, Function, and Bioinformatics, Vol. 84
  • DOI: 10.1002/prot.25064

A multilayer evaluation approach for protein structure prediction and model quality assessment
journal, January 2011

  • Zhang, Jingfen; Wang, Qingguo; Vantasin, Kittinun
  • Proteins: Structure, Function, and Bioinformatics, Vol. 79, Issue S10
  • DOI: 10.1002/prot.23184

Toward High-Resolution de Novo Structure Prediction for Small Proteins
journal, September 2005


Improved model quality assessment using ProQ2
journal, January 2012


ModFOLD6: an accurate web server for the global and local quality estimation of 3D protein models
journal, April 2017

  • Maghrabi, Ali H. A.; McGuffin, Liam J.
  • Nucleic Acids Research, Vol. 45, Issue W1
  • DOI: 10.1093/nar/gkx332

A global machine learning based scoring function for protein structure prediction: Global Knowledge-Based Function for Protein Structure
journal, November 2013

  • Faraggi, Eshel; Kloczkowski, Andrzej
  • Proteins: Structure, Function, and Bioinformatics, Vol. 82, Issue 5
  • DOI: 10.1002/prot.24454

conSSert: Consensus SVM Model for Accurate Prediction of Ordered Secondary Structure
journal, February 2016

  • Kieslich, Chris A.; Smadbeck, James; Khoury, George A.
  • Journal of Chemical Information and Modeling, Vol. 56, Issue 3
  • DOI: 10.1021/acs.jcim.5b00566

A large-scale experiment to assess protein structure prediction methods
journal, November 1995

  • Moult, John; Pedersen, Jan T.; Judson, Richard
  • Proteins: Structure, Function, and Genetics, Vol. 23, Issue 3
  • DOI: 10.1002/prot.340230303

Critical assessment of methods of protein structure prediction (CASP) - round x: Critical Assessment of Structure Prediction
journal, December 2013

  • Moult, John; Fidelis, Krzysztof; Kryshtafovych, Andriy
  • Proteins: Structure, Function, and Bioinformatics, Vol. 82
  • DOI: 10.1002/prot.24452

The Protein Data Bank
journal, January 2000


GOAP: A Generalized Orientation-Dependent, All-Atom Statistical Potential for Protein Structure Prediction
journal, October 2011


The ModFOLD server for the quality assessment of protein structural models
journal, January 2008


Crystal structure of a monomeric retroviral protease solved by protein folding game players
journal, September 2011

  • Khatib, Firas; DiMaio, Frank; Cooper, Seth
  • Nature Structural & Molecular Biology, Vol. 18, Issue 10
  • DOI: 10.1038/nsmb.2119

The HHpred interactive server for protein homology detection and structure prediction
journal, July 2005

  • Soding, J.; Biegert, A.; Lupas, A. N.
  • Nucleic Acids Research, Vol. 33, Issue Web Server
  • DOI: 10.1093/nar/gki408

Creating a gateway that enables large-scale science coopetition: LARGE-SCALE SCIENCE COOPETITION
journal, April 2014

  • Crivelli, Silvia; Dooley, Rion; Holmes, Raquell
  • Concurrency and Computation: Practice and Experience, Vol. 27, Issue 2
  • DOI: 10.1002/cpe.3270

    Works referencing / citing this record:

    GalaxyRefine2: simultaneous refinement of inaccurate local regions and overall protein structure
    journal, April 2019

    • Lee, Gyu Rie; Won, Jonghun; Heo, Lim
    • Nucleic Acids Research, Vol. 47, Issue W1
    • DOI: 10.1093/nar/gkz288

    GalaxyRefine2: simultaneous refinement of inaccurate local regions and overall protein structure
    journal, April 2019

    • Lee, Gyu Rie; Won, Jonghun; Heo, Lim
    • Nucleic Acids Research, Vol. 47, Issue W1
    • DOI: 10.1093/nar/gkz288

    A nonlinear support vector machine‐based feature selection approach for fault detection and diagnosis: Application to the Tennessee Eastman process
    journal, January 2019

    • Onel, Melis; Kieslich, Chris A.; Pistikopoulos, Efstratios N.
    • AIChE Journal, Vol. 65, Issue 3
    • DOI: 10.1002/aic.16497

    Introduction of a bounded penalty function in contact‐assisted simulations of protein structures to omit false restraints
    journal, April 2019

    • Lubecka, Emilia A.; Liwo, Adam
    • Journal of Computational Chemistry
    • DOI: 10.1002/jcc.25847

    Assessment of protein model structure accuracy estimation in CASP13: Challenges in the era of deep learning
    journal, August 2019

    • Won, Jonghun; Baek, Minkyung; Monastyrskyy, Bohdan
    • Proteins: Structure, Function, and Bioinformatics, Vol. 87, Issue 12
    • DOI: 10.1002/prot.25804

    Computational re-design of protein structures to improve solubility
    journal, July 2019


    Analysis validation has been neglected in the Age of Reproducibility
    journal, December 2018


    Grouping of complex substances using analytical chemistry data: A framework for quantitative evaluation and visualization
    journal, October 2019