An investigation of irreproducibility in maximum likelihood phylogenetic inference

Shen, Xing-Xing; Li, Yuanning; Hittinger, Chris Todd; Chen, Xue-xin; Rokas, Antonis

doi:10.1038/s41467-020-20005-6

An investigation of irreproducibility in maximum likelihood phylogenetic inference

Journal Article · Sun Nov 29 23:00:00 EST 2020 · Nature Communications

DOI:https://doi.org/10.1038/s41467-020-20005-6· OSTI ID:1764993

^[1]; ^[2]; ^[3]; ^[4]; ^[2]

Zhejiang Univ., Hangzhou (China); Vanderbilt University Department of Biological Sciences
Vanderbilt Univ., Nashville, TN (United States)
Univ. of Wisconsin, Madison, WI (United States)
Zhejiang Univ., Hangzhou (China)

Phylogenetic trees are essential for studying biology, but their reproducibility under identical parameter settings remains unexplored. Here, we find that 3515 (18.11%) IQ-TREE-inferred and 1813 (9.34%) RAxML-NG-inferred maximum likelihood (ML) gene trees are topologically irreproducible when executing two replicates (Run1 and Run2) for each of 19,414 gene alignments in 15 animal, plant, and fungal phylogenomic datasets. Notably, coalescent-based ASTRAL species phylogenies inferred from Run1 and Run2 sets of individual gene trees are topologically irreproducible for 9/15 phylogenomic datasets, whereas concatenation-based phylogenies inferred twice from the same supermatrix are reproducible. Our simulations further show that irreproducible phylogenies are more likely to be incorrect than reproducible phylogenies. These results suggest that a considerable fraction of single-gene ML trees may be irreproducible. Increasing reproducibility in ML inference will benefit from providing analyses’ log files, which contain typically reported parameters (e.g., program, substitution model, number of tree searches) but also typically unreported ones (e.g., random starting seed number, number of threads, processor type).

View Accepted Manuscript (DOE)

Research Organization:: Great Lakes Bioenergy Research Center, Madison, WI (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER)

Grant/Contract Number:: SC0018409

OSTI ID:: 1764993

Journal Information:: Nature Communications, Journal Name: Nature Communications Journal Issue: 1 Vol. 11; ISSN 2041-1723

Publisher:: Nature Publishing GroupCopyright Statement

Country of Publication:: United States

Language:: English

References (66)

Reproducibility in Chemical Research Bergman, Robert G.; Danheiser, Rick L. Angewandte Chemie International Edition, Vol. 55, Issue 41 https://doi.org/10.1002/anie.201606591	journal	August 2016
Comparison of phylogenetic trees Robinson, D. F.; Foulds, L. R. Mathematical Biosciences, Vol. 53, Issue 1-2 https://doi.org/10.1016/0025-5564(81)90043-2	journal	February 1981
Comparison of labeled trees with valency three Robinson, D. F. Journal of Combinatorial Theory, Series B, Vol. 11, Issue 2 https://doi.org/10.1016/0095-8956(71)90020-7	journal	October 1971
Tempo and Mode of Genome Evolution in the Budding Yeast Subphylum Shen, Xing-Xing; Opulente, Dana A.; Kominek, Jacek Cell, Vol. 175, Issue 6 https://doi.org/10.1016/j.cell.2018.10.023	journal	November 2018
Horizontal Transfer of a Large and Highly Toxic Secondary Metabolic Gene Cluster between Fungi Slot, Jason C.; Rokas, Antonis Current Biology, Vol. 21, Issue 2 https://doi.org/10.1016/j.cub.2010.12.020	journal	January 2011
Irreproducibility in Preclinical Biomedical Research: Perceptions, Uncertainties, and Knowledge Gaps Jarvis, Michael F.; Williams, Michael Trends in Pharmacological Sciences, Vol. 37, Issue 4 https://doi.org/10.1016/j.tips.2015.12.001	journal	April 2016
An Integrated Perspective on Phylogenetic Workflows Guang, August; Zapata, Felipe; Howison, Mark Trends in Ecology & Evolution, Vol. 31, Issue 2 https://doi.org/10.1016/j.tree.2015.12.007	journal	February 2016
Transparency in Ecology and Evolution: Real Problems, Real Solutions Parker, Timothy H.; Forstmeier, Wolfgang; Koricheva, Julia Trends in Ecology & Evolution, Vol. 31, Issue 9 https://doi.org/10.1016/j.tree.2016.07.002	journal	September 2016
Coalescence vs. concatenation: Sophisticated analyses vs. first principles applied to rooting the angiosperms Simmons, Mark P.; Gatesy, John Molecular Phylogenetics and Evolution, Vol. 91 https://doi.org/10.1016/j.ympev.2015.05.011	journal	October 2015
Data archiving is a good investment Piwowar, Heather A.; Vision, Todd J.; Whitlock, Michael C. Nature, Vol. 473, Issue 7347 https://doi.org/10.1038/473285a	journal	May 2011
Missing data mean holes in tree of life Drew, Bryan T. Nature, Vol. 493, Issue 7432 https://doi.org/10.1038/493305f	journal	January 2013
1,500 scientists lift the lid on reproducibility Baker, Monya Nature, Vol. 533, Issue 7604 https://doi.org/10.1038/533452a	journal	May 2016
The case for open computer programs Ince, Darrel C.; Hatton, Leslie; Graham-Cumming, John Nature, Vol. 482, Issue 7386 https://doi.org/10.1038/nature10836	journal	February 2012
Mutational heterogeneity in cancer and the search for new cancer-associated genes Lawrence, Michael S.; Stojanov, Petar; Polak, Paz Nature, Vol. 499, Issue 7457 https://doi.org/10.1038/nature12213	journal	June 2013
A new view of the tree of life Hug, Laura A.; Baker, Brett J.; Anantharaman, Karthik Nature Microbiology, Vol. 1, Issue 5 https://doi.org/10.1038/nmicrobiol.2016.48	journal	April 2016
Explosive diversification of marine fishes at the Cretaceous–Palaeogene boundary Alfaro, Michael E.; Faircloth, Brant C.; Harrington, Richard C. Nature Ecology & Evolution, Vol. 2, Issue 4 https://doi.org/10.1038/s41559-018-0494-6	journal	March 2018
Releasing uncurated datasets is essential for reproducible phylogenomics Salomaki, Eric D.; Eme, Laura; Brown, Matthew W. Nature Ecology & Evolution, Vol. 4, Issue 11 https://doi.org/10.1038/s41559-020-01296-w	journal	September 2020
Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015 Camerer, Colin F.; Dreber, Anna; Holzmeister, Felix Nature Human Behaviour, Vol. 2, Issue 9 https://doi.org/10.1038/s41562-018-0399-z	journal	August 2018
The evolution of language families is shaped by the environment beyond neutral drift Bentz, Christian; Dediu, Dan; Verkerk, Annemarie Nature Human Behaviour, Vol. 2, Issue 11 https://doi.org/10.1038/s41562-018-0457-6	journal	November 2018
Open is not enough Chen, Xiaoli; Dallmeier-Tiessen, Sünje; Dasler, Robin Nature Physics, Vol. 15, Issue 2 https://doi.org/10.1038/s41567-018-0342-2	journal	November 2018
Ancient hepatitis B viruses from the Bronze Age to the Medieval period Mühlemann, Barbara; Jones, Terry C.; Damgaard, Peter de Barros Nature, Vol. 557, Issue 7705 https://doi.org/10.1038/s41586-018-0097-z	journal	May 2018
One thousand plant transcriptomes and the phylogenomics of green plants No authors listed Nature, Vol. 574, Issue 7780, p. 679-685 https://doi.org/10.1038/s41586-019-1693-2	journal	October 2019
An Approximately Unbiased Test of Phylogenetic Tree Selection Shimodaira, Hidetoshi Systematic Biology, Vol. 51, Issue 3 https://doi.org/10.1080/10635150290069913	journal	May 2002
Seq-Gen: an application for the Monte Carlo simulation of DNA sequence evolution along phylogenetic trees Rambaut, Andrew; Grass, Nicholas C. Bioinformatics, Vol. 13, Issue 3 https://doi.org/10.1093/bioinformatics/13.3.235	journal	January 1997
phangorn: phylogenetic analysis in R Schliep, Klaus Peter Bioinformatics, Vol. 27, Issue 4 https://doi.org/10.1093/bioinformatics/btq706	journal	December 2010
Inferring duplications, losses, transfers and incomplete lineage sorting with nonbinary species trees Stolzer, Maureen; Lai, Han; Xu, Minli Bioinformatics, Vol. 28, Issue 18 https://doi.org/10.1093/bioinformatics/bts386	journal	September 2012
RAxML version 8: a tool for phylogenetic analysis and post-analysis of large phylogenies Stamatakis, Alexandros Bioinformatics, Vol. 30, Issue 9 https://doi.org/10.1093/bioinformatics/btu033	journal	January 2014
ASTRAL: genome-scale coalescent-based species tree estimation Mirarab, S.; Reaz, R.; Bayzid, Md. S. Bioinformatics, Vol. 30, Issue 17 https://doi.org/10.1093/bioinformatics/btu462	journal	August 2014
ape 5.0: an environment for modern phylogenetics and evolutionary analyses in R Paradis, Emmanuel; Schliep, Klaus Bioinformatics, Vol. 35, Issue 3 https://doi.org/10.1093/bioinformatics/bty633	journal	July 2018
RAxML-NG: a fast, scalable and user-friendly tool for maximum likelihood phylogenetic inference Kozlov, Alexey M.; Darriba, Diego; Flouri, Tomáš Bioinformatics, Vol. 35, Issue 21 https://doi.org/10.1093/bioinformatics/btz305	journal	May 2019
IQ-TREE: A Fast and Effective Stochastic Algorithm for Estimating Maximum-Likelihood Phylogenies Nguyen, Lam-Tung; Schmidt, Heiko A.; von Haeseler, Arndt Molecular Biology and Evolution, Vol. 32, Issue 1 https://doi.org/10.1093/molbev/msu300	journal	November 2014
Evaluating Fast Maximum Likelihood-Based Phylogenetic Programs Using Empirical Phylogenomic Data Sets Zhou, Xiaofan; Shen, Xing-Xing; Hittinger, Chris Todd Molecular Biology and Evolution, Vol. 35, Issue 2 https://doi.org/10.1093/molbev/msx302	journal	November 2017
MEGA X: Molecular Evolutionary Genetics Analysis across Computing Platforms Kumar, Sudhir; Stecher, Glen; Li, Michael Molecular Biology and Evolution, Vol. 35, Issue 6 https://doi.org/10.1093/molbev/msy096	journal	May 2018
ModelTest-NG: A New and Scalable Tool for the Selection of DNA and Protein Evolutionary Models Darriba, Diego; Posada, David; Kozlov, Alexey M. Molecular Biology and Evolution, Vol. 37, Issue 1 https://doi.org/10.1093/molbev/msz189	journal	August 2019
The Multispecies Coalescent Model Outperforms Concatenation Across Diverse Phylogenomic Data Sets Jiang, Xiaodong; Edwards, Scott V.; Liu, Liang Systematic Biology, Vol. 69, Issue 4 https://doi.org/10.1093/sysbio/syaa008	journal	February 2020
New Algorithms and Methods to Estimate Maximum-Likelihood Phylogenies: Assessing the Performance of PhyML 3.0 Guindon, Stéphane; Dufayard, Jean-François; Lefort, Vincent Systematic Biology, Vol. 59, Issue 3 https://doi.org/10.1093/sysbio/syq010	journal	March 2010
Ultraconserved Elements Anchor Thousands of Genetic Markers Spanning Multiple Evolutionary Timescales Faircloth, Brant C.; McCormack, John E.; Crawford, Nicholas G. Systematic Biology, Vol. 61, Issue 5 https://doi.org/10.1093/sysbio/sys004	journal	January 2012
Anchored Hybrid Enrichment for Massively High-Throughput Phylogenomics Lemmon, Alan R.; Emme, Sandra A.; Lemmon, Emily Moriarty Systematic Biology, Vol. 61, Issue 5 https://doi.org/10.1093/sysbio/sys049	journal	July 2012
The Phylogenetic Likelihood Library Flouri, T.; Izquierdo-Carrasco, F.; Darriba, D. Systematic Biology, Vol. 64, Issue 2 https://doi.org/10.1093/sysbio/syu084	journal	October 2014
Impacts of Terraces on Phylogenetic Inference Sanderson, Michael J.; McMahon, Michelle M.; Stamatakis, Alexandros Systematic Biology, Vol. 64, Issue 5 https://doi.org/10.1093/sysbio/syv024	journal	May 2015
Efficient computation of the phylogenetic likelihood function on multi-gene alignments and multi-core architectures Stamatakis, Alexandros; Ott, Michael Philosophical Transactions of the Royal Society B: Biological Sciences, Vol. 363, Issue 1512 https://doi.org/10.1098/rstb.2008.0163	journal	October 2008
Exon capture phylogenomics: efficacy across scales of divergence Bragg, Jason G.; Potter, Sally; Bi, Ke Molecular Ecology Resources, Vol. 16, Issue 5 https://doi.org/10.1111/1755-0998.12449	journal	August 2015
Dissecting the basis of novel trait evolution in a radiation with widespread phylogenetic discordance Wu, Meng; Kostyun, Jamie L.; Hahn, Matthew W. Molecular Ecology, Vol. 27, Issue 16 https://doi.org/10.1111/mec.14780	journal	July 2018
Language Phylogenies Reveal Expansion Pulses and Pauses in Pacific Settlement Gray, R. D.; Drummond, A. J.; Greenhill, S. J. Science, Vol. 323, Issue 5913 https://doi.org/10.1126/science.1166858	journal	January 2009
Genome-Wide Comparison of Medieval and Modern Mycobacterium leprae Schuenemann, V. J.; Singh, P.; Mendum, T. A. Science, Vol. 341, Issue 6142 https://doi.org/10.1126/science.1238286	journal	June 2013
Reproducibility McNutt, M. Science, Vol. 343, Issue 6168 https://doi.org/10.1126/science.1250475	journal	January 2014
Estimating the reproducibility of psychological science No authors listed Science, Vol. 349, Issue 6251 https://doi.org/10.1126/science.aac4716	journal	August 2015
Evaluating replicability of laboratory experiments in economics Camerer, C. F.; Dreber, A.; Forsell, E. Science, Vol. 351, Issue 6280 https://doi.org/10.1126/science.aaf0918	journal	March 2016
Reproducibility in Scientific Computing Ivie, Peter; Thain, Douglas ACM Computing Surveys, Vol. 51, Issue 3 https://doi.org/10.1145/3186266	journal	July 2018
State-of the art methodologies dictate new standards for phylogenetic analysis Anisimova, Maria; Liberles, David A.; Philippe, Hervé BMC Evolutionary Biology, Vol. 13, Issue 1 https://doi.org/10.1186/1471-2148-13-161	journal	January 2013
Sharing and re-use of phylogenetic trees (and associated data) to facilitate synthesis Stoltzfus, Arlin; O'Meara, Brian; Whitacre, Jamie BMC Research Notes, Vol. 5, Issue 1 https://doi.org/10.1186/1756-0500-5-574	journal	October 2012
ASTRAL-III: polynomial time species tree reconstruction from partially resolved gene trees Zhang, Chao; Rabiee, Maryam; Sayyari, Erfan BMC Bioinformatics, Vol. 19, Issue S6 https://doi.org/10.1186/s12859-018-2129-y	journal	May 2018
OrthoFinder: phylogenetic orthology inference for comparative genomics Emms, David M.; Kelly, Steven Genome Biology, Vol. 20, Issue 1 https://doi.org/10.1186/s13059-019-1832-y	journal	November 2019
Lost Branches on the Tree of Life Drew, Bryan T.; Gazis, Romina; Cabezas, Patricia PLoS Biology, Vol. 11, Issue 9 https://doi.org/10.1371/journal.pbio.1001636	journal	September 2013
Troubleshooting Public Data Archiving: Suggestions to Increase Participation Roche, Dominique G.; Lanfear, Robert; Binning, Sandra A. PLoS Biology, Vol. 12, Issue 1 https://doi.org/10.1371/journal.pbio.1001779	journal	January 2014
Public Data Archiving in Ecology and Evolution: How Well Are We Doing? Roche, Dominique G.; Kruuk, Loeske E. B.; Lanfear, Robert PLOS Biology, Vol. 13, Issue 11 https://doi.org/10.1371/journal.pbio.1002295	journal	November 2015
Extensive loss of cell-cycle and DNA repair genes in an ancient lineage of bipolar budding yeasts Steenwyk, Jacob L.; Opulente, Dana A.; Kominek, Jacek PLOS Biology, Vol. 17, Issue 5 https://doi.org/10.1371/journal.pbio.3000255	journal	May 2019
Why Most Published Research Findings Are False Ioannidis, John P. A. PLoS Medicine, Vol. 2, Issue 8 https://doi.org/10.1371/journal.pmed.0020124	journal	August 2005
FastTree 2 – Approximately Maximum-Likelihood Trees for Large Alignments Price, Morgan N.; Dehal, Paramvir S.; Arkin, Adam P. PLoS ONE, Vol. 5, Issue 3 https://doi.org/10.1371/journal.pone.0009490	journal	March 2010
A study of the impact of data sharing on article citations using journal policies as a natural experiment Christensen, Garret; Dafoe, Allan; Miguel, Edward PLOS ONE, Vol. 14, Issue 12 https://doi.org/10.1371/journal.pone.0225883	journal	December 2019
Molecular Systematics Moore, Debra S.; Hillis, David M.; Moritz, Craig Copeia, Vol. 1996, Issue 4 https://doi.org/10.2307/1447682	journal	December 1996
Pitfalls in supermatrix phylogenomics Philippe, Hervé; Vienne, Damien M. de; Ranwez, Vincent European Journal of Taxonomy, Issue 283 https://doi.org/10.5852/ejt.2017.283	journal	February 2017
An investigation of irreproducibility in maximum likelihood phylogenetic inference Shen, Xing-Xing figshare https://doi.org/10.6084/m9.figshare.11917770	dataset	January 2020
Public data archiving in ecology and evolution: how well are we doing? Roche, Dominique; Kruuk, Loeske; Lanfear, Robert figshare https://doi.org/10.6084/m9.figshare.1393269	dataset	January 2015
ASTRAL-III: polynomial time species tree reconstruction from partially resolved gene trees Zhang, Chao; Rabiee, Maryam; Sayyari, Erfan figshare https://doi.org/10.6084/m9.figshare.c.4093409.v1	collection	January 2018
A study of the impact of data sharing on article citations using journal policies as a natural experiment Christensen, Garret; Dafoe, Allan; Miguel, Edward Harvard Dataverse https://doi.org/10.7910/dvn/ortjt5	dataset	January 2020

Similar Records

The Influence of the Number of Tree Searches on Maximum Likelihood Inference in Phylogenomics

Journal Article · Fri Jun 28 00:00:00 EDT 2024 · Systematic Biology · OSTI ID:2406411

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
Phylogenetics
Phylogeny

An investigation of irreproducibility in maximum likelihood phylogenetic inference

Citation Formats

References (66)

Similar Records

Related Subjects