Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits

Azodi, Christina B.; Bolger, Emily; McCarren, Andrew; Roantree, Mark; de los Campos, Gustavo; Shiu, Shin-Han

doi:10.1534/g3.119.400498

Title: Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits

Abstract

The usefulness of genomic prediction in crop and livestock breeding programs has prompted efforts to develop new and improved genomic prediction algorithms, such as artificial neural networks and gradient tree boosting. However, the performance of these algorithms has not been compared in a systematic manner using a wide range of datasets and models. Using data of 18 traits across six plant species with different marker densities and training population sizes, we compared the performance of six linear and six non-linear algorithms. First, we found that hyperparameter selection was necessary for all non-linear algorithms and that feature selection prior to model training was critical for artificial neural networks when the markers greatly outnumbered the number of training lines. Across all species and trait combinations, no one algorithm performed best, however predictions based on a combination of results from multiple algorithms (i.e., ensemble predictions) performed consistently well. While linear and non-linear algorithms performed best for a similar number of traits, the performance of non-linear algorithms vary more between traits. Although artificial neural networks did not perform best for any trait, we identified strategies (i.e., feature selection, seeded starting weights) that boosted their performance to near the level of other algorithms. Our resultsmore »« less

Authors:

^[1]; Bolger, Emily ^[2];

^[2]; Roantree, Mark ^[2];

^[1];

^[1]

Michigan State Univ., East Lansing, MI (United States)
Dublin City Univ. (Ireland)

Publication Date:: Wed Sep 18 00:00:00 EDT 2019

Research Org.:: Great Lakes Bioenergy Research Center (GLBRC), Madison, WI (United States); Univ. of Wisconsin, Madison, WI (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Biological and Environmental Research (BER)

OSTI Identifier:: 1637329

Grant/Contract Number:: SC0018409

Resource Type:: Accepted Manuscript

Journal Name:: G3

Additional Journal Information:: Journal Volume: 9; Journal Issue: 11; Journal ID: ISSN 2160-1836

Publisher:: Genetics Society of America

Country of Publication:: United States

Language:: English

Subject:: 59 BASIC BIOLOGICAL SCIENCES; genomic selection; artificial neural network; genotype-to-phenotype; genomic prediction; genpred; shared data resources

Citation Formats


                    Azodi, Christina B., Bolger, Emily, McCarren, Andrew, Roantree, Mark, de los Campos, Gustavo, and Shiu, Shin-Han. Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits.  United States: N. p., 2019. 
Web.  doi:10.1534/g3.119.400498.

Copy to clipboard


                    Azodi, Christina B., Bolger, Emily, McCarren, Andrew, Roantree, Mark, de los Campos, Gustavo, & Shiu, Shin-Han. Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits.  United States.  https://doi.org/10.1534/g3.119.400498

Copy to clipboard


                    Azodi, Christina B., Bolger, Emily, McCarren, Andrew, Roantree, Mark, de los Campos, Gustavo, and Shiu, Shin-Han. Wed .  
"Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits".  United States.  https://doi.org/10.1534/g3.119.400498.  https://www.osti.gov/servlets/purl/1637329.

Copy to clipboard


                    
@article{osti_1637329,

  title        = {Benchmarking Parametric and Machine Learning Models for Genomic Prediction of Complex Traits},

  author       = {Azodi, Christina B. and Bolger, Emily and McCarren, Andrew and Roantree, Mark and de los Campos, Gustavo and Shiu, Shin-Han},

  abstractNote = {The usefulness of genomic prediction in crop and livestock breeding programs has prompted efforts to develop new and improved genomic prediction algorithms, such as artificial neural networks and gradient tree boosting. However, the performance of these algorithms has not been compared in a systematic manner using a wide range of datasets and models. Using data of 18 traits across six plant species with different marker densities and training population sizes, we compared the performance of six linear and six non-linear algorithms. First, we found that hyperparameter selection was necessary for all non-linear algorithms and that feature selection prior to model training was critical for artificial neural networks when the markers greatly outnumbered the number of training lines. Across all species and trait combinations, no one algorithm performed best, however predictions based on a combination of results from multiple algorithms (i.e., ensemble predictions) performed consistently well. While linear and non-linear algorithms performed best for a similar number of traits, the performance of non-linear algorithms vary more between traits. Although artificial neural networks did not perform best for any trait, we identified strategies (i.e., feature selection, seeded starting weights) that boosted their performance to near the level of other algorithms. Our results highlight the importance of algorithm selection for the prediction of trait values.},

  doi          = {10.1534/g3.119.400498},

  journal      = {G3},

  number       = 11,

  volume       = 9,

  place        = {United States},

  year         = {Wed Sep 18 00:00:00 EDT 2019},

  month        = {Wed Sep 18 00:00:00 EDT 2019}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1534/g3.119.400498

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 68 works

Citation information provided by
Web of Science

Figures / Tables:

Figure 1: Algorithms used and compared in past GP studies and algorithms and data included in the GP benchmark. (A) Number of times a GP algorithm was utilized (diagonal) or directly compared to other GP algorithms (lower triangle) out of 91 publications published between 2012-2018 (Table S1). GP algorithms weremore »

All figures and tables (5 total)

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Predicting Quantitative Traits With Regression Models for Dense Molecular Markers and Pedigree
journal, March 2009

de los Campos, Gustavo; Naya, Hugo; Gianola, Daniel
Genetics, Vol. 182, Issue 1
DOI: 10.1534/genetics.109.101501

Genomic-Assisted Prediction of Genetic Value With Semiparametric Procedures
journal, April 2006

Gianola, Daniel; Fernando, Rohan L.; Stella, Alessandra
Genetics, Vol. 173, Issue 3
DOI: 10.1534/genetics.105.049510

Genomic selection of agronomic traits in hybrid rice using an NCII population
journal, May 2018

Xu, Yang; Wang, Xin; Ding, Xiaowen
Rice, Vol. 11, Issue 1
DOI: 10.1186/s12284-018-0223-4

Recurrent Neural Networks for Sequential Phenotype Prediction in Genomics
conference, December 2015

Pouladi, Farhad; Salehinejad, Hojjat; Gilani, Amir Mohammad
2015 International Conference on Developments of E-Systems Engineering (DeSE)
DOI: 10.1109/DeSE.2015.52

Semi-parametric genomic-enabled prediction of genetic values using reproducing kernel Hilbert spaces methods
journal, August 2010

De Los Campos, Gustavo; Gianola, Daniel; Rosa, Guilherme J. M.
Genetics Research, Vol. 92, Issue 4
DOI: 10.1017/S0016672310000285

Extensive Genetic Diversity is Present within North American Switchgrass Germplasm
journal, January 2018

Evans, Joseph; Sanciangco, Millicent D.; Lau, Kin H.
The Plant Genome, Vol. 11, Issue 1
DOI: 10.3835/plantgenome2017.06.0055

Application of neural networks with back-propagation to genome-enabled prediction of complex traits in Holstein-Friesian and German Fleckvieh cattle
journal, March 2015

Ehret, Anita; Hochstuhl, David; Gianola, Daniel
Genetics Selection Evolution, Vol. 47, Issue 1
DOI: 10.1186/s12711-015-0097-5

Applications of Machine Learning Methods to Genomic Selection in Breeding Wheat for Rust Resistance
journal, July 2018

González-Camacho, Juan Manuel; Ornella, Leonardo; Pérez-Rodríguez, Paulino
The Plant Genome, Vol. 11, Issue 2
DOI: 10.3835/plantgenome2017.11.0104

Insights into the Maize Pan-Genome and Pan-Transcriptome
journal, January 2014

Hirsch, Candice N.; Foerster, Jillian M.; Johnson, James M.
The Plant Cell, Vol. 26, Issue 1
DOI: 10.1105/tpc.113.119982

Genome-Enabled Prediction Models for Yield Related Traits in Chickpea
journal, November 2016

Roorkiwal, Manish; Rathore, Abhishek; Das, Roma R.
Frontiers in Plant Science, Vol. 7
DOI: 10.3389/fpls.2016.01666

A deep convolutional neural network approach for predicting phenotypes from genotypes
journal, August 2018

Ma, Wenlong; Qiu, Zhixu; Song, Jie
Planta, Vol. 248, Issue 5
DOI: 10.1007/s00425-018-2976-9

Genome-wide prediction of discrete traits using bayesian regressions and machine learning
journal, February 2011

González-Recio, Oscar; Forni, Selma
Genetics Selection Evolution, Vol. 43, Issue 1
DOI: 10.1186/1297-9686-43-7

Genetic architecture of complex traits in plants
journal, April 2007

Holland, J.
Current Opinion in Plant Biology, Vol. 10, Issue 2
DOI: 10.1016/j.pbi.2007.01.003

A Ranking Approach to Genomic Selection
journal, June 2015

Blondel, Mathieu; Onogi, Akio; Iwata, Hiroyoshi
PLOS ONE, Vol. 10, Issue 6
DOI: 10.1371/journal.pone.0128570

Genome-enabled prediction of genetic values using radial basis function neural networks
journal, May 2012

González-Camacho, J. M.; de los Campos, G.; Pérez, P.
Theoretical and Applied Genetics, Vol. 125, Issue 4
DOI: 10.1007/s00122-012-1868-9

LASSO with cross-validation for genomic selection
journal, December 2009

Usai, M. Graziano; Goddard, Mike E.; Hayes, Ben J.
Genetics Research, Vol. 91, Issue 6
DOI: 10.1017/S0016672309990334

Whole-Genome Regression and Prediction Methods Applied to Plant and Animal Breeding
journal, June 2012

de los Campos, Gustavo; Hickey, John M.; Pong-Wong, Ricardo
Genetics, Vol. 193, Issue 2
DOI: 10.1534/genetics.112.143313

Early Stopping - But When?
book, January 1998

Prechelt, Lutz
Lecture Notes in Computer Science
DOI: 10.1007/3-540-49430-8_3

Regularization and variable selection via the elastic net
journal, April 2005

Zou, Hui; Hastie, Trevor
Journal of the Royal Statistical Society: Series B (Statistical Methodology), Vol. 67, Issue 2
DOI: 10.1111/j.1467-9868.2005.00503.x

Dominance and Epistasis Interactions Revealed as Important Variants for Leaf Traits of Maize NAM Population
journal, June 2018

Monir, Md. M.; Zhu, Jun
Frontiers in Plant Science, Vol. 9
DOI: 10.3389/fpls.2018.00627

Genomic Selection and Association Mapping in Rice (Oryza sativa): Effect of Trait Genetic Architecture, Training Population Composition, Marker Number and Statistical Model on Accuracy of Rice Genomic Selection in Elite, Tropical Rice Breeding Lines
journal, February 2015

Spindel, Jennifer; Begum, Hasina; Akdemir, Deniz
PLOS Genetics, Vol. 11, Issue 2
DOI: 10.1371/journal.pgen.1004982

Optimising Genomic Selection in Wheat: Effect of Marker Density, Population Size and Population Structure on Prediction Accuracy
journal, July 2018

Norman, Adam; Taylor, Julian; Edwards, James
G3: Genes|Genomes|Genetics, Vol. 8, Issue 9
DOI: 10.1534/g3.118.200311

Efficiency of multi-trait, indirect, and trait-assisted genomic selection for improvement of biomass sorghum
journal, December 2017

Fernandes, Samuel B.; Dias, Kaio O. G.; Ferreira, Daniel F.
Theoretical and Applied Genetics, Vol. 131, Issue 3
DOI: 10.1007/s00122-017-3033-y

Accuracy of breeding values of 'unrelated' individuals predicted by dense SNP genotyping
journal, June 2009

Meuwissen, Theo HE
Genetics Selection Evolution, Vol. 41, Issue 1
DOI: 10.1186/1297-9686-41-35

Genomic selection accuracies within and between environments and small breeding groups in white spruce
journal, January 2014

Beaulieu, Jean; Doerksen, Trevor K.; MacKay, John
BMC Genomics, Vol. 15, Issue 1
DOI: 10.1186/1471-2164-15-1048

Genomic Selection for Crop Improvement
journal, January 2009

Heffner, Elliot L.; Sorrells, Mark E.; Jannink, Jean-Luc
Crop Science, Vol. 49, Issue 1
DOI: 10.2135/cropsci2008.08.0512

Handling limited datasets with neural networks in medical applications: A small-data approach
journal, January 2017

Shaikhina, Torgyn; Khovanova, Natalia A.
Artificial Intelligence in Medicine, Vol. 75
DOI: 10.1016/j.artmed.2016.12.003

Diversity and population structure of northern switchgrass as revealed through exome capture sequencing
journal, November 2015

Evans, Joseph; Crisovan, Emily; Barry, Kerrie
The Plant Journal, Vol. 84, Issue 4
DOI: 10.1111/tpj.13041

SMURF: Genomic mapping of fungal secondary metabolite clusters
journal, September 2010

Khaldi, Nora; Seifuddin, Fayaz T.; Turner, Geoff
Fungal Genetics and Biology, Vol. 47, Issue 9
DOI: 10.1016/j.fgb.2010.06.003