SpaRC: scalable sequence clustering using Apache Spark

Shi, Lizhen; Meng, Xiandong; Tseng, Elizabeth; Mascagni, Michael; Wang, Zhong; Birol, Inanc

doi:10.1093/bioinformatics/bty733

SpaRC: scalable sequence clustering using Apache Spark

Journal Article · Thu Aug 23 00:00:00 EDT 2018 · Bioinformatics

DOI:https://doi.org/10.1093/bioinformatics/bty733· OSTI ID:1542383

Shi, Lizhen ^[1]; Meng, Xiandong ^[2]; Tseng, Elizabeth ^[3]; Mascagni, Michael ^[1]; ^[4]; Birol, Inanc ^[5]

Florida State Univ., Tallahassee, FL (United States)
USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Pacific Biosciences Inc., Menlo Park, CA (United States)
USDOE Joint Genome Institute (JGI), Walnut Creek, CA (United States); Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California at Merced,CA (United States)
Canada's Michael Smith Genome Sciences Centre, Vancouver, BC (Canada); Univ. of British Columbia, Vancouver, BC (Canada); Simon Fraser Univ., Burnaby, BC (Canada)

MOTIVATION: Whole genome shotgun based next-generation transcriptomics and metagenomics studies often generate 100-1000 GB sequence data derived from tens of thousands of different genes or microbial species. Assembly of these data sets requires tradeoffs between scalability and accuracy. Current assembly methods optimized for scalability often sacrifice accuracy and vice versa. An ideal solution would both scale and produce optimal accuracy for individual genes or genomes.

RESULTS: In this work we describe an Apache Spark-based scalable sequence clustering application, SparkReadClust (SpaRC), that partitions reads based on their molecule of origin to enable downstream assembly optimization. SpaRC produces high clustering performance on transcriptomes and metagenomes from both short and long read sequencing technologies. It reaches near-linear scalability with input data size and number of compute nodes. SpaRC can run on both cloud computing and HPC environments without modification while delivering similar performance. Our findings demonstrate that SpaRC provides a scalable solution for clustering billions of reads from next-generation sequencing experiments, and Apache Spark represents a cost-effective solution with rapid development/deployment cycles for similar large-scale sequence data analysis problems.

AVAILABILITY AND IMPLEMENTATION: https://bitbucket.org/berkeleylab/jgi-sparc.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Biological and Environmental Research (BER) (SC-23)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1542383

Alternate ID(s):: OSTI ID: 1471135

Journal Information:: Bioinformatics, Journal Name: Bioinformatics Journal Issue: 5 Vol. 35; ISSN 1367-4803

Publisher:: Oxford University PressCopyright Statement

Country of Publication:: United States

Language:: English

References (29)

An Eulerian path approach to DNA fragment assembly Pevzner, P. A.; Tang, H.; Waterman, M. S. Proceedings of the National Academy of Sciences, Vol. 98, Issue 17 https://doi.org/10.1073/pnas.171285098	journal	August 2001
Pregel: a system for large-scale graph processing Malewicz, Grzegorz; Austern, Matthew H.; Bik, Aart J. C. Proceedings of the 2010 international conference on Management of data - SIGMOD '10 https://doi.org/10.1145/1807167.1807184	conference	January 2010
A case study of tuning MapReduce for efficient Bioinformatics in the cloud Shi, Lizhen; Wang, Zhong; Yu, Weikuan Parallel Computing, Vol. 61 https://doi.org/10.1016/j.parco.2016.10.002	journal	January 2017
Assembly algorithms for next-generation sequencing data Miller, Jason R.; Koren, Sergey; Sutton, Granger Genomics, Vol. 95, Issue 6 https://doi.org/10.1016/j.ygeno.2010.03.001	journal	June 2010
Detection of low-abundance bacterial strains in metagenomic datasets by eigengenome partitioning Cleary, Brian; Brito, Ilana Lauren; Huang, Katherine Nature Biotechnology, Vol. 33, Issue 10 https://doi.org/10.1038/nbt.3329	journal	September 2015
Critical Assessment of Metagenome Interpretation—a benchmark of metagenomics software Sczyrba, Alexander; Hofmann, Peter; Belmann, Peter Nature Methods, Vol. 14, Issue 11 https://doi.org/10.1038/nmeth.4458	journal	October 2017
Metagenomics: DNA sequencing of environmental samples Tringe, Susannah Green; Rubin, Edward M. Nature Reviews Genetics, Vol. 6, Issue 11 https://doi.org/10.1038/nrg1709	journal	October 2005
Next-generation transcriptome assembly Martin, Jeffrey A.; Wang, Zhong Nature Reviews Genetics, Vol. 12, Issue 10 https://doi.org/10.1038/nrg3068	journal	September 2011
Next generation sequencing data of a defined microbial mock community Singer, Esther; Andreopoulos, Bill; Bowers, Robert M. Scientific Data, Vol. 3, Issue 1 https://doi.org/10.1038/sdata.2016.81	journal	September 2016
A near complete snapshot of the Zea mays seedling transcriptome revealed from ultra-deep sequencing Martin, Jeffrey A.; Johnson, Nicole V.; Gross, Stephen M. Scientific Reports, Vol. 4, Issue 1 https://doi.org/10.1038/srep04519	journal	March 2014
Tackling soil diversity with the assembly of large, complex metagenomes Howe, Adina Chuang; Jansson, Janet K.; Malfatti, Stephanie A. Proceedings of the National Academy of Sciences, Vol. 111, Issue 13 https://doi.org/10.1073/pnas.1402564111	journal	March 2014
DIME: A Novel Framework for De Novo Metagenomic Sequence Assembly Guo, Xuan; Yu, Ning; Ding, Xiaojun Journal of Computational Biology, Vol. 22, Issue 2 https://doi.org/10.1089/cmb.2014.0251	journal	February 2015
A fast, lock-free approach for efficient parallel counting of occurrences of k-mers Marçais, Guillaume; Kingsford, Carl Bioinformatics, Vol. 27, Issue 6 https://doi.org/10.1093/bioinformatics/btr011	journal	January 2011
MetaCluster 5.0: a two-round binning approach for metagenomic data for low-abundance species in a noisy sample Wang, Y.; Leung, H. C. M.; Yiu, S. M. Bioinformatics, Vol. 28, Issue 18 https://doi.org/10.1093/bioinformatics/bts397	journal	September 2012
DSK: k-mer counting with very low memory usage Rizk, G.; Lavenier, D.; Chikhi, R. Bioinformatics, Vol. 29, Issue 5 https://doi.org/10.1093/bioinformatics/btt020	journal	January 2013
KMC 2: fast and resource-frugal k-mer counting Deorowicz, Sebastian; Kokot, Marek; Grabowski, Szymon Bioinformatics, Vol. 31, Issue 10 https://doi.org/10.1093/bioinformatics/btv022	journal	January 2015
MEGAHIT: an ultra-fast single-node solution for large and complex metagenomics assembly via succinct de Bruijn graph Li, Dinghua; Liu, Chi-Man; Luo, Ruibang Bioinformatics, Vol. 31, Issue 10 https://doi.org/10.1093/bioinformatics/btv033	journal	January 2015
Minimap and miniasm: fast mapping and de novo assembly for noisy long sequences Li, Heng Bioinformatics, Vol. 32, Issue 14 https://doi.org/10.1093/bioinformatics/btw152	journal	March 2016
Biospark: scalable analysis of large numerical datasets from biological simulations and experiments using Hadoop and Spark Klein, Max; Sharma, Rati; Bohrer, Chris H. Bioinformatics, Vol. 33, Issue 2 https://doi.org/10.1093/bioinformatics/btw614	journal	September 2016
Accurate and comprehensive sequencing of personal genomes Ajay, S. S.; Parker, S. C. J.; Ozel Abaan, H. Genome Research, Vol. 21, Issue 9 https://doi.org/10.1101/gr.123638.111	journal	July 2011
Methane yield phenotypes linked to differential gene expression in the sheep rumen microbiome Shi, Weibing; Moon, Christina D.; Leahy, Sinead C. Genome Research, Vol. 24, Issue 9 https://doi.org/10.1101/gr.168245.113	journal	June 2014
metaSPAdes: a new versatile metagenomic assembler Nurk, Sergey; Meleshko, Dmitry; Korobeynikov, Anton Genome Research, Vol. 27, Issue 5 https://doi.org/10.1101/gr.213959.116	journal	March 2017
Near linear time algorithm to detect community structures in large-scale networks Raghavan, Usha Nandini; Albert, Réka; Kumara, Soundar Physical Review E, Vol. 76, Issue 3 https://doi.org/10.1103/PhysRevE.76.036106	journal	September 2007
Metagenomic Discovery of Biomass-Degrading Genes and Genomes from Cow Rumen Hess, M.; Sczyrba, A.; Egan, R. Science, Vol. 331, Issue 6016 https://doi.org/10.1126/science.1200387	journal	January 2011
Structure and function of the global ocean microbiome Sunagawa, S.; Coelho, L. P.; Chaffron, S. Science, Vol. 348, Issue 6237 https://doi.org/10.1126/science.1261359	journal	May 2015
Counting the Uncountable: Statistical Approaches to Estimating Microbial Diversity Hughes, J. B.; Hellmann, J. J.; Ricketts, T. H. Applied and Environmental Microbiology, Vol. 67, Issue 10 https://doi.org/10.1128/AEM.67.10.4399-4406.2001	journal	October 2001
A framework for space-efficient read clustering in metagenomic samples Alanko, Jarno; Cunial, Fabio; Belazzougui, Djamal BMC Bioinformatics, Vol. 18, Issue S3 https://doi.org/10.1186/s12859-017-1466-6	journal	March 2017
SparkBLAST: scalable BLAST processing using in-memory operations de Castro, Marcelo Rodrigo; Tostes, Catherine dos Santos; Dávila, Alberto M. R. BMC Bioinformatics, Vol. 18, Issue 1 https://doi.org/10.1186/s12859-017-1723-8	journal	June 2017
Widespread Polycistronic Transcripts in Fungi Revealed by Single-Molecule mRNA Sequencing Gordon, Sean P.; Tseng, Elizabeth; Salamov, Asaf PLOS ONE, Vol. 10, Issue 7 https://doi.org/10.1371/journal.pone.0132628	journal	July 2015

Cited By (1)

Computational Strategies for Scalable Genomics Analysis Shi, Lizhen; Wang, Zhong Genes, Vol. 10, Issue 12 https://doi.org/10.3390/genes10121017	journal	December 2019

Similar Records

MetaBAT, an efficient tool for accurately reconstructing single genomes from complex microbial communities

Journal Article · Wed Dec 31 23:00:00 EST 2014 · PeerJ · OSTI ID:1257449

De novo Nanopore read quality improvement using deep learning

Journal Article · Tue Nov 05 23:00:00 EST 2019 · BMC Bioinformatics · OSTI ID:1581387

DECA: scalable XHMM exome copy-number variant calling with ADAM and Apache Spark

Journal Article · Fri Oct 11 00:00:00 EDT 2019 · BMC Bioinformatics · OSTI ID:1618535

Related Subjects

59 BASIC BIOLOGICAL SCIENCES

SpaRC: scalable sequence clustering using Apache Spark

Citation Formats

References (29)

Cited By (1)

Similar Records

Related Subjects