Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Lin, Han; Su, Zhichao; Meng, Xiandong; Jin, Xu; Wang, Zhong; Han, Wenting; An, Hong; Chi, Mengxian; Wu, Zheng

doi:10.1007/s10766-017-0524-z

Title: Combining Hadoop with MPI to Solve Metagenomics Problems that are both Data- and Compute-intensive

Journal Article · 07 October 2017 · International Journal of Parallel Programming

DOI: https://doi.org/10.1007/s10766-017-0524-z · OSTI ID:1532334

^[1]; Su, Zhichao ^[1]; Meng, Xiandong ^[2]; Jin, Xu ^[1]; Wang, Zhong ^[2]; Han, Wenting ^[1]; An, Hong ^[1]; Chi, Mengxian ^[1]; Wu, Zheng ^[1]

Univ. of Science and Technology of China, Hefei (China)
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Metagenomics, the study of all microbial species cohabitants in an environment, usually produces large amount of sequence data varying from several GBs to a few TBs. Analyzing metagenomics data includes both data-intensive and compute-intensive steps, making the entire process hard to scale. Here we aim to optimize a metagenomics application that partitions the shortgun metagenomics sequences based on their species of origin. Our solution combines MapReduce-based BioPig analytic toolkit with MPI to provide scalability in respective to both data and compute. We also made some improvements to the existing BioPig toolkit by using simplified data types and compressed k-mer storage. These optimizations leads up to 193× speedup for the computing-intensive step and 9.6× speedup over the entire pipeline. Our optimized application is also capable of processing datasets that are 16 times larger on the same hardware platform. These conclusions indicate integrating heterogeneous technologies such as Hadoop and MPI is quite efficient to solve large genomics problems that are both data-intensive and compute-intensive.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC); National Key Research and Development Program of China

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1532334

Journal Information:: International Journal of Parallel Programming, Vol. 46, Issue 4; ISSN 0885-7458

Publisher:: SpringerCopyright Statement

Country of Publication:: United States

Language:: English

References (20)

MapReduce: simplified data processing on large clusters Dean, Jeffrey; Ghemawat, Sanjay; Mehta, Brijesh Communications of the ACM, Vol. 51, Issue 1 https://doi.org/10.1145/1327452.1327492	journal	January 2008
A Map-Reduce Framework for Clustering Metagenomes Rasheed, Zeehasham; Rangwala, Huzefa 2013 IEEE International Symposium on Parallel & Distributed Processing, Workshops and Phd Forum (IPDPSW) https://doi.org/10.1109/IPDPSW.2013.100	conference	May 2013
DIME: A Novel Framework for De Novo Metagenomic Sequence Assembly Guo, Xuan; Yu, Ning; Ding, Xiaojun Journal of Computational Biology, Vol. 22, Issue 2 https://doi.org/10.1089/cmb.2014.0251	journal	February 2015
Apache hadoop performance-tuning methodologies and best practices Joshi, Shrinivas B. Proceedings of the third joint WOSP/SIPEW international conference on Performance Engineering - ICPE '12 https://doi.org/10.1145/2188286.2188323	conference	January 2012
HPC-ABDS High Performance Computing Enhanced Apache Big Data Stack Fox, Geoffrey C.; Qiu, Judy; Kamburugamuve, Supun 2015 15th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (CCGrid) https://doi.org/10.1109/CCGrid.2015.122	conference	May 2015
DataMPI: Extending MPI to Hadoop-Like Big Data Computing Lu, Xiaoyi; Liang, Fan; Wang, Bing 2014 IEEE International Parallel & Distributed Processing Symposium (IPDPS), 2014 IEEE 28th International Parallel and Distributed Processing Symposium https://doi.org/10.1109/IPDPS.2014.90	conference	May 2014
Sequencing technologies — the next generation Metzker, Michael L. Nature Reviews Genetics, Vol. 11, Issue 1 https://doi.org/10.1038/nrg2626	journal	December 2009
Big Data Analytics in the Cloud: Spark on Hadoop vs MPI/OpenMP on Beowulf Reyes-Ortiz, Jorge L.; Oneto, Luca; Anguita, Davide Procedia Computer Science, Vol. 53 https://doi.org/10.1016/j.procs.2015.07.286	journal	January 2015
Connected Components in MapReduce and Beyond Kiveris, Raimondas; Lattanzi, Silvio; Mirrokni, Vahab Proceedings of the ACM Symposium on Cloud Computing - SOCC '14 https://doi.org/10.1145/2670979.2670997	conference	January 2014
Performance evaluation and tuning of BioPig for genomic analysis Shi, Lizhen; Wang, Zhong; Yu, Weikuan Proceedings of the 2015 International Workshop on Data-Intensive Scalable Computing Systems - DISCS '15 https://doi.org/10.1145/2831244.2831252	conference	January 2015
Efficiency of a Good But Not Linear Set Union Algorithm Tarjan, Robert Endre Journal of the ACM, Vol. 22, Issue 2 https://doi.org/10.1145/321879.321884	journal	April 1975
MRONLINE: MapReduce online performance tuning Li, Min; Zeng, Liangzhao; Meng, Shicong Proceedings of the 23rd international symposium on High-performance parallel and distributed computing - HPDC '14 https://doi.org/10.1145/2600212.2600229	conference	January 2014
Pig latin: a not-so-foreign language for data processing Olston, Christopher; Reed, Benjamin; Srivastava, Utkarsh Proceedings of the 2008 ACM SIGMOD international conference on Management of data - SIGMOD '08 https://doi.org/10.1145/1376616.1376726	conference	January 2008
OpenMP: an industry standard API for shared-memory programming Dagum, L.; Menon, R. IEEE Computational Science and Engineering, Vol. 5, Issue 1 https://doi.org/10.1109/99.660313	journal	January 1998
Metagenomic Discovery of Biomass-Degrading Genes and Genomes from Cow Rumen Hess, M.; Sczyrba, A.; Egan, R. Science, Vol. 331, Issue 6016 https://doi.org/10.1126/science.1200387	journal	January 2011
A high-performance, portable implementation of the MPI message passing interface standard Gropp, William; Lusk, Ewing; Doss, Nathan Parallel Computing, Vol. 22, Issue 6 https://doi.org/10.1016/0167-8191(96)00024-5	journal	September 1996
Bridging the gap between HPC and big data frameworks Anderson, Michael; Smith, Shaden; Sundaram, Narayanan Proceedings of the VLDB Endowment, Vol. 10, Issue 8 https://doi.org/10.14778/3090163.3090168	journal	April 2017
BioPig: a Hadoop-based analytic toolkit for large-scale sequence data Nordberg, H.; Bhatia, K.; Wang, K. Bioinformatics, Vol. 29, Issue 23 https://doi.org/10.1093/bioinformatics/btt528	journal	September 2013
Next-generation sequencing: big data meets high performance computing Schmidt, Bertil; Hildebrandt, Andreas Drug Discovery Today, Vol. 22, Issue 4 https://doi.org/10.1016/j.drudis.2017.01.014	journal	April 2017
Apache Hadoop YARN: yet another resource negotiator Vavilapalli, Vinod Kumar; Seth, Siddharth; Saha, Bikas Proceedings of the 4th annual Symposium on Cloud Computing - SOCC '13 https://doi.org/10.1145/2523616.2523633	conference	January 2013