BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis

Bhatia, Karan; Wang, Zhong

Title: BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis

Full Record
Other Related Research

Abstract

Next Generation sequencing is producing ever larger data sizes with a growth rate outpacing Moore's Law. The data deluge has made many of the current sequenceanalysis tools obsolete because they do not scale with data. Here we present BioPig, a collection of cloud computing tools to scale data analysis and management. Pig is aflexible data scripting language that uses Apache's Hadoop data structure and map reduce framework to process very large data files in parallel and combine the results.BioPig extends Pig with capability with sequence analysis. We will show the performance of BioPig on a variety of bioinformatics tasks, including screeningsequence contaminants, Illumina QA/QC, and gene discovery from metagenome data sets using the Rumen metagenome as an example.

Authors:: Bhatia, Karan; Wang, Zhong

Publication Date:: Tue Mar 22 00:00:00 EDT 2011

Research Org.:: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)

Sponsoring Org.:: Genomics Division

OSTI Identifier:: 1050659

Report Number(s):: LBNL-4784E-Poster
TRN: US201218%%871

DOE Contract Number:: DE-AC02-05CH11231

Resource Type:: Conference

Resource Relation:: Conference: 6th Annual DOE JGI User Meeting, Walnut Creek, CA, 3/22-24

Country of Publication:: United States

Language:: English

Subject:: 59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICAL METHODS AND COMPUTING; 99 GENERAL AND MISCELLANEOUS//MATHEMATICS, COMPUTING, AND INFORMATION SCIENCE; CLOUDS; DATA ANALYSIS; GENES; MANAGEMENT; PERFORMANCE; RUMINANTS; STOMACH; STRUCTURAL CHEMICAL ANALYSIS; BioPig, cloud computing, kmers, ngrams, sequence analysis

Citation Formats


                    Bhatia, Karan, and Wang, Zhong. BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis.  United States: N. p., 2011. 
        Web.

Copy to clipboard


                    Bhatia, Karan, & Wang, Zhong. BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis.  United States.

Copy to clipboard


                    Bhatia, Karan, and Wang, Zhong. 2011.  
        "BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis".  United States.  https://www.osti.gov/servlets/purl/1050659.

Copy to clipboard


                    
@article{osti_1050659,

  title        = {BioPig: Developing Cloud Computing Applications for Next-Generation Sequence Analysis},

  author       = {Bhatia, Karan and Wang, Zhong},

  abstractNote = {Next Generation sequencing is producing ever larger data sizes with a growth rate outpacing Moore's Law. The data deluge has made many of the current sequenceanalysis tools obsolete because they do not scale with data. Here we present BioPig, a collection of cloud computing tools to scale data analysis and management. Pig is aflexible data scripting language that uses Apache's Hadoop data structure and map reduce framework to process very large data files in parallel and combine the results.BioPig extends Pig with capability with sequence analysis. We will show the performance of BioPig on a variety of bioinformatics tasks, including screeningsequence contaminants, Illumina QA/QC, and gene discovery from metagenome data sets using the Rumen metagenome as an example.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1050659},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {Tue Mar 22 00:00:00 EDT 2011},

  month        = {Tue Mar 22 00:00:00 EDT 2011}

}

Copy to clipboard

Conference:

View Conference (3.20 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Next generation sequencing data of a defined microbial mock community

Journal Article Singer, Esther; Andreopoulos, Bill; Bowers, Robert; ... - Scientific Data

Generating sequence data of a defined community composed of organisms with complete reference genomes is indispensable for the benchmarking of new genome sequence analysis methods, including assembly and binning tools. Moreover the validation of new sequencing library protocols and platforms to assess critical components such as sequencing errors and biases relies on such datasets. We here report the next generation metagenomic sequence data of a defined mock community (Mock Bacteria ARchaea Community; MBARC-26), composed of 23 bacterial and 3 archaeal strains with finished genomes. These strains span 10 phyla and 14 classes, a range of GC contents, genome sizes, repeatmore »« less
Cited by 61
https://doi.org/10.1038/sdata.2016.81

Full Text Available
A case study of tuning MapReduce for efficient Bioinformatics in the cloud

Journal Article Shi, Lizhen; Wang, Zhong; Yu, Weikuan; ... - Parallel Computing

The combination of the Hadoop MapReduce programming model and cloud computing allows biological scientists to analyze next-generation sequencing (NGS) data in a timely and cost-effective manner. Cloud computing platforms remove the burden of IT facility procurement and management from end users and provide ease of access to Hadoop clusters. However, biological scientists are still expected to choose appropriate Hadoop parameters for running their jobs. More importantly, the available Hadoop tuning guidelines are either obsolete or too general to capture the particular characteristics of bioinformatics applications. In this paper, we aim to minimize the cloud computing cost spent on bioinformatics datamore »« less
Cited by 9
https://doi.org/10.1016/j.parco.2016.10.002

Full Text Available
Deactivation and decommissioning web log analysis using big data technology - 15710

Conference Joshi, Santosh; Upadhyay, Himanshu; Lagos, Leonel

The D and D Knowledge Management Information Tool (D and D KM- IT) is a web-based knowledge management information tool built for the D and D user community. Big data is a massive volume of structured and unstructured datasets which is so large that it's difficult to process using traditional database techniques. Web logs are the repository of files which are generated automatically for any operation on the web site. Web log files generated from the D and D KM-IT will be processed using the Apache Hadoop Framework to extract meaningful data. Hadoop is an open-source software framework for storingmore »« less
A Secure Web Application Providing Public Access to High-Performance Data Intensive Scientific Resources - ScalaBLAST Web Application

Conference Curtis, Darren; Peterson, Elena; Oehmen, Chris

This work presents the ScalaBLAST Web Application (SWA), a web based application implemented using the PHP script language, MySQL DBMS, and Apache web server under a GNU/Linux platform. SWA is an application built as part of the Data Intensive Computer for Complex Biological Systems (DICCBS) project at the Pacific Northwest National Laboratory (PNNL). SWA delivers accelerated throughput of bioinformatics analysis via high-performance computing through a convenient, easy-to-use web interface. This approach greatly enhances emerging fields of study in biology such as ontology-based homology, and multiple whole genome comparisons which, in the absence of a tool like SWA, require a heroicmore »« less
An overview of the Hadoop/MapReduce/HBase framework and its current applications in bioinformatics

Journal Article Taylor, Ronald - BMC Bioinformatics, 11(Suppl 12):S1

Bioinformatics researchers are increasingly confronted with analysis of ultra large-scale data sets, a problem that will only increase at an alarming rate in coming years. Recent developments in open source software, that is, the Hadoop project and associated software, provide a foundation for scaling to petabyte scale data warehouses on Linux clusters, providing fault-tolerant parallelized analysis on such data using a programming style named MapReduce. An overview is given of the current usage within the bioinformatics community of Hadoop, a top-level Apache Software Foundation project, and of associated open source software projects. The concepts behind Hadoop and the associated HBasemore »« less
https://doi.org/10.1186/1471-2105-11-S12-S1

Similar Records