Title: Blazing Signature Filter: a library for fast pairwise similarity comparisons

Abstract

Identifying similarities between datasets is a fundamental task in data mining and has become an integral part of modern scientific investigation. Whether the task is to identify co-expressed genes in large-scale expression surveys or to predict combinations of gene knockouts which would elicit a similar phenotype, the underlying computational task is often a multi-dimensional similarity test. As datasets continue to grow, improvements to the efficiency, sensitivity or specificity of such computation will have broad impacts as it allows scientists to more completely explore the wealth of scientific data. A significant practical drawback of large-scale data mining is the vast majority of pairwise comparisons are unlikely to be relevant, meaning that they do not share a signature of interest. It is therefore essential to efficiently identify these unproductive comparisons as rapidly as possible and exclude them from more time-intensive similarity calculations. The Blazing Signature Filter (BSF) is a highly efficient pairwise similarity algorithm which enables extensive data mining within a reasonable amount of time. The algorithm transforms datasets into binary metrics, allowing it to utilize the computationally efficient bit operators and provide a coarse measure of similarity. As a result, the BSF can scale to high dimensionality and rapidly filter unproductivemore » pairwise comparison. Furthermore, two bioinformatics applications of the tool are presented to demonstrate the ability to scale to billions of pairwise comparisons and the usefulness of this approach.« less

Authors:

Lee, Joon -Yong ^[1]; Fujimoto, Grant M. ^[1]; Wilson, Ryan ^[1]; Wiley, H. Steven ^[1]; Payne, Samuel H.  ^[1]

• Search DOE PAGES for author "Payne, Samuel H."
• Search DOE PAGES for ORCID "0000-0002-8351-1994"

---

+ Show Author Affiliations

1. Pacific Northwest National Lab. (PNNL), Richland, WA (United States)

Publication Date:

Mon Jun 11 00:00:00 EDT 2018

Research Org.:

Pacific Northwest National Laboratory (PNNL), Richland, WA (United States)

Sponsoring Org.:

USDOE

OSTI Identifier:

1455280

Report Number(s):

PNNL-SA-126956
Journal ID: ISSN 1471-2105; 453060036

Grant/Contract Number:  

AC05-76RL01830

Resource Type:

Accepted Manuscript

Journal Name:

BMC Bioinformatics

Additional Journal Information:

Journal Volume: 19; Journal Issue: 1; Journal ID: ISSN 1471-2105

Publisher:

BioMed Central

Country of Publication:

United States

Language:

English

Subject:

59 BASIC BIOLOGICAL SCIENCES; 97 MATHEMATICS AND COMPUTING; Pairwise similarity comparison; Filtering; Large-scale data mining