Parallel membership queries on very large scientific data sets using bitmap indexes

Yildiz, Beytullah; Wu, Kesheng; Byna, Suren; Shoshani, Arie

doi:10.1002/cpe.5157

Parallel membership queries on very large scientific data sets using bitmap indexes

Journal Article · Sat Aug 10 04:00:00 EDT 2019 · Concurrency and Computation. Practice and Experience

DOI:https://doi.org/10.1002/cpe.5157· OSTI ID:1503658

Yildiz, Beytullah; Wu, Kesheng; Byna, Suren; Shoshani, Arie

© 2019 John Wiley & Sons, Ltd. Many scientific applications produce very large amounts of data as advances in hardware fuel computing and experimental facilities. Managing and analyzing massive quantities of scientific data is challenging as data are often stored in specific formatted files, such as HDF5 and NetCDF, which do not offer appropriate search capabilities. In this research, we investigated a special class of search capability, called membership query, to identify whether queried elements of a set are members of an attribute. Attributes that naturally have classification values appear frequently in scientific domains such as category and object type as well as in daily life such as zip code and occupation. Because classification attribute values are discrete and require random data access, performing a membership query on a large scientific data set creates challenges. We applied bitmap indexing and parallelization to membership queries to overcome these challenges. Bitmap indexing provides high performance not only for low cardinality attributes but also for high cardinality attributes, such as floating-point variables, electric charge, or momentum in a particle physics data set, due to compression algorithms such as Word-Aligned Hybrid. We conducted experiments, in a highly parallelized environment, on data obtained from a particle accelerator model and a synthetic data set.

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC), Advanced Scientific Computing Research (ASCR) (SC-21)

DOE Contract Number:: AC02-05CH11231

OSTI ID:: 1503658

Journal Information:: Concurrency and Computation. Practice and Experience, Journal Name: Concurrency and Computation. Practice and Experience Journal Issue: 15 Vol. 31; ISSN 1532-0626

Country of Publication:: United States

Language:: English

References (33)

Database technology for decision support systems Chaudhuri, S.; Dayal, U.; Ganti, V. Computer, Vol. 34, Issue 12 https://doi.org/10.1109/2.970575	journal	January 2001
Bloofi: Multidimensional Bloom filters Crainiceanu, Adina; Lemire, Daniel Information Systems, Vol. 54 https://doi.org/10.1016/j.is.2015.01.002	journal	December 2015
An efficient bitmap encoding scheme for selection queries Chan, Chee-Yong; Ioannidis, Yannis E. Proceedings of the 1999 ACM SIGMOD international conference on Management of data - SIGMOD '99 https://doi.org/10.1145/304182.304201	conference	January 1999
SQL server column store indexes Larson, Per-Åke; Clinciu, Cipri; Hanson, Eric N. Proceedings of the 2011 international conference on Management of data - SIGMOD '11 https://doi.org/10.1145/1989323.1989448	conference	January 2011
Toward a modular and efficient distribution for Web service handlers: TOWARD A MODULAR AND EFFICIENT DISTRIBUTION FOR WEB SERVICE HANDLERS Yildiz, Beytullah; Fox, Geoffrey C. Concurrency and Computation: Practice and Experience, Vol. 25, Issue 3 https://doi.org/10.1002/cpe.2854	journal	June 2012
Parallel I/O, analysis, and visualization of a trillion particle simulation Byna, Surendra; Chou, Jerry; Rubel, Oliver 2012 SC - International Conference for High Performance Computing, Networking, Storage and Analysis, 2012 International Conference for High Performance Computing, Networking, Storage and Analysis https://doi.org/10.1109/SC.2012.92	conference	November 2012
Analyses of multi-level and multi-component compressed bitmap indexes Wu, Kesheng; Shoshani, Arie; Stockinger, Kurt ACM Transactions on Database Systems, Vol. 35, Issue 1 https://doi.org/10.1145/1670243.1670245	journal	February 2010
Multi-resolution bitmap indexes for scientific data Sinha, Rishi Rakesh; Winslett, Marianne ACM Transactions on Database Systems, Vol. 32, Issue 3 https://doi.org/10.1145/1272743.1272746	journal	August 2007
Optimizing bitmap indices with efficient compression Wu, Kesheng; Otoo, Ekow J.; Shoshani, Arie ACM Transactions on Database Systems, Vol. 31, Issue 1 https://doi.org/10.1145/1132863.1132864	journal	March 2006
A survey of bitmap index compression algorithms for Big Data No authors listed Tsinghua Science and Technology, Vol. 20, Issue 1 https://doi.org/10.1109/TST.2015.7040519	journal	February 2015
Position list word aligned hybrid: optimizing space and performance for compressed bitmaps Deliège, François; Pedersen, Torben Bach Proceedings of the 13th International Conference on Extending Database Technology - EDBT '10 https://doi.org/10.1145/1739041.1739071	conference	January 2010
A memory efficient reachability data structure through bit vector compression van Schaik, Sebastiaan J.; de Moor, Oege Proceedings of the 2011 international conference on Management of data - SIGMOD '11 https://doi.org/10.1145/1989323.1989419	conference	January 2011
Space efficient bitmap indexing Koudas, Nick Proceedings of the ninth international conference on Information and knowledge management - CIKM '00 https://doi.org/10.1145/354756.354819	conference	January 2000
An Orchestration for Distributed Web Service Handlers Yildiz, Beytullah; Fox, Geoffrey; Pallickara, Shrideep 2008 3rd International Conference on internet and Web Applications and Services (ICIW), 2008 Third International Conference on Internet and Web Applications and Services https://doi.org/10.1109/ICIW.2008.55	conference	June 2008
Multi-level bitmap indexes for flash memory storage Wu, Kesheng; Madduri, Kamesh; Canon, Shane Proceedings of the Fourteenth International Database Engineering & Applications Symposium on - IDEAS '10 https://doi.org/10.1145/1866480.1866497	conference	January 2010
Strategies for processing ad hoc queries on large data warehouses Stockinger, Kurt; Wu, Kesheng; Shoshani, Arie Proceedings of the 5th ACM international workshop on Data Warehousing and OLAP - DOLAP '02 https://doi.org/10.1145/583890.583901	conference	January 2002
Queries and concept learning Angluin, Dana Machine Learning, Vol. 2, Issue 4 https://doi.org/10.1007/BF00116828	journal	April 1988
Parallel index and query for large scale data analysis Chou, Jerry; Ryne, Rob D.; Howison, Mark Proceedings of 2011 International Conference for High Performance Computing, Networking, Storage and Analysis on - SC '11 https://doi.org/10.1145/2063384.2063424	conference	January 2011
Optimizing candidate check costs for bitmap indices Rotem, Doron; Stockinger, Kurt; Wu, Kesheng Proceedings of the 14th ACM international conference on Information and knowledge management - CIKM '05 https://doi.org/10.1145/1099554.1099718	conference	January 2005
Sorting improves word-aligned bitmap indexes Lemire, Daniel; Kaser, Owen; Aouiche, Kamel Data & Knowledge Engineering, Vol. 69, Issue 1 https://doi.org/10.1016/j.datak.2009.08.006	journal	January 2010
Locality-Sensitive Bloom Filter for Approximate Membership Query Hua, Yu; Xiao, Bin; Veeravalli, Bharadwaj IEEE Transactions on Computers, Vol. 61, Issue 6 https://doi.org/10.1109/TC.2011.108	journal	June 2012
The Square Kilometre Array Dewdney, P. E.; Hall, P. J.; Schilizzi, R. T. Proceedings of the IEEE, Vol. 97, Issue 8 https://doi.org/10.1109/JPROC.2009.2021005	journal	August 2009
Bitmap index design and evaluation Chan, Chee-Yong; Ioannidis, Yannis E. Proceedings of the 1998 ACM SIGMOD international conference on Management of data - SIGMOD '98 https://doi.org/10.1145/276304.276336	conference	January 1998
On the Performance of Bitmap Indices for High Cardinality Attributes Wu, Kesheng; Otoo, Ekow; Shoshani, Arie Proceedings 2004 VLDB Conference https://doi.org/10.1016/B978-012088469-8.50006-1	book	January 2004
An Efficient Multi-Component Indexing Embedded Bitmap Compression for Data Reorganization Sharma, Yashvardhan; Goyal, Navneet Information Technology Journal, Vol. 7, Issue 1 https://doi.org/10.3923/itj.2008.160.164	journal	January 2008
High resolution simulation of beam dynamics in electron linacs for x-ray free electron lasers Qiang, J.; Ryne, R. D.; Venturini, M. Physical Review Special Topics - Accelerators and Beams, Vol. 12, Issue 10 https://doi.org/10.1103/PhysRevSTAB.12.100702	journal	October 2009
Parallel in situ indexing for data-intensive computing Kim, Jinoh; Abbasi, Hasan; Chacon, Luis 2011 IEEE Symposium on Large Data Analysis and Visualization (LDAV) https://doi.org/10.1109/LDAV.2011.6092319	conference	October 2011
Improved Bitmap Indexing Strategy for Data Warehouses Goyal, Navneet; Zaveri, Susheel Kumar; Sharma, Yashvardhan 9th International Conference on Information Technology (ICIT'06) https://doi.org/10.1109/ICIT.2006.59	conference	December 2006
Indexing and Parallel Query Processing Support for Visualizing Climate Datasets Su, Yu; Agrawal, Gagan; Woodring, Jonathan 2012 41st International Conference on Parallel Processing (ICPP) https://doi.org/10.1109/ICPP.2012.33	conference	September 2012
Scatter Bitmap: Space-Time Efficient Bitmap Indexing for Equality and Membership Queries Vanichayobon, Sirirut; Manfuekphan, Jarin; Gruenwald, Le 2006 IEEE Conference on Cybernetics and Intelligent Systems https://doi.org/10.1109/ICCIS.2006.252354	conference	November 2006
FastBit: interactively searching massive data Wu, K.; Ahern, S.; Bethel, E. W. Journal of Physics: Conference Series, Vol. 180 https://doi.org/10.1088/1742-6596/180/1/012053	journal	July 2009
Optimizing fastquery performance on lustre file system Lin, Kuan-Wu; Byna, Surendra; Chou, Jerry Proceedings of the 25th International Conference on Scientific and Statistical Database Management - SSDBM https://doi.org/10.1145/2484838.2484853	conference	January 2013
An overview of data warehousing and OLAP technology Chaudhuri, Surajit; Dayal, Umeshwar ACM SIGMOD Record, Vol. 26, Issue 1 https://doi.org/10.1145/248603.248616	journal	March 1997

Similar Records

Evaluation Strategies for Bitmap Indices with Binning

Conference · Thu Jun 03 00:00:00 EDT 2004 · OSTI ID:861196

HDF5-FastQuery: Accelerating Complex Queries on HDF Datasets usingFast Bitmap Indices

Conference · Wed Mar 29 23:00:00 EST 2006 · OSTI ID:881620

Efficient binning for bitmap indices on high-cardinality attributes

Technical Report · Tue Nov 16 23:00:00 EST 2004 · OSTI ID:841113

Parallel membership queries on very large scientific data sets using bitmap indexes

Citation Formats

References (33)

Similar Records

Related Subjects