FastBit: Interactively Searching Massive Data

Conference · Tue Jun 23 00:00:00 EDT 2009

OSTI ID:964373

Wu, Kesheng; Ahern, Sean; Bethel, E Wes; Chen, Jacqueline; Childs, Hank; Cormier-Michel, Estelle; Geddes, Cameron; Gu, Junmin; Hagen, Hans; Hamann, Bernd; Koegler, Wendy; Lauret, Jerome; Meredith, Jeremy; Messmer, Peter; Otoo, Ekow; Perevoztchikov, Victor; Poskanzer, Arthur; Prabhat,; Rubel, Oliver; Shoshani, Arie ; Sim, Alexander; Stockinger, Kurt; Weber, Gunther; Zhang, Wei-Ming

As scientific instruments and computer simulations produce more and more data, the task of locating the essential information to gain insight becomes increasingly difficult. FastBit is an efficient software tool to address this challenge. In this article, we present a summary of the key underlying technologies, namely bitmap compression, encoding, and binning. Together these techniques enable FastBit to answer structured (SQL) queries orders of magnitude faster than popular database systems. To illustrate how FastBit is used in applications, we present three examples involving a high-energy physics experiment, a combustion simulation, and an accelerator simulation. In each case, FastBit significantly reduces the response time and enables interactive exploration on terabytes of data.

• Research Organization: Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
• Sponsoring Organization: Computational Research Division
• DOE Contract Number: DE-AC02-05CH11231
• OSTI ID: 964373
• Report Number(s): LBNL-2164E
• Resource Relation: Conference: SciDAC 2009, San Diego, CA, June 14-18, 2009
• Country of Publication: United States
• Language: English

Related Subjects

97
FastBit
bitmap index
data analysis
data warehousing