Efficient Analysis of Live and Historical Streaming Data and itsApplication to Cybersecurity
Applications that query data streams in order to identifytrends, patterns, or anomalies can often benefit from comparing the livestream data with archived historical stream data. However, searching thishistorical data in real time has been considered so far to beprohibitively expensive. One of the main bottlenecks is the update costsof the indices over the archived data. In this paper, we address thisproblem by using our highly-efficient bitmap indexing technology (calledFastBit) and demonstrate that the index update operations aresufficiently efficient for this bottleneck to be removed. We describe ourprototype system based on the TelegraphCQ streaming query processor andthe FastBit bitmap index. We present a detailed performance evaluation ofour system using a complex query workload for analyzing real networktraffic data. The combined system uses TelegraphCQ to analyze streams oftraffic information and FastBit to correlate current behaviors withhistorical trends. We demonstrate that our system can simultaneouslyanalyze (1) live streams with high data rates and (2) a large repositoryof historical stream data.
- Research Organization:
- Ernest Orlando Lawrence Berkeley NationalLaboratory, Berkeley, CA (US)
- Sponsoring Organization:
- USDOE Director. Office of Science. Advanced ScientificComputing Research
- DOE Contract Number:
- AC02-05CH11231
- OSTI ID:
- 920351
- Report Number(s):
- LBNL--61080; BnR: KJ0101030
- Country of Publication:
- United States
- Language:
- English
Similar Records
Using Bitmap Indexing Technology for Combined Numerical and TextQueries
FastBit Reference Manual