skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1

Abstract

Handling large streaming data is essential for various applications such as network traffic analysis, social networks, energy cost trends, and environment modeling. However, it is in general intractable to store, compute, search, and retrieve large streaming data. This software addresses a fundamental issue, which is to reduce the size of large streaming data and still obtain accurate statistical analysis. As an example, when a high-speed network such as 100 Gbps network is monitored, the collected measurement data rapidly grows so that polynomial time algorithms (e.g., Gaussian processes) become intractable. One possible solution to reduce the storage of vast amounts of measured data is to store a random sample, such as one out of 1000 network packets. However, such static sampling methods (linear sampling) have drawbacks: (1) it is not scalable for high-rate streaming data, and (2) there is no guarantee of reflecting the underlying distribution. In this software, we implemented a dynamic sampling algorithm, based on the recent technology from the relational dynamic bayesian online locally exchangeable measures, that reduces the storage of data records in a large scale, and still provides accurate analysis of large streaming data. The software can be used for both online and offline data records.

Authors:
; ;
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE
Contributing Org.:
Lawrence Berkeley National Laboratory
OSTI Identifier:
1240674
Report Number(s):
IDEALEM; 004653IBMPC00
R&D Project: 830404000; 2016-045
DOE Contract Number:  
AC02-05CH11231
Resource Type:
Software
Software Revision:
00
Software Package Number:
004653
Software CPU:
IBMPC
Source Code Available:
Yes
Country of Publication:
United States

Citation Formats

Sim, Alex, Lee, Dongeun, and Wu, K. John. Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1. Computer software. Vers. 00. USDOE. 4 Mar. 2016. Web.
Sim, Alex, Lee, Dongeun, & Wu, K. John. (2016, March 4). Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1 (Version 00) [Computer software].
Sim, Alex, Lee, Dongeun, and Wu, K. John. Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1. Computer software. Version 00. March 4, 2016.
@misc{osti_1240674,
title = {Implementation of Dynamic Extensible Adaptive Locally Exchangeable Measures (IDEALEM) v 0.1, Version 00},
author = {Sim, Alex and Lee, Dongeun and Wu, K. John},
abstractNote = {Handling large streaming data is essential for various applications such as network traffic analysis, social networks, energy cost trends, and environment modeling. However, it is in general intractable to store, compute, search, and retrieve large streaming data. This software addresses a fundamental issue, which is to reduce the size of large streaming data and still obtain accurate statistical analysis. As an example, when a high-speed network such as 100 Gbps network is monitored, the collected measurement data rapidly grows so that polynomial time algorithms (e.g., Gaussian processes) become intractable. One possible solution to reduce the storage of vast amounts of measured data is to store a random sample, such as one out of 1000 network packets. However, such static sampling methods (linear sampling) have drawbacks: (1) it is not scalable for high-rate streaming data, and (2) there is no guarantee of reflecting the underlying distribution. In this software, we implemented a dynamic sampling algorithm, based on the recent technology from the relational dynamic bayesian online locally exchangeable measures, that reduces the storage of data records in a large scale, and still provides accurate analysis of large streaming data. The software can be used for both online and offline data records.},
doi = {},
url = {https://www.osti.gov/biblio/1240674}, year = {2016},
month = {3},
note =
}

Software:
To order this software, request consultation services, or receive further information, please fill out the following request.

Save / Share:

To receive further information, fill out the request form below. OSTI staff will begin to process an order for scientific and technical software once the signed site license agreement is received. You may also reach us by email at: .

Software Request

(required)
(required)
(required)
(required)
(required)
(required)
(required)
(required)