skip to main content
OSTI.GOV title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Big Data Enhancement to the Integrated Environmental Quality Sensing System

Abstract

There is a lack of cost-effective and easy-to-use tools and services for complex data streams that are commonly referred to as Big Data. Consequently, Big Data creates a scalability challenge for environmental system solutions that use traditional relational databases to perform organization, retrieval, analysis, sharing, and integration with modeling. This project demonstrated the feasibility of using NoSQL databases for efficient handling of large sets of structured and unstructured data to allow for better data management and analysis. A document-based database added flexibility in data modeling and proved to be highly scalable. A graph learning algorithm was developed to capture high-level relationships amongst sensor data and found patterns for extracting meaningful dependency. Machine learning techniques were also applied to data for detection, recognition and prediction of data patterns. This helped identify redundancy in data collection and optimization of sensor placement. Understanding typical workflows also helps in developing analytics tools that automate manual processes. To achieve this, IWT worked with domain specialists to understand how sensor data is used to analyze contaminant plumes and make assessments to support remedial decisions. Choosing proper data representation is also very important when visualizing Big Data. After evaluation of multiple visualization tools, IWT decided to buildmore » a tool on top of the kepler.gl framework, which is an open-source geospatial analysis toolbox. It is a data-agnostic, high-performance, web-based application for visual exploration of large-scale geolocation data sets. This tool was customized to plot millions of sensor data points and enabled easy visual exploration of large-scale data sets without compromising performance of the system.« less

Authors:
; ; ; ;
Publication Date:
Research Org.:
Innovative Wireless Technologies, Inc.
Sponsoring Org.:
USDOE Office of Science (SC)
Contributing Org.:
Pacific Northwest National Laboratory
OSTI Identifier:
1489806
Report Number(s):
DOE-IWT-18596
81.049
DOE Contract Number:  
SC0018596
Type / Phase:
SBIR (Phase I)
Resource Type:
Technical Report
Country of Publication:
United States
Language:
English
Subject:
54 ENVIRONMENTAL SCIENCES; Integrated Environmental Quality Sensor; Environmental Sensors; BigData; Big Data; Analytics; Data Analytics; Sensor Analytics

Citation Formats

Prabhu, Swathi, Dechant, Tim, Koleszar, Luke, Stephens, Cara, and Colling, Jeremiah. Big Data Enhancement to the Integrated Environmental Quality Sensing System. United States: N. p., 2019. Web.
Prabhu, Swathi, Dechant, Tim, Koleszar, Luke, Stephens, Cara, & Colling, Jeremiah. Big Data Enhancement to the Integrated Environmental Quality Sensing System. United States.
Prabhu, Swathi, Dechant, Tim, Koleszar, Luke, Stephens, Cara, and Colling, Jeremiah. Tue . "Big Data Enhancement to the Integrated Environmental Quality Sensing System". United States.
@article{osti_1489806,
title = {Big Data Enhancement to the Integrated Environmental Quality Sensing System},
author = {Prabhu, Swathi and Dechant, Tim and Koleszar, Luke and Stephens, Cara and Colling, Jeremiah},
abstractNote = {There is a lack of cost-effective and easy-to-use tools and services for complex data streams that are commonly referred to as Big Data. Consequently, Big Data creates a scalability challenge for environmental system solutions that use traditional relational databases to perform organization, retrieval, analysis, sharing, and integration with modeling. This project demonstrated the feasibility of using NoSQL databases for efficient handling of large sets of structured and unstructured data to allow for better data management and analysis. A document-based database added flexibility in data modeling and proved to be highly scalable. A graph learning algorithm was developed to capture high-level relationships amongst sensor data and found patterns for extracting meaningful dependency. Machine learning techniques were also applied to data for detection, recognition and prediction of data patterns. This helped identify redundancy in data collection and optimization of sensor placement. Understanding typical workflows also helps in developing analytics tools that automate manual processes. To achieve this, IWT worked with domain specialists to understand how sensor data is used to analyze contaminant plumes and make assessments to support remedial decisions. Choosing proper data representation is also very important when visualizing Big Data. After evaluation of multiple visualization tools, IWT decided to build a tool on top of the kepler.gl framework, which is an open-source geospatial analysis toolbox. It is a data-agnostic, high-performance, web-based application for visual exploration of large-scale geolocation data sets. This tool was customized to plot millions of sensor data points and enabled easy visual exploration of large-scale data sets without compromising performance of the system.},
doi = {},
journal = {},
number = ,
volume = ,
place = {United States},
year = {2019},
month = {1}
}

Technical Report:
This technical report may be released as soon as January 8, 2023
Other availability
Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that may hold this item. Keep in mind that many technical reports are not cataloged in WorldCat.

Save / Share: