Big Data Enhancement to the Integrated Environmental Quality Sensing System
There is a lack of cost-effective and easy-to-use tools and services for complex data streams that are commonly referred to as Big Data. Consequently, Big Data creates a scalability challenge for environmental system solutions that use traditional relational databases to perform organization, retrieval, analysis, sharing, and integration with modeling. This project demonstrated the feasibility of using NoSQL databases for efficient handling of large sets of structured and unstructured data to allow for better data management and analysis. A document-based database added flexibility in data modeling and proved to be highly scalable. A graph learning algorithm was developed to capture high-level relationships amongst sensor data and found patterns for extracting meaningful dependency. Machine learning techniques were also applied to data for detection, recognition and prediction of data patterns. This helped identify redundancy in data collection and optimization of sensor placement. Understanding typical workflows also helps in developing analytics tools that automate manual processes. To achieve this, IWT worked with domain specialists to understand how sensor data is used to analyze contaminant plumes and make assessments to support remedial decisions. Choosing proper data representation is also very important when visualizing Big Data. After evaluation of multiple visualization tools, IWT decided to build a tool on top of the kepler.gl framework, which is an open-source geospatial analysis toolbox. It is a data-agnostic, high-performance, web-based application for visual exploration of large-scale geolocation data sets. This tool was customized to plot millions of sensor data points and enabled easy visual exploration of large-scale data sets without compromising performance of the system.
- Research Organization:
- Innovative Wireless Technologies, Inc.
- Sponsoring Organization:
- USDOE Office of Science (SC)
- Contributing Organization:
- Pacific Northwest National Laboratory
- DOE Contract Number:
- SC0018596
- OSTI ID:
- 1489806
- Type / Phase:
- SBIR (Phase I)
- Report Number(s):
- DOE-IWT-18596; 81.049
- Country of Publication:
- United States
- Language:
- English
Similar Records
A Magnetoencephalographic/Encephalographic (MEG/EEG) Brain-Computer Interface Driver for Interactive iOS Mobile Videogame Applications Utilizing the Hadoop Ecosystem, MongoDB, and Cassandra NoSQL Databases
Query and Visualization of extremely large network datasets over the web using Quadtree based KML Regional Network Links