Detecting outliers in streaming time series data from ARM distributed sensors

Lu, Yuping; Kumar, Jitendra; Collier, Nathaniel O.; Krishna, Bhargavi; Langston, Michael A.

Title: Detecting outliers in streaming time series data from ARM distributed sensors

Full Record
Other Related Research

Abstract

The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliersthrough a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site andmore »« less

Authors:

Lu, Yuping ^[1];

^[1];

^[1]

ORNL

Publication Date:: 2018-11-01

Research Org.:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Org.:: USDOE Office of Science (SC), Biological and Environmental Research (BER)

OSTI Identifier:: 1491320

DOE Contract Number:: AC05-00OR22725

Resource Type:: Conference

Resource Relation:: Conference: IEEE International Conference on Data Mining Workshops - Singapore, , Singapore - 11/17/2018 10:00:00 AM-11/20/2018 10:00:00 AM

Country of Publication:: United States

Language:: English

Citation Formats


                    Lu, Yuping, Kumar, Jitendra, Collier, Nathaniel O., Krishna, Bhargavi, and Langston, Michael A.. Detecting outliers in streaming time series data from ARM distributed sensors.  United States: N. p., 2018. 
        Web.

Copy to clipboard


                    Lu, Yuping, Kumar, Jitendra, Collier, Nathaniel O., Krishna, Bhargavi, & Langston, Michael A.. Detecting outliers in streaming time series data from ARM distributed sensors.  United States.

Copy to clipboard


                    Lu, Yuping, Kumar, Jitendra, Collier, Nathaniel O., Krishna, Bhargavi, and Langston, Michael A.. 2018.  
        "Detecting outliers in streaming time series data from ARM distributed sensors".  United States.  https://www.osti.gov/servlets/purl/1491320.

Copy to clipboard


                    
@article{osti_1491320,

  title        = {Detecting outliers in streaming time series data from ARM distributed sensors},

  author       = {Lu, Yuping and Kumar, Jitendra and Collier, Nathaniel O. and Krishna, Bhargavi and Langston, Michael A.},

  abstractNote = {The Atmospheric Radiation Measurement (ARM) Data Center at ORNL collects data from a number of permanent and mobile facilities around the globe. The data is then ingested to create high level scientific products. High frequency streaming measurements from sensors and radar instruments at ARM sites require high degree of accuracy to enable rigorous study of atmospheric processes. Outliers in collected data are common due to instrument failure or extreme weather events. Thus, it is critical to identify and flag them. We employed multiple univariate, multivariate and time series techniques for outlier detection methods and studied their effectiveness. First, we examined Pearson correlation coefficient which is used to measure the pairwise correlations between variables. Singular Spectrum Analysis (SSA) was applied to detect outliers by removing the anticipated annual and seasonal cycles from the signal to accentuate anomalies. K-means was applied for multivariate examination of data from collection of sensors to identify any deviation from expected and known patterns and identify abnormal observation. The Pearson correlation coefficient, SSA and K-means methods were later combined together in a framework to detect outliersthrough a range of checks. We applied the developed method to data from meteorological sensors at ARM Southern Great Plains site and validated against existing database of known data quality issues.},

  doi          = {},

  url          = {https://www.osti.gov/biblio/1491320},
  journal      = {},
number       = ,

  volume       = ,

  place        = {United States},

  year         = {2018},

  month        = {11}

}

Copy to clipboard

Conference:

View Conference (0.66 MB)

Other availability

Please see Document Availability for additional information on obtaining the full-text document. Library patrons may search WorldCat to identify libraries that hold this conference proceeding.

Save / Share:

Export Metadata

Save to My Library

Similar records in OSTI.GOV collections:

Real-time statistical quality control and ARM

Conference Blough, D

An important component of the Atmospheric Radiation Measurement (ARM) Program is real-time quality control of data obtained from meteorological instruments. It is the goal of the ARM program to enhance the predictive capabilities of global circulation models by incorporating in them more detailed information on the radiative characteristics of the earth's atmosphere. To this end, a number of Cloud and Radiation Testbeds (CART's) will be built at various locations worldwide. Each CART will consist of an array of instruments designed to collect radiative data. The large amount of data obtained from these instruments necessitates real-time processing in order to flagmore »« less
Full Text Available
Real-time statistical quality control and ARM

Conference Blough, D

An important component of the Atmospheric Radiation Measurement (ARM) Program is real-time quality control of data obtained from meteorological instruments. It is the goal of the ARM program to enhance the predictive capabilities of global circulation models by incorporating in them more detailed information on the radiative characteristics of the earth`s atmosphere. To this end, a number of Cloud and Radiation Testbeds (CART`s) will be built at various locations worldwide. Each CART will consist of an array of instruments designed to collect radiative data. The large amount of data obtained from these instruments necessitates real-time processing in order to flagmore »« less
Full Text Available
Analysis and Calibration of CRF Raman Lidar Cloud Liquid Water Measurements

Technical Report Turner, D; Whiteman, D; Russo, F

The Atmospheric Radiation Measurement (ARM) Raman lidar (RL), located at the Southern Great Plains (SGP) Climate Research Facility (CRF), is a unique state-of-the-art active remote sensor that is able to measure profiles of water vapor, aerosol, and cloud properties at high temporal and vertical resolution throughout the diurnal cycle. In October 2005, the capability of the RL was extended by the addition of a new detection channel that is sensitive to the Raman scattering of liquid water. This new channel permits the system, in theory, to measure profiles of liquid water content (LWC) by the RL. To our knowledge, themore »« less
https://doi.org/10.2172/918736

Full Text Available
An in-situ device to monitor root-soil-microbe interactions. Phase I Final Report

Technical Report Windle, Jesse

This project aimed to show that light emitting diodes (LEDs) and light detecting diodes could be used to measure root system architecture (RSA) and soil composition. Neither RSA nor soil composition are easy to measure, currently. Shovelomics, the state-of-the-art for measuring RSA, is laborious and destructive. Mid-infrared spectroscopy, the gold standard for assaying soil organic matter (SOM), is costly and not easily miniaturized. In contrast, an LED-based approach to measuring RSA or SOM would be affordable, non-destructive, and easily deployed in a field setting.The project had two parts, one to show that LED-based sensors could detect roots in soil, andmore »« less
Fraud detection in medicare claims: A multivariate outlier detection approach

Conference Burr, T; Hale, C; Kantor, M

We apply traditional and customized multivariate outlier detection methods to detect fraud in medicare claims. We use two sets of 11 derived features, and one set of the 22 combined features. The features are defined so that fraudulent medicare providers should tend to have larger features values than non-fraudulent providers. Therefore we have an apriori direction ({open_quotes}large values{close_quotes}) in high dimensional feature space to search for the multivariate outliers. We focus on three issues: (1) outlier masking (Example: the presence of one outlier can make it difficult to detect a second outlier), (2) the impact of having an apriori directionmore »« less
Full Text Available

Similar Records