Search Menu
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information
Search Scholarly Publications

Title: Imputation for multisource data with comparison and assessment techniques

Abstract

Missing data are prevalent issue in analyses involving data collection. The problem of missing data is exacerbated for multisource analysis, where data from multiple sensors are combined to arrive at a single conclusion. In this scenario, it is more likely to occur and can lead to discarding a large amount of data collected; however, the information from observed sensors can be leveraged to estimate those values not observed. We propose two methods for imputation of multisource data, both of which take advantage of potential correlation between data from different sensors, through ridge regression and a state-space model. These methods, as well as the common median imputation, are applied to data collected from a variety of sensors monitoring an experimental facility. Performance of imputation methods is compared with the mean absolute deviation; however, rather than using this metric to solely rank themethods,we also propose an approach to identify significant differences. Imputation techniqueswill also be assessed by their ability to produce appropriate confidence intervals, through coverage and length, around the imputed values. Finally, performance of imputed datasets is compared with a marginalized dataset through a weighted k-means clustering. In general, we found that imputation through a dynamic linearmodel tended to be themore » most accurate and to produce the most precise confidence intervals, and that imputing the missing values and down weighting them with respect to observed values in the analysis led to the most accurate performance.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [1]
+ Show Author Affiliations
  1. Los Alamos National Lab. (LANL), Los Alamos, NM (United States)
Publication Date:
Research Org.:
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Sponsoring Org.:
USDOE
OSTI Identifier:
1416279
Report Number(s):
LA-UR-17-23333
Journal ID: ISSN 1524-1904
Grant/Contract Number:  
AC52-06NA25396
Resource Type:
Accepted Manuscript
Journal Name:
Applied Stochastic Models in Business and Industry
Additional Journal Information:
Journal Volume: 34; Journal Issue: 1; Journal ID: ISSN 1524-1904
Publisher:
Wiley
Country of Publication:
United States
Language:
English
Subject:
97 MATHEMATICS AND COMPUTING; Imputation; assessment; multisource data; dynamic linear model; comparison; k-means clustering

Citation Formats

Casleton, Emily Michele, Osthus, David Allen, and Van Buren, Kendra Lu. Imputation for multisource data with comparison and assessment techniques. United States: N. p., 2017. Web. doi:10.1002/asmb.2299.
Copy to clipboard
Casleton, Emily Michele, Osthus, David Allen, & Van Buren, Kendra Lu. Imputation for multisource data with comparison and assessment techniques. United States. https://doi.org/10.1002/asmb.2299
Copy to clipboard
Casleton, Emily Michele, Osthus, David Allen, and Van Buren, Kendra Lu. Wed . "Imputation for multisource data with comparison and assessment techniques". United States. https://doi.org/10.1002/asmb.2299. https://www.osti.gov/servlets/purl/1416279.
Copy to clipboard
@article{osti_1416279,
title = {Imputation for multisource data with comparison and assessment techniques},
author = {Casleton, Emily Michele and Osthus, David Allen and Van Buren, Kendra Lu},
abstractNote = {Missing data are prevalent issue in analyses involving data collection. The problem of missing data is exacerbated for multisource analysis, where data from multiple sensors are combined to arrive at a single conclusion. In this scenario, it is more likely to occur and can lead to discarding a large amount of data collected; however, the information from observed sensors can be leveraged to estimate those values not observed. We propose two methods for imputation of multisource data, both of which take advantage of potential correlation between data from different sensors, through ridge regression and a state-space model. These methods, as well as the common median imputation, are applied to data collected from a variety of sensors monitoring an experimental facility. Performance of imputation methods is compared with the mean absolute deviation; however, rather than using this metric to solely rank themethods,we also propose an approach to identify significant differences. Imputation techniqueswill also be assessed by their ability to produce appropriate confidence intervals, through coverage and length, around the imputed values. Finally, performance of imputed datasets is compared with a marginalized dataset through a weighted k-means clustering. In general, we found that imputation through a dynamic linearmodel tended to be the most accurate and to produce the most precise confidence intervals, and that imputing the missing values and down weighting them with respect to observed values in the analysis led to the most accurate performance.},
doi = {10.1002/asmb.2299},
journal = {Applied Stochastic Models in Business and Industry},
number = 1,
volume = 34,
place = {United States},
year = {Wed Dec 27 00:00:00 EST 2017},
month = {Wed Dec 27 00:00:00 EST 2017}
}
Copy to clipboard
Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record
https://doi.org/10.1002/asmb.2299
Copyright Statement
Other availability
Search WorldCat Search WorldCat to find libraries that may hold this journal
Citation Metrics:
Cited by: 1 work
Citation information provided by
Web of Science

Figures / Tables:

Figure 1 Figure 1: Sensor placement and example subsets of the raw data from each type of sensor.
All figures and tables (17 total)
Save / Share:
Export Metadata
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.

Works referenced in this record:

An Improved Adaptive Background Mixture Model for Real-time Tracking with Shadow Detection
book, January 2002

The Bayesian elastic net
journal, March 2010

Foci and Bases of Employee Commitment: Implications for job Performance.
journal, April 1996

  • Becker, T. E.; Billings, R. S.; Eveleth, D. M.
  • Academy of Management Journal, Vol. 39, Issue 2
  • DOI: 10.2307/256788

An Entropy Weighting k-Means Algorithm for Subspace Clustering of High-Dimensional Sparse Data
journal, August 2007

  • Jing, Liping; Ng, Michael K.; Huang, Joshua Zhexue
  • IEEE Transactions on Knowledge and Data Engineering, Vol. 19, Issue 8
  • DOI: 10.1109/TKDE.2007.1048

Methods for imputation of missing values in air quality data sets
journal, June 2004

Missing value estimation methods for DNA microarrays
journal, June 2001

Ridge Regression: Biased Estimation for Nonorthogonal Problems
journal, February 1970

A Bayesian Study of the Error-in-Variables Model
journal, August 1981

  • Reilly, Park M.; Patino-Leal, Hugo
  • Technometrics, Vol. 23, Issue 3
  • DOI: 10.2307/1267784

An analysis of four missing data treatment methods for supervised learning
journal, May 2003

  • Batista, Gustavo E. A. P. A.; Monard, Maria Carolina
  • Applied Artificial Intelligence, Vol. 17, Issue 5-6
  • DOI: 10.1080/713827181

Bi-level multi-source learning for heterogeneous block-wise missing data
journal, November 2014

Regularization Paths for Generalized Linear Models via Coordinate Descent
journal, January 2010

  • Friedman, Jerome; Hastie, Trevor; Tibshirani, Robert
  • Journal of Statistical Software, Vol. 33, Issue 1
  • DOI: 10.18637/jss.v033.i01

Sequential Imputations and Bayesian Missing Data Problems
journal, March 1994

  • Kong, Augustine; Liu, Jun S.; Wong, Wing Hung
  • Journal of the American Statistical Association, Vol. 89, Issue 425
  • DOI: 10.1080/01621459.1994.10476469

Spectral methods for imputation of missing air quality data
journal, December 2015

  • Moshenberg, Shai; Lerner, Uri; Fishbain, Barak
  • Environmental Systems Research, Vol. 4, Issue 1
  • DOI: 10.1186/s40068-015-0052-z

An Introduction to Statistical Learning
book, January 2013

Review: A gentle introduction to imputation of missing values
journal, October 2006

  • Donders, A. Rogier T.; van der Heijden, Geert J. M. G.; Stijnen, Theo
  • Journal of Clinical Epidemiology, Vol. 59, Issue 10
  • DOI: 10.1016/j.jclinepi.2006.01.014

The Calculation of Posterior Distributions by Data Augmentation
journal, June 1987

Sensor fusion potential exploitation-innovative architectures and illustrative applications
journal, January 1997

  • Dasarathy, B. V.
  • Proceedings of the IEEE, Vol. 85, Issue 1
  • DOI: 10.1109/5.554206

An introduction to multisensor data fusion
journal, January 1997

  • Hall, D. L.; Llinas, J.
  • Proceedings of the IEEE, Vol. 85, Issue 1
  • DOI: 10.1109/5.554205
    Sort by:
    [ × clear filter / sort ]

    Figures / Tables found in this record:

    Figure 1 (p. 5)figure
    Table I (p. 5)table
    Figure 2 (p. 6)figure
    Figure 3 (p. 6)figure
    Figure 4 (p. 7)figure
    Figure 5 (p. 8)figure
    Figure 6 (p. 9)figure
    Table II (p. 10)table
    Figure 7 (p. 11)figure
    Figure 8 (p. 12)figure
    Figure 9 (p. 13)figure
    Table III (p. 13)table
    Figure 10 (p. 14)figure
    Figure 11 (p. 15)figure
    Figure 12 (p. 15)figure
    Figure 13 (p. 16)figure
    Table IV (p. 16)table
      [ × clear filter / sort ]
      Figures/Tables have been extracted from DOE-funded journal article accepted manuscripts.

      Similar Records in DOE PAGES and OSTI.GOV collections: