The value of human data annotation for machine learning based anomaly detection in environmental systems

Russo, Stefania; Besmer, Michael D.; Blumensaat, Frank; Bouffard, Damien; Disch, Andy; Hammes, Frederik; Hess, Angelika; Lürig, Moritz; Matthews, Blake; Minaudo, Camille; Morgenroth, Eberhard; Tran-Khac, Viet; Villez, Kris

doi:10.1016/j.watres.2021.117695

Title: The value of human data annotation for machine learning based anomaly detection in environmental systems

Journal Article · Mon Sep 27 00:00:00 EDT 2021 · Water Research

DOI:https://doi.org/10.1016/j.watres.2021.117695· OSTI ID:1827039

^[1];

^[2];

^[1];

^[3];

^[3]; Hammes, Frederik ^[3]; Hess, Angelika ^[1];

^[4]; Matthews, Blake ^[3];

^[5];

^[1]; Tran-Khac, Viet ^[6]; Villez, Kris ^[7]

Swiss Federal Institute of Aquatic Science and Technology (Eawag), Dübendorf (Switzerland); Eidgenoessische Technische Hochschule (ETH), Zurich (Switzerland)
onCyt Microbiology AG, Zürich (Switzerland)
Swiss Federal Institute of Aquatic Science and Technology (Eawag), Dübendorf (Switzerland)
Swiss Federal Institute of Aquatic Science and Technology (Eawag), Dübendorf (Switzerland); Lund Univ. (Sweden)
Ecole Polytechnique Federale Lausanne (Switzerland)
Université Savoie Mont Blanc, Thonon-les-Bains (France)
Swiss Federal Institute of Aquatic Science and Technology (Eawag), Dübendorf (Switzerland); Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Anomaly detection is the process of identifying unexpected data samples in datasets. Automated anomaly detection is either performed using supervised machine learning models, which require a labelled dataset for their calibration, or unsupervised models, which do not require labels. While academic research has produced a vast array of tools and machine learning models for automated anomaly detection, the research community focused on environmental systems still lacks a comparative analysis that is simultaneously comprehensive, objective, and systematic. This knowledge gap is addressed for the first time in this study, where 15 different supervised and unsupervised anomaly detection models are evaluated on 5 different environmental datasets from engineered and natural aquatic systems. To this end, anomaly detection performance, labelling efforts, as well as the impact of model and algorithm tuning are taken into account. As a result, our analysis reveals the relative strengths and weaknesses of the different approaches in an objective manner without bias for any particular paradigm in machine learning. Most importantly, our results show that expert-based data annotation is extremely valuable for anomaly detection based on machine learning.

View Accepted Manuscript (DOE)

Cite

Export

Save

Research Organization:: Oak Ridge National Lab. (ORNL), Oak Ridge, TN (United States)

Sponsoring Organization:: USDOE; Eawag Discretionary Funds

Grant/Contract Number:: AC05-00OR22725; 5221.00492.012.02

OSTI ID:: 1827039

Journal Information:: Water Research, Vol. 206; ISSN 0043-1354

Publisher:: ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (32)

Functional unfold principal component analysis for automatic plant-based stress detection in grapevine Baert, Annelies; Villez, Kris; Steppe, Kathy Functional Plant Biology, Vol. 39, Issue 6 https://doi.org/10.1071/FP12007	journal	January 2012
One-class classification: taxonomy of study and review of techniques Khan, Shehroz S.; Madden, Michael G. The Knowledge Engineering Review, Vol. 29, Issue 3 https://doi.org/10.1017/S026988891300043X	journal	January 2014
Fault detection in a real wastewater plant using parameter-estimation techniques Fuente, M. J.; Vega, P.; Zarrop, M. Control Engineering Practice, Vol. 4, Issue 8 https://doi.org/10.1016/0967-0661(96)00107-4	journal	August 1996
Real-time remote monitoring of small-scaled biological wastewater treatment plants by a multivariate statistical process control and neural network-based software sensors Lee, Min Woo; Hong, Sung Hun; Choi, Hyeoksun Process Biochemistry, Vol. 43, Issue 10 https://doi.org/10.1016/j.procbio.2008.06.002	journal	October 2008
Clustering and Support Vector Regression for Water Demand Forecasting and Anomaly Detection Candelieri, Antonio Water, Vol. 9, Issue 3 https://doi.org/10.3390/w9030224	journal	March 2017
Environmental Data Science Gibert, Karina; Horsburgh, Jeffery S.; Athanasiadis, Ioannis N. Environmental Modelling & Software, Vol. 106 https://doi.org/10.1016/j.envsoft.2018.04.005	journal	August 2018
Transforming data into knowledge for improved wastewater treatment operation: A critical review of techniques Corominas, Ll.; Garrido-Baserba, M.; Villez, K. Environmental Modelling & Software, Vol. 106 https://doi.org/10.1016/j.envsoft.2017.11.023	journal	August 2018
Performance evaluation of fault detection methods for wastewater treatment processes Corominas, Lluís; Villez, Kris; Aguado, Daniel Biotechnology and Bioengineering, Vol. 108, Issue 2 https://doi.org/10.1002/bit.22953	journal	November 2010
Multivariate and multiscale monitoring of wastewater treatment operation Rosen, C.; Lennox, J. A. Water Research, Vol. 35, Issue 14 https://doi.org/10.1016/S0043-1354(01)00069-0	journal	October 2001
The Fourth-Revolution in the Water Sector Encounters the Digital Revolution Garrido-Baserba, Manel; Corominas, Lluís; Cortés, Ulises Environmental Science & Technology, Vol. 54, Issue 8 https://doi.org/10.1021/acs.est.9b04251	journal	March 2020
Advanced monitoring of water systems using in situ measurement stations: data validation and fault detection Alferes, Janelcy; Tik, Sovanna; Copp, John Water Science and Technology, Vol. 68, Issue 5 https://doi.org/10.2166/wst.2013.302	journal	September 2013
Self-Organizing Maps Application in a Remote Water Quality Monitoring System Postolache, O. A.; Girao, P. M. B. S.; Pereira, J. M. D. IEEE Transactions on Instrumentation and Measurement, Vol. 54, Issue 1 https://doi.org/10.1109/TIM.2004.834583	journal	February 2005
A framework for automated anomaly detection in high frequency water-quality data from in situ sensors Leigh, Catherine; Alsibai, Omar; Hyndman, Rob J. Science of The Total Environment, Vol. 664 https://doi.org/10.1016/j.scitotenv.2019.02.085	journal	May 2019
An introduction to ROC analysis Fawcett, Tom Pattern Recognition Letters, Vol. 27, Issue 8 https://doi.org/10.1016/j.patrec.2005.10.010	journal	June 2006
A domain-independent methodology to analyze IoT data streams in real-time. A proof of concept implementation for anomaly detection from environmental data Trilles, Sergio; Belmonte, Òscar; Schade, Sven International Journal of Digital Earth, Vol. 10, Issue 1 https://doi.org/10.1080/17538947.2016.1209583	journal	July 2016
Interactive effects of foundation species on ecosystem functioning and stability in response to disturbance Narwani, Anita; Reyes, Marta; Pereira, Aaron Louis Proceedings of the Royal Society B: Biological Sciences, Vol. 286, Issue 1913 https://doi.org/10.1098/rspb.2019.1857	journal	October 2019
Fault detection in a real-time monitoring network for water quality in the lagoon of Venice (Italy) Ciavatta, S.; Pastres, R.; Lin, Z. Water Science and Technology, Vol. 50, Issue 11 https://doi.org/10.2166/wst.2004.0670	journal	December 2004
Anomaly detection in streaming environmental sensor data: A data-driven modeling approach Hill, David J.; Minsker, Barbara S. Environmental Modelling & Software, Vol. 25, Issue 9 https://doi.org/10.1016/j.envsoft.2009.08.010	journal	September 2010
Monitoring of a sequencing batch reactor using adaptive multiblock principal component analysis Lee, Dae Sung; Vanrolleghem, Peter A. Biotechnology and Bioengineering, Vol. 82, Issue 4 https://doi.org/10.1002/bit.10589	journal	March 2003
Active learning for anomaly detection in environmental data Russo, Stefania; Lürig, Moritz; Hao, Wenjin Environmental Modelling & Software, Vol. 134 https://doi.org/10.1016/j.envsoft.2020.104869	journal	December 2020
Combining multiway principal component analysis (MPCA) and clustering for efficient data mining of historical data sets of SBR processes Villez, Kris; Ruiz, Magda; Sin, Gürkan Water Science and Technology, Vol. 57, Issue 10 https://doi.org/10.2166/wst.2008.143	journal	May 2008
River Flooding Forecasting and Anomaly Detection Based on Deep Learning Miau, Scott; Hung, Wei-Hsi IEEE Access, Vol. 8 https://doi.org/10.1109/ACCESS.2020.3034875	journal	January 2020
Online flow cytometry reveals microbial dynamics influenced by concurrent natural and operational events in groundwater used for drinking water treatment Besmer, Michael D.; Epting, Jannis; Page, Rebecca M. Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep38462	journal	December 2016
Anomaly Detection in Environmental Monitoring Networks [Application Notes] Bezdek, James; Rajasegarar, Sutharshan; Moshtaghi, Masud IEEE Computational Intelligence Magazine, Vol. 6, Issue 2 https://doi.org/10.1109/MCI.2011.940751	journal	May 2011
Multivariate SPC of a sequencing batch reactor for wastewater treatment Aguado, D.; Ferrer, A.; Ferrer, J. Chemometrics and Intelligent Laboratory Systems, Vol. 85, Issue 1 https://doi.org/10.1016/j.chemolab.2006.05.003	journal	January 2007
Anomaly detection: A survey Chandola, Varun; Banerjee, Arindam; Kumar, Vipin ACM Computing Surveys, Vol. 41, Issue 3, p. 1-58 https://doi.org/10.1145/1541880.1541882	journal	July 2009
A comparison of multiway regression and scaling methods Gurden, Stephen P.; Westerhuis, Johan A.; Bro, Rasmus Chemometrics and Intelligent Laboratory Systems, Vol. 59, Issue 1-2 https://doi.org/10.1016/S0169-7439(01)00168-X	journal	November 2001
The feasibility of automated online flow cytometry for in-situ monitoring of microbial dynamics in aquatic ecosystems Besmer, Michael D.; Weissbrodt, David G.; Kratochvil, Bradley E. Frontiers in Microbiology, Vol. 5 https://doi.org/10.3389/fmicb.2014.00265	journal	June 2014
Abrupt Event Monitoring for Water Environment System Based on KPCA and SVM Ni, Jianjun; Zhang, Chuanbiao; Ren, Li IEEE Transactions on Instrumentation and Measurement, Vol. 61, Issue 4 https://doi.org/10.1109/TIM.2011.2173000	journal	April 2012
A nonlinear kernel Gaussian mixture model based inferential monitoring approach for fault detection and diagnosis of chemical processes Yu, Jie Chemical Engineering Science, Vol. 68, Issue 1 https://doi.org/10.1016/j.ces.2011.10.011	journal	January 2012
Multivariate statistical monitoring of continuous wastewater treatment plants Aguado, Daniel; Rosen, Christian Engineering Applications of Artificial Intelligence, Vol. 21, Issue 7 https://doi.org/10.1016/j.engappai.2007.08.004	journal	October 2008
Adaptive multiscale principal component analysis for on-line monitoring of a sequencing batch reactor Lee, Dae Sung; Park, Jong Moon; Vanrolleghem, Peter A. Journal of Biotechnology, Vol. 116, Issue 2 https://doi.org/10.1016/j.jbiotec.2004.10.012	journal	March 2005

Similar Records

Process Anomaly Detection for Sparsely Labeled Events in Nuclear Power Plants

Technical Report · Wed Sep 01 00:00:00 EDT 2021 · OSTI ID:1827039

Farber, Jacob A.; Al Rashdan, Ahmad Y.; Abdel-Khalik, Hany; +4 more

Active Learning for Anomaly Detection in Environmental data

Journal Article · Tue Sep 15 00:00:00 EDT 2020 · Environmental Modelling and Software · OSTI ID:1827039

Russo, Stefania; Lürig, Moritz; Hao, Wenjin; +2 more

The Livermore Brain: Massive Deep Learning Networks Enabled by High Performance Computing

Technical Report · Tue Nov 29 00:00:00 EST 2016 · OSTI ID:1827039

Chen, Barry Y.

Related Subjects

54 ENVIRONMENTAL SCIENCES
machine learning
anomaly detection
environmental systems
labels

Title: The value of human data annotation for machine learning based anomaly detection in environmental systems

Citation Formats

References (32)

Similar Records

Related Subjects