Learning from crowds with variational Gaussian processes

Ruiz, Pablo; Morales-Alvarez, Pablo; Molina, Rafael; Katsaggelos, Aggelos

doi:10.1016/j.patcog.2018.11.021

Title: Learning from crowds with variational Gaussian processes

Abstract

Solving a supervised learning problem requires to label a training set. This task is traditionally performed by an expert, who provides a label for each sample. The proliferation of social web services (e.g., Amazon Mechanical Turk) has introduced an alternative crowdsourcing approach. Anybody with a computer can register in one of these services and label, either partially or completely, a dataset. The effort of labeling is then shared between a great number of annotators. However, this approach introduces scientifically challenging problems such as combining the unknown expertise of the annotators, handling disagreements on the annotated samples, or detecting the existence of spammer and adversarial annotators. All these problems require probabilistic sound solutions which go beyond the naive use of majority voting plus classical classification methods. In this work we introduce a new crowdsourcing model and inference procedure which trains a Gaussian Process classifier using the noisy labels provided by the annotators. Variational Bayes inference is used to estimate all unknowns. The proposed model can predict the class of new samples and assess the expertise of the involved annotators. Moreover, the Bayesian treatment allows for a solid uncertainty quantification. Since when predicting the class of a new sample we might havemore »« less

Authors:

Ruiz, Pablo ^[1]; Morales-Alvarez, Pablo ^[2]; Molina, Rafael ^[2];

^[1]

Northwestern University, Evanston, IL (United States)
University of Granada, Granada (Spain)

Publication Date:: Tue Nov 20 00:00:00 EST 2018

Research Org.:: Northwestern Univ., Evanston, IL (United States)

Sponsoring Org.:: USDOE National Nuclear Security Administration (NNSA); Spanish Ministry of Economy and Competitiveness; University of Granada; La Caixa Banking Foundation

OSTI Identifier:: 1488416

Alternate Identifier(s):: OSTI ID: 1636753

Grant/Contract Number:: NA0002520; DPI2016-77869-C2-2-R

Resource Type:: Accepted Manuscript

Journal Name:: Pattern Recognition

Additional Journal Information:: Journal Volume: 88; Journal Issue: C; Journal ID: ISSN 0031-3203

Publisher:: Elsevier

Country of Publication:: United States

Language:: English

Subject:: 97 MATHEMATICS AND COMPUTING; crowdsourcing; classification; gaussian processes; Bayesian modeling; variational inference

Citation Formats


                    Ruiz, Pablo, Morales-Alvarez, Pablo, Molina, Rafael, and Katsaggelos, Aggelos. Learning from crowds with variational Gaussian processes.  United States: N. p., 2018. 
Web.  doi:10.1016/j.patcog.2018.11.021.

Copy to clipboard


                    Ruiz, Pablo, Morales-Alvarez, Pablo, Molina, Rafael, & Katsaggelos, Aggelos. Learning from crowds with variational Gaussian processes.  United States.  https://doi.org/10.1016/j.patcog.2018.11.021

Copy to clipboard


                    Ruiz, Pablo, Morales-Alvarez, Pablo, Molina, Rafael, and Katsaggelos, Aggelos. Tue .  
"Learning from crowds with variational Gaussian processes".  United States.  https://doi.org/10.1016/j.patcog.2018.11.021.  https://www.osti.gov/servlets/purl/1488416.

Copy to clipboard


                    
@article{osti_1488416,

  title        = {Learning from crowds with variational Gaussian processes},

  author       = {Ruiz, Pablo and Morales-Alvarez, Pablo and Molina, Rafael and Katsaggelos, Aggelos},

  abstractNote = {Solving a supervised learning problem requires to label a training set. This task is traditionally performed by an expert, who provides a label for each sample. The proliferation of social web services (e.g., Amazon Mechanical Turk) has introduced an alternative crowdsourcing approach. Anybody with a computer can register in one of these services and label, either partially or completely, a dataset. The effort of labeling is then shared between a great number of annotators. However, this approach introduces scientifically challenging problems such as combining the unknown expertise of the annotators, handling disagreements on the annotated samples, or detecting the existence of spammer and adversarial annotators. All these problems require probabilistic sound solutions which go beyond the naive use of majority voting plus classical classification methods. In this work we introduce a new crowdsourcing model and inference procedure which trains a Gaussian Process classifier using the noisy labels provided by the annotators. Variational Bayes inference is used to estimate all unknowns. The proposed model can predict the class of new samples and assess the expertise of the involved annotators. Moreover, the Bayesian treatment allows for a solid uncertainty quantification. Since when predicting the class of a new sample we might have access to some annotations for it, we also show how our method can naturally incorporate this additional information. Furthermore, a comprehensive experimental section evaluates the proposed method with synthetic and real experiments, showing that it consistently outperforms other state-of-the-art crowdsourcing approaches.},

  doi          = {10.1016/j.patcog.2018.11.021},

  journal      = {Pattern Recognition},

  number       = C,

  volume       = 88,

  place        = {United States},

  year         = {Tue Nov 20 00:00:00 EST 2018},

  month        = {Tue Nov 20 00:00:00 EST 2018}

}

Copy to clipboard

Journal Article:

Free Publicly Available Full Text

Accepted Manuscript (Publisher)

Accepted Manuscript (DOE)

Publisher's Version of Record

https://doi.org/10.1016/j.patcog.2018.11.021

Other availability

Search WorldCat to find libraries that may hold this journal

Citation Metrics:

Cited by: 21 works

Citation information provided by
Web of Science

Save / Share:

Export Metadata

Save to My Library

Works referenced in this record:

Handling data irregularities in classification: Foundations, trends, and future challenges
journal, September 2018

Das, Swagatam; Datta, Shounak; Chaudhuri, Bidyut B.
Pattern Recognition, Vol. 81
DOI: 10.1016/j.patcog.2018.03.008

Active cleaning of label noise
journal, March 2016

Ekambaram, Rajmadhan; Fefilatyev, Sergiy; Shreve, Matthew
Pattern Recognition, Vol. 51
DOI: 10.1016/j.patcog.2015.09.020

Learning from crowdsourced labeled data: a survey
journal, July 2016

Zhang, Jing; Wu, Xindong; Sheng, Victor S.
Artificial Intelligence Review, Vol. 46, Issue 4
DOI: 10.1007/s10462-016-9491-9

Learning from multiple annotators with varying expertise
journal, October 2013

Yan, Yan; Rosales, Rómer; Fung, Glenn
Machine Learning, Vol. 95, Issue 3
DOI: 10.1007/s10994-013-5412-1

Joint Data Filtering and Labeling Using Gaussian Processes and Alternating Direction Method of Multipliers
journal, July 2016

Ruiz, Pablo; Molina, Rafael; Katsaggelos, Aggelos K.
IEEE Transactions on Image Processing, Vol. 25, Issue 7
DOI: 10.1109/TIP.2016.2558472

Remote Sensing Image Classification With Large-Scale Gaussian Processes
journal, February 2018

Morales-Alvarez, Pablo; Perez-Suay, Adrian; Molina, Rafael
IEEE Transactions on Geoscience and Remote Sensing, Vol. 56, Issue 2
DOI: 10.1109/TGRS.2017.2758922

Learning Supervised Topic Models for Classification and Regression from Crowds
journal, December 2017

Rodrigues, Filipe; Lourenco, Mariana; Ribeiro, Bernardete
IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 39, Issue 12
DOI: 10.1109/TPAMI.2017.2648786

Learning with privileged information for multi-Label classification
journal, September 2018

Wang, Shangfei; Chen, Shiyu; Chen, Tanfang
Pattern Recognition, Vol. 81
DOI: 10.1016/j.patcog.2018.03.033

Labelling strategies for hierarchical multi-label classification techniques
journal, August 2016

Triguero, Isaac; Vens, Celine
Pattern Recognition, Vol. 56
DOI: 10.1016/j.patcog.2016.02.017

Sloop: A pattern retrieval engine for individual animal identification
journal, April 2015

Duyck, James; Finn, Chelsea; Hutcheon, Andy
Pattern Recognition, Vol. 48, Issue 4
DOI: 10.1016/j.patcog.2014.07.017

Citizen crowds and experts: observer variability in image-based plant phenotyping
journal, February 2018

Giuffrida, M. Valerio; Chen, Feng; Scharr, Hanno
Plant Methods, Vol. 14, Issue 1
DOI: 10.1186/s13007-018-0278-7

A global dataset of crowdsourced land cover and land use reference data
journal, June 2017

Fritz, Steffen; See, Linda; Perger, Christoph
Scientific Data, Vol. 4, Issue 1
DOI: 10.1038/sdata.2017.75

AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
journal, May 2016

Albarqouni, Shadi; Baur, Christoph; Achilles, Felix
IEEE Transactions on Medical Imaging, Vol. 35, Issue 5
DOI: 10.1109/TMI.2016.2528120

Gravity Spy: integrating advanced LIGO detector characterization, machine learning, and citizen science
journal, February 2017

Zevin, M.; Coughlin, S.; Bahaadini, S.
Classical and Quantum Gravity, Vol. 34, Issue 6
DOI: 10.1088/1361-6382/aa5cea

Learning kernel logistic regression in the presence of class label noise
journal, November 2014

Bootkrajang, Jakramate; Kabán, Ata
Pattern Recognition, Vol. 47, Issue 11
DOI: 10.1016/j.patcog.2014.05.007

Learning from multiple annotators: Distinguishing good from random labelers
journal, September 2013

Rodrigues, Filipe; Pereira, Francisco; Ribeiro, Bernardete
Pattern Recognition Letters, Vol. 34, Issue 12
DOI: 10.1016/j.patrec.2013.05.012

Musical genre classification of audio signals
journal, July 2002

Tzanetakis, G.; Cook, P.
IEEE Transactions on Speech and Audio Processing, Vol. 10, Issue 5
DOI: 10.1109/TSA.2002.800560

Similar Records in DOE PAGES and OSTI.GOV collections:

Scalable and efficient learning from crowds with Gaussian processes

Journal Article Morales-Álvarez, Pablo ; Ruiz, Pablo ; Santos-Rodríguez, Raúl ; ... - Information Fusion

Over the last few years, multiply-annotated data has become a very popular source of information. Online platforms such as Amazon Mechanical Turk have revolutionized the labelling process needed for any classification task, sharing the effort between a number of annotators (instead of the classical single expert). This crowdsourcing approach has introduced new challenging problems, such as handling disagreements on the annotated samples or combining the unknown expertise of the annotators. Probabilistic methods, such as Gaussian Processes (GP), have proven successful to model this new crowdsourcing scenario. However, GPs do not scale up well with the training set size, which makesmore »« less
Cited by 11
https://doi.org/10.1016/j.inffus.2018.12.008

Full Text Available
Learning from Crowds by Modeling Common Confusions

Conference Wang, Hongning ; Chu, Zhendong ; Ma, Jing

Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low cost. However, the annotation quality of annotators varies considerably, which imposes new challenges in learning a high-quality model from the crowdsourced annotations. In this work, we provide a new perspective to decompose annotation noise into common noise and individual noise and differentiate the source of confusion based on instance difficulty and annotator expertise on a per-instance-annotator basis. We realize this new crowdsourcing model by an end-to-end learning solution with two types of noise adaptation layers: one is shared across annotators to capture their commonly sharedmore »« less
Full Text Available
Improve Learning from Crowds via Generative Augmentation

Conference Chu, Zhendong ; Wang, Hongning - Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining

Crowdsourcing provides an efficient label collection schema for supervised machine learning. However, to control annotation cost, each instance in the crowdsourced data is typically annotated by a small number of annotators. This creates a sparsity issue and limits the quality of machine learning models trained on such data. In this paper, we study how to handle sparsity in crowdsourced data using data augmentation. Specifically, we propose to directly learn a classifier by augmenting the raw sparse annotations. We implement two principles of high-quality augmentation using Generative Adversarial Networks: 1) the generated annotations should follow the distribution of authentic ones, whichmore »« less
https://doi.org/10.1145/3447548.3467409

Full Text Available
Bayesian coherent analysis of in-spiral gravitational wave signals with a detector network

Journal Article Veitch, J ; School of Physics and Astronomy, Cardiff University, Cardiff CF24 3AA ; Vecchio, A - Physical Review. D, Particles Fields

The present operation of the ground-based network of gravitational-wave laser interferometers in enhanced configuration and the beginning of the construction of second-generation (or advanced) interferometers with planned observation runs beginning by 2015 bring the search for gravitational waves into a regime where detection is highly plausible. The development of techniques that allow us to discriminate a signal of astrophysical origin from instrumental artefacts in the interferometer data and to extract the full range of information are therefore some of the primary goals of the current work. Here we report the details of a Bayesian approach to the problem of inferencemore »« less
https://doi.org/10.1103/PHYSREVD.81.062003
Joint Data Filtering and Labeling Using Gaussian Processes and Alternating Direction Method of Multipliers

Journal Article Ruiz, Pablo ; Molina, Rafael ; Katsaggelos, Aggelos K. - IEEE Transactions on Image Processing

Here, sequence labeling aims at assigning a label to every sample of a signal (or pixel of an image) while considering the sequentiality (or vicinity) of the samples. To perform this task, many works in the literature first filter and then label the data. Unfortunately, the filtering, which is performed independently from the labeling, is far from optimal and frequently makes the latter task harder. In this paper, a novel approach that trains a Gaussian process classifier and estimates the coefficients of an optimal filter jointly is presented. The new approach, based on Bayesian modeling and alternating direction method ofmore »« less
https://doi.org/10.1109/TIP.2016.2558472

Similar Records

Title: Learning from crowds with variational Gaussian processes

Abstract

Citation Formats

Handling data irregularities in classification: Foundations, trends, and future challenges journal, September 2018

Active cleaning of label noise journal, March 2016

Learning from crowdsourced labeled data: a survey journal, July 2016

Learning from multiple annotators with varying expertise journal, October 2013

Joint Data Filtering and Labeling Using Gaussian Processes and Alternating Direction Method of Multipliers journal, July 2016

Remote Sensing Image Classification With Large-Scale Gaussian Processes journal, February 2018

Learning Supervised Topic Models for Classification and Regression from Crowds journal, December 2017

Learning with privileged information for multi-Label classification journal, September 2018

Labelling strategies for hierarchical multi-label classification techniques journal, August 2016

Sloop: A pattern retrieval engine for individual animal identification journal, April 2015

Citizen crowds and experts: observer variability in image-based plant phenotyping journal, February 2018

A global dataset of crowdsourced land cover and land use reference data journal, June 2017

AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images journal, May 2016

Gravity Spy: integrating advanced LIGO detector characterization, machine learning, and citizen science journal, February 2017

Learning kernel logistic regression in the presence of class label noise journal, November 2014

Learning from multiple annotators: Distinguishing good from random labelers journal, September 2013

Musical genre classification of audio signals journal, July 2002

Handling data irregularities in classification: Foundations, trends, and future challenges
journal, September 2018

Active cleaning of label noise
journal, March 2016

Learning from crowdsourced labeled data: a survey
journal, July 2016

Learning from multiple annotators with varying expertise
journal, October 2013

Joint Data Filtering and Labeling Using Gaussian Processes and Alternating Direction Method of Multipliers
journal, July 2016

Remote Sensing Image Classification With Large-Scale Gaussian Processes
journal, February 2018

Learning Supervised Topic Models for Classification and Regression from Crowds
journal, December 2017

Learning with privileged information for multi-Label classification
journal, September 2018

Labelling strategies for hierarchical multi-label classification techniques
journal, August 2016

Sloop: A pattern retrieval engine for individual animal identification
journal, April 2015

Citizen crowds and experts: observer variability in image-based plant phenotyping
journal, February 2018

A global dataset of crowdsourced land cover and land use reference data
journal, June 2017

AggNet: Deep Learning From Crowds for Mitosis Detection in Breast Cancer Histology Images
journal, May 2016

Gravity Spy: integrating advanced LIGO detector characterization, machine learning, and citizen science
journal, February 2017

Learning kernel logistic regression in the presence of class label noise
journal, November 2014

Learning from multiple annotators: Distinguishing good from random labelers
journal, September 2013

Musical genre classification of audio signals
journal, July 2002