Learning from Crowds by Modeling Common Confusions

Conference · Tue Feb 09 04:00:00 EST 2021

OSTI ID:1822656

Wang, Hongning ^[1]; Chu, Zhendong ^[2]; Ma, Jing ^[2]

---

1. University of Virginia; Department of Computer Science, University of Virginia
2. University of Virginia

Crowdsourcing provides a practical way to obtain large amounts of labeled data at a low cost. However, the annotation quality of annotators varies considerably, which imposes new challenges in learning a high-quality model from the crowdsourced annotations. In this work, we provide a new perspective to decompose annotation noise into common noise and individual noise and differentiate the source of confusion based on instance difficulty and annotator expertise on a per-instance-annotator basis. We realize this new crowdsourcing model by an end-to-end learning solution with two types of noise adaptation layers: one is shared across annotators to capture their commonly shared confusions, and the other one is pertaining to each annotator to realize individual confusion. To recognize the source of noise in each annotation, we use an auxiliary network to choose from the two noise adaptation layers with respect to both instances and annotators. Extensive experiments on both synthesized and real-world benchmarks demonstrate the effectiveness of our proposed common noise adaptation solution.

• Research Organization: University of Virginia
• Sponsoring Organization: USDOE Office of Energy Efficiency and Renewable Energy (EERE), Energy Efficiency Office. Building Technologies Office
• DOE Contract Number: EE0008227
• OSTI ID: 1822656
• Report Number(s): DOE-UVA-0008227-6
• Country of Publication: United States
• Language: English

Related Subjects

96 KNOWLEDGE MANAGEMENT AND PRESERVATION
Learning of cost, reliability, and skill of label, learning human values and preferences, semi-supervised learning, probabilistic graphical models