Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety
Abstract
Construction workplace hazard detection requires engineers to analyze scenes manually against many safety rules, which is time-consuming, labor-intensive, and error-prone. Computer vision algorithms are yet to achieve reliable discrimination of anomalous and benign object relations underpinning safety violation detections. Recently developed deep learning-based computer vision algorithms need tens of thousands of images, including labels of the safety rules violated, in order to train deep-learning networks for acquiring spatiotemporal reasoning capacity in complex workplaces. Such training processes need human experts to label images and indicate whether the relationship between the worker, resource, and equipment in the scenes violate spatiotemporal arrangement rules for safe and productive operations. False alarms in those manual labels (labeling no-violation images as having violations) can significantly mislead the machine learning process and result in computer vision models that produce inaccurate hazard detections. Compared with false alarms, another type of mislabels, false negatives (labeling images having violations as “no violations”), seem to have fewer impacts on the reliability of the trained computer vision models.
- Authors:
-
- Arizona State University, Tempe, AZ (United States). Del E. Webb School of Construction
- Tsinghua Univ., Beijing (China)
- Texas A & M Univ., College Station, TX (United States)
- Arizona State Univ., Tempe, AZ (United States)
- Publication Date:
- Research Org.:
- Arizona State Univ., Tempe, AZ (United States)
- Sponsoring Org.:
- USDOE Office of Nuclear Energy (NE)
- OSTI Identifier:
- 1801215
- Alternate Identifier(s):
- OSTI ID: 1570882
- Grant/Contract Number:
- NE0008403
- Resource Type:
- Accepted Manuscript
- Journal Name:
- Advanced Engineering Informatics
- Additional Journal Information:
- Journal Volume: 42; Journal Issue: C; Journal ID: ISSN 1474-0346
- Publisher:
- Elsevier
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 97 MATHEMATICS AND COMPUTING; Computer Science; Engineering; Crowdsourcing; Construction safety; Image annotation; Bayesian network model; Safety inspection
Citation Formats
Wang, Yanyu, Liao, Pin-Chao, Zhang, Cheng, Ren, Yi, Sun, Xinlu, and Tang, Pingbo. Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety. United States: N. p., 2019.
Web. doi:10.1016/j.aei.2019.101001.
Wang, Yanyu, Liao, Pin-Chao, Zhang, Cheng, Ren, Yi, Sun, Xinlu, & Tang, Pingbo. Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety. United States. https://doi.org/10.1016/j.aei.2019.101001
Wang, Yanyu, Liao, Pin-Chao, Zhang, Cheng, Ren, Yi, Sun, Xinlu, and Tang, Pingbo. Fri .
"Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety". United States. https://doi.org/10.1016/j.aei.2019.101001. https://www.osti.gov/servlets/purl/1801215.
@article{osti_1801215,
title = {Crowdsourced reliable labeling of safety-rule violations on images of complex construction scenes for advanced vision-based workplace safety},
author = {Wang, Yanyu and Liao, Pin-Chao and Zhang, Cheng and Ren, Yi and Sun, Xinlu and Tang, Pingbo},
abstractNote = {Construction workplace hazard detection requires engineers to analyze scenes manually against many safety rules, which is time-consuming, labor-intensive, and error-prone. Computer vision algorithms are yet to achieve reliable discrimination of anomalous and benign object relations underpinning safety violation detections. Recently developed deep learning-based computer vision algorithms need tens of thousands of images, including labels of the safety rules violated, in order to train deep-learning networks for acquiring spatiotemporal reasoning capacity in complex workplaces. Such training processes need human experts to label images and indicate whether the relationship between the worker, resource, and equipment in the scenes violate spatiotemporal arrangement rules for safe and productive operations. False alarms in those manual labels (labeling no-violation images as having violations) can significantly mislead the machine learning process and result in computer vision models that produce inaccurate hazard detections. Compared with false alarms, another type of mislabels, false negatives (labeling images having violations as “no violations”), seem to have fewer impacts on the reliability of the trained computer vision models.},
doi = {10.1016/j.aei.2019.101001},
journal = {Advanced Engineering Informatics},
number = C,
volume = 42,
place = {United States},
year = {Fri Oct 18 00:00:00 EDT 2019},
month = {Fri Oct 18 00:00:00 EDT 2019}
}
Free Publicly Available Full Text
Publisher's Version of Record
Other availability
Search WorldCat to find libraries that may hold this journal
Cited by: 13 works
Citation information provided by
Web of Science
Web of Science
Save to My Library
You must Sign In or Create an Account in order to save documents to your library.
Works referenced in this record:
Structured Analysis (SA): A Language for Communicating Ideas
journal, January 1977
- Ross, D. T.
- IEEE Transactions on Software Engineering, Vol. SE-3, Issue 1
Crowdsourcing the creation of image segmentation algorithms for connectomics
journal, November 2015
- Arganda-Carreras, Ignacio; Turaga, Srinivas C.; Berger, Daniel R.
- Frontiers in Neuroanatomy, Vol. 9
Implementation of safety management systems in Hong Kong construction industry – A safety practitioner's perspective
journal, February 2018
- Yiu, Nicole S. N.; Sze, N. N.; Chan, Daniel W. M.
- Journal of Safety Research, Vol. 64
The Mechanism of how Design Failures cause Unsafe Behavior: The Cognitive Reliability and Error Analysis Method (CREAM)
journal, January 2016
- Liao, Pin-Chao; Luo, Xintong; Wang, Tao
- Procedia Engineering, Vol. 145
Crowdsourcing as a Model for Problem Solving: An Introduction and Cases
journal, February 2008
- Brabham, Daren C.
- Convergence: The International Journal of Research into New Media Technologies, Vol. 14, Issue 1
Crowdsourcing—Harnessing the Masses to Advance Health and Medicine, a Systematic Review
journal, July 2013
- Ranard, Benjamin L.; Ha, Yoonhee P.; Meisel, Zachary F.
- Journal of General Internal Medicine, Vol. 29, Issue 1
Hardhat-Wearing Detection for Enhancing On-Site Safety of Construction Workers
journal, September 2015
- Park, Man-Woo; Elsafty, Nehad; Zhu, Zhenhua
- Journal of Construction Engineering and Management, Vol. 141, Issue 9
Limitations of Majority Agreement in Crowdsourced Image Interpretation
journal, March 2016
- Salk, Carl F.; Sturn, Tobias; See, Linda
- Transactions in GIS, Vol. 21, Issue 2
Safety Activity Analysis Framework to Evaluate Safety Performance in Construction
journal, March 2017
- Awolusi, Ibukun G.; Marks, Eric D.
- Journal of Construction Engineering and Management, Vol. 143, Issue 3
Designing performance analysis and IDEF0 for enterprise modelling in BPR
journal, March 2002
- Kim, Soung-Hie; Jang, Ki-Jin
- International Journal of Production Economics, Vol. 76, Issue 2
Ontology-Based Representation and Reasoning Framework for Supporting Job Hazard Analysis
journal, November 2011
- Wang, Han-Hsiang; Boukamp, Frank
- Journal of Computing in Civil Engineering, Vol. 25, Issue 6
When Crowdsourcing Fails: A Study of Expertise on Crowdsourced Design Evaluation
journal, March 2015
- Burnap, Alex; Ren, Yi; Gerth, Richard
- Journal of Mechanical Design, Vol. 137, Issue 3
ImageNet Large Scale Visual Recognition Challenge
journal, April 2015
- Russakovsky, Olga; Deng, Jia; Su, Hao
- International Journal of Computer Vision, Vol. 115, Issue 3
Image Understanding using vision and reasoning through Scene Description Graph
journal, August 2018
- Aditya, Somak; Yang, Yezhou; Baral, Chitta
- Computer Vision and Image Understanding, Vol. 173
Contributing factors in construction accidents
journal, July 2005
- Haslam, R. A.; Hide, S. A.; Gibb, A. G. F.
- Applied Ergonomics, Vol. 36, Issue 4
A vision-based motion capture and recognition framework for behavior-based safety management
journal, November 2013
- Han, SangUk; Lee, SangHyun
- Automation in Construction, Vol. 35
Classification in the Presence of Label Noise: A Survey
journal, May 2014
- Frenay, Benoit; Verleysen, Michel
- IEEE Transactions on Neural Networks and Learning Systems, Vol. 25, Issue 5
Tracking-based 3D human skeleton extraction from stereo video camera toward an on-site safety and ergonomic analysis
journal, July 2016
- Liu, Meiyin; Han, SangUk; Lee, SangHyun
- Construction Innovation, Vol. 16, Issue 3
Continuous localization of construction workers via integration of detection and tracking
journal, December 2016
- Park, Man-Woo; Brilakis, Ioannis
- Automation in Construction, Vol. 72
The Value of Crowdsourcing: Can Users Really Compete with Professionals in Generating New Product Ideas?: The Value of Crowdsourcing
journal, January 2012
- Poetz, Marion K.; Schreier, Martin
- Journal of Product Innovation Management, Vol. 29, Issue 2
Evaluation on crowdsourcing research: Current status and future direction
journal, April 2012
- Zhao, Yuxiang; Zhu, Qinghua
- Information Systems Frontiers, Vol. 16, Issue 3
EcoRacer: Game-Based Optimal Electric Vehicle Design and Driver Control Using Human Players
journal, May 2016
- Ren, Yi; Bayrak, Alparslan Emrah; Papalambros, Panos Y.
- Journal of Mechanical Design, Vol. 138, Issue 6
Computer vision techniques for construction safety and health monitoring
journal, April 2015
- Seo, JoonOh; Han, SangUk; Lee, SangHyun
- Advanced Engineering Informatics, Vol. 29, Issue 2
Crowdsourcing in Computer Vision
journal, January 2014
- Kovashka, Adriana; Russakovsky, Olga; Fei-Fei, Li
- Foundations and Trends® in Computer Graphics and Vision, Vol. 10, Issue 3
The Path to Path-Traced Movies
journal, January 2014
- Christensen, Per H.; Jarosz, Wojciech
- Foundations and Trends® in Computer Graphics and Vision, Vol. 10, Issue 2
Proactive behavior-based system for controlling safety risks in urban highway construction megaprojects
journal, November 2018
- Li, Yongkui; Hu, Yi; Xia, Bo
- Automation in Construction, Vol. 95
Crowdsourcing Construction Activity Analysis from Jobsite Video Streams
journal, November 2015
- Liu, Kaijian; Golparvar-Fard, Mani
- Journal of Construction Engineering and Management, Vol. 141, Issue 11
A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure
journal, April 2015
- Koch, Christian; Georgieva, Kristina; Kasireddy, Varun
- Advanced Engineering Informatics, Vol. 29, Issue 2
Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors
journal, July 2013
- Memarzadeh, Milad; Golparvar-Fard, Mani; Niebles, Juan Carlos
- Automation in Construction, Vol. 32
Incentives for Effort in Crowdsourcing Using the Peer Truth Serum
journal, July 2016
- Radanovic, Goran; Faltings, Boi; Jurca, Radu
- ACM Transactions on Intelligent Systems and Technology, Vol. 7, Issue 4