Toward the validation of crowdsourced experiments for lightness perception

Stark, Emily N.; Turton, Terece L.; Miller, Jonah; Barenholtz, Elan; Hong, Sang; Bujack, Roxana

doi:10.1371/journal.pone.0315853

Toward the validation of crowdsourced experiments for lightness perception

Journal Article · Mon Dec 23 00:00:00 EST 2024 · PLoS ONE

DOI:https://doi.org/10.1371/journal.pone.0315853· OSTI ID:2483632

^[1]; Turton, Terece L. ^[2]; Miller, Jonah ^[2]; Barenholtz, Elan ^[3]; Hong, Sang ^[3]; Bujack, Roxana ^[2]

Los Alamos National Laboratory (LANL), Los Alamos, NM (United States); Florida Atlantic Univ., Boca Raton, FL (United States)
Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)
Florida Atlantic Univ., Boca Raton, FL (United States)

Crowdsource platforms have been used to study a range of perceptual stimuli such as the graphical perception of scatterplots and various aspects of human color perception. Given the lack of control over a crowdsourced participant’s experimental setup, there are valid concerns on the use of crowdsourcing for color studies as the perception of the stimuli is highly dependent on the stimulus presentation. Here, we propose that the error due to a crowdsourced experimental design can be effectively averaged out because the crowdsourced experiment can be accommodated by the Thurstonian model as the convolution of two normal distributions, one that is perceptual in nature and one that captures the error due to variability in stimulus presentation. Based on this, we provide a mathematical estimate for the sample size needed to produce a crowdsourced experiment with the same power as the corresponding in-person study. We tested this claim by replicating a large-scale, crowdsourced study of human lightness perception with a diverse sample with a highly controlled, in-person study with a sample taken from psychology undergraduates. Our claim was supported by the replication of the results from the latter. These findings suggest that, with sufficient sample size, color vision studies may be completed online, giving access to a larger and more representative sample. With this framework at hand, experimentalists have the validation that choosing either many online participants or few in person participants will not sacrifice the impact of their results.

View Journal Article

Research Organization:: Los Alamos National Laboratory (LANL), Los Alamos, NM (United States)

Sponsoring Organization:: USDOE Laboratory Directed Research and Development (LDRD) Program; USDOE National Nuclear Security Administration (NNSA)

Grant/Contract Number:: 89233218CNA000001; 20200512ECR

OSTI ID:: 2483632

Alternate ID(s):: OSTI ID: 2500864

Report Number(s):: LA-UR-22-22545

Journal Information:: PLoS ONE, Vol. 19, Issue 12; ISSN 1932-6203

Publisher:: Public Library of ScienceCopyright Statement

Country of Publication:: United States

Language:: English

References (19)

Modeling Color Difference for Visualization Design Szafir, Danielle Albers IEEE Transactions on Visualization and Computer Graphics, Vol. 24, Issue 1 https://doi.org/10.1109/TVCG.2017.2744359	journal	January 2018
Ranking Visualizations of Correlation Using Weber's Law Harrison, Lane; Yang, Fumeng; Franconeri, Steven IEEE Transactions on Visualization and Computer Graphics, Vol. 20, Issue 12 https://doi.org/10.1109/TVCG.2014.2346979	journal	December 2014
Graphical Perception: Theory, Experimentation, and Application to the Development of Graphical Methods Cleveland, William S.; McGill, Robert Journal of the American Statistical Association, Vol. 79, Issue 387 https://doi.org/10.1080/01621459.1984.10478080	journal	September 1984
College sophomores in the laboratory: Influences of a narrow data base on social psychology's view of human nature. Sears, David O. Journal of Personality and Social Psychology, Vol. 51, Issue 3 https://doi.org/10.1037/0022-3514.51.3.515	journal	September 1986
Maximum likelihood estimation of difference scaling functions for suprathreshold judgments Teti, Emily S.; Turton, Terece L.; Miller, Jonah M. Journal of Vision, Vol. 22, Issue 10 https://doi.org/10.1167/jov.22.10.9	journal	September 2022
Conducting perception research over the internet: a tutorial review Woods, Andy T.; Velasco, Carlos; Levitan, Carmel A. PeerJ, Vol. 3 https://doi.org/10.7717/peerj.1058	journal	July 2015
Memory modulates color appearance Hansen, Thorsten; Olkkonen, Maria; Walter, Sebastian Nature Neuroscience, Vol. 9, Issue 11 https://doi.org/10.1038/nn1794	journal	October 2006
Amazon's Mechanical Turk: A New Source of Inexpensive, Yet High-Quality, Data? Buhrmester, Michael; Kwang, Tracy; Gosling, Samuel D. Perspectives on Psychological Science, Vol. 6, Issue 1 https://doi.org/10.1177/1745691610393980	journal	January 2011
An Easy Way to Show Memory Color Effects Witzel, Christoph i-Perception, Vol. 7, Issue 5 https://doi.org/10.1177/2041669516663751	journal	August 2016
College Sophomores in the Laboratory Redux: Influences of a Narrow Data Base on Social Psychology's View of the Nature of Prejudice Henry, P. J. Psychological Inquiry, Vol. 19, Issue 2 https://doi.org/10.1080/10478400802049936	journal	May 2008
Color sequences for univariate maps: theory, experiments and principles Ware, C. IEEE Computer Graphics and Applications, Vol. 8, Issue 5 https://doi.org/10.1109/38.7760	journal	September 1988
Crowdsourcing graphical perception: using mechanical turk to assess visualization design Heer, Jeffrey; Bostock, Michael Proceedings of the 28th international conference on Human factors in computing systems - CHI '10 https://doi.org/10.1145/1753326.1753357	conference	January 2010
The Perception of Correlation in Scatterplots Rensink, Ronald A.; Baldridge, Gideon Computer Graphics Forum, Vol. 29, Issue 3 https://doi.org/10.1111/j.1467-8659.2009.01694.x	journal	June 2010
The World Wide “Gamma” Moroney, Nathan; Beretta, Giordano Color and Imaging Conference, Vol. 18, Issue 1 https://doi.org/10.2352/CIC.2010.18.1.art00001	journal	January 2010
The non-Riemannian nature of perceptual color space Bujack, Roxana; Teti, Emily; Miller, Jonah Proceedings of the National Academy of Sciences, Vol. 119, Issue 18 https://doi.org/10.1073/pnas.2119753119	journal	April 2022
G*Power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences Faul, Franz; Erdfelder, Edgar; Lang, Albert-Georg Behavior Research Methods, Vol. 39, Issue 2 https://doi.org/10.3758/BF03193146	journal	May 2007
Versus—A tool for evaluating visualizations and image quality using a 2AFC methodology Vuong, Jenny; Kaur, Sandeep; Heinrich, Julian Visual Informatics, Vol. 2, Issue 4 https://doi.org/10.1016/j.visinf.2018.12.003	journal	December 2018
The Generalizability of Survey Experiments Mullinix, Kevin J.; Leeper, Thomas J.; Druckman, James N. Journal of Experimental Political Science, Vol. 2, Issue 2 https://doi.org/10.1017/XPS.2015.19	journal	January 2015
Measuring and Modeling the Feature Detection Threshold Functions of Colormaps Ware, Colin; Turton, Terece L.; Bujack, Roxana IEEE Transactions on Visualization and Computer Graphics, Vol. 25, Issue 9 https://doi.org/10.1109/TVCG.2018.2855742	journal	September 2019

Similar Records

Spatial Brightness Perception of Trichromatic Stimuli

Journal Article · Fri Nov 16 00:00:00 EST 2012 · LEUKOS - The Journal of the Illuminating Engineering Society of North America, 9(2):89-108 · OSTI ID:1130733

Royer, Michael P.; Houser, Kevin W.

Are media reports able to cause somatic symptoms attributed to WiFi radiation? An experimental test of the negative expectation hypothesis

Journal Article · Sat Jul 15 00:00:00 EDT 2017 · Environmental Research · OSTI ID:22708006

Bräscher, Anne-Kathrin; Raymaekers, Koen; Van den Bergh, Omer; +1 more

Related Subjects

97 MATHEMATICS AND COMPUTING

Toward the validation of crowdsourced experiments for lightness perception

Citation Formats

References (19)

Similar Records

Related Subjects