Advanced Search

Browse by Discipline

Scientific Societies

E-print Alerts

Add E-prints

E-print Network

  Advanced Search  

Cue Integration in Categorical Tasks: Insights from Audio-Visual Speech Perception

Summary: Cue Integration in Categorical Tasks: Insights from
Audio-Visual Speech Perception
Vikranth Rao Bejjanki*, Meghan ClayardsĄ
, David C. Knill, Richard N. Aslin
Department of Brain and Cognitive Sciences, University of Rochester, Rochester, New York, United States of America
Previous cue integration studies have examined continuous perceptual dimensions (e.g., size) and have shown that human
cue integration is well described by a normative model in which cues are weighted in proportion to their sensory reliability,
as estimated from single-cue performance. However, this normative model may not be applicable to categorical perceptual
dimensions (e.g., phonemes). In tasks defined over categorical perceptual dimensions, optimal cue weights should depend
not only on the sensory variance affecting the perception of each cue but also on the environmental variance inherent in
each task-relevant category. Here, we present a computational and experimental investigation of cue integration in a
categorical audio-visual (articulatory) speech perception task. Our results show that human performance during audio-visual
phonemic labeling is qualitatively consistent with the behavior of a Bayes-optimal observer. Specifically, we show that the
participants in our task are sensitive, on a trial-by-trial basis, to the sensory uncertainty associated with the auditory and
visual cues, during phonemic categorization. In addition, we show that while sensory uncertainty is a significant factor in
determining cue weights, it is not the only one and participants' performance is consistent with an optimal model in which
environmental, within category variability also plays a role in determining cue weights. Furthermore, we show that in our
task, the sensory variability affecting the visual modality during cue-combination is not well estimated from single-cue
performance, but can be estimated from multi-cue performance. The findings and computational principles described here


Source: Aslin, Richard N. - Department of Brain and Cognitive Sciences, University of Rochester


Collections: Biology and Medicine