A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy

Kell, Alexander J.E.; Yamins, Daniel L.K.; Shook, Erica N.; Norman-Haignere, Sam V.; McDermott, Josh H.

doi:10.1016/j.neuron.2018.03.044

A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy

Journal Article · Thu Apr 19 00:00:00 EDT 2018 · Neuron

DOI:https://doi.org/10.1016/j.neuron.2018.03.044· OSTI ID:1538638

Kell, Alexander J.E. ^[1]; Yamins, Daniel L.K. ^[2]; Shook, Erica N. ^[3]; Norman-Haignere, Sam V. ^[3]; McDermott, Josh H. ^[4]

Massachusetts Institute of Technology (MIT), Cambridge, MA (United States); DOE/OSTI
Stanford University, CA (United States); Stanford Neurosciences Institute, Stanford, CA (United States)
Massachusetts Institute of Technology (MIT), Cambridge, MA (United States)
Massachusetts Institute of Technology (MIT), Cambridge, MA (United States); Harvard University, Cambridge, MA (United States)

A core goal of auditory neuroscience is to build quantitative models that predict cortical responses to natural sounds. Reasoning that a complete model of auditory cortex must solve ecologically relevant tasks, we optimized hierarchical neural networks for speech and music recognition. The best-performing network contained separate music and speech pathways following early shared processing, potentially replicating human cortical organization. The network performed both tasks as well as humans and exhibited human-like errors despite not being optimized to do so, suggesting common constraints on network and human performance. The network predicted fMRI voxel responses substantially better than traditional spectrotemporal filter models throughout auditory cortex. It also provided a quantitative signature of cortical representational hierarchy—primary and non-primary responses were best predicted by intermediate and late network layers, respectively. Here the results suggest that task optimization provides a powerful set of tools for modeling sensory systems.

View Accepted Manuscript (DOE)

Research Organization:: Krell Institute, Ames, IA (United States); Massachusetts Institute of Technology (MIT), Cambridge, MA (United States)

Sponsoring Organization:: USDOE Office of Science (SC); NVIDIA Corporation; National Institutes of Health (NIH); McDonnell Scholar Award; National Science Foundation (NSF)

Grant/Contract Number:: FG02-97ER25308

OSTI ID:: 1538638

Journal Information:: Neuron, Journal Name: Neuron Journal Issue: 3 Vol. 98; ISSN 0896-6273

Publisher:: Cell Press - ElsevierCopyright Statement

Country of Publication:: United States

Language:: English

References (67)

Human Primary Auditory Cortex: Cytoarchitectonic Subdivisions and Mapping into a Spatial Reference System Morosan, P.; Rademacher, J.; Schleicher, A. NeuroImage, Vol. 13, Issue 4 https://doi.org/10.1006/nimg.2000.0715	journal	April 2001
Intrahemispheric cortico-cortical connections of the human auditory cortex Cammoun, Leila; Thiran, Jean Philippe; Griffa, Alessandra Brain Structure and Function, Vol. 220, Issue 6 https://doi.org/10.1007/s00429-014-0872-z	journal	August 2014
Derivation of auditory filter shapes from notched-noise data Glasberg, Brian R.; Moore, Brian C. J. Hearing Research, Vol. 47, Issue 1-2 https://doi.org/10.1016/0378-5955(90)90170-T	journal	August 1990
Serial and parallel processing in the primate auditory cortex revisited Recanzone, Gregg H.; Cohen, Yale E. Behavioural Brain Research, Vol. 206, Issue 1 https://doi.org/10.1016/j.bbr.2009.08.015	journal	January 2010
Music listening engages specific cortical regions within the temporal lobes: Differences between musicians and non-musicians Angulo-Perkins, Arafat; Aubé, William; Peretz, Isabelle Cortex, Vol. 59 https://doi.org/10.1016/j.cortex.2014.07.013	journal	October 2014
Reservoir computing approaches to recurrent neural network training Lukoševičius, Mantas; Jaeger, Herbert Computer Science Review, Vol. 3, Issue 3 https://doi.org/10.1016/j.cosrev.2009.03.005	journal	August 2009
Sound Categories Are Represented as Distributed Patterns in the Human Auditory Cortex Staeren, Noël; Renvall, Hanna; De Martino, Federico Current Biology, Vol. 19, Issue 6 https://doi.org/10.1016/j.cub.2009.01.066	journal	March 2009
Locating the initial stages of speech–sound processing in human temporal cortex Uppenkamp, Stefan; Johnsrude, Ingrid S.; Norris, Dennis NeuroImage, Vol. 31, Issue 3 https://doi.org/10.1016/j.neuroimage.2006.01.004	journal	July 2006
Tonotopic organization of human auditory cortex Humphries, Colin; Liebenthal, Einat; Binder, Jeffrey R. NeuroImage, Vol. 50, Issue 3 https://doi.org/10.1016/j.neuroimage.2010.01.046	journal	April 2010
Encoding and decoding in fMRI Naselaris, Thomas; Kay, Kendrick N.; Nishimoto, Shinji NeuroImage, Vol. 56, Issue 2 https://doi.org/10.1016/j.neuroimage.2010.07.073	journal	May 2011
FreeSurfer Fischl, Bruce NeuroImage, Vol. 62, Issue 2 https://doi.org/10.1016/j.neuroimage.2012.01.021	journal	August 2012
Seeing it all: Convolutional network layers map the function of the human visual system Eickenberg, Michael; Gramfort, Alexandre; Varoquaux, Gaël NeuroImage, Vol. 152 https://doi.org/10.1016/j.neuroimage.2016.10.001	journal	May 2017
Reduction of Information Redundancy in the Ascending Auditory Pathway Chechik, Gal; Anderson, Michael J.; Bar-Yosef, Omer Neuron, Vol. 51, Issue 3 https://doi.org/10.1016/j.neuron.2006.06.030	journal	August 2006
Sound Texture Perception via Statistics of the Auditory Periphery: Evidence from Sound Synthesis McDermott, Josh H.; Simoncelli, Eero P. Neuron, Vol. 71, Issue 5 https://doi.org/10.1016/j.neuron.2011.06.032	journal	September 2011
Distinct Cortical Pathways for Music and Speech Revealed by Hypothesis-Free Voxel Decomposition Norman-Haignere, Sam; Kanwisher, Nancy G.; McDermott, Josh H. Neuron, Vol. 88, Issue 6 https://doi.org/10.1016/j.neuron.2015.11.035	journal	December 2015
A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons Zipser, David; Andersen, Richard A. Nature, Vol. 331, Issue 6158 https://doi.org/10.1038/331679a0	journal	February 1988
Network model of shape-from-shading: neural function arises from both receptive and projective fields Lehky, Sidney R.; Sejnowski, Terrence J. Nature, Vol. 333, Issue 6172 https://doi.org/10.1038/333452a0	journal	June 1988
Deep learning LeCun, Yann; Bengio, Yoshua; Hinton, Geoffrey Nature, Vol. 521, Issue 7553 https://doi.org/10.1038/nature14539	journal	May 2015
Natural speech reveals the semantic maps that tile human cerebral cortex Huth, Alexander G.; de Heer, Wendy A.; Griffiths, Thomas L. Nature, Vol. 532, Issue 7600 https://doi.org/10.1038/nature17637	journal	April 2016
Unraveling the principles of auditory cortical processing: can we learn from the visual system? King, Andrew J.; Nelken, Israel Nature Neuroscience, Vol. 12, Issue 6 https://doi.org/10.1038/nn.2308	journal	May 2009
Maps and streams in the auditory cortex: nonhuman primates illuminate human speech processing Rauschecker, Josef P.; Scott, Sophie K. Nature Neuroscience, Vol. 12, Issue 6 https://doi.org/10.1038/nn.2331	journal	May 2009
Categorical speech representation in human superior temporal gyrus Chang, Edward F.; Rieger, Jochem W.; Johnson, Keith Nature Neuroscience, Vol. 13, Issue 11 https://doi.org/10.1038/nn.2641	journal	October 2010
Summary statistics in auditory perception McDermott, Josh H.; Schemitsch, Michael; Simoncelli, Eero P. Nature Neuroscience, Vol. 16, Issue 4 https://doi.org/10.1038/nn.3347	journal	February 2013
The cortical analysis of speech-specific temporal structure revealed by responses to sound quilts Overath, Tobias; McDermott, Josh H.; Zarate, Jean Mary Nature Neuroscience, Vol. 18, Issue 6 https://doi.org/10.1038/nn.4021	journal	May 2015
Using goal-driven deep learning models to understand sensory cortex Yamins, Daniel L. K.; DiCarlo, James J. Nature Neuroscience, Vol. 19, Issue 3 https://doi.org/10.1038/nn.4244	journal	February 2016
Hierarchical and asymmetric temporal sensitivity in human auditory cortices Boemio, Anthony; Fromm, Stephen; Braun, Allen Nature Neuroscience, Vol. 8, Issue 3 https://doi.org/10.1038/nn1409	journal	February 2005
Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence Cichy, Radoslaw Martin; Khosla, Aditya; Pantazis, Dimitrios Scientific Reports, Vol. 6, Issue 1 https://doi.org/10.1038/srep27755	journal	June 2016
Spectro-temporal modulation transfer function of single voxels in the human auditory cortex measured with high-resolution fMRI Schonwiesner, M.; Zatorre, R. J. Proceedings of the National Academy of Sciences, Vol. 106, Issue 34 https://doi.org/10.1073/pnas.0907682106	journal	August 2009
Neural latencies across auditory cortex of macaque support a dorsal stream supramodal timing advantage in primates Camalier, C. R.; D'Angelo, W. R.; Sterbing-D'Angelo, S. J. Proceedings of the National Academy of Sciences, Vol. 109, Issue 44 https://doi.org/10.1073/pnas.1206387109	journal	October 2012
Performance-optimized hierarchical models predict neural responses in higher visual cortex Yamins, D. L. K.; Hong, H.; Cadieu, C. F. Proceedings of the National Academy of Sciences, Vol. 111, Issue 23 https://doi.org/10.1073/pnas.1403112111	journal	May 2014
Subdivisions of auditory cortex and processing streams in primates Kaas, J. H.; Hackett, T. A. Proceedings of the National Academy of Sciences, Vol. 97, Issue 22 https://doi.org/10.1073/pnas.97.22.11793	journal	October 2000
Neural Substrates of Phonemic Perception Liebenthal, Einat; Binder, Jeffrey R.; Spitzer, Stephanie M. Cerebral Cortex, Vol. 15, Issue 10 https://doi.org/10.1093/cercor/bhi040	journal	February 2005
Hierarchical Organization of Human Auditory Cortex: Evidence from Acoustic Invariance in the Response to Intelligible Speech Okada, Kayoko; Rong, Feng; Venezia, Jon Cerebral Cortex, Vol. 20, Issue 10 https://doi.org/10.1093/cercor/bhp318	journal	January 2010
Speech versus Song: Multiple Pitch-Sensitive Areas Revealed by a Naturally Occurring Musical Illusion Tierney, Adam; Dick, Fred; Deutsch, Diana Cerebral Cortex, Vol. 23, Issue 2 https://doi.org/10.1093/cercor/bhs003	journal	February 2012
Hierarchical Organization of Auditory and Motor Representations in Speech Perception: Evidence from Searchlight Similarity Analysis Evans, Samuel; Davis, Matthew H. Cerebral Cortex, Vol. 25, Issue 12 https://doi.org/10.1093/cercor/bhv136	journal	July 2015
Learning Transferable Architectures for Scalable Image Recognition Zoph, Barret; Vasudevan, Vijay; Shlens, Jonathon 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2018.00907	conference	June 2018
CNN Features Off-the-Shelf: An Astounding Baseline for Recognition Razavian, Ali Sharif; Azizpour, Hossein; Sullivan, Josephine 2014 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) https://doi.org/10.1109/CVPRW.2014.131	conference	June 2014
Tandem-STRAIGHT: A temporally stable power spectral representation for periodic signals and applications to interference-free spectrum, F0, and aperiodicity estimation Kawahara, H.; Morise, M.; Takahashi, T. 2008 IEEE International Conference on Acoustics, Speech and Signal Processing https://doi.org/10.1109/ICASSP.2008.4518514	conference	March 2008
CNN architectures for large-scale audio classification Hershey, Shawn; Chaudhuri, Sourish; Ellis, Daniel P. W. 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) https://doi.org/10.1109/ICASSP.2017.7952132	conference	March 2017
Robust Formant Tracking for Continuous Speech With Speaker Variability Mustafa, K.; Bruce, I. C. IEEE Transactions on Audio, Speech and Language Processing, Vol. 14, Issue 2 https://doi.org/10.1109/TSA.2005.855840	journal	March 2006
Multiresolution spectrotemporal analysis of complex sounds Chi, Taishih; Ru, Powen; Shamma, Shihab A. The Journal of the Acoustical Society of America, Vol. 118, Issue 2 https://doi.org/10.1121/1.1945807	journal	August 2005
Modeling auditory processing of amplitude modulation. I. Detection and masking with narrow-band carriers Dau, Torsten; Kollmeier, Birger; Kohlrausch, Armin The Journal of the Acoustical Society of America, Vol. 102, Issue 5 https://doi.org/10.1121/1.420344	journal	November 1997
"Who" Is Saying "What"? Brain-Based Decoding of Human Voice and Speech Formisano, Elia; De Martino, Federico; Bonte, Milene Science, Vol. 322, Issue 5903 https://doi.org/10.1126/science.1164318	journal	November 2008
Phonetic Feature Encoding in Human Superior Temporal Gyrus Mesgarani, Nima; Cheung, Connie; Johnson, Keith Science, Vol. 343, Issue 6174 https://doi.org/10.1126/science.1245994	journal	February 2014
Processing of complex sounds in the macaque nonprimary auditory cortex Rauschecker, J.; Tian, B.; Hauser, M. Science, Vol. 268, Issue 5207 https://doi.org/10.1126/science.7701330	journal	April 1995
Spectrotemporal Receptive Fields in the Lemniscal Auditory Thalamus and Cortex Miller, Lee M.; Escabí, Monty A.; Read, Heather L. Journal of Neurophysiology, Vol. 87, Issue 1 https://doi.org/10.1152/jn.00395.2001	journal	January 2002
Receptive field dimensionality increases from the auditory midbrain to cortex Atencio, Craig A.; Sharpee, Tatyana O.; Schreiner, Christoph E. Journal of Neurophysiology, Vol. 107, Issue 10 https://doi.org/10.1152/jn.01025.2011	journal	May 2012
Spectro-Temporal Response Field Characterization With Dynamic Ripples in Ferret Primary Auditory Cortex Depireux, Didier A.; Simon, Jonathan Z.; Klein, David J. Journal of Neurophysiology, Vol. 85, Issue 3 https://doi.org/10.1152/jn.2001.85.3.1220	journal	March 2001
Hierarchical Organization of the Human Auditory Cortex Revealed by Functional Magnetic Resonance Imaging Wessinger, C. M.; VanMeter, J.; Tian, B. Journal of Cognitive Neuroscience, Vol. 13, Issue 1 https://doi.org/10.1162/089892901564108	journal	January 2001
Learning Midlevel Auditory Codes from Natural Sound Statistics Młynarski, Wiktor; McDermott, Josh H. Neural Computation, Vol. 30, Issue 3 https://doi.org/10.1162/neco_a_01048	journal	March 2018
The Psychophysics Toolbox Brainard, David H. Spatial Vision, Vol. 10, Issue 4 https://doi.org/10.1163/156856897X00357	journal	January 1997
A High-Throughput Screening Approach to Discovering Good Forms of Biologically Inspired Visual Representation Pinto, Nicolas; Doukhan, David; DiCarlo, James J. PLoS Computational Biology, Vol. 5, Issue 11 https://doi.org/10.1371/journal.pcbi.1000579	journal	November 2009
Sparse Codes for Speech Predict Spectrotemporal Receptive Fields in the Inferior Colliculus Carlson, Nicole L.; Ming, Vivienne L.; DeWeese, Michael Robert PLoS Computational Biology, Vol. 8, Issue 7 https://doi.org/10.1371/journal.pcbi.1002594	journal	July 2012
Music in Our Ears: The Biological Bases of Musical Timbre Perception Patil, Kailash; Pressnitzer, Daniel; Shamma, Shihab PLoS Computational Biology, Vol. 8, Issue 11 https://doi.org/10.1371/journal.pcbi.1002759	journal	November 2012
Encoding of Natural Sounds at Multiple Spectral and Temporal Resolutions in the Human Auditory Cortex Santoro, Roberta; Moerel, Michelle; De Martino, Federico PLoS Computational Biology, Vol. 10, Issue 1 https://doi.org/10.1371/journal.pcbi.1003412	journal	January 2014
Deep Supervised, but Not Unsupervised, Models May Explain IT Cortical Representation Khaligh-Razavi, Seyed-Mahdi; Kriegeskorte, Nikolaus PLoS Computational Biology, Vol. 10, Issue 11 https://doi.org/10.1371/journal.pcbi.1003915	journal	November 2014
Cortical Representation of Natural Complex Sounds: Effects of Acoustic Features and Auditory Object Category Leaver, A. M.; Rauschecker, J. P. Journal of Neuroscience, Vol. 30, Issue 22 https://doi.org/10.1523/JNEUROSCI.0296-10.2010	journal	June 2010
The Consequences of Response Nonlinearities for Interpretation of Spectrotemporal Receptive Fields Christianson, G. Björn; Sahani, Maneesh; Linden, Jennifer F. The Journal of Neuroscience, Vol. 28, Issue 2 https://doi.org/10.1523/JNEUROSCI.1775-07.2007	journal	January 2008
Cortical Pitch Regions in Humans Respond Primarily to Resolved Harmonics and Are Located in Specific Tonotopic Regions of Anterior Auditory Cortex Norman-Haignere, Sam; Kanwisher, Nancy; McDermott, Josh H. The Journal of Neuroscience, Vol. 33, Issue 50 https://doi.org/10.1523/JNEUROSCI.2880-13.2013	journal	December 2013
The Hierarchical Cortical Organization of Human Speech Processing de Heer, Wendy A.; Huth, Alexander G.; Griffiths, Thomas L. The Journal of Neuroscience, Vol. 37, Issue 27 https://doi.org/10.1523/JNEUROSCI.3267-16.2017	journal	June 2017
Deep Neural Networks Reveal a Gradient in the Complexity of Neural Representations across the Ventral Stream Guclu, U.; van Gerven, M. A. J. Journal of Neuroscience, Vol. 35, Issue 27 https://doi.org/10.1523/JNEUROSCI.5023-14.2015	journal	July 2015
Rapid Synaptic Depression Explains Nonlinear Modulation of Spectro-Temporal Tuning in Primary Auditory Cortex by Natural Stimuli David, S. V.; Mesgarani, N.; Fritz, J. B. Journal of Neuroscience, Vol. 29, Issue 11 https://doi.org/10.1523/JNEUROSCI.5249-08.2009	journal	March 2009
The Proof and Measurement of Association between Two Things Spearman, C. The American Journal of Psychology, Vol. 15, Issue 1 https://doi.org/10.2307/1412159	journal	January 1904
The design for the wall street journal-based CSR corpus Paul, Douglas B.; Baker, Janet M. Proceedings of the workshop on Speech and Natural Language - HLT '91 https://doi.org/10.3115/1075527.1075614	conference	January 1992
Measuring the Performance of Neural Models Schoppe, Oliver; Harper, Nicol S.; Willmore, Ben D. B. Frontiers in Computational Neuroscience, Vol. 10 https://doi.org/10.3389/fncom.2016.00010	journal	February 2016
Segregation of Vowels and Consonants in Human Auditory Cortex: Evidence for Distributed Hierarchical Organization Obleser, Jonas; Leaver, Amber M.; VanMeter, John Frontiers in Psychology, Vol. 1 https://doi.org/10.3389/fpsyg.2010.00232	journal	January 2010
Headphone screening to facilitate web-based auditory experiments Woods, Kevin J. P.; Siegel, Max H.; Traer, James Attention, Perception, & Psychophysics, Vol. 79, Issue 7 https://doi.org/10.3758/s13414-017-1361-2	journal	July 2017

Cited By (2)

Deep neuroethology of a virtual rodent Merel, Josh; Aldarondo, Diego; Marshall, Jesse arXiv https://doi.org/10.48550/arxiv.1911.09451	preprint	January 2019
Predictive Coding Can Do Exact Backpropagation on Convolutional and Recurrent Neural Networks Salvatori, Tommaso; Song, Yuhang; Lukasiewicz, Thomas arXiv https://doi.org/10.48550/arxiv.2103.03725	preprint	January 2021

Similar Records

Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions

Journal Article · Tue Dec 12 23:00:00 EST 2023 · PLoS Biology (Online) · OSTI ID:2229937

Invariance to background noise as a signature of non-primary auditory cortex

Journal Article · Mon Sep 02 00:00:00 EDT 2019 · Nature Communications · OSTI ID:1610345

Related Subjects

59 BASIC BIOLOGICAL SCIENCES
auditory cortex
convolutional neural network
deep learning
deep neural network
encoding models
fMRI
hierarchy
human auditory cortex
natural sounds
word recognition

A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy

Citation Formats

References (67)

Cited By (2)

Similar Records

Related Subjects