Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex

Livezey, Jesse A.; Bouchard, Kristofer E.; Chang, Edward F.

doi:10.1371/journal.pcbi.1007091

Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex

Journal Article · Mon Sep 16 00:00:00 EDT 2019 · PLoS Computational Biology (Online)

DOI:https://doi.org/10.1371/journal.pcbi.1007091· OSTI ID:1572056

^[1]; ^[1]; ^[2]

Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
Univ. of California, San Francisco, CA (United States)

A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.

View Accepted Manuscript (DOE)

Research Organization:: Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)

Sponsoring Organization:: USDOE Office of Science (SC)

Grant/Contract Number:: AC02-05CH11231

OSTI ID:: 1572056

Journal Information:: PLoS Computational Biology (Online), Journal Name: PLoS Computational Biology (Online) Journal Issue: 9 Vol. 15; ISSN 1553-7358

Publisher:: Public Library of ScienceCopyright Statement

Country of Publication:: United States

Language:: English

References (62)

Cortical gamma responses: Searching high and low Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J. International Journal of Psychophysiology, Vol. 79, Issue 1 https://doi.org/10.1016/j.ijpsycho.2010.10.013	journal	January 2011
Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids Ramsey, N. F.; Salari, E.; Aarnoutse, E. J. NeuroImage, Vol. 180 https://doi.org/10.1016/j.neuroimage.2017.10.011	journal	October 2018
Let the Rhythm Guide You: Non-invasive Tracking of Cortical Communication Channels Gross, Joachim Neuron, Vol. 89, Issue 2 https://doi.org/10.1016/j.neuron.2016.01.001	journal	January 2016
Brain–computer interfaces for communication and control Wolpaw, Jonathan R.; Birbaumer, Niels; McFarland, Dennis J. Clinical Neurophysiology, Vol. 113, Issue 6 https://doi.org/10.1016/s1388-2457(02)00057-3	journal	June 2002
A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons Zipser, David; Andersen, Richard A. Nature, Vol. 331, Issue 6158 https://doi.org/10.1038/331679a0	journal	February 1988
Functional organization of human sensorimotor cortex for speech articulation Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith Nature, Vol. 495, Issue 7441 https://doi.org/10.1038/nature11911	journal	February 2013
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T. Nature Biotechnology, Vol. 33, Issue 8 https://doi.org/10.1038/nbt.3300	journal	July 2015
Perceptual restoration of masked speech in human cortex Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J. Nature Communications, Vol. 7, Issue 1 https://doi.org/10.1038/ncomms13619	journal	December 2016
Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P. Nature Communications, Vol. 6, Issue 1 https://doi.org/10.1038/ncomms8169	journal	May 2015
Propagating waves mediate information transfer in the motor cortex Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G. Nature Neuroscience, Vol. 9, Issue 12 https://doi.org/10.1038/nn1802	journal	November 2006
The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes Buzsáki, György; Anastassiou, Costas A.; Koch, Christof Nature Reviews Neuroscience, Vol. 13, Issue 6 https://doi.org/10.1038/nrn3241	journal	May 2012
Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies Canolty, R. T.; Ganguly, K.; Kennerley, S. W. Proceedings of the National Academy of Sciences, Vol. 107, Issue 40 https://doi.org/10.1073/pnas.1008306107	journal	September 2010
Direct classification of all American English phonemes using signals from functional speech motor cortex Mugler, Emily M.; Patton, James L.; Flint, Robert D. Journal of Neural Engineering, Vol. 11, Issue 3 https://doi.org/10.1088/1741-2560/11/3/035015	journal	May 2014
Decoding flexion of individual fingers using electrocorticographic signals in humans Kubánek, J.; Miller, K. J.; Ojemann, J. G. Journal of Neural Engineering, Vol. 6, Issue 6 https://doi.org/10.1088/1741-2560/6/6/066001	journal	October 2009
Decoding spoken words using local field potentials recorded from the cortical surface Kellis, Spencer; Miller, Kai; Thomson, Kyle Journal of Neural Engineering, Vol. 7, Issue 5 https://doi.org/10.1088/1741-2560/7/5/056007	journal	September 2010
Using the electrocorticographic speech network to control a brain–computer interface in humans Leuthardt, Eric C.; Gaona, Charles; Sharma, Mohit Journal of Neural Engineering, Vol. 8, Issue 3 https://doi.org/10.1088/1741-2560/8/3/036004	journal	April 2011
Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement Wulsin, D. F.; Gupta, J. R.; Mani, R. Journal of Neural Engineering, Vol. 8, Issue 3 https://doi.org/10.1088/1741-2560/8/3/036015	journal	April 2011
Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans Pei, Xiaomei; Barbour, Dennis L.; Leuthardt, Eric C. Journal of Neural Engineering, Vol. 8, Issue 4 https://doi.org/10.1088/1741-2560/8/4/046028	journal	July 2011
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event- related desynchronization Crone, N. Brain, Vol. 121, Issue 12 https://doi.org/10.1093/brain/121.12.2271	journal	December 1998
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band Crone, N. Brain, Vol. 121, Issue 12 https://doi.org/10.1093/brain/121.12.2301	journal	December 1998
Somatic Motor and Sensory Representation in the Cerebral Cortex of man as Studied by Electrical Stimulation Penfield, Wilder; Boldrey, Edwin Brain, Vol. 60, Issue 4 https://doi.org/10.1093/brain/60.4.389	journal	January 1937
Networks for approximation and learning Poggio, T.; Girosi, F. Proceedings of the IEEE, Vol. 78, Issue 9 https://doi.org/10.1109/5.58326	journal	January 1990
Emergence of Invariance and Disentanglement in Deep Representations Achille, Alessandro; Soatto, Stefano 2018 Information Theory and Applications Workshop (ITA) https://doi.org/10.1109/ita.2018.8503149	conference	February 2018
Single-trial spike trains in parietal cortex reveal discrete steps during decision-making Latimer, K. W.; Yates, J. L.; Meister, M. L. R. Science, Vol. 349, Issue 6244 https://doi.org/10.1126/science.aaa4056	journal	July 2015
Note on Information Transfer Rates in Human Communication Reed, Charlotte M.; Durlach, Nathaniel I. Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5 https://doi.org/10.1162/105474698565893	journal	October 1998
Spike-triggered neural characterization Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C. Journal of Vision, Vol. 6, Issue 4 https://doi.org/10.1167/6.4.13	journal	February 2006
Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex Ray, Supratim; Maunsell, John H. R. PLoS Biology, Vol. 9, Issue 4 https://doi.org/10.1371/journal.pbio.1000610	journal	April 2011
Power-Law Scaling in the Brain Surface Electric Potential Miller, Kai J.; Sorensen, Larry B.; Ojemann, Jeffrey G. PLoS Computational Biology, Vol. 5, Issue 12 https://doi.org/10.1371/journal.pcbi.1000609	journal	December 2009
A Wireless Brain-Machine Interface for Real-Time Speech Synthesis Guenther, Frank H.; Brumberg, Jonathan S.; Wright, E. Joseph PLoS ONE, Vol. 4, Issue 12 https://doi.org/10.1371/journal.pone.0008218	journal	December 2009
Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier Steyrl, David; Scherer, Reinhold; Faller, Josef Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1 https://doi.org/10.1515/bmt-2014-0117	journal	February 2016
Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex Bouchard, K. E.; Chang, E. F. Journal of Neuroscience, Vol. 34, Issue 38 https://doi.org/10.1523/jneurosci.1219-14.2014	journal	September 2014
Spectral Changes in Cortical Surface Potentials during Motor Movement Miller, K. J.; Leuthardt, E. C.; Schalk, G. Journal of Neuroscience, Vol. 27, Issue 9 https://doi.org/10.1523/jneurosci.3886-06.2007	journal	February 2007
Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans Miller, K. J.; Zanos, S.; Fetz, E. E. Journal of Neuroscience, Vol. 29, Issue 10 https://doi.org/10.1523/jneurosci.5506-08.2009	journal	March 2009
Electrocorticographic representations of segmental features in continuous speech Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter Frontiers in Human Neuroscience, Vol. 09 https://doi.org/10.3389/fnhum.2015.00097	journal	February 2015
Scikit-learn: Machine Learning in Python Pedregosa, Fabian; Varoquaux, Gaël; Gramfort, Alexandre arXiv https://doi.org/10.48550/arxiv.1201.0490	text	January 2012
Convergent Learning: Do different neural networks learn the same representations? Li, Yixuan; Yosinski, Jason; Clune, Jeff arXiv https://doi.org/10.48550/arxiv.1511.07543	preprint	January 2015
Opening the Black Box of Deep Neural Networks via Information Shwartz-Ziv, Ravid; Tishby, Naftali arXiv https://doi.org/10.48550/arxiv.1703.00810	preprint	January 2017
Brain-to-text: Decoding spoken phrases from phone representations in the brain Herff, C.; Heger, D.; De Pesters, A. Karlsruhe https://doi.org/10.5445/ir/1000049750	text	January 2015
Visualizing and Understanding Convolutional Networks Zeiler, Matthew D.; Fergus, Rob Computer Vision – ECCV 2014 https://doi.org/10.1007/978-3-319-10590-1_53	book	January 2014
Comparison of neuronal responses in primate inferior-temporal cortex and feed-forward deep neural network model with regard to information processing of faces Matsumoto, Narihisa; Mototake, Yoh-ichi; Kawano, Kenji Journal of Computational Neuroscience, Vol. 49, Issue 3 https://doi.org/10.1007/s10827-021-00778-5	journal	February 2021
Brain–computer interfaces for communication and control Wolpaw, Jonathan R.; Birbaumer, Niels; McFarland, Dennis J. Clinical Neurophysiology, Vol. 113, Issue 6 https://doi.org/10.1016/S1388-2457(02)00057-3	journal	June 2002
Event-related EEG/MEG synchronization and desynchronization: basic principles Pfurtscheller, G.; Lopes da Silva, F. H. Clinical Neurophysiology, Vol. 110, Issue 11 https://doi.org/10.1016/S1388-2457(99)00141-8	journal	November 1999
Beta-band oscillations—signalling the status quo? Engel, Andreas K.; Fries, Pascal Current Opinion in Neurobiology, Vol. 20, Issue 2 https://doi.org/10.1016/j.conb.2010.02.015	journal	April 2010
Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas Michalareas, Georgios; Vezoli, Julien; van Pelt, Stan Neuron, Vol. 89, Issue 2 https://doi.org/10.1016/j.neuron.2015.12.018	journal	January 2016
Performance-optimized hierarchical models predict neural responses in higher visual cortex Yamins, D. L. K.; Hong, H.; Cadieu, C. F. Proceedings of the National Academy of Sciences, Vol. 111, Issue 23 https://doi.org/10.1073/pnas.1403112111	journal	May 2014
Enhanced Higgs Boson to τ + τ − Search with Deep Learning Baldi, P.; Sadowski, P.; Whiteson, D. Physical Review Letters, Vol. 114, Issue 11 https://doi.org/10.1103/PhysRevLett.114.111801	journal	March 2015
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) https://doi.org/10.1109/CVPR.2016.90	conference	June 2016
Feature extraction with stacked autoencoders for epileptic seizure detection Supratak, Akara 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) https://doi.org/10.1109/EMBC.2014.6944546	conference	August 2014
Pattern learning with deep neural networks in EMG-based speech recognition Wand, Michael; Schultz, Tanja 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) https://doi.org/10.1109/EMBC.2014.6944550	conference	August 2014
Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography Bouchard, Kristofer E.; Chang, Edward F. 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC) https://doi.org/10.1109/EMBC.2014.6945185	conference	August 2014
Emergence of Invariance and Disentanglement in Deep Representations Achille, Alessandro; Soatto, Stefano 2018 Information Theory and Applications Workshop (ITA) https://doi.org/10.1109/ITA.2018.8503149	conference	February 2018
Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates Carmena, Jose M.; Lebedev, Mikhail A.; Crist, Roy E. PLoS Biology, Vol. 1, Issue 2 https://doi.org/10.1371/journal.pbio.0000042	journal	October 2003
Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex Bouchard, K. E.; Chang, E. F. Journal of Neuroscience, Vol. 34, Issue 38 https://doi.org/10.1523/JNEUROSCI.1219-14.2014	journal	September 2014
Spectral-Temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds Theunissen, Frédéric E.; Sen, Kamal; Doupe, Allison J. The Journal of Neuroscience, Vol. 20, Issue 6 https://doi.org/10.1523/JNEUROSCI.20-06-02315.2000	journal	March 2000
Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans Manning, J. R.; Jacobs, J.; Fried, I. Journal of Neuroscience, Vol. 29, Issue 43 https://doi.org/10.1523/JNEUROSCI.2041-09.2009	journal	October 2009
Do We Know What the Early Visual System Does? Carandini, M. Journal of Neuroscience, Vol. 25, Issue 46 https://doi.org/10.1523/JNEUROSCI.3726-05.2005	journal	November 2005
Spectral Changes in Cortical Surface Potentials during Motor Movement Miller, K. J.; Leuthardt, E. C.; Schalk, G. Journal of Neuroscience, Vol. 27, Issue 9 https://doi.org/10.1523/JNEUROSCI.3886-06.2007	journal	February 2007
Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans Miller, K. J.; Zanos, S.; Fetz, E. E. Journal of Neuroscience, Vol. 29, Issue 10 https://doi.org/10.1523/JNEUROSCI.5506-08.2009	journal	March 2009
Speech reconstruction from human auditory cortex with deep neural networks Yang, Minda; Sheth, Sameer A.; Schevon, Catherine A. Interspeech 2015 https://doi.org/10.21437/Interspeech.2015-294	conference	September 2015
Exploring how deep neural networks form phonemic categories Nagamine, Tasha; Seltzer, Michael L.; Mesgarani, Nima Interspeech 2015 https://doi.org/10.21437/Interspeech.2015-422	conference	September 2015
Brain-to-text: decoding spoken phrases from phone representations in the brain Herff, Christian; Heger, Dominic; de Pesters, Adriana Frontiers in Neuroscience, Vol. 9 https://doi.org/10.3389/fnins.2015.00217	journal	June 2015
Deep Residual Learning for Image Recognition He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing arXiv https://doi.org/10.48550/arxiv.1512.03385	preprint	January 2015

Cited By (3)

Speech synthesis from ECoG using densely connected 3D convolutional neural networks Angrick, Miguel; Herff, Christian; Mugler, Emily Journal of Neural Engineering, Vol. 16, Issue 3 https://doi.org/10.1088/1741-2552/ab0c59	journal	April 2019
Decoding Movement From Electrocorticographic Activity: A Review Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander Frontiers in Neuroinformatics, Vol. 13 https://doi.org/10.3389/fninf.2019.00074	journal	December 2019
Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis Stavisky, Sergey D.; Willett, Francis R.; Wilson, Guy H. eLife, Vol. 8 https://doi.org/10.7554/elife.46015	journal	December 2019

Similar Records

A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy

Journal Article · Wed Apr 18 20:00:00 EDT 2018 · Neuron · OSTI ID:1538638

Deep learning approaches for neural decoding across architectures and recording modalities

Journal Article · Mon Dec 28 19:00:00 EST 2020 · Briefings in Bioinformatics · OSTI ID:1826323

Human Sensorimotor Cortex Control of Directly Measured Vocal Tract Movements during Vowel Production

Journal Article · Tue Mar 20 20:00:00 EDT 2018 · Journal of Neuroscience · OSTI ID:1485087

Related Subjects

60 APPLIED LIFE SCIENCES

Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex

Citation Formats

References (62)

Cited By (3)

Similar Records

Related Subjects