Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex
Abstract
A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speechmore »
- Authors:
-
- Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
- Univ. of California, San Francisco, CA (United States)
- Publication Date:
- Research Org.:
- Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
- Sponsoring Org.:
- USDOE Office of Science (SC)
- OSTI Identifier:
- 1572056
- Grant/Contract Number:
- AC02-05CH11231
- Resource Type:
- Accepted Manuscript
- Journal Name:
- PLoS Computational Biology (Online)
- Additional Journal Information:
- Journal Name: PLoS Computational Biology (Online); Journal Volume: 15; Journal Issue: 9; Journal ID: ISSN 1553-7358
- Publisher:
- Public Library of Science
- Country of Publication:
- United States
- Language:
- English
- Subject:
- 60 APPLIED LIFE SCIENCES
Citation Formats
Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F.. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States: N. p., 2019.
Web. doi:10.1371/journal.pcbi.1007091.
Livezey, Jesse A., Bouchard, Kristofer E., & Chang, Edward F.. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States. https://doi.org/10.1371/journal.pcbi.1007091
Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F.. Mon .
"Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex". United States. https://doi.org/10.1371/journal.pcbi.1007091. https://www.osti.gov/servlets/purl/1572056.
@article{osti_1572056,
title = {Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex},
author = {Livezey, Jesse A. and Bouchard, Kristofer E. and Chang, Edward F.},
abstractNote = {A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.},
doi = {10.1371/journal.pcbi.1007091},
journal = {PLoS Computational Biology (Online)},
number = 9,
volume = 15,
place = {United States},
year = {Mon Sep 16 00:00:00 EDT 2019},
month = {Mon Sep 16 00:00:00 EDT 2019}
}
Web of Science
Works referenced in this record:
Electrocorticographic representations of segmental features in continuous speech
journal, February 2015
- Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter
- Frontiers in Human Neuroscience, Vol. 09
Decoding spoken words using local field potentials recorded from the cortical surface
journal, September 2010
- Kellis, Spencer; Miller, Kai; Thomson, Kyle
- Journal of Neural Engineering, Vol. 7, Issue 5
Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas
journal, January 2016
- Michalareas, Georgios; Vezoli, Julien; van Pelt, Stan
- Neuron, Vol. 89, Issue 2
Using the electrocorticographic speech network to control a brain–computer interface in humans
journal, April 2011
- Leuthardt, Eric C.; Gaona, Charles; Sharma, Mohit
- Journal of Neural Engineering, Vol. 8, Issue 3
Let the Rhythm Guide You: Non-invasive Tracking of Cortical Communication Channels
journal, January 2016
- Gross, Joachim
- Neuron, Vol. 89, Issue 2
Brain-to-text: Decoding spoken phrases from phone representations in the brain
text, January 2015
- Herff, C.; Heger, D.; De Pesters, A.
- Karlsruhe
Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
journal, September 2010
- Canolty, R. T.; Ganguly, K.; Kennerley, S. W.
- Proceedings of the National Academy of Sciences, Vol. 107, Issue 40
Event-related EEG/MEG synchronization and desynchronization: basic principles
journal, November 1999
- Pfurtscheller, G.; Lopes da Silva, F. H.
- Clinical Neurophysiology, Vol. 110, Issue 11
Propagating waves mediate information transfer in the motor cortex
journal, November 2006
- Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G.
- Nature Neuroscience, Vol. 9, Issue 12
Functional organization of human sensorimotor cortex for speech articulation
journal, February 2013
- Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith
- Nature, Vol. 495, Issue 7441
Spectral-Temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds
journal, March 2000
- Theunissen, Frédéric E.; Sen, Kamal; Doupe, Allison J.
- The Journal of Neuroscience, Vol. 20, Issue 6
Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans
journal, July 2011
- Pei, Xiaomei; Barbour, Dennis L.; Leuthardt, Eric C.
- Journal of Neural Engineering, Vol. 8, Issue 4
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015
- Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
- Nature Biotechnology, Vol. 33, Issue 8
Decoding flexion of individual fingers using electrocorticographic signals in humans
journal, October 2009
- Kubánek, J.; Miller, K. J.; Ojemann, J. G.
- Journal of Neural Engineering, Vol. 6, Issue 6
Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier
journal, February 2016
- Steyrl, David; Scherer, Reinhold; Faller, Josef
- Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1
A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons
journal, February 1988
- Zipser, David; Andersen, Richard A.
- Nature, Vol. 331, Issue 6158
Note on Information Transfer Rates in Human Communication
journal, October 1998
- Reed, Charlotte M.; Durlach, Nathaniel I.
- Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5
Spectral Changes in Cortical Surface Potentials during Motor Movement
journal, February 2007
- Miller, K. J.; Leuthardt, E. C.; Schalk, G.
- Journal of Neuroscience, Vol. 27, Issue 9
Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans
journal, October 2009
- Manning, J. R.; Jacobs, J.; Fried, I.
- Journal of Neuroscience, Vol. 29, Issue 43
Spike-triggered neural characterization
journal, February 2006
- Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C.
- Journal of Vision, Vol. 6, Issue 4
Brain–computer interfaces for communication and control
journal, June 2002
- Wolpaw, Jonathan R.; Birbaumer, Niels; McFarland, Dennis J.
- Clinical Neurophysiology, Vol. 113, Issue 6
Networks for approximation and learning
journal, January 1990
- Poggio, T.; Girosi, F.
- Proceedings of the IEEE, Vol. 78, Issue 9
The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
journal, May 2012
- Buzsáki, György; Anastassiou, Costas A.; Koch, Christof
- Nature Reviews Neuroscience, Vol. 13, Issue 6
The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
journal, May 2012
- Buzsáki, György; Anastassiou, Costas A.; Koch, Christof
- Nature Reviews Neuroscience, Vol. 13, Issue 6
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event- related desynchronization
journal, December 1998
- Crone, N.
- Brain, Vol. 121, Issue 12
Do We Know What the Early Visual System Does?
journal, November 2005
- Carandini, M.
- Journal of Neuroscience, Vol. 25, Issue 46
Single-trial spike trains in parietal cortex reveal discrete steps during decision-making
journal, July 2015
- Latimer, K. W.; Yates, J. L.; Meister, M. L. R.
- Science, Vol. 349, Issue 6244
Power-Law Scaling in the Brain Surface Electric Potential
journal, December 2009
- Miller, Kai J.; Sorensen, Larry B.; Ojemann, Jeffrey G.
- PLoS Computational Biology, Vol. 5, Issue 12
Perceptual restoration of masked speech in human cortex
journal, December 2016
- Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.
- Nature Communications, Vol. 7, Issue 1
Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
journal, April 2011
- Ray, Supratim; Maunsell, John H. R.
- PLoS Biology, Vol. 9, Issue 4
Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates
journal, October 2003
- Carmena, Jose M.; Lebedev, Mikhail A.; Crist, Roy E.
- PLoS Biology, Vol. 1, Issue 2
Direct classification of all American English phonemes using signals from functional speech motor cortex
journal, May 2014
- Mugler, Emily M.; Patton, James L.; Flint, Robert D.
- Journal of Neural Engineering, Vol. 11, Issue 3
Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex
journal, May 2015
- Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P.
- Nature Communications, Vol. 6, Issue 1
Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement
journal, April 2011
- Wulsin, D. F.; Gupta, J. R.; Mani, R.
- Journal of Neural Engineering, Vol. 8, Issue 3
Decoding flexion of individual fingers using electrocorticographic signals in humans
journal, October 2009
- Kubánek, J.; Miller, K. J.; Ojemann, J. G.
- Journal of Neural Engineering, Vol. 6, Issue 6
Opening the Black Box of Deep Neural Networks via Information
preprint, January 2017
- Shwartz-Ziv, Ravid; Tishby, Naftali
- arXiv
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band
journal, December 1998
- Crone, N.
- Brain, Vol. 121, Issue 12
A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
journal, December 2009
- Guenther, Frank H.; Brumberg, Jonathan S.; Wright, E. Joseph
- PLoS ONE, Vol. 4, Issue 12
Decoding spoken words using local field potentials recorded from the cortical surface
journal, September 2010
- Kellis, Spencer; Miller, Kai; Thomson, Kyle
- Journal of Neural Engineering, Vol. 7, Issue 5
Propagating waves mediate information transfer in the motor cortex
journal, November 2006
- Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G.
- Nature Neuroscience, Vol. 9, Issue 12
Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography
conference, August 2014
- Bouchard, Kristofer E.; Chang, Edward F.
- 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier
journal, February 2016
- Steyrl, David; Scherer, Reinhold; Faller, Josef
- Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1
Performance-optimized hierarchical models predict neural responses in higher visual cortex
journal, May 2014
- Yamins, D. L. K.; Hong, H.; Cadieu, C. F.
- Proceedings of the National Academy of Sciences, Vol. 111, Issue 23
Brain-to-text: Decoding spoken phrases from phone representations in the brain
text, January 2015
- Herff, C.; Heger, D.; De Pesters, A.
- Karlsruhe
Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans
journal, March 2009
- Miller, K. J.; Zanos, S.; Fetz, E. E.
- Journal of Neuroscience, Vol. 29, Issue 10
Scikit-learn: Machine Learning in Python
text, January 2012
- Pedregosa, Fabian; Varoquaux, Gaël; Gramfort, Alexandre
- arXiv
Using the electrocorticographic speech network to control a brain–computer interface in humans
journal, April 2011
- Leuthardt, Eric C.; Gaona, Charles; Sharma, Mohit
- Journal of Neural Engineering, Vol. 8, Issue 3
Functional organization of human sensorimotor cortex for speech articulation
journal, February 2013
- Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith
- Nature, Vol. 495, Issue 7441
Brain-to-text: decoding spoken phrases from phone representations in the brain
journal, June 2015
- Herff, Christian; Heger, Dominic; de Pesters, Adriana
- Frontiers in Neuroscience, Vol. 9
Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids
journal, October 2018
- Ramsey, N. F.; Salari, E.; Aarnoutse, E. J.
- NeuroImage, Vol. 180
Direct classification of all American English phonemes using signals from functional speech motor cortex
journal, May 2014
- Mugler, Emily M.; Patton, James L.; Flint, Robert D.
- Journal of Neural Engineering, Vol. 11, Issue 3
Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement
journal, April 2011
- Wulsin, D. F.; Gupta, J. R.; Mani, R.
- Journal of Neural Engineering, Vol. 8, Issue 3
Spike-triggered neural characterization
journal, February 2006
- Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C.
- Journal of Vision, Vol. 6, Issue 4
Cortical gamma responses: Searching high and low
journal, January 2011
- Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J.
- International Journal of Psychophysiology, Vol. 79, Issue 1
Note on Information Transfer Rates in Human Communication
journal, October 1998
- Reed, Charlotte M.; Durlach, Nathaniel I.
- Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5
A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons
journal, February 1988
- Zipser, David; Andersen, Richard A.
- Nature, Vol. 331, Issue 6158
Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex
journal, September 2014
- Bouchard, K. E.; Chang, E. F.
- Journal of Neuroscience, Vol. 34, Issue 38
Comparison of neuronal responses in primate inferior-temporal cortex and feed-forward deep neural network model with regard to information processing of faces
journal, February 2021
- Matsumoto, Narihisa; Mototake, Yoh-ichi; Kawano, Kenji
- Journal of Computational Neuroscience, Vol. 49, Issue 3
Cortical gamma responses: Searching high and low
journal, January 2011
- Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J.
- International Journal of Psychophysiology, Vol. 79, Issue 1
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. II. Event-related synchronization in the gamma band
journal, December 1998
- Crone, N.
- Brain, Vol. 121, Issue 12
A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
journal, December 2009
- Guenther, Frank H.; Brumberg, Jonathan S.; Wright, E. Joseph
- PLoS ONE, Vol. 4, Issue 12
Functional mapping of human sensorimotor cortex with electrocorticographic spectral analysis. I. Alpha and beta event- related desynchronization
journal, December 1998
- Crone, N.
- Brain, Vol. 121, Issue 12
Feature extraction with stacked autoencoders for epileptic seizure detection
conference, August 2014
- Supratak, Akara
- 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
Deep Residual Learning for Image Recognition
preprint, January 2015
- He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
- arXiv
Enhanced Higgs Boson to Search with Deep Learning
journal, March 2015
- Baldi, P.; Sadowski, P.; Whiteson, D.
- Physical Review Letters, Vol. 114, Issue 11
Convergent Learning: Do different neural networks learn the same representations?
preprint, January 2015
- Li, Yixuan; Yosinski, Jason; Clune, Jeff
- arXiv
Networks for approximation and learning
journal, January 1990
- Poggio, T.; Girosi, F.
- Proceedings of the IEEE, Vol. 78, Issue 9
Emergence of Invariance and Disentanglement in Deep Representations
conference, February 2018
- Achille, Alessandro; Soatto, Stefano
- 2018 Information Theory and Applications Workshop (ITA)
Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex
journal, May 2015
- Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P.
- Nature Communications, Vol. 6, Issue 1
Deep Residual Learning for Image Recognition
conference, June 2016
- He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
- 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
Somatic Motor and Sensory Representation in the Cerebral Cortex of man as Studied by Electrical Stimulation
journal, January 1937
- Penfield, Wilder; Boldrey, Edwin
- Brain, Vol. 60, Issue 4
Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
journal, April 2011
- Ray, Supratim; Maunsell, John H. R.
- PLoS Biology, Vol. 9, Issue 4
Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
journal, September 2010
- Canolty, R. T.; Ganguly, K.; Kennerley, S. W.
- Proceedings of the National Academy of Sciences, Vol. 107, Issue 40
Single-trial spike trains in parietal cortex reveal discrete steps during decision-making
journal, July 2015
- Latimer, K. W.; Yates, J. L.; Meister, M. L. R.
- Science, Vol. 349, Issue 6244
Power-Law Scaling in the Brain Surface Electric Potential
journal, December 2009
- Miller, Kai J.; Sorensen, Larry B.; Ojemann, Jeffrey G.
- PLoS Computational Biology, Vol. 5, Issue 12
Beta-band oscillations—signalling the status quo?
journal, April 2010
- Engel, Andreas K.; Fries, Pascal
- Current Opinion in Neurobiology, Vol. 20, Issue 2
Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015
- Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
- Nature Biotechnology, Vol. 33, Issue 8
Pattern learning with deep neural networks in EMG-based speech recognition
conference, August 2014
- Wand, Michael; Schultz, Tanja
- 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans
journal, July 2011
- Pei, Xiaomei; Barbour, Dennis L.; Leuthardt, Eric C.
- Journal of Neural Engineering, Vol. 8, Issue 4
Perceptual restoration of masked speech in human cortex
journal, December 2016
- Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.
- Nature Communications, Vol. 7, Issue 1
Electrocorticographic representations of segmental features in continuous speech
journal, February 2015
- Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter
- Frontiers in Human Neuroscience, Vol. 09
Speech reconstruction from human auditory cortex with deep neural networks
conference, September 2015
- Yang, Minda; Sheth, Sameer A.; Schevon, Catherine A.
- Interspeech 2015
Exploring how deep neural networks form phonemic categories
conference, September 2015
- Nagamine, Tasha; Seltzer, Michael L.; Mesgarani, Nima
- Interspeech 2015
Works referencing / citing this record:
Decoding Movement From Electrocorticographic Activity: A Review
journal, December 2019
- Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
- Frontiers in Neuroinformatics, Vol. 13
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
journal, April 2019
- Angrick, Miguel; Herff, Christian; Mugler, Emily
- Journal of Neural Engineering, Vol. 16, Issue 3
Speech synthesis from ECoG using densely connected 3D convolutional neural networks
journal, April 2019
- Angrick, Miguel; Herff, Christian; Mugler, Emily
- Journal of Neural Engineering, Vol. 16, Issue 3
Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis
journal, December 2019
- Stavisky, Sergey D.; Willett, Francis R.; Wilson, Guy H.
- eLife, Vol. 8
Decoding Movement From Electrocorticographic Activity: A Review
journal, December 2019
- Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
- Frontiers in Neuroinformatics, Vol. 13