DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex

Abstract

A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speechmore » production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [2]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
  2. Univ. of California, San Francisco, CA (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Laboratory (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1572056
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
PLoS Computational Biology (Online)
Additional Journal Information:
Journal Name: PLoS Computational Biology (Online); Journal Volume: 15; Journal Issue: 9; Journal ID: ISSN 1553-7358
Publisher:
Public Library of Science
Country of Publication:
United States
Language:
English
Subject:
60 APPLIED LIFE SCIENCES

Citation Formats

Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F.. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States: N. p., 2019. Web. doi:10.1371/journal.pcbi.1007091.
Livezey, Jesse A., Bouchard, Kristofer E., & Chang, Edward F.. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States. https://doi.org/10.1371/journal.pcbi.1007091
Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F.. Mon . "Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex". United States. https://doi.org/10.1371/journal.pcbi.1007091. https://www.osti.gov/servlets/purl/1572056.
@article{osti_1572056,
title = {Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex},
author = {Livezey, Jesse A. and Bouchard, Kristofer E. and Chang, Edward F.},
abstractNote = {A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.},
doi = {10.1371/journal.pcbi.1007091},
journal = {PLoS Computational Biology (Online)},
number = 9,
volume = 15,
place = {United States},
year = {Mon Sep 16 00:00:00 EDT 2019},
month = {Mon Sep 16 00:00:00 EDT 2019}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 14 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Electrocorticographic representations of segmental features in continuous speech
journal, February 2015

  • Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter
  • Frontiers in Human Neuroscience, Vol. 09
  • DOI: 10.3389/fnhum.2015.00097

Decoding spoken words using local field potentials recorded from the cortical surface
journal, September 2010


Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas
journal, January 2016


Using the electrocorticographic speech network to control a brain–computer interface in humans
journal, April 2011


Brain-to-text: Decoding spoken phrases from phone representations in the brain
text, January 2015


Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
journal, September 2010

  • Canolty, R. T.; Ganguly, K.; Kennerley, S. W.
  • Proceedings of the National Academy of Sciences, Vol. 107, Issue 40
  • DOI: 10.1073/pnas.1008306107

Event-related EEG/MEG synchronization and desynchronization: basic principles
journal, November 1999


Propagating waves mediate information transfer in the motor cortex
journal, November 2006

  • Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G.
  • Nature Neuroscience, Vol. 9, Issue 12
  • DOI: 10.1038/nn1802

Functional organization of human sensorimotor cortex for speech articulation
journal, February 2013

  • Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith
  • Nature, Vol. 495, Issue 7441
  • DOI: 10.1038/nature11911

Spectral-Temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds
journal, March 2000


Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans
journal, July 2011


Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015

  • Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
  • Nature Biotechnology, Vol. 33, Issue 8
  • DOI: 10.1038/nbt.3300

Decoding flexion of individual fingers using electrocorticographic signals in humans
journal, October 2009


Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier
journal, February 2016

  • Steyrl, David; Scherer, Reinhold; Faller, Josef
  • Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1
  • DOI: 10.1515/bmt-2014-0117

A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons
journal, February 1988

  • Zipser, David; Andersen, Richard A.
  • Nature, Vol. 331, Issue 6158
  • DOI: 10.1038/331679a0

Note on Information Transfer Rates in Human Communication
journal, October 1998

  • Reed, Charlotte M.; Durlach, Nathaniel I.
  • Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5
  • DOI: 10.1162/105474698565893

Spectral Changes in Cortical Surface Potentials during Motor Movement
journal, February 2007


Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans
journal, October 2009


Spike-triggered neural characterization
journal, February 2006

  • Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C.
  • Journal of Vision, Vol. 6, Issue 4
  • DOI: 10.1167/6.4.13

Brain–computer interfaces for communication and control
journal, June 2002


Networks for approximation and learning
journal, January 1990

  • Poggio, T.; Girosi, F.
  • Proceedings of the IEEE, Vol. 78, Issue 9
  • DOI: 10.1109/5.58326

The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
journal, May 2012

  • Buzsáki, György; Anastassiou, Costas A.; Koch, Christof
  • Nature Reviews Neuroscience, Vol. 13, Issue 6
  • DOI: 10.1038/nrn3241

The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
journal, May 2012

  • Buzsáki, György; Anastassiou, Costas A.; Koch, Christof
  • Nature Reviews Neuroscience, Vol. 13, Issue 6
  • DOI: 10.1038/nrn3241

Do We Know What the Early Visual System Does?
journal, November 2005


Single-trial spike trains in parietal cortex reveal discrete steps during decision-making
journal, July 2015


Power-Law Scaling in the Brain Surface Electric Potential
journal, December 2009


Perceptual restoration of masked speech in human cortex
journal, December 2016

  • Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.
  • Nature Communications, Vol. 7, Issue 1
  • DOI: 10.1038/ncomms13619

Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
journal, April 2011


Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates
journal, October 2003


Direct classification of all American English phonemes using signals from functional speech motor cortex
journal, May 2014


Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex
journal, May 2015

  • Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P.
  • Nature Communications, Vol. 6, Issue 1
  • DOI: 10.1038/ncomms8169

Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement
journal, April 2011


Decoding flexion of individual fingers using electrocorticographic signals in humans
journal, October 2009


Opening the Black Box of Deep Neural Networks via Information
preprint, January 2017


A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
journal, December 2009


Decoding spoken words using local field potentials recorded from the cortical surface
journal, September 2010


Propagating waves mediate information transfer in the motor cortex
journal, November 2006

  • Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G.
  • Nature Neuroscience, Vol. 9, Issue 12
  • DOI: 10.1038/nn1802

Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography
conference, August 2014

  • Bouchard, Kristofer E.; Chang, Edward F.
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6945185

Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier
journal, February 2016

  • Steyrl, David; Scherer, Reinhold; Faller, Josef
  • Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1
  • DOI: 10.1515/bmt-2014-0117

Performance-optimized hierarchical models predict neural responses in higher visual cortex
journal, May 2014

  • Yamins, D. L. K.; Hong, H.; Cadieu, C. F.
  • Proceedings of the National Academy of Sciences, Vol. 111, Issue 23
  • DOI: 10.1073/pnas.1403112111

Brain-to-text: Decoding spoken phrases from phone representations in the brain
text, January 2015


Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans
journal, March 2009


Scikit-learn: Machine Learning in Python
text, January 2012


Using the electrocorticographic speech network to control a brain–computer interface in humans
journal, April 2011


Functional organization of human sensorimotor cortex for speech articulation
journal, February 2013

  • Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith
  • Nature, Vol. 495, Issue 7441
  • DOI: 10.1038/nature11911

Brain-to-text: decoding spoken phrases from phone representations in the brain
journal, June 2015


Decoding spoken phonemes from sensorimotor cortex with high-density ECoG grids
journal, October 2018


Direct classification of all American English phonemes using signals from functional speech motor cortex
journal, May 2014


Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement
journal, April 2011


Spike-triggered neural characterization
journal, February 2006

  • Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C.
  • Journal of Vision, Vol. 6, Issue 4
  • DOI: 10.1167/6.4.13

Cortical gamma responses: Searching high and low
journal, January 2011

  • Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J.
  • International Journal of Psychophysiology, Vol. 79, Issue 1
  • DOI: 10.1016/j.ijpsycho.2010.10.013

Note on Information Transfer Rates in Human Communication
journal, October 1998

  • Reed, Charlotte M.; Durlach, Nathaniel I.
  • Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5
  • DOI: 10.1162/105474698565893

A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons
journal, February 1988

  • Zipser, David; Andersen, Richard A.
  • Nature, Vol. 331, Issue 6158
  • DOI: 10.1038/331679a0

Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex
journal, September 2014


Comparison of neuronal responses in primate inferior-temporal cortex and feed-forward deep neural network model with regard to information processing of faces
journal, February 2021

  • Matsumoto, Narihisa; Mototake, Yoh-ichi; Kawano, Kenji
  • Journal of Computational Neuroscience, Vol. 49, Issue 3
  • DOI: 10.1007/s10827-021-00778-5

Cortical gamma responses: Searching high and low
journal, January 2011

  • Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J.
  • International Journal of Psychophysiology, Vol. 79, Issue 1
  • DOI: 10.1016/j.ijpsycho.2010.10.013

A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
journal, December 2009


Feature extraction with stacked autoencoders for epileptic seizure detection
conference, August 2014

  • Supratak, Akara
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6944546

Deep Residual Learning for Image Recognition
preprint, January 2015


Enhanced Higgs Boson to τ + τ Search with Deep Learning
journal, March 2015


Convergent Learning: Do different neural networks learn the same representations?
preprint, January 2015


Networks for approximation and learning
journal, January 1990

  • Poggio, T.; Girosi, F.
  • Proceedings of the IEEE, Vol. 78, Issue 9
  • DOI: 10.1109/5.58326

Emergence of Invariance and Disentanglement in Deep Representations
conference, February 2018

  • Achille, Alessandro; Soatto, Stefano
  • 2018 Information Theory and Applications Workshop (ITA)
  • DOI: 10.1109/ita.2018.8503149

Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex
journal, May 2015

  • Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P.
  • Nature Communications, Vol. 6, Issue 1
  • DOI: 10.1038/ncomms8169

Deep Residual Learning for Image Recognition
conference, June 2016

  • He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • DOI: 10.1109/CVPR.2016.90

Somatic Motor and Sensory Representation in the Cerebral Cortex of man as Studied by Electrical Stimulation
journal, January 1937


Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
journal, April 2011


Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
journal, September 2010

  • Canolty, R. T.; Ganguly, K.; Kennerley, S. W.
  • Proceedings of the National Academy of Sciences, Vol. 107, Issue 40
  • DOI: 10.1073/pnas.1008306107

Single-trial spike trains in parietal cortex reveal discrete steps during decision-making
journal, July 2015


Power-Law Scaling in the Brain Surface Electric Potential
journal, December 2009


Beta-band oscillations—signalling the status quo?
journal, April 2010


Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015

  • Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
  • Nature Biotechnology, Vol. 33, Issue 8
  • DOI: 10.1038/nbt.3300

Pattern learning with deep neural networks in EMG-based speech recognition
conference, August 2014

  • Wand, Michael; Schultz, Tanja
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6944550

Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans
journal, July 2011


Perceptual restoration of masked speech in human cortex
journal, December 2016

  • Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.
  • Nature Communications, Vol. 7, Issue 1
  • DOI: 10.1038/ncomms13619

Electrocorticographic representations of segmental features in continuous speech
journal, February 2015

  • Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter
  • Frontiers in Human Neuroscience, Vol. 09
  • DOI: 10.3389/fnhum.2015.00097

Speech reconstruction from human auditory cortex with deep neural networks
conference, September 2015


Exploring how deep neural networks form phonemic categories
conference, September 2015


Works referencing / citing this record:

Decoding Movement From Electrocorticographic Activity: A Review
journal, December 2019

  • Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
  • Frontiers in Neuroinformatics, Vol. 13
  • DOI: 10.3389/fninf.2019.00074

Speech synthesis from ECoG using densely connected 3D convolutional neural networks
journal, April 2019

  • Angrick, Miguel; Herff, Christian; Mugler, Emily
  • Journal of Neural Engineering, Vol. 16, Issue 3
  • DOI: 10.1088/1741-2552/ab0c59

Speech synthesis from ECoG using densely connected 3D convolutional neural networks
journal, April 2019

  • Angrick, Miguel; Herff, Christian; Mugler, Emily
  • Journal of Neural Engineering, Vol. 16, Issue 3
  • DOI: 10.1088/1741-2552/ab0c59

Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis
journal, December 2019

  • Stavisky, Sergey D.; Willett, Francis R.; Wilson, Guy H.
  • eLife, Vol. 8
  • DOI: 10.7554/elife.46015

Decoding Movement From Electrocorticographic Activity: A Review
journal, December 2019

  • Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
  • Frontiers in Neuroinformatics, Vol. 13
  • DOI: 10.3389/fninf.2019.00074