skip to main content
DOE PAGES title logo U.S. Department of Energy
Office of Scientific and Technical Information

Title: Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex

Abstract

A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speechmore » production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.« less

Authors:
ORCiD logo [1]; ORCiD logo [1]; ORCiD logo [2]
  1. Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States); Univ. of California, Berkeley, CA (United States)
  2. Univ. of California, San Francisco, CA (United States)
Publication Date:
Research Org.:
Lawrence Berkeley National Lab. (LBNL), Berkeley, CA (United States)
Sponsoring Org.:
USDOE Office of Science (SC)
OSTI Identifier:
1572056
Grant/Contract Number:  
AC02-05CH11231
Resource Type:
Accepted Manuscript
Journal Name:
PLoS Computational Biology (Online)
Additional Journal Information:
Journal Name: PLoS Computational Biology (Online); Journal Volume: 15; Journal Issue: 9; Journal ID: ISSN 1553-7358
Publisher:
Public Library of Science
Country of Publication:
United States
Language:
English
Subject:
60 APPLIED LIFE SCIENCES

Citation Formats

Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States: N. p., 2019. Web. https://doi.org/10.1371/journal.pcbi.1007091.
Livezey, Jesse A., Bouchard, Kristofer E., & Chang, Edward F. Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex. United States. https://doi.org/10.1371/journal.pcbi.1007091
Livezey, Jesse A., Bouchard, Kristofer E., and Chang, Edward F. Mon . "Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex". United States. https://doi.org/10.1371/journal.pcbi.1007091. https://www.osti.gov/servlets/purl/1572056.
@article{osti_1572056,
title = {Deep learning as a tool for neural data analysis: Speech classification and cross-frequency coupling in human sensorimotor cortex},
author = {Livezey, Jesse A. and Bouchard, Kristofer E. and Chang, Edward F.},
abstractNote = {A fundamental challenge in neuroscience is to understand what structure in the world is represented in spatially distributed patterns of neural activity from multiple single-trial measurements. This is often accomplished by learning a simple, linear transformations between neural features and features of the sensory stimuli or motor task. While successful in some early sensory processing areas, linear mappings are unlikely to be ideal tools for elucidating nonlinear, hierarchical representations of higher-order brain areas during complex tasks, such as the production of speech by humans. Here, we apply deep networks to predict produced speech syllables from a dataset of high gamma cortical surface electric potentials recorded from human sensorimotor cortex. We find that deep networks had higher decoding prediction accuracy compared to baseline models. Having established that deep networks extract more task relevant information from neural data sets relative to linear models (i.e., higher predictive accuracy), we next sought to demonstrate their utility as a data analysis tool for neuroscience. We first show that deep network's confusions revealed hierarchical latent structure in the neural data, which recapitulated the underlying articulatory nature of speech motor control. We next broadened the frequency features beyond high-gamma and identified a novel high-gamma-to-beta coupling during speech production. Finally, we used deep networks to compare task-relevant information in different neural frequency bands, and found that the high-gamma band contains the vast majority of information relevant for the speech prediction task, with little-to-no additional contribution from lower-frequency amplitudes. Together, these results demonstrate the utility of deep networks as a data analysis tool for basic and applied neuroscience.},
doi = {10.1371/journal.pcbi.1007091},
journal = {PLoS Computational Biology (Online)},
number = 9,
volume = 15,
place = {United States},
year = {2019},
month = {9}
}

Journal Article:
Free Publicly Available Full Text
Publisher's Version of Record

Citation Metrics:
Cited by: 3 works
Citation information provided by
Web of Science

Save / Share:

Works referenced in this record:

Spectral-Temporal Receptive Fields of Nonlinear Auditory Neurons Obtained Using Natural Sounds
journal, March 2000


Do We Know What the Early Visual System Does?
journal, November 2005


Spike-triggered neural characterization
journal, February 2006

  • Schwartz, Odelia; Pillow, Jonathan W.; Rust, Nicole C.
  • Journal of Vision, Vol. 6, Issue 4
  • DOI: 10.1167/6.4.13

Networks for approximation and learning
journal, January 1990

  • Poggio, T.; Girosi, F.
  • Proceedings of the IEEE, Vol. 78, Issue 9
  • DOI: 10.1109/5.58326

Deep Residual Learning for Image Recognition
conference, June 2016

  • He, Kaiming; Zhang, Xiangyu; Ren, Shaoqing
  • 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
  • DOI: 10.1109/CVPR.2016.90

Random forests in non-invasive sensorimotor rhythm brain-computer interfaces: a practical and convenient non-linear classifier
journal, February 2016

  • Steyrl, David; Scherer, Reinhold; Faller, Josef
  • Biomedical Engineering / Biomedizinische Technik, Vol. 61, Issue 1
  • DOI: 10.1515/bmt-2014-0117

Modeling electroencephalography waveforms with semi-supervised deep belief nets: fast classification and anomaly measurement
journal, April 2011


Pattern learning with deep neural networks in EMG-based speech recognition
conference, August 2014

  • Wand, Michael; Schultz, Tanja
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6944550

Feature extraction with stacked autoencoders for epileptic seizure detection
conference, August 2014

  • Supratak, Akara
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6944546

A back-propagation programmed network that simulates response properties of a subset of posterior parietal neurons
journal, February 1988

  • Zipser, David; Andersen, Richard A.
  • Nature, Vol. 331, Issue 6158
  • DOI: 10.1038/331679a0

Performance-optimized hierarchical models predict neural responses in higher visual cortex
journal, May 2014

  • Yamins, D. L. K.; Hong, H.; Cadieu, C. F.
  • Proceedings of the National Academy of Sciences, Vol. 111, Issue 23
  • DOI: 10.1073/pnas.1403112111

Functional organization of human sensorimotor cortex for speech articulation
journal, February 2013

  • Bouchard, Kristofer E.; Mesgarani, Nima; Johnson, Keith
  • Nature, Vol. 495, Issue 7441
  • DOI: 10.1038/nature11911

Control of Spoken Vowel Acoustics and the Influence of Phonetic Context in Human Speech Sensorimotor Cortex
journal, September 2014


Neural decoding of spoken vowels from human sensory-motor cortex with high-density electrocorticography
conference, August 2014

  • Bouchard, Kristofer E.; Chang, Edward F.
  • 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC)
  • DOI: 10.1109/EMBC.2014.6945185

Direct classification of all American English phonemes using signals from functional speech motor cortex
journal, May 2014


Electrocorticographic representations of segmental features in continuous speech
journal, February 2015

  • Lotte, Fabien; Brumberg, Jonathan S.; Brunner, Peter
  • Frontiers in Human Neuroscience, Vol. 09
  • DOI: 10.3389/fnhum.2015.00097

A Wireless Brain-Machine Interface for Real-Time Speech Synthesis
journal, December 2009


Using the electrocorticographic speech network to control a brain–computer interface in humans
journal, April 2011


Decoding spoken words using local field potentials recorded from the cortical surface
journal, September 2010


Decoding vowels and consonants in spoken and imagined words using electrocorticographic signals in humans
journal, July 2011


Brain-to-text: decoding spoken phrases from phone representations in the brain
journal, June 2015


The origin of extracellular fields and currents — EEG, ECoG, LFP and spikes
journal, May 2012

  • Buzsáki, György; Anastassiou, Costas A.; Koch, Christof
  • Nature Reviews Neuroscience, Vol. 13, Issue 6
  • DOI: 10.1038/nrn3241

Alpha-Beta and Gamma Rhythms Subserve Feedback and Feedforward Influences among Human Visual Cortical Areas
journal, January 2016


Propagating waves mediate information transfer in the motor cortex
journal, November 2006

  • Rubino, Doug; Robbins, Kay A.; Hatsopoulos, Nicholas G.
  • Nature Neuroscience, Vol. 9, Issue 12
  • DOI: 10.1038/nn1802

Large-scale spatiotemporal spike patterning consistent with wave propagation in motor cortex
journal, May 2015

  • Takahashi, Kazutaka; Kim, Sanggyun; Coleman, Todd P.
  • Nature Communications, Vol. 6, Issue 1
  • DOI: 10.1038/ncomms8169

Oscillatory phase coupling coordinates anatomically dispersed functional cell assemblies
journal, September 2010

  • Canolty, R. T.; Ganguly, K.; Kennerley, S. W.
  • Proceedings of the National Academy of Sciences, Vol. 107, Issue 40
  • DOI: 10.1073/pnas.1008306107

Event-related EEG/MEG synchronization and desynchronization: basic principles
journal, November 1999


Beta-band oscillations—signalling the status quo?
journal, April 2010


Single-trial spike trains in parietal cortex reveal discrete steps during decision-making
journal, July 2015


Brain–computer interfaces for communication and control
journal, June 2002


Decoding flexion of individual fingers using electrocorticographic signals in humans
journal, October 2009


Predicting the sequence specificities of DNA- and RNA-binding proteins by deep learning
journal, July 2015

  • Alipanahi, Babak; Delong, Andrew; Weirauch, Matthew T.
  • Nature Biotechnology, Vol. 33, Issue 8
  • DOI: 10.1038/nbt.3300

Enhanced Higgs Boson to τ + τ Search with Deep Learning
journal, March 2015


Decoupling the Cortical Power Spectrum Reveals Real-Time Representation of Individual Finger Movements in Humans
journal, March 2009


Broadband Shifts in Local Field Potential Power Spectra Are Correlated with Single-Neuron Spiking in Humans
journal, October 2009


Spectral Changes in Cortical Surface Potentials during Motor Movement
journal, February 2007


Perceptual restoration of masked speech in human cortex
journal, December 2016

  • Leonard, Matthew K.; Baud, Maxime O.; Sjerps, Matthias J.
  • Nature Communications, Vol. 7, Issue 1
  • DOI: 10.1038/ncomms13619

Note on Information Transfer Rates in Human Communication
journal, October 1998

  • Reed, Charlotte M.; Durlach, Nathaniel I.
  • Presence: Teleoperators and Virtual Environments, Vol. 7, Issue 5
  • DOI: 10.1162/105474698565893

Learning to Control a Brain–Machine Interface for Reaching and Grasping by Primates
journal, October 2003


Cortical gamma responses: Searching high and low
journal, January 2011

  • Crone, Nathan E.; Korzeniewska, Anna; Franaszczuk, Piotr J.
  • International Journal of Psychophysiology, Vol. 79, Issue 1
  • DOI: 10.1016/j.ijpsycho.2010.10.013

Different Origins of Gamma Rhythm and High-Gamma Activity in Macaque Visual Cortex
journal, April 2011


Power-Law Scaling in the Brain Surface Electric Potential
journal, December 2009


Emergence of Invariance and Disentanglement in Deep Representations
conference, February 2018

  • Achille, Alessandro; Soatto, Stefano
  • 2018 Information Theory and Applications Workshop (ITA)
  • DOI: 10.1109/ITA.2018.8503149

    Works referencing / citing this record:

    Decoding Movement From Electrocorticographic Activity: A Review
    journal, December 2019

    • Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
    • Frontiers in Neuroinformatics, Vol. 13
    • DOI: 10.3389/fninf.2019.00074

    Decoding Movement From Electrocorticographic Activity: A Review
    journal, December 2019

    • Volkova, Ksenia; Lebedev, Mikhail A.; Kaplan, Alexander
    • Frontiers in Neuroinformatics, Vol. 13
    • DOI: 10.3389/fninf.2019.00074

    Neural ensemble dynamics in dorsal motor cortex during speech in people with paralysis
    journal, December 2019

    • Stavisky, Sergey D.; Willett, Francis R.; Wilson, Guy H.
    • eLife, Vol. 8
    • DOI: 10.7554/elife.46015